The Deep End | Paul Venezia | TAG: Networks

MORE ENTRIES

TAG: Networks

April 14, 2008 | Comments: (0)

A few months ago, I wrote about a ComCast Business circuit that had yet to be installed, despite the fact that the contract had been signed and initial fees paid six months prior. It was a sad tale, and one without resolution -- until Friday.

Yes, on Friday, April 11th, just over eight months after the contract was signed, the circuit went live. Although ComCast had claimed several times during those months that the circuit would be installed "in two weeks", the last time they made this claim, they actually delivered -- two weeks to the day after hearing those words yet again, packets were passed, source routing was configured, and there was much rejoicing.

I've long thought that cable and DSL services make for solid overflow Internet bandwidth for many companies. While the tried-and-true T1 is still the best bet for reliable, synchronous bandwidth, they're too expensive for basic Internet browsing these days, with the relatively low cost of high-bandwidth cable and DSL circuits. Many firewalls can support multiple egress links, and many switches can handle the source routing tasks necessary to use two different circuits -- heck, in the absence of either of those, a really low-spec Linux box can handle the source routing -- so it makes all kinds of sense to go this route.

Of course, that's assuming that it doesn't take the better part of a year to install, but I would hope that is the exception and not the rule.

Posted by Paul Venezia on April 14, 2008 09:29 AM

February 12, 2008 | Comments: (0)

Let the games begin

It seems that Nick Farrell over at The Inquirer isn't so thrilled by my MacBook Air review. Actually, he doesn't really mention the review, opting instead to summarize the sidebar with additional commentary. To clarify a few of his points:

o- Yep, it took five hours to do the whole migration. The first 30 minutes were problematic, but the rest of the time was the two systems transferring 50GB of files via 100Mbit Ethernet without supervision.
o- The Air didn't crash -- the Migration Assistant application crashed.
o- I bought the Air myself.
o- "Fanboy" seems to be a favorite expression of someone who doesn't like to see positive comments about something they don't like. I gave the Air a "Very Good" rating, and it earned it. If it had integrated 3G and a realistic 5 hours of battery life, it might have made it to "Excellent".
o- Isn't it odd that although I'm apparently a "hack" trying to put positive spin on Apple's products, I decided to write an entire sidebar about a negative experience?

I suggest that Nick read the whole review as well as my blog comments. I'd be delighted to see him run that though his fun-house mirror.

UPDATE: Interesting. All the comments on the Inquirer post just disappeared right after I submitted one.
UPDATE: They're back, sans my comment. Curious.

Yet another UPDATE: I might suggest that anyone interested in this topic read the actual review, and my companion blog post, not just the sidebar. I wouldn't want anyone to be embarrassingly misinformed -- it's bad for the knees.

Posted by Paul Venezia on February 12, 2008 11:22 AM

January 30, 2008 | Comments: (0)

O Verizon, how I loathe thee

Why must this be so difficult, so painful? Why must you spurn me at every opportunity, causing me to rend my clothing and speak in tongues? This hold you have over me is distressing... O Verizon, how I loathe thee.

You tempt me with promises of on-line account management, of security, let leave me hanging with Byzantine confirmation methods and completely unintelligible voice recordings of temporary PIN numbers. You email me validation codes that don't work, serve me ASP.NET pages that look and function like it's 1998, and yet STILL, you won't let me check my bill on-line.

Why must it be so? Why must you insist that you call my home phone with a temporary PIN thats read in a sampled voice? A voice that makes the letters D,E,G,P,V, and Z all sound alike? How many possible combinations must I try before I'm granted access to my own account, an account that I had full access to only weeks ago? It seems like so long -- so long since I found your website even moderately useful. No, I fear that the deeper feeling is gone, edged away from true apathy by a breathtaking barrage of useless and completely non-functional verification steps. It didn't have to be this way. You could have shown even an inkling of competence -- I would have forgiven, I would have tolerated you for a little while longer...

Now, I know not what will become of me. Perhaps I will finally convert all my lines to Time Warner Digital Phone. But wait! I cannot! You have me in an impossible position because I have DSL!

O Verizon... why can't I quit you?

Posted by Paul Venezia on January 30, 2008 12:27 PM

January 21, 2008 | Comments: (0)

Thinking about a ComCast business circuit? Think again.

This is unfortunately a tale of woe. There are no names in this story to protect the innocent, but rest assured, dear reader, this is completely true.

Over six months ago, a signed contract and $1,000 in build-out fees were sent to the ComCast Business division. This was to be a business-class circuit with a single fixed IP address to handle overflow from existing T1 circuits. ComCast already had copper and fiber in the immediate area, and simply needed to run it into the building, which was one of two brand-new multi-tenant office buildings in an office park -- a third building was under construction. The contract terms and fees has taken nearly two months to sort out, with an email trail that smacked more of Abbot and Costello than a business deal, including blatant, repeated, and honestly puzzling misspellings of the company name, forgotten details, and wrong paperwork. But that's in the past right? Pen has been put to paper, the contracts are signed, money has changed hands, and the waiting begins.

It's not just waiting though. The word 'waiting' evokes a passive tone, but this wasn't exactly passive. It took ComCast months to send a request to the building management to do the install, even though they claimed that the building had never responded. After another month of wrangling to get the paperwork, it was signed and returned the same day. During this time, not one, but *two* requests were made by ComCast that the company resend the original contract to ComCast, since they lost it. Twice. Yes, they lost their own contract. Twice. Following that, there was a spark of action -- a site visit from a third-party wiring contractor. The anticipation generated from that visit was palpable but for naught, since several months later the situation hadn't changed. All the network hardware had been ordered, arrived, and configured, waiting for the one cable to make it functional -- yet there was nothing there, and nothing on the horizon. ComCast is a deadzone -- a complete lack of communication. Repeated emails go unanswered. Phone calls terminate without any voicemail or actual live person. A black hole.

Meanwhile, existing Internet circuits are running 80-90% utilization and complaints are mounting. Telling people that more bandwidth had been ordered over six months ago simply doesn't work anymore.

So the moral of this story is that ComCast Business has nothing to do with business -- in fact, it might just be a figment of our collective imaginations... except that $1,000 check was very real.

Posted by Paul Venezia on January 21, 2008 11:19 AM

July 25, 2007 | Comments: (0)

Yet another reason the geek shall inherit the Earth

The old saying is just one letter off -- it's not the meek, it's the geek. I'm sitting in Portland Airport in Oregon waiting for a redeye, and the wifi is up but the DHCP server is dead. PDX is an enlightened airport, offering free WiFi for the entire airport (when it works) and I have a few hours to kill before the soul-sucking cross-country flight, and I really needed to get some work done. So, where most of the world would have given up in frustration, I simply ran tcpdump on my WiFi interface, found traffic on the local net which told me what subnet was in use, and I picked an ip high in that range, guessed at the default route (it was .1), specified a public DNS server I run, and voila, I'm posting this entry.

Between that and the knowledge required to successfully configure Bluetooth contact list synchronization, us geeks have it all tied up.

Posted by Paul Venezia on July 25, 2007 12:23 AM

June 08, 2007 | Comments: (0)

Making APC network cards play nice with Active Directory

Since I'm flush with APC gear at the moment, I've been working with the network management devices (obviously, since I wrote Cacti and Nagios plugins for most of them). One of the features of most of the hardware is the ability to use local or RADIUS authentication. Obviously, with a large number of devices, using centralized authentication is not just a good idea, it's the only way to fly. Unfortunately, none of the NMCs have straight-up LDAP auth, but they will do RADIUS. Thus, using Microsoft's IAS (Internet Authentication Service) -- their interpretation of a RADIUS server, it's possible to do auth to AD via RADIUS though it's not exactly straightfoward.

APC has a knowledgebase document that describes a RADIUS implementation using FreeRADIUS. I'm a big FreeRADIUS fan, having used it in countless UNIX and dialup scenarios. However, it won't authenticate to AD without some serious gyrations, and it's simpler to use IAS. In order to use IAS, however, custom attributes need to be defined, otherwise logins may work but the admin bit will not be set and administration of the devices will be impossible. Here's the remedy.

Install IAS on a Windows server and register it with AD. Then, define a client. The client should be configured with any friendly name, and the IP of the NMC card in the APC device. Set the vendor to RADIUS Standard, and define the shared secret you'll be using. Next, define a new Remote Access Policy. Add some conditions, such as Windows-Groups matches "DOMAIN\Domain Admins" AND Client-IP-Address matches "172.16.1.*", which would cause this policy to only grant access if the user account is in the Domain Admins group, and the client IP falls within the 172.16.1 network. It's a good idea to restrict access in this way, since IAS RADIUS policies stop on the first match -- if you have multiple policies, you want them to be specific to their task.

Once this is done, some custom attributes have to be defined to grant admin privs to the user logging into the APC gear. Click Edit Profile with in the policy properties page. Click the Advanced tab, and Add an attribute. Select Vendor-Specific, and then click Add. Click Enter Vendor Code, and enter 318 into the text field to the right. Then, click Yes, it conforms, and then click Configure Attribute. Set the Vendor Assigned Attribute Number to 1, Attribute Format to "Decimal", and the Value to 1. This specifies that the value of attribute number 1, with Vendor Code 318 should be a decimal (integer) and be set to 1. The other possible values are 2, which denotes a device login, or 3, which is a read-only login, and is the default if this attribute is not defined. Note that by defining another policy that might match on a different set of groups, and setting this value to "3" will result in those users getting read-only access to the same devices, which might be handy.

Click OK a few times to get back to the Profile settings dialog, and select Authentication. make sure CHAP and PAP are selected, then click Apply and OK. Make sure that "Grant remote access permission" is selected, and click OK again. At this point, the server should be configured properly for the one RADIUS client originally specified. I usually stop and restart the IAS server at this point, since I've seen it act oddly when this isn't done.

Now, log into the APC device and configure RADIUS authentication. This is usually found on the Administration tab, under Security/Remote Users. Note that this has been tested with APC app module v3.3.1, and aos 3.3.4. If you haven't updated to this revision of the firmware on your devices, it's well worth the time to do so before continuing. Prior revisions may not even support RADIUS auth, for instance.

Fill in the server IP address and shared secret, then test the authentication. Assuming it works, set the authentication selection to "RADIUS, then Local Authentication", log out and then log back in with an account that matches the Remote Access policy defined on the IAS server. You should be in like Flynn.

I've run into some issues with using Network PowerChute with RADIUS authentication -- namely, the Network PowerChute service will not properly authenticate to a UPS NMC card that's configured for RADIUS auth, even with valid RADIUS credentials. I'm working with APC on a solution to that issue now, but I've tested this configuration with APC managed rack PDUs and air units without issue.

Posted by Paul Venezia on June 8, 2007 12:33 PM

June 01, 2007 | Comments: (0)

Introducing ASAP - Automated Switchport Access Provisioning

...or something like that.

ASAP is a PHP/Perl application that automates switchport VLAN assignments for Cisco switches.

The good stuff:

o- Web and telnet interface
o- Can handle multiple switches across multiple sites
o- All switch interaction is via SNMP
o- Forces selected switchport down/up, causing Windows systems to automatically release/renew a DHCP lease
o- Prunes the ARP table following a VLAN modification
o- Can prevent certain subnets and IP addresses from being modified
o- Can be easily used by end-users
o- Admin interface provides instant system location by hostname, IP or MAC address
o- LDAP authentication for admin interface
o- MySQL logging
o- Basic reporting tools
o- Has been tested on CatOS and IOS with Cisco 6500-series and 3500-series switches

The bad stuff:

o- Rudimentary reporting needs work
o- Unsure of scalability. Sites with dozens of switches may require code tweaks
o- Hasn't been tested on several switch classes
o- Configuration could be more straightforward

Overview:

Once a network has been built and is fully operational, the vast majority of configuration tasks are simple VLAN assignments. Usually, these assignments happen only once, when a workstation is first introduced into the network, but in lab environments, VLAN assignments can occur constantly. ASAP was designed to remove the burden of system switchport location and VLAN modification from IT, and allow general users to easily perform these changes. Alternatively, ASAP can be configured to only allow admin access, and given a MAC address, IP address, or hostname, a specific system's current switchport can be located and modified without telnetting to a switch, and with an audit trail.

I originally wrote this right before I moved two large sites from one building to another. Each site had over 800 switchports and I was lazy enough to not want to deal with VLAN assignments. I wrote the ASAP Web and telnet applications, and placed every switchport into a VLAN with ACLs preventing access to any internal resources other than the Linux server running the apps, a dhcpd and a wildcard DNS server. Thus, whenever a client is plugged into an unknown switchport, they're given an IP in the "deadzone" range, and any Web site they try to visit brings up the ASAP app. They can then select the appropriate VLAN for their system, and 45 seconds later, they're fully up and running, without rebooting. *nix systems that don't run a GUI interface can also do their own VLAN assignments via the telnet application. Telnetting to the IP/hostname of the ASAP server brings up a CLI version of the Web application.

Both apps are autonomous, and can be configured independently. This permits greater modularity as well as security, since it's likely that the Web app will be used in a general corporate setting, where the telnet app will probably be used more in a lab setting. Both the Web and telnet apps log to a common MySQL database.

The apps work by determining the IP address of the connecting client, then polling each switch in turn until the IP, MAC address, switch, and switchport information is determined. Then, if the IP doesn't match a denysubnets definition, and all necessary info has been gathered, the user can select from a list of VLANs, and the app will change that specific port to the new VLAN, disable and re-enable the port (Web app only), and remove the original ARP entry from the router. With most browsers, the user will be sent to a "Please wait..." page that will refresh after 45 seconds showing that all is well. In the background, the switchport has been changed to the right VLAN, and the port disable/enable action forces Windows and Mac systems to release/renew their DHCP lease. This forces the system into the correct VLAN without requiring any user interaction or reboots. Note that the telnet app does not perform the disable/enable action though it could certainly be coded to do so.

To date, this code has been used by several hundred people to change several hundred switchports, but needs testing in lots of other settings. There are probably bugs that will be triggered by older IOS/CatOS revisions, among other things.

Configuration:

Configuration is relatively manual for now. Read through the asapd.pl and index.php files to configure the application. The most important bits are obviously the switch IP/SNMP settings, denysubnet definitions, and other site information. Note that you'll have to manually pull the VLAN index numbers from your switches. Info on how to do this is in the script comments. The included asap.sql file should be imported into a new database and general access granted to a username/pass pair matching that found in the db.inc file and the asapd.pl file via mysql -u root -p < ./asap.sql and an accompanying grant all privileges on asap.* to asap@localhost identified by 'passwd'. The help.html file can be modified to show whatever help info you wish on the main app page. login.php needs to be modified with the appropriate LDAP/AD configuration matching your site. It's currently built for non-anonymous binding to a normal Windows 2003 AD server.

So there's a bit of work to do to configure the app, but if you're at all familiar with Perl, PHP, and MySQL, it shouldn't take more than a few minutes.

Troubleshooting:

There are no debugging facilities to speak of. Since real men debug with print statements, that's what you'll find. Enjoy.

If there's enough interest in this tool, I'll put more time into tightening up the configuration and reporting, and work on any bugs that might get dug up. Either way, if you're using ASAP in your network, I'd love to hear about it. You can find the code linked below.

Download asap-0.0.1.tgz

Posted by Paul Venezia on June 1, 2007 02:44 PM

May 10, 2007 | Comments: (0)

More on broadband banditry

Yesterday, I posted about six things that need to change. One of them was entitled "Broadband Bandits", where I basically denounced broadband companies' artificially limited bandwidth options. After re-reading it, I think I need to clarify a few things.

Certainly, these companies aren't in this business for wholly altruistic purposes -- they're in it to make money. That's the whole idea. The problem that I have with most broadband offerings is that they're specifically designed to limit end-user options without any reasonable alternative. Most areas with broadband access have one or two options, and they're generally both playing this game.

One of the major issues is the ridiculously limited upstream bandwidth provided in most residential packages. For $50 a month, I would expect to get better than 39KB/s uploading images to Flickr, videos to YouTube, pictures to my eBay auctions, and when sending email attachments. Unfortunately this is rarely the case, since upstream bandwidth has been squeezed as low as possible.

Even RoadRunner, a company that does not generally limit users' bandwidth, nor block well-known ports, delivers 5Mb/384k service with their standard package. I tested a freshly-installed RoadRunner line the other day, and found that it's just barely possible to get 5Mb down, with the 384k upstream completely maxed out with TCP ACKs. Other companies do the same thing, offering a 15:1 up/down ratio service that can just barely reach those levels, hampered by the upstream caps.

The DOCSIS cable standard isn't synchronous. Current DOCSIS installations based on the 2.0 standard are capable of delivering 38Mbit/s downstream and 27Mb/s upstream to a group of modems. A small neighborhood would have this bandwidth split between any number of modems, and using the law of averages, most users will get their rated download speeds. But notice that the 2.0 standard's down/up ratio is roughly 5:3. This doesn't coincide with the 15:1 ratio found in most broadband plans. Some offerings in the US and Canada are nearly 20:1. This doesn't jive with the capabilities of DOCSIS, so there's no technical reason why these plans exist. Upstream data is subjected to higher noise levels across a cable plant, but that doesn't justify the low caps found nearly everywhere.

The new DOCSIS 3.0 standard is very new and hasn't been widely adopted yet, but was designed to give FIOS a run for its money, offering 160Mb/s downstream and 120Mbit/s upstream to the same number of modems. Again, we see a similar down/up ratio in play.

I've seen many commercials for broadband service showing a fellow sitting in his kitchen with a laptop, telling his wife he can't go to the mall because he has to finish some work. Suddenly, we see a screenshot showing a "Done" dialog box, and voila, due to the power of XYZ's broadband service, the lucky fellow can go to the mall and relax on the hard wooden benches outside Bed Bath and Beyond. The problem here is that the ad specifically targets those people that can telecommute, without mentioning that if he was uploading a PowerPoint presentation, he'd be sitting there for a long, long time... assuming that the provider hasn't blocked IPSec and he can actually connect to the corporate network in the first place.

Consumer broadband needs to change. It needs to provide at least a 5:3 down/up ratio as part of the standard package for a reasonable price. I know dozens of broadband users that would gladly trade a few Mb downstream for a few Mb upstream, and this trend is only going to grow. Fears of illicit filesharing and copyright infringement be damned -- you can't penalize a captive audience for something they might do.

Posted by Paul Venezia on May 10, 2007 11:42 AM

May 09, 2007 | Comments: (0)

Six things that need to change

Although I'm generally able to see both sides of an argument, there exists a short list of issues that I just can't comprehend. These are those issues.

1) The RIAA's war on its customers
This one has been going on so long as to almost be accepted. Of course, that's their plan. The vast amount of money being poured into lawyers, lobbyists, and scare tactics by the RIAA would have been more than enough to rework their long-deceased business model into something for the next generation. For an industry that was built upon pushing the envelope, they certainly can't seem to think outside the CD case. The heavy lobbying in Florida that has resulted in the used CD market there receiving stricter controls than the gun market is just one tiny example.

The RIAA is certainly under attack from every angle -- piracy, slowing CD sales, a massive increase in self-produced music, and flagging interest in marquee acts -- but nearly all of that is their own fault. Instead of embracing the new market, they've been trying to kill it by shipping CDs with rootkits masquerading as DRM schemes, producing lawsuits by the bushel, apparently destroying Internet radio, and projecting an overall public persona that falls somewhere between Al Capone and Stalin. It's just ludicrous.

But then, this is the industry best described in a misquote to Hunter S Thompson: "The music business is a cruel and shallow money trench, a long plastic hallway where thieves and pimps run free, and good men die like dogs. There's also a negative side." His original words were actually describing TV broadcasting, but the sentiment prevails.

2) Broadband Bandits (Update: More on this topic can be found here)
Comcast is the easy target on this one, but there are many perpetrators of this travesty. You know who you are. More importantly, your customers know who you are, and will jump ship in an instant if given the chance. With most of the competition buried in the backyard, and a weakened FCC sitting idly by, Comcast, Verizon, and many other providers are ramping up prices and dropping service levels. They're also applying voodoo AUP interpretations to cut off paying customers that go over some amorphous limit. Many of these companies come from a delivery-only background, where they deliver the signal, and the customer passively accepts it, such as cable TV. Back in the day, this was largely true of the Internet -- Web servers existed in datacenters, ISPs, and universities, and most content was text and the occasional picture. With Flickr, YouTube, MySpace, and the advent of simple videoconferencing, end users are much more apt to be sending nearly as much as they receive, yet most broadband connections are still ridiculously asynchronous. I just ordered Verizon DSL to provide a backup circuit. $30/mo for a 3Mb/768k circuit. This means that uploading a few 5 megapixel photos will take me roughly 3 minutes, and completely obliterate that 3Mb/s download rate due to upstream congestion, even though I'm not downloading anything.

There are a few reasons that most of these wildly unbalanced plans exist. Contracts with peering partners generally dictate up/down ratios to be maintained (eg, saving the ISP money). They also prevent customers from using videoconferencing and VoIP technologies to their full potential, resulting in poor performance. This forces the customer to only use approved methods of communication (eg, paying the carrier more per month). And lastly, they've always been that way, right?

As a sideline to all this nonsense, many carriers go so far as to block well known ports, such as Web, IPSec, and SMTP ports to residential lines. True, most people aren't running Web servers from their house, but lots of them are just trying to connect to the corporate VPN. To do that, you need a business-level contract for way more money per month and usually lower bandwidth. What a bargain.

Certainly, not all carriers act this way. Comcast and Verizon DSL are famous for it, but Time Warner's RoadRunner seems to be above this chicanery, at least so far. If AT&T wasn't dismantled nearly 25 years ago, we'd still be renting our phones from Ma Bell for $20 a month, and our telecommunications infrastructure would be the best the third-world had to offer. At least Verizon is offering FIOS in some areas, yet I know of entire communities that have no broadband whatsoever. Wasn't there a Universal Service initiative started over a decade ago? Note as you read that page, you see "The Federal-State Joint Board on Universal Service recommended that the Federal Communications Commission take immediate action to rein in explosive growth in high-cost universal service support disbursements. The Joint Board is also seeking comment on proposals for long-term, comprehensive reform of the high-cost program. 5/1/07." This is because we've gotten nothing for a whole lot of something.

Just ask a South Korean how much they spend on the 100Mbit Internet circuit in their house. CNet was talking about 20Mbit links, universal video-on-demand on the cheap back in 2004. Not much has changed in three years, except their average bandwidth has increased fivefold. Heck, just ask them about the Internet service to their mobile phones -- it beats anything in the US by far. This brings me to number three.

3) The US is a mobile communications wasteland
Crazy, indecipherable "plans", "anytime minutes", $0.10 per text message, $0.003 per KB (read that any way you want), and current phones that were cutting edge in Europe when John Paul II was still wandering around the Vatican. That's the state of mobile connectivity in the US today. I've heard more than a few foreigners describe a trip to a T-Mobile store as "like visiting a cellphone museum". Given what they're used to in Europe and Asia, I have little doubt this is true. Wireless carriers in the US have been raking in money hand over fist for the past five years, riding the cellphone boom as high as it will go. During all this, they've been slowly doling out features to their users like cake to the starving, while the rest of the world runs circles around us.

The pending release of Apple's iPhone may spark something here, just as the iPod blew the portable MP3 player market apart. Hey, has Steve Jobs ever made a mistake?

4) Airport Wifi
This one's personal. I understand that fleecing business travelers for $10 or so during a flight delay is part of the business model, but even crack dealers give away the first few tastes. Can't we get 30 minutes free, and a reasonable hourly rate thereafter? I can't believe that any airport Wifi installation hasn't already paid for itself a hundred times over. I'll continue to hold Manchester Airport up as a shining example -- wide coverage, free service, no splash page. It's just beautiful.

5) Spam and the Windows Protection Racket
This one will never disappear, but it can be marginalized. If thousands and thousands of compromised Windows systems were to be patched, replaced, or burned in effigy, the volume of spam worldwide would be drastically reduced. Couple this to viruses, adware, malware, and so on, and there's very little that your PC can't do -- your taxes, spreadsheets, Web surfing, and spamming the bejeezus out of thousands of people. I think we may be near the top of a Bell curve on that one. Vista is more secure than XP (which isn't saying much) but the sheer numbers of wide-open Windows systems on the Internet will necessarily begin to decline due to hardware failure, if nothing else. If the replacements are tougher to compromise, then the spam levels will abate somewhat, as will other nefarious afflictions of the digital age, and we'll all be a little safer and saner.

Of course, if Windows were suddenly secure, it would directly affect the revenue of hundreds of smaller software vendors hawking Windows protection applications, but I can't feel too bad for Symantec or McAfee -- they'll survive.

6) Oops! I lost your ID. My bad.
Every week or so, we hear about the theft of another million identities from a laptop or network intrusion. Sometimes it's a corporation, sometimes a university, or sometimes the federal government. Sometimes it's your ID, sometimes it's mine. Pretty soon, it'll be nearly everyone that's ever had a credit card, applied for a loan, opened a bank account, or was simply assigned a social security number.

There are no formal penalties for this invasive personal intrusion, and some companies simply don't tell anyone that the event occurred. If a company doesn't have adequate security and lets a few hundred thousand database records flap in the wind, the victim will at best spend days straightening out a credit mess and changing all their accounts to new numbers. At worst, they'll lose money, their credit rating, and maybe even their job through no fault of their own. If a department store chains' physical security was so lax as to have their customers violently mugged en masse simply for being in one of their stores, you can bet they wouldn't be in business any more. What would be worse would be the poor people that got mugged because they were in a different store, but that store told the muggers they were there. Identity theft isn't much different -- since your ID is bought and sold to whomever, without your approval.

We need accountability for data security lapses of this magnitude, plain and simple. We only get one identity, and when it has been dragged through the mud it can take years to recover, and sometimes it's impossible. Unfortunately, it will take new laws and stiff penalties to see any change here, since it's apparently more cost effective to throw your customers under the bus (see number one, above).

It's obvious that the US is going through a period of massive change, largely related to the presence of the Internet and the forces that can exert some influence on it. Some of these issues may be just growing pains, but some of them may be cancer. Thus, it's very important that we not shortchange our technological future for short-term economic and bureaucratic issues. We've sold our society to the electron, and we'll be beholden to anyone who wields it better than we do.

Posted by Paul Venezia on May 9, 2007 02:58 PM

March 12, 2007 | Comments: (0)

Where's Waldo? Locating the OID you need.

A few days ago I decided to write a little Cisco-centric SNMP query/modify tool. I didn't need or want anything beyond simply finding the switch and switchport a MAC or IP address was plugged into, and to be able to set that port to another VLAN, and then enable/disable the port to force the system to renew it's DHCP lease. Most of the OIDs I needed were simple to find, others not so for some reason. Here's my short list:

Pull the MAC address table: .1.3.6.1.2.1.17.4.3.1.1
o- Used in conjuction with community@vlan syntax.

Pull the bridge port number table: .1.3.6.1.2.1.17.4.3.1.2

Find the ifIndex number: .1.3.6.1.2.1.17.1.4.1.2.<bridge port number>

Find the assigned VLAN: .1.3.6.1.4.1.9.9.68.1.2.2.1.2.<ifIndex>

Find the real port name: .1.3.6.1.2.1.31.1.1.1.1.<ifIndex>

Set a port to another VLAN: .1.3.6.1.4.1.9.9.68.1.2.2.1.2.<ifIndex> integer <VLAN ID>

Enable/disable a switchport: .1.3.6.1.2.1.2.2.1.7.379.<ifIndex> integer [ 1 = enabled | 2 = disabled ]

I'm still writing this tool, so there's sure to be more in the near future.

Posted by Paul Venezia on March 12, 2007 07:22 PM

December 27, 2006 | Comments: (0)

Your PSTN and you: Linksys SPA-3102 and Asterisk

So another piece of my Asterisk/TrixBox puzzle was completed today -- or rather, almost completed. I received the Linksys SPA-3102 FXO/FXS SIP ATA, which will be the bridge between Asterisk and one of the incoming POTS lines to the lab. I probably should have ordered the SPA-3000, since I really don't need the routing/NAT capabilities present in the SPA-3102, but then again, it's a lab. I have it working inbound and outbound with Asterisk, and it's also driving a few analog phones that are referenced as a single addressable Asterisk extension. It was a bit of a struggle, but not much. Some notes on this follow:

If you don't need the routing/NAT features of the SPA-3102, you need to initially configure it with a link on the Ethernet port, change the LAN Network Service to bridge, move the Ethernet link to the WAN port, dial #### from a phone connected to the FXS port, then dial 7932#, then 1 to enable the Web server on the WAN port.

On the voice side, you can get the SPA-3102 to trunk inbound and outbound to Asterisk without bothering to create individual extensions for the FXO and SIP ports. My trunk config is simple:



auth=md5

context=from-pstn

dtmfmode=rfc2833

fromuser=spa3102

host=172.16.32.123

insecure=very

port=5061

secret=spa3102

type=peer

username=spa3102

The Asterisk server is set under Proxy, and the username/password is referenced in the SPA-3102 PSTN Line Subscriber Info section. You will show a failed registration on the PSTN line, but as long as you've flagged Make Call without Reg and Ans Call without Reg as Yes, it will work. I'd rather have this setup than cluttered extensions on Asterisk that aren't used.

Generating DIDs from the PSTN for inbound routes on Asterisk is non-obvious. This is handled as a dialplan within the SPA-3102. For instance, (S0<:4155551212) as Dial Plan 2 under PSTN Line with a corresponding flag to use DP2 under PSTN-to-VoIP Setup will generate a 4155551212 DID to Asterisk for an incoming call. Obviously, you can put whatever you want in that field.

Speaking of dialplans, the best FXS dialplan for the SPA-3102 and Asterisk (at least my configuration) is

(*x|*xx|xxx|[3469]11|0|00|[2-9]xxxxxx|1xxx[2-9]xxxxxxS0|xxxxxxxxxxxx.)

This speeds up recognition of internal extension and feature code dialing, as well as just about everything else. With this dialplan, calls from the FXS port are routed to Asterisk, which has outbound routing configured to prefer the SIP trunks over the PSTN lines for long-distance, and the PSTN lines for all local calling, including emergency dialing.

Call Waiting is a problem, and one I haven't solved yet. With the SPA-3102 not ringing through to the FXS port on an inbound call, Asterisk takes over and handles the ring distribution perfectly. However, I can't correctly generate a single flash to the PSTN when another call comes in. I get the CW notification beeps fine on both SIP and analog extensions, but hitting the switch hook simply brings me to the Asterisk dialtone for call forwarding, parking, etc. There's no clear method of generating that flash to the PSTN line to kick over to the other call. I just discovered this problem an hour ago, and am probably going to code something into Asterisk to respond to a *3 or other unused feature code to provide a flash to the PSTN. That's rather non-optimal, however. It would be nice to be able to trim the switch hook response timings within Asterisk (which may be possible, but I've not found it yet) and then do the same on the SPA-3102, which would let me configure the SPA-3102 to interpret the flash before Asterisk... but that would mean that other feature codes would be inoperable. Catch-22. The other option is to let the SPA-3102 ring through to the FXS port, which should let CW work on the analog phones -- but that still leaves the SIP extensions in the dark as far as call-waiting. Otherwise, CWCID works with the analog phones, and the call quality is great throughout.

Aside from a few little pieces, the SPA-3102 is a great unit for the $90 price. The firmware is very complete in terms of tweaking just about every setting, and each port (FXO, FXS, SIP) can be treated as a unique entity, which makes it perfect for this purpose. I did upgrade to the latest firmware (5.1.5GWa as of this writing) without issue, though it requires a Windows system for the upgrade. The Web interface is a little wonky at times, but is otherwise fast and responsive. I especially like the fail-open configuration -- if power is out, the FXS port is bridged to the FXO port so normal calling is still possible.

I'm expecting another one of these for another PSTN line, and possibly a Digium Iaxy to play with. Suffice it to say, I'm getting into Asterisk in a big way.

Posted by Paul Venezia on December 27, 2006 05:07 PM

December 19, 2006 | Comments: (0)

Year end lab notes part 3: Thrilla in Manilla

And at last we reach the last installment of the year-end lab notes for 2006. The previous posts dealt with infrastructure, servers, and network gear. I'm going to add a few notes to those categories, and finish up with workstations, testing tools, apps, PBXes and ISPs.

Workstations
My main workstation is the homegrown Ultra 40 I built in May, running Fedora Core 5 x86_64. After six months of constant (ab)use, it's humming right along. At some point I'll update it to FC6, but not until it's absolutely required. I've found FC5 x86_64 to be stable and reliable, with only a few dependency glitches when installing packages with yum. I'm also using several quasi-compatible repos, so most of those are probably on me anyway.

The raw performance of this workstation is not to be dismissed. I'm routinely pushing the upper limits of the 8GB of RAM present in the system, as well as the disks. It's not just a pretty face, it's a solid workhorse.

As far as laptops go, with one exception, they're all G4 PowerBooks or MacBook Pros. I just received a new 17" MacBook Pro, and my initial impressions are positive. The screen is surprisingly light, and the hinges don't seem to be as tough as on my 15" PowerBook, which has caused a few issues, but performance-wise, it's a keeper. As Tom Yager told me awhile ago, once you get a 17" laptop, you can't go back. I think he's right.

The one exception is a Dell Latitude D800 running FC6. This laptop has definitely not had an easy life. It has traveled probably 50k miles in the air -- via FedEx -- as well as many thousands more a carry-on, been left running for weeks at a time in labs all over the country, and at one point in its life it was drafted into service as the core of an entire L3 city network when the production core switch developed a hardware problem. The embedded gigabit Broadcom NIC actually did surprisingly well at handling the load, as did the FC3 build that it was running at the time.

ISPs
My lab is a bit limited in terms of locations and available services. Actually, very limited. Cable is my only non-TDM option, so RoadRunner it is. In the three-plus years the business-class service has been in, it's been exceedingly reliable and fast. The prices are right, the bandwidth is guaranteed, and the reality is that I've found that nothing else is required. It would be nice to see FIOS or even DSL availability to provide a backup circuit, but given RoadRunner's track record, I'm not worried.

Testing Gear
Whenever I need to pull in some heavy network testing gear, I make a call to Spirent and get a SmartBits chassis with whatever modules I need. The SmartBits are powerful, reliable, and infinitely configurable, almost to a fault. For lower-end testing, I simply write small tools in perl, or use nuttcp, netperf, IOMeter, or any of a dozen other open-source network testing tools. On the storage side, IOMeter, bonnie/bonnie++ and good ol' dd are all that's really required. Also, I'm currently evaluating a Shunra VE network testing appliance, which I'm very interested in using for some upcoming tests.

Apps
I don't really have a stable of go-to applications, I tend to use whatever is available on whatever system I happen to be using. I do plenty of writing in OpenOffice.org and Microsoft Word, but also in vim. I do all of my coding in vim. I use Apple Mail, Evolution and Thunderbird just about equally, as well as SquirrelMail for a Web email client. Firefox and/or Camino are the browsers of choice, and iChat and Gaim handle IM nicely. I do have to give props to ecto, which is one app that I definitely rely on, since I do all my blogging from a MacBook or PowerBook, and ecto makes it obscenely simple.

PBX
If you've been reading my Asterisk posts recently, you know that the voice side of the lab is in flux right now. I'll keep up my notes on that as it progresses.

If anyone has specific questions on hardware or lab infrastructure, feel free to drop me a note and I'll address it. Also, in the very likely event that I've forgotten something, we might just have a part four.

Posted by Paul Venezia on December 19, 2006 06:01 PM

December 15, 2006 | Comments: (0)

More on Asterisk and the Cisco 7970

After some tweaking, I believe I have everything working the way I want it with my new TrixBox/Asterisk system. To recap, the actual Asterisk server is a Trixbox 1.2.3 build running on a VMware virtual server, linking to a SIP trunk from BroadVoice. Internal phones have now been expanded to include a few Cisco 7960 sets as well as the 7970.

Following my earlier posting about getting the 7970 to function with Asterisk, TrixBox founder Andrew Gillis sent me a note about the Endpoint Manager feature in TrixBox, which will create Cisco phone configuration files on the fly. In TrixBox 1.2.3, there's support for the XML configuration file format that drives the 7970's SIP firmware, as well as older phones such as the 7940 and 7960. I'd seen this in my initial perusal of TrixBox, but didn't immediately see that the configuration files were available anywhere, so I built my own. Of course, TrixBox assumes that it will be the TFTP server for the phones, so it dumps the configuration files into /tftpboot on the TrixBox server itself. Well, if I'd looked there to begin with, I wouldn't know half of what I now know about the configuration of these phones... and the phone setup would have been much quicker.

The configuration files generated by TrixBox for the 7970 will work (and seem to have cleared up the continuous ring problem, though I'm not sure exactly why yet), but still need manual intervention to configure some extended functions of the phone, such as multiple line and speed dial settings. Also, the Services and Phonebook menus on the 7970 wouldn't work -- they wouldn't even make a HTTP request to the server. It appears that with the <webAccess> parameter set to '1', the phone's internal Web server and client are disabled. Setting this to '0' seems to have enabled both functions, and the phonebook and services menus are now populated on the phone. In addition, there's a debug statement present in /var/www/html/cisco/services/PhoneDirectory.php that will cause XML parsing errors. Comment out the echo $Query statement near the bottom of that script to fix that problem. I also noted that Cisco has released a new rev of the SIP firmware for the 7970, version 8.2(0) Unfortunately, it seems flawed right out of the box, as the load file that comes with the firmware references the wrong filenames. Specifically, it begins the firmware upgrade process, and looks for a file named Jar70sip.8-2-0-55.sbn, but that file doesn't exist in the firmware zip file. Presumably, it's looking for the file jar70sip.8-2-0-55.sbn, which is present. I began to rename the file to permit the firmware upgrade to continue, but then thought better of it. If Cisco can't seem to get the names of their own files straight in a production firmware release, what other bugs might be in this new version? I punted and am still running 8.0(3).

As far as the 7960s go, they're a whole different beast than the 7970. The configuration file syntax is arguably easier, and they don't have the 7970's habit of going into a reboot death spiral when there are XML parsing problems on the configuration file. They also work flawlessly with Asterisk.

So at the moment I have five extensions on my Asterisk box with a single SIP gateway. I'm going to be dropping in a Linksys SPA-3102 to act as both an FXS and FXO analog adapter for a regular POTS line, and run through the dialplans on both that and the Asterisk box to bring at least one POTS line into the mix, along with an analog phone. If that goes well, the other lines will get the same treatment. The SPA-3101 seems to have the right featureset, including internal dialplans to shift calls from the analog phones to Asterisk first, then the connected POTS line should the Asterisk server be unavailable. Also, in the event of power loss to the SPA-3101, it will fail open, providing the analog phone with a passthrough to the POTS service. Hopefully, it will arrive early next week.

In a perfect world, I'll have everything running through Asterisk with several POTS lines and at least one SIP trunk, a smart dialplan to prefer certain trunks over others depending on the number dialed, and failover to POTS. It'll be a little bit of a project, but every step of the way, I'm more impressed with Asterisk and TrixBox.

Posted by Paul Venezia on December 15, 2006 02:42 PM

December 11, 2006 | Comments: (0)

Year end lab notes part 2: Rumble in the Jungle

Continuing the year-end series I started a few days ago, I'm going to detail some of the networking and infrastructure components of my lab, with special emphasis on what I rely on day after day.

Networking
I've been reviewing Dell's PowerConnect switching line since its inception a few years ago, and have yet to be disappointed with one of their switches. They're not top-end Cisco gear, but they also don't carry top-end prices, and that means quite a lot in both lab and SMB infrastructures. Make no mistake, there's some big Cisco iron in the lab as well, including a 6509, and a 4506, switch. Those switches are used in production as well as forming the baseline for all switch testing. However, as part of my testing of the newest PowerConnect, the 6248 (look for the review in an upcoming InfoWorld issue), I pulled the 6509 out of the core, replacing it with the 6248. The lab network is fully L3 switched, so this put the full weight of the lab through the new Dell L3 switch, and I have yet to pull it out. I might even leave it in place for awhile. Since I did this a month or so ago, the new core has switched petabytes of data without missing a beat. The long-term testing I've been doing with older PowerConnect switches has also gone well -- I haven't had a single hardware problem with any of those switches to date, from the 6024 to the 3424P that runs PoE to all the phones.

The configuration file syntax has changed significantly with the 6248 vs its' predecessor, the 6024. Gone are the scattered commands and repetitive port configurations, and in their place is more in line with true Cisco IOS configs, with each port granted a separate place in the config for individual parameters. It's much, much better.

The Cisco gear hasn't had an easy life, like most of the lab gear, but lives up to Cisco's reputation for reliable, dependable, high-performance switching products. As a general rule of thumb, you can't go wrong buying Cisco switches -- even though you might spend a bit too much. Both the Cisco and Dell switches coexist peacefully, with PAgP aggregate links and 802.1q trunking. No muss, no fuss, it just works.

Wireless
There isn't a significant WiFi implementation in the lab, just a few Apple Airport Extremes and actually a few Linksys 802.11G units. They work well and are largely set-and-forget.

Firewall
The entire lab is firewalled by IPCop 1.4.10 running on an elderly Dell Optiplex GX110. It's a very small footprint workstation-class system with a PIII 667Mhz CPU, 128MB of RAM, a few 10/100 NICs and a CF-to-IDE adapter with a 512MB CompactFlash card. No moving parts other than the CPU and case fans, and voila, a stable, reliable, configurable, open-source firewall. IPCop really is extremely easy and featureful. Currently, my IPCop box is doing some rudimentary QoS for SIP calls, providing an OpenVPN endpoint for when I'm on the road, and running several nailed-up IPSec VPN connections to a variety of other gear, including PIX firewalls and SonicWall firewalls. This system has been in place for five years now without missing a beat. I even had time to put together a gkrellm package for IPCop to let me watch network I/O and stats on the firewall in real-time from my workstation.

Racks, PDUs and UPSes
American Power Conversion (APC), all the way. The abuse that the racks in the lab receive is generally far beyond what any normal production infrastructure would endure, with servers being racked, unracked, and re-racked on a constant basis. The APC enclosures that I have in the lab withstand all of that without even a scratch (really). There's plenty of sidewall cable-routing space, and the 0U networked PDUs make remote powercycling simple.

The UPSes are all APC as well. I don't have a hard-wired UPS, so the lab is served by several SmartUPS 2200XLs and a few smaller units. I do have several dead UPSes still hanging around, including a few older APC models whose batteries gave up the ghost, but the Tripp-Lite unit is a doorstop now, even though the batteries aren't dead, as are the lower-end Belkin units. In addition, when the generator kicks in, it tends to produce occasional harmonics in the lab power service, which none of the other units dealt with very well. The APC units would trigger alarms during these instances, but with the easily-adjustable sensitivity control at the rear of the units, it became a non-issue. Also, the SNMP support is great, providing my Cacti-tweaking habit with plenty of fodder.

As far as the generator goes, it's a Guardian LP unit, straight from Home Depot, with a 200 amp automatic transfer switch. It's saved my bacon several times, though it hasn't been called into service in 10 months or so. Winter's definitely coming, however, and so I have every belief that it'll see some action soon. In the interim, it runs weekly 10-minute exercise cycles that have been keeping it in shape.

KVM
I just installed a Raritan Dominion KX432 to take over for a few smaller Raritan units, including a 10-year-old Master Console IIx. The Dominion series' density is high, the physical units are small, and they have all the features I need in an IP KVM, all contained in a single unit that doesn't require an additional physical server like the Avocent units. Their Java-based console app seals this deal, since I work on Linux and Mac OS X workstations. I've found that the console app can be somewhat flaky over higher-latency connections, but still usable, although working with the networking prefs can improve performance. As a front-of-rack KVM, the Dominion is as solid as any that Raritian has produced... and my recently-retired Master Console IIx can attest to that reputation.

The next part of this series will be workstation hardware and OSes, monitors, phones, and the rest of the little bits. Stay tuned.

Posted by Paul Venezia on December 11, 2006 06:27 PM

December 08, 2006 | Comments: (0)

Asterisk, VMware, a Cisco 7970 and a few hours

Since I seem to be feeling rather talkative today, I figured I'd post a bit on the latest little bit of geekery I committed this week. All noted configuration file examples can be downloaded from here.

I've had a Cisco 7970 running SCCP firmware in the lab for a little, but suddenly it lacked a Cisco CallManager box to talk to when I shipped out some test gear. The 7970 is simply a gorgeous phone (although the SIP code could use some help) and I have a deep-seated requirement to use a cool piece of hardware whenever possible, I decided to hook that phone up to my Broadvoice SIP account instead of my Zyxel 2000W V2 WiFi SIP phone. Theoretically, it should have been simple -- Broadvoice offers a TFTP server for Cisco 7940 and 7960 phones, so it should just be a matter of telling the phone where to find a config file and voila.

Not so.

The 7970 changed all the rules of Cisco IP phone configurations. The new config format is XML, versus the prior variable=value style configuration. This presented a problem. I could take some time and whack away using guessed values for the phone's configuration until I was able to peer with my Broadvoice account, or I could set up an Asterisk system in the lab to switch that phone and the Zyxel, and peer with Broadvoice for outbound calling.

It certainly wasn't as simple as it should be, but it's all working -- well, most of it. At the moment, if a call comes into the SIP line, both phones (extensions) ring as they should given the call group configuration. The Cisco phone, however, never stops ringing. If left unanswered, it will bleat until someone gets annoyed enough to "answer" the non-existent call. I haven't fleshed that bug out yet, but then, I've spent a total of two hours on this since Tuesday, so I'll consider myself ahead of the game.

There's lots of bits and pieces of 7970 information floating about on the Internet. Kerry Garrison's excellent Cisco 7970 tutorial was brief, but formed the kernel of the working configuration I now have. Configuration file hacking on the 7970 isn't all it's cracked up to be. I've found several interesting tidbits supporting this, such as the fact that the phone will not error noticeably during boot when it fails to parse a configuration file pulled from the TFTP server, it will just use the last config file it had. This gets interesting, since it fails to correctly parse an XML configuration file where the <phoneLabel> tag contains a value with a space character in it. Apparently, you can only name your phone Cher, Prince, or Twiggy. When this occurs, the phone does pull the file from the TFTP server, so packet sniffing will still show the file transfer, but there's no quick way of knowing if the phone is running from a cached or current config. Also, apparently some phones and/or firmware revs will request files with different capitalization from the TFTP server at boot. XMLDefault.cnf.xml, XmlDefault.cnf.xml and xmlDefault.CNF.XML may appear similar to the casual observer, but they're very different files.

I really like the 7970, though. The voice quality is superb, and it just looks fantastic on a desk, but the SIP firmware needs some polish, including a functional message waiting light. Since Cisco isn't so interested in keeping up with the SIP firmware vs. their proprietary SCCP version, that might take awhile.

But back to my PBX build. I downloaded Trixbox 2.0 beta 1 at first, but had problems with the phone connecting to the assigned extension. Backing off to v1.2.3 with Asterisk v1.2.12.1 brought the phone up properly, but that was only after playing with different 7970 SIP firmware revs, settling on version 8.0.3. Trixbox is just dead easy. Download the CentOS-based ISO, install, log in. Beautiful.

Loading firmware on the 7970 is interesting, as the phone will always query a TFTP server at boot, and will load a new firmware image from the same server if the XML configuration file includes a version that the phone isn't running. It's a relatively odd way to do it, but it does work once you've figured out exactly what the phone wants to see in that file, and have determined that the Cisco firmware release .cop file is just a tar.gz. Toss everything in that file into your TFTP root, reference the version.loads file in the phone's XML configuration (without the .loads) and reboot the phone. It should update the firmware on the resulting boot.

So with the mix of Trixbox 1.2.3 running as a VM on a VMware Server host, and Cisco 7970 SIP firmware 8.0.3, I had a functional system. Now, all I needed was to connect with Broadvoice to seal the deal. They do provide instructions on their website for configuring Asterisk, but they appear to either be wrong or outdated. They're close, but not all the way there. My working sip.conf entry for Broadvoice is simple:

pedantic=no [Broadvoice] username=603XXXXXXX user=phone type=peer secret=que? insecure=very host=sip.broadvoice.com fromuser=603XXXXXXX fromdomain=sip.broadvoice.com dtmfmode=inband dtmf=inband context=from-pstn canreinvite=no authname=603XXXXXXX

The registration line is too:
register=603XXXXXXX@sip.broadvoice.com:que?:603XXXXXXX@sip.broadvoice.com/603XXXXXXX

Their docs refer to an extension designation at the end of the registration string which is puzzling. Using the phonenumber/accountID does the trick. Also, they reference context=from-broadvoice when it apparently should be context=from-pstn to permit proper inbound calling to function. I'm not entirely sure of the background here, and my Asterisk experience is limited to the few hours I had this week, so I might be missing something.

When configured as a trunk via FreePBX or manually, this configuration should work -- it does for me. You may need to add the pedantic=no to the top of sip.conf. Also, if you have a currently registered device with Broadvoice, you might want to leave it off for an hour or more before trying to peer with them from a new device or PBX to let the previous connection timeout. This is pure rumor to me at the moment, however. YMMV.

My working 7970 SIP configuration file, SEP0014DEADBEEF.cnf.xml, is based on Kerry Garrison's example with a few modifications, including NTP sync parameters. It references the 8.0.3 firmware rev in the <loadInformation> tag. Obviously, this file will need to be renamed to use your phone's MAC address.

All told, I'm a little unsure of where to take this from here. I can call between extensions, ring in and out from both, and I will probably add another Cisco phone, probably a 7960, to the mix shortly, but the future is slightly uncertain. I don't trust this setup enough to move production voice lines to it, especially with the little continuous ring problem, but it definitely works well enough for casual use at the moment. I spent 30 minutes chatting with InfoWorld Editor-in-Chief Steve Fox through it the other day and nearly an hour with Oliver Rist tonight, and the call quality was nearly perfect with no discernible lag or dropouts, which was a shame since I can only take so much of Oliver at one go. All this was with no QoS anywhere, least of all my egress points. Not bad at all.

Oh, and if you create a directory off the TFTP root named (at least on this rev) /Desktops/320x212x12 and toss in some .png files and a List.xml file referencing them, you can put custom backgrounds on your phone. Hip.

Posted by Paul Venezia on December 8, 2006 05:39 PM

December 08, 2006 | Comments: (0)

Year end lab notes

Over the next week or so I'm going to be discussing what's in my test lab, both the good and bad, and what I use on a daily basis vs what gets shelved. In any working lab -- especially one testing all sorts of new gear -- there's a foundation. That foundation includes racks, UPSes, KVMs, switches, servers, and basically everything that a standard datacenter would have, but all geared to be modified and changed constantly to facilitate hardware and software testing.

In essence, this means that the foundation lab gear gets much more abuse than standard datacenter gear -- for instance, the racks are constantly changing, with new gear getting racked and older hardware re-racked, switch configurations change all the time, and servers are built with multiple operating systems more times than I can count. In short, over and above the hardware and software that gets reviewed, there's some really rough real-world testing occurring constantly, and that part of things rarely gets any press. So, I'm going to start that right now, starting with servers, storage, and operating systems.

Servers
There's an old lady of the lab. Actually, there's two, but the other one is a switch that I'll get to in a later post. The elderly yet rock-solid HP ML370 sitting in the corner of the lab has proven to be a wise investment. It's a G2 model, two 2.8Ghz Intel XEON CPUs, a few gigabit NICs and 4GB of RAM. There's no RAID controller, just a mix of 18GB, 36GB, 72GB, and 147GB drives in the six U320 drive bays. It began life running Red Hat AS 2.1, and was then upgraded to RHEL3 quite some time ago. It hasn't been upgraded since due largely to lack of time and need. Among other things, it was running GSX Server for over a year, just recently upgraded to VMware Server 1.0. There are only a few VMs running on it, but they include the lab's main domain controller, and currently a VirtualCenter 2.0 server. Otherwise, this server is used to generate loads of varying types to hardware under test, from Web loads to SMB torture testing, network throughput testing, and on, and on. I'm probably jinxing it now, but in the several years it's been running, it hasn't hiccuped once, nor lost a disk. Also, it's never been backed up.

Hmmm. I'll be right back.

Okay. I feel slightly better now that it's backing up to nearline storage.

The other lab stalwarts aren't quite as aged as the ML370, but are put under heavy load constantly and have again proven to be 100% reliable no matter what the conditions. These servers include a Newisys N4300 with four dual-core Operons, an HP DL585 G1, also with four dual-core Opterons, and a SunFire T2000 running the Sun SPARC T1 eight-core processor. Each of these servers has proven their worth over and over throughout the time they've been chained to the proverbial lab plow. The DL585 in particular has been a champ, reassuring me that the Server class Technology of the Year award I gave it last year was well earned. That server has run nearly every modern OS and functioned as a virtualization platform under VMware and Virtuozzo, and served very well under extremely heavy load and focused attempts to break either it, or the software running on it. There are other servers in the lab, of course, but these three stand out as true workhorses. If I'd written this post yesterday, I would have included my Apple XServe in this list, but after spending a month without being run, it suddenly refuses to power up.

And, oddly, a workstation is in the mix. The lab's DNS, DHCP, and PXE infrastructure is all centered around a six-year-old HP Karak XU800 with a single PIII 866Mhz CPU, a pair of software mirrored 40GB PATA drives and 512MB of RAMBUS RAM running Fedora Core 3. It's simply a case of ain't broke, don't fix it. By serving as the PXE TFTP host, that box has launched a thousand OS builds, if not many more.

Operating Systems
The lab is built mainly on three OSes. Four, if you count VMware. In no particular order, they are Linux, FreeBSD and Mac OS X. The Macs are the workstations in the form of several laptops and a few G4 PowerMacs, and Linux and FreeBSD power just about everything else, from storage, mail, DNS, DHCP, testing platforms, and so forth. Windows boxes are built as needed for testing, but none remain as permanent fixtures other than two that run under VMware, providing AD services for a few other systems. There are disk sets with Windows x64 builds for the Opteron servers, but they're only called into play when the test calls for it. Otherwise, it's Linux or FreeBSD.

Storage
The main point of storage in the lab at the moment is a SNAPServer 18000 with around 1.5TB of SATA storage. It's not the fastest storage device out there, nor the most elegant, but it works, and it works well. Accompanying it in the lab right now are a NetApp StoreVault with around 3.5TB of SATA storage, and a EqualLogic PS3800XV with a few TB in SAS drives. Ever since they posted the best numbers in the six-way iSCSI test I ran last year, EqualLogic has impressed me, and this unit seems to share the high points of the PS200E I tested with the added bonus of more space and faster disks. The full review of this unit will be in an upcoming InfoWorld issue.

Also driving some disk in the lab is a custom-built server running a 3Ware 9500S-8 SATA controller with around one TB of storage. That's running Fedora Core 3 and currently holds all the OS installer roots, updates, various testing tools, and more ISO images than you can shake a stick at.

So there you have a little insight into the inner workings of my test lab. Still to come are my notes on racks, UPSes, KVMs, switching, routing, firewalls, and more. Stay tuned.

Posted by Paul Venezia on December 8, 2006 06:24 AM

September 30, 2006 | Comments: (0)

A prayer for my modem

So after years of diligent service for several masters, my sister's iBook G3 800 finally died. It was quick and painful. I ordered her a new MacBook with a 100GB disk and 1GB of RAM, and it became the first x86 Mac I've used to date (I'm rather stodgy that way sometimes, and am waiting for the Merom upgrades to the Mac Book Pro line). Aesthetically, this MacBook is quite charming, and certainly fast. However, after booting it up past the initial setup, software update ran... and required 664.5MB in updates to be downloaded. This included were everything from a few small security fixes to the iLife updates, Java, and the 10.4.8 update (300MB all by itself).

Luckily, I have a 10Mb connection, and I routinely get 1.2MB/s from Apple, so the updates are flying down as I type this, but God help you if you're on dialup. That's just under 28 hours with a solid 56k connection.

Posted by Paul Venezia on September 30, 2006 03:35 PM

August 15, 2006 | Comments: (0)

Sound MPLS route redistribution

I usually file things like this under "duh", but a recent conversation with a network architect made me think that it's worth noting in public space.

I recently had a problem related to dynamic route distribution in an MPLS environment. Several remote sites were served via a Paetec MPLS network, all sites falling under a common major net, 192.168.0.0/16, with DMZ and edge networks built on another major private net. In a frame-relay or PTP network, routing would easily be handled by EIGRP or OSPF, or even static routes, and moving subnets from one site to another would be as simple as referencing them in the routing table of a core switch or router at the target site. In an MPLS environment, however, the routers connecting the LAN to the MPLS network are controlled by the provider, and the nature of MPLS dictates that these routers have explicit routes that are propagated among the participating MPLS routers via BGP.

The problem was that in order to be able to move subnets around between sites, for instance in order to accommodate the failure of an inbound Internet circuit at one site by moving VPN subnets to another site, the provider would need to be in on the configuration, and that can hardly be considered a workable proposition in an emergency. So, I built an OSPF area at each site that was redistributing connected and static subnets, and had the MPLS provider configure their router to participate in that area, and redistribute routes learned via OSPF into BGP. With this in place, adding a route statement to any router at any site causes that route to be propagated throughout the MPLS network, and then into the OSPF area at each site, including defroute information. Turning the routing around now is absurdly simple and doesn't require the involvement of the provider at any level. Prior to this, all routes were static, with subnet of the major net assigned to each site, such as 192.168.0.0/20, 192.168.64.0/20 and so on. Now, it's as granular as necessary.

If you're going to undertake this configuration, be sure that you properly assign loopback interfaces on the core routers and switches with the highest IP in the subnet assigned to that site to ensure that it becomes the DR/BDR interface and the status of any physical interface doesn't interfere with proper OSPF operation.

Posted by Paul Venezia on August 15, 2006 06:14 PM

June 03, 2006 | Comments: (0)

Where is that code anyway?

More than a few readers sent me notes asking about the availability of the code that I mentioned in last week's Enterprise Hacks feature. I wrote two pieces in that article, the first one regarding timezone settings for thin clients, and the second on using Perl, PHP, and MySQL to map Windows shares and permissions across an enterprise network. The code for the first is easy, since I posted the code back on my blog back in 2003.

The share mapping code will be trickier. For one, it's fairly involved, including Win32 and Linux-based perl scripts, PHP scripts, and a database schema. Also, I'll have to ensure that the code is distributable. If I can get all these pieces together for that app, I'll post it here, but be forewarned -- it was written for one specific purpose, and for one specific network. It should be easily portable to other networks, but it'll have to be screened for security information and probably dressed up a bit. Stay tuned -- I can't promise anything, but I'll try.

Posted by Paul Venezia on June 3, 2006 09:54 AM

May 23, 2006 | Comments: (0)

OpenVPN for IPCop

Since I was playing around with my IPCop firewall anyway to do the gkrellmd work, I decided to upgrade it to 1.4.10 and install the ZERINA OpenVPN addon. Even though this isn't an official IPCop addon, it works very well, has a simple installer, and integrates very nicely with the IPCop Web UI. After generating all the PKI information, including the client certs, I installed Tunnelblick 3.0RC2 for OS X on my PowerBook. The OpenVPN addon is so complete that it will actually generate a zipfile containing a valid OpenVPN configuration for connecting to the firewall as well as the client PKS12 certificate right from the IPCop Web UI. I pulled this down, tossed it in ~pvenezia/Library/openvpn and fired up Tunnelblick. No go on the first try with a rather bizarre error claiming "unroutable packet received" from the IPCop system. Then I realized that the time on my firewall was off by over an hour, which would cause problems with the certs. I set the time and configured NTP time sync, and tried again. Bam -- instant secure access with more than a bit of panache. For those running Windows, check out the nicely detailed howto, including Windows client setup.

Posted by Paul Venezia on May 23, 2006 12:09 PM

May 22, 2006 | Comments: (0)

gkrellmd for IPCop

The thought occurred to me the other day that it might be cool to have a gkrellm monitor on my main workstation displaying throughput on my IPCop firewall. I couldn't find a gkrellmd addon for IPCop, so I put one together. This is based on gkrellm-daemon 2.2.5 and includes the necessary glib2 2.4.7 libraries.

Instructions are in the tarball, but essentially you just tar zxf gkrellm-server-ipcop1.4.tgz at the root dir of your IPCop system, edit /etc/gkrellmd.conf and /etc/rc.d/rc.local to run /usr/bin/gkrellmd -d. It's been tested on IPCop 1.4.10, but should work on any 1.4 release. You can monitor all the normal info, as well as specific IPSec connections.

Grab gkrellm-server-ipcop1.4.tgz here.

Posted by Paul Venezia on May 22, 2006 04:14 PM

May 18, 2006 | Comments: (0)