Geeks in Paradise | Brian Chee » June 2006

June 15, 2006 | Comments: (0)

Asking the right question

TAGS: Testing notes

When you design a test you always have to keep asking yourself: "are you asking the right question?" We just did a quick test of an aggregating tap for a friend...

Whenever you dive into testing, you really need to keep your eye on the ball. Are you really asking the right question? Yesterday we had a visiting professor bring us an aggregating copper tap and asked us to either confirm or deny its operational specifications. My first question to her was: "just what are we trying to prove or disprove?" which in turn led to about a two hour discussion on just how taps work, the overhead inherent to layer 2 versus layer 3 traffic, switch overhead (VLAN Tags, ospf, etc) and a general discussion on trying to determine the correct questions to ask.

In her case, she was interested in proving/disproving that aggregating taps aren't the magic wand that network forensic folks think they are. She went on to say that many auditors just slap a tap into place and think they have "Perry Mason" working for them. However, considering that in this case we had a full duplex 100mb/sec tap aggregating bi-directional traffic onto a single instrument port she postulated that there existed circumstances that would put this device into question.

We started asking whether latency was an issue, since increased latency would also change the relationship of time stamps on those packets and we also had to ask about just how was the buffer in that passive tap handled. So the question came down to asking at what point did the tap start dropping packets, and as the load was increased, what did that do to latency?

So we fired up our spanky new Spirent Test Center and dropped three ports down to 100mb (gig capable) and with the help of the Spirent Honolulu office engineering staff we figured out how to get the ports to light up without a switch in the mix. (We didn't want a switch to add buffering and flow control, nor did we want the extra latency)

The good news is that the tap did work as expected. Increase the frame size, rate and duration and in a relatively Logarithmic fasion the tap started dropping packets and latency increased.

However, the right question wasn't whether the tap would drop packets once the sum throughput in both directions exceeded 100mb/sec going to the test instrument; but rather what is the load curve so that we could predict when the forensic device could no longer be trusted to have captured the evidence. Her premise was that under certain conditions you can no longer trust that the aggregating tap is giving you a reliable picture of the flow under forensic study.

We also started asking just how we'd actually be able to tell that the forensic study was in trouble. Switch statistics tend to fall into a couple catagories with sFlow and NetFlow being the two 1000lbs gorillas. Each is based upon sampling at different points in the switch and sample under different conditions. The question we started asking was in regards to how fast switch traffic could burst past the tap's oversubscription rate and whether sampling statistics would be able to catch the event. Which then led to a discussion on just what is good enough?

Anyway, I'll save that for a bit later since we started a whole new discussion on how much we can trust network forensics when it starts coming down to compliance. The base problem is that the courts have yet to determine what is good enough and Sarbanes-Oxley and its ilk really haven't determine that either.

So I'm really looking forward to her academic paper on this, and I was really glad that Neal Allen of Fluke Networks sent her in my direction. Neal is one of those folks that likes to ask the hard questions about network devices. As an example: he was the person that pointed out to me the effects that a 10mb/sec half duplex device can do some mighty strange things to your fancy new 100mb/sec or 1000mb/sec network.

TTFN...
/brian chee

Posted by Brian Chee on June 15, 2006 10:54 AM

June 14, 2006 | Comments: (0)

A bit of ANCL History

TAGS: Lab Freaks

The Advanced Network Computing Laboratory (ANCL) was founded in 1995 by a combined effort of Brian Chee, Oliver Rist and Wayne Rash while we were all with Communications Week Magazine. I was serving with the GSA Office of Information Security (secure data+video+voice) and met with Wayne Rash for drinks while he was covering CommNet. Now at our permanent home with the University of Hawaii School of Ocean and Earth Sciences and Technology (SOEST) the Advanced Network Computing Laboratory is one of the largest labs in the InfoWorld stable, while also giving university students real world experience with cutting edge technology.

Oliver Rist just happened to join him and being a wonderful October in Washington DC we all got rained upon. While escaping into a hotel bar we had a conversation about wanting to do comparitive product reviews in Hawaii while sipping Pina Coladas under a coconut tree. (thanks Oliver) The short version is that I played straight man to Oliver's wise crack and soon I was calling a friend at the University of Hawaii.

Oliver then started looking seriously at this concept and after doing LOTS of spreadsheets on the actual cost of doing reviews, found that doing testing in Hawaii wasn't a boondoggle afterall. His cost comparison actually showed a significant cost savings over alternative sites elsewhere in the US. Approximately 8 months later we rolled in the door of the UH Computer Science department with $8 million bucks worth of ATM gear from folks like FORE Systems, Cabletron, NewBridge and 3Com. This set the stage for an arms race between magazines on who could do the biggest and baddest comparitive reviews in the industry.

10 years later we've gone from Communications Week to Internet Week and now InfoWorld. The really cool part is that InfoWorld's management has given the Reviews group the leeway to continuously set the pace for large scale enterprise simulations. From the very first ATM shootout at Communications Week, to the current Triple-Play simulation (data+video+voice) in the current 10gig Enterprise Switch shootout, ANCL and InfoWorld are leading the pack.

Speaking of which, our goal is to create simulations that are solutions based rather than speeds and feeds. No IS manager in their right mind would dare run their infrastructure anywhere near capacity, and we've heard the feedback. We actually send out test invitations that closely resemble a request for proposal that our readers might send. We fine tune it with feedback from the vendors to create the proverbial level playing field. We also work in close cooperation with the test equipment vendors to design the test methodology (now published in Lab Rat Magazine) based upon statistics sent by readers and associates around the world. We take this "real world" information and do "what if" scenarios sized to the target audience. (Large enterprise, SMB, SOBO, SOHO, etc) We consistently make every effort to mix in real data streams (ie. broadcast quality MPEG, SIP PBX's, etc) along with the synthetic data. This real data approach gives readers a way to identify closer with our simulations. We also tend to layer in the data streams all at the same time instead of one at a time like other reviews. Afterall, who runs ONLY data or ONLY voice on their network now? The watch word is convergence and we follow trends that our readers follow.

So, do you have a scenario that you'd like to share? We'd love to hear about it. Willing to share your firewall stats, ACL's or router stats? We'll keep it confidential and add it to the data we use to size our simulations. Have a bone to pick? Let me hear it, and believe me..we'll listen. Hey, we've even taken RFP's from some friends and ran with it. The point is that we can't possibly experience the world's variety of enterprise configurations and would really love to hear from you (either publically on this blog or privately and I'll remove company references if you wish)so that we can continue to help you hack away the spin and get down to reality.

Heck, we even have some special agreements with equipment vendors to see their product roadmaps under NDA so that we can tune our editorial calendar to coincide with new product releases. We scoop the rest of the pack because we listen.

Feel free to drop me a line. If you make it to the Interop Trade Show, come visit me at the NOC...I've been on the NOC team since 1995 and beebop between the wireless team lead, addressing (yup, a class "A" is a challenge) voice+video, etc...I also strongly encourage you to spend a bit of time at the InteropLABs where we demonstrate "not yet baked" technologies, and give you the chance to play around and talk to the folks that most likely sit on the standards committees.

/brian chee

Posted by Brian Chee on June 14, 2006 11:01 AM

June 08, 2006 | Comments: (0)

Hi. My Name is Brian

TAGS: Lab Freaks

I founded the lab about 10 years ago in hopes of creating an environment where Computer Science students could get some industry experience to balance their ivory tower curriculum. What a concept, students that can be immediately valuable to a corporation, but with the firm science foundation with which to innovate.

The Advanced Network Computing Laboratory (ANCL) creates enterprise simulations based upon data sent by friends and associates around the world. We then use these examples to do "what if" scenarios sized to the target audience. Our main goal is to put ourselves into the shoes of our readers and to sweat the details of keeping the comparisons on a level playing field.

A bit of history...
I got involved pretty early in networking with a job as a student helper on the AlohaNET project run by Norman Abramson, PhD at the University of Hawaii. This DARPA funded project produced several papers, one of which was used by Bob Metcalf of Xerox PARC as the basis for Ethernet. I finally graduated and went to work for Xerox as an interfacing specialist, then moved on to a regional distributor to become one of the first ten Novell Certified Instructors outside of Novell itself. With stops along the way with folks like Fujitsu, I landed at the General Services Administration Office of Information Security in the '90s doing secure data/video/voice communications systems. I'm now the Director of the Advanced Network Computing Laboratory at the University of Hawaii Manoa Campus in the School of Ocean and Earth Sciences and Technology (SOEST). As the largest research unit at UH I get to work on projects like upgrading the fiber optic networks on our research ships (www.soest.hawaii.edu/agor26) like the Kilo Moana (top 10 most fantastic vessels in the world according to the Discovery Channel's SuperShips show ( http://www.parthenonentertainment.com/Pages/Programs/In%20Production/superships.htm ) and suboceanic cables connecting underwater sensor nodes back to the university.

Posted by Brian Chee on June 8, 2006 11:44 AM