Geeks in Paradise | Brian Chee » Asking the right question

June 15, 2006 | Comments: (0)

Asking the right question

TAGS: Testing notes

When you design a test you always have to keep asking yourself: "are you asking the right question?" We just did a quick test of an aggregating tap for a friend...

Whenever you dive into testing, you really need to keep your eye on the ball. Are you really asking the right question? Yesterday we had a visiting professor bring us an aggregating copper tap and asked us to either confirm or deny its operational specifications. My first question to her was: "just what are we trying to prove or disprove?" which in turn led to about a two hour discussion on just how taps work, the overhead inherent to layer 2 versus layer 3 traffic, switch overhead (VLAN Tags, ospf, etc) and a general discussion on trying to determine the correct questions to ask.

In her case, she was interested in proving/disproving that aggregating taps aren't the magic wand that network forensic folks think they are. She went on to say that many auditors just slap a tap into place and think they have "Perry Mason" working for them. However, considering that in this case we had a full duplex 100mb/sec tap aggregating bi-directional traffic onto a single instrument port she postulated that there existed circumstances that would put this device into question.

We started asking whether latency was an issue, since increased latency would also change the relationship of time stamps on those packets and we also had to ask about just how was the buffer in that passive tap handled. So the question came down to asking at what point did the tap start dropping packets, and as the load was increased, what did that do to latency?

So we fired up our spanky new Spirent Test Center and dropped three ports down to 100mb (gig capable) and with the help of the Spirent Honolulu office engineering staff we figured out how to get the ports to light up without a switch in the mix. (We didn't want a switch to add buffering and flow control, nor did we want the extra latency)

The good news is that the tap did work as expected. Increase the frame size, rate and duration and in a relatively Logarithmic fasion the tap started dropping packets and latency increased.

However, the right question wasn't whether the tap would drop packets once the sum throughput in both directions exceeded 100mb/sec going to the test instrument; but rather what is the load curve so that we could predict when the forensic device could no longer be trusted to have captured the evidence. Her premise was that under certain conditions you can no longer trust that the aggregating tap is giving you a reliable picture of the flow under forensic study.

We also started asking just how we'd actually be able to tell that the forensic study was in trouble. Switch statistics tend to fall into a couple catagories with sFlow and NetFlow being the two 1000lbs gorillas. Each is based upon sampling at different points in the switch and sample under different conditions. The question we started asking was in regards to how fast switch traffic could burst past the tap's oversubscription rate and whether sampling statistics would be able to catch the event. Which then led to a discussion on just what is good enough?

Anyway, I'll save that for a bit later since we started a whole new discussion on how much we can trust network forensics when it starts coming down to compliance. The base problem is that the courts have yet to determine what is good enough and Sarbanes-Oxley and its ilk really haven't determine that either.

So I'm really looking forward to her academic paper on this, and I was really glad that Neal Allen of Fluke Networks sent her in my direction. Neal is one of those folks that likes to ask the hard questions about network devices. As an example: he was the person that pointed out to me the effects that a 10mb/sec half duplex device can do some mighty strange things to your fancy new 100mb/sec or 1000mb/sec network.

TTFN...
/brian chee

Posted by Brian Chee on June 15, 2006 10:54 AM

RATE THIS ARTICLE: