IT Troubleshooter | Harper Mann » December 2005

December 15, 2005 | Comments: (0)

Catching Sasser and Zotob at InteropNET with Network Physics

TAGS: Networking

Dwight Barker, Director Product Management from Network Physics shared with me today his story of looking for virus attacks across InteropNET. Dwight is part of the volunteer NOC team and has been busy the past few days tracking all the TCP flows from begining to end across the network. Using the Network Physics MP2000 appliance running the Netsensory OS, he's been able to time stamp a bunch of key metrics along the way and track failed connections and potential viruses.

Many of the machines running on the InteropNET, including servers on InteropNET and the laptops exhibitors and attendees bring to the show are typically behind firewalls and virus scanning systems when on their home networks. When these machines connect to the network at Interop, they're now on the public Internet without any protection and are vulnerable to attack.

Worms like Zotob, blaster and Sasser, exploit well known wide area file services on ports 135, 139 and 445 ports in Windows servers and clients. Once a worm gets on a system, it really only knows its own IP address. So worms start their attacks by blasting out attempted connections to any and all subclass addresses on the network.

In Dwight's case today he noticed a suspect with address 130.128.80.60 attempting and failing connections on many 130.128.x addresses. He was able to use one of the built in templates, or what the company calls "Insights" to look for and chart extreme spikes in attempted connections.

Each of the show booths at Interop are assigned a business group in the Network Physics system by IP address, so the spikes can be viewed by exhibitor. Dwight can aggregate the failed connections over time and catch the spikes that indicate denial of service attach or vulnerability attack. In one case a single IP was flooding the network port 135. The offending IP address turned out to be a laptop in a vendor booth infected with a Sasser worm. The InteropNET team tracked down the booth by IP address and to the exhibitor's surprise should up to find the specific laptop and clean it.

During the course of two days, the InteropNET team caught five infected machines and walked the aisles to clean machines.

Posted by Michael Baum on December 15, 2005 07:03 PM

December 14, 2005 | Comments: (0)

The IT Troubleshooter @ Interop NYC

TAGS: IT Community, IT Community, Networking

This week the IT Troubleshooter is hanging out at the InteropNET network operations center (NOC) in New York. You can check out the NOC through one of the four live webcams. InteropNET is the the world's largest temporary network, built as a living lab of best-of-breed technologies. Fourteen vendors including AGN, APC, Avaya, Citrix, Computer Associates, Cyclades, Extreme Networks, Fluke Networks, Infoblox, Network General, Gigamon Systems, Juniper, Splunk and Quest have commited gear, software and people to run internet access for the Interop trade show.

The NOC consists of twenty crack system administrators and a handful of help desk support staff working together to set-up, manage and troubleshoot the InteropNET systems. A total of 60 people make up the complete InteropNET team.

The network installation starts on-site at 6:00 am Friday and is delivered by Monday in time for the beginning of the show. Troubleshooting in this type of just-in-time environment can be pretty arduous given the timeframes and the range of technologies to integrate. The network uses 6000 feet of fiber and 20,000 feet of CAT 5 cable. The 802.11a/b/g wireless network has to cover about 200,000 square feet. Internet access to the outside world is a 45Mb DS3 circuit. To read more about the InteropNET mission and the journey towards it's exisitence read an interview with Glenn Evans, Interop's Lead Network Engineer.

These savvy system administrators have some great stories about cutting edge troubleshooting problems and techniques. Over the next few days I'll be bringing you some of their tales. Stay tuned.

Posted by Michael Baum on December 14, 2005 09:27 AM

December 09, 2005 | Comments: (0)

The Ultimate Troublshooting Meet-up

TAGS: IT Community

This week marks the 19th annual meeting of ubber systems administrators at LISA '05, the Large System Installation Administration Conference.

Talk about a troubleshooter's delight. For those of you who have never been, it really is the best gathering of bright IT minds on the planet. Five days of technical sessions, demonstrations, BOFs and constant geek talk. And of course the weather in California is really a treat given what it's like in Boston and New York this week.

This year LISA will has "Solve My Problem Boards" distributed throughout the conference site in a real world version of constant collaboration that goes on within the data center.

Posted by Michael Baum on December 9, 2005 11:54 AM

December 05, 2005 | Comments: (0)

Managing the Logic Layer is a Headache, Will it Become a Virtual Migraine?

TAGS: None

As an industry, systems management has really grown up on physical infrastructure management, making sure that physical components are up and running. This has been no small task, with the explosion of firewalls, routers, commodity servers, devices on the "edge," and the many other subcomponents that make up the physical layer in today's enterprise datacenter.

But experts are starting to point out that the challenge of managing the logical infrastructure is now the new battle for IT professionals. From J2EE and LAMP application stacks, to the proliferation of open source, to new directions like SOA and virtualization -- the degree of complexity in the logical layer is exploding.

Think of how many different touch points your applications and services have within your infrastructure. IBM's own autonomic computing research points to 25-40 sources of data for the typical enterprise application. Now multiply each one of those sources by the number of load balanced physical devices and servers that produce IT data. No doubt the unending software sprawl of logical components and their dependencies make managing the physical stuff now look trivial. In fact, just two weeks ago IBM announced they have acquired Collation, a company to help them map all those interdependencies. See the InfoWorld story here.

Figuring out the actual runtime logic to troubleshoot systems, measure performance or whatever it is you're trying to do, is still a very human-intensive task performed. We utilize blunt instruments like grep and Perl and awk and sed, not modern kind of tools like you'd expect to find. And if you think its hard now, just wait for virtualization to arrive at a data center near you.

According to Jonathan Eunice, principal analyst at Illuminata:

"Virtualization is supposed to make things simpler-and it does. But along the way, it adds to the explosion of layers, components, and issues that IT managers and administrators have to deal with. The other big datacenter trends, service-oriented architecture (SOA) and modular infrastructure, are much the same. In the big picture, they're Big Wins. Indeed, Really Big Wins. But tactically, they bring in a lot of new stuff to absorb, understand, and integrate. There's a real problem of skills. Most shops don't have years of experience running IT in a virtualized, componentized, service-led way. And so tools and techniques that help "cut to the chase"-that is, reduce time-to-correct operations, and thus, time-to-value-are golden."

Thanks Jonathan for your thoughts.

Do you have a virtualization troubleshooting scenario to share? Write me at thebaum@splunk.com.

Posted by Michael Baum on December 5, 2005 05:00 AM