No Silver Bullet

I'm working on my weekly InfoWorld column (this one will run in print and online on March 8) and I'm referencing an essay from Frederick Brooks (of "Mythical Man-Month" fame) entitled "No Silver Bullet: Essence and Accidents of Software Engineering."

You just have to read this. I've read it many times before and referenced it in a column on web services two years ago, but the essay continues to amaze me. Although it was written eighteen years ago, the content still rings true. Just a sample:

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract in that such a conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

Amen. Be sure to read the rest.

It's all about scale

My discussions at the O'Reilly Emerging Technology Conference today have all revolved around scale.

First, I ran into Sam Gassel (old bio here), who architected the backend for CNN.com in the early days and scaled it to amazing levels as web traffic grew. CNN.com was the most demanding web environment I've ever seen -- we're talking about a site that got 1 billion page views in a single month, and Sam was in the middle of all of it. Sam is now a Technology Fellow at CNN and continues to spend his time thinking about scaling issues. I learned a lot of lessons about scaling web sites from Sam and fondly remember Sam telling vendors that we would melt their routers, switches, OS'es, and load balancers (he was nearly always right). Whenever we crush a network device in the InfoWorld Test Center, I think of Sam. Sam, you need to start a blog to impart your hard-earned wisdom to the rest of us! [Update: I made a correction to make it clear that Sam is a Technology Fellow for CNN, not Time Warner, the parent company. - CD]

I continue to be impressed with Salesforce.com (disclosure: InfoWorld is a salesforce.com customer -- actually I think this makes me more qualified to comment on it). To a large degree, Salesforce's innovations have gradually turned my early skepticism about web services into enthusiasm.

Today, I had a great conversation with Adam Gross about their sforce API and some of the activity around it in their community and on sourceforge. I'm not ready to go into the details of what we discussed, but I'm planning to do some coding on my own within the sforce API to link their data more tightly into some of our business processes. This work will likely be of broader use to Salesforce.com customers. Stay tuned.

O'Reilly Emerging Technology conference

I'm headed off to the O'Reilly Emerging Technology Conference today. This is the first O'Reilly conference I've been to since 2000, when I presented in the business track about building a media company (Salon.com) with open source technologies. The third slide in my presentation is certainly a relic of a past era -- note that Time Warner's market cap was $148B at the time of my presentation and now it's $79B.

Linux, OS X, and cowboy coding

Frank Steele has an interesting piece about running a business on OS X and Linux, but it's also about simplicity versus complexity and "real" software development versus scripting (something I touched on last year that got a lot of discussion going on Slashdot).

I worked with Frank back in my CNN.com and CNNSI.com days so his lessons are familiar and a lot of what I write about here at InfoWorld was learned in those days. The development group I hired and managed to build, launch, and maintain CNNSI.com could have won a few cowboy coding rodeos had they existed -- but then again, the web was new and we were blazing new trails by solving problems that had never been solved before, so a little cowboy bravado was probably in order.

Testing the kernel

As I tell my editors, sometimes I get too busy being a CTO to actually write about being a CTO. This is definitely one of those times, so my blog output is going to be light for a bit.

In the meantime, if you haven't seen our package about the Linux 2.6 kernel, check it out. Paul Venezia of our Test Center got his hands dirty testing the kernel, but received help from the top of the Linux food chain, working it into his story about how the kernel is built:

During the course of my kernel testing, I ran into a few interesting anomalies with the v2.6-test kernel series and posted my findings and thoughts to the list. The first post detailed an issue regarding poor per-character performance of the v2.6.0-test8 kernel. Less than 30 minutes later, Linus Torvalds had responded to my post and offered some suggestions, followed by Bill Rugolsky Jr. and Andrew Morton adding insight. Within six hours, the problem had been reliably reproduced by a number of core Linux kernel developers, and traced to an issue with a library linked by the NPTL (Native POSIX Thread Library) code. The problem was fixed by Ulrich Drepper of Red Hat and incorporated into the next release.

A few days later, I noticed another performance issue. The Samba tests under the v2.6 kernel were ridiculously slow. The numbers I was seeing on the Intel Xeon system were truly terrible. I had previously tested the lower-level kernel subsystems on this server and seen no problems with performance, but the higher-level tests showed a significant bottleneck somewhere. Another post to the LKML [Linux Kernel Mailing List], followed by responses from Torvalds and Morton, helped me identify the source of the problem.

That's just one story. Also in the package, you'll find an interview with Andrew Morton (the kernel maintainer), the actual testing of the kernel in our Test Center, and plenty more.