I like to consider myself a patient person. After all, when you spend as much time as I do spouting off about this issue or that you're bound to take some flak.
When the criticism comes from an impassioned individual with a differing viewpoint I do my best to take it in stride - why get steamed over some silly ad-hominem attack? However, when the source of the criticism is Microsoft - and when the accusations being leveled are both baseless and deliberately misleading - I do take offense.
Case in point: Microsoft's "rebuttal" of sorts to a recent series of controversial entries over at the exo.performance.network blog site. It seams my colleagues at www.xpnet.com hit a nerve when they wrote-up their findings regarding Windows Vista performance and how it sucks compared to Windows XP, especially with Service Pack 3 installed. Microsoft's response, as presented in a blog entry by Windows Vista team member Nick White, was to attack the credibility of the test script. And not with any hard data or technical insight (they have none), but rather with an accelerated video clip and some generalized ranting about what constitutes a "real" benchmark.
So, why do I care? Simple: I wrote the test script that they're attacking.
Some Background
OfficeBench is a classic linear test script that uses OLE Automation to drive Microsoft Word, Excel, PowerPoint and Internet Explorer through a series of typical end user productivity tasks. These include creating and formatting a compound document, accompanying spreadsheet (with charts) and PowerPoint presentation. It's a relatively short script and was designed primarily for testing in hard to benchmark environments, like heavily loaded PCs/workstations and virtualized or multi-user desktop platforms (i.e. the places that break traditional scripts, like BapCo). As such, it is highly robust, having been proven over nearly 8 years of continuous use throughout the greater IT community.
I wrote OfficeBench while working under contract to Intel's Desktop Architecture Labs (DAL). While there, we used it to test everything from the (at the time) brand new Pentium III and Pentium 4 CPUs to Windows Terminal server and Microsoft's "IntelliMirror" configuration management framework. On numerous occasions, OfficeBench provided the ammunition Intel needed convince its customers that the enterprise desktop would need an ever-increasing pool of MIPs to counter Windows' ever-expanding appetite for CPU cycles. We were the guys Intel turned to when they needed an answer, and we have continued to evolve OfficeBench and the rest of the Clarity Studio toolset to meet the ongoing challenge of documenting Microsoft's development largesse.
A "Hit" Job?
To be honest, I'm not surprised that Microsoft responded to the exo.blog postsings. After all, the truth hurts, and the last thing Microsoft wants people to focus on right now is how slow and bloated their new flagship desktop OS has become. I also know that the folks at the exo.blog have already done a fine job of defending the script's technical underpinnings (thanks, guys). No, what really steams my clams is the fact that Microsoft tried to dismiss OfficeBench without so much as a shred of technical data to support their claims. That's FUD mongering at its worst.
So, in conclusion: To defend my integrity, and also to expose the lie that is Nick White's "hit piece," I hereby FORMALLY CHALLENGE MICROSOFT to prove that OfficeBench, as executed by the exo.performance.network research staff, is not a valid measurement of cross-platform, cross-version performance under Windows and Office.
In the meantime, I encourage all of my readers to visit www.xpnet.com, register for a free Portal site account and download OfficeBench (part of DMS Clarity Studio). Then see for yourselves whether a test script that runs unmodified across every major (NT-based) version of Windows and Office is not a valid tool for comparing multi-generational performance.
Posted by Randall Kennedy on December 3, 2007 08:08 AM
