Ahead of the Curve | Tom Yager » TAG: Green benchmarks

April 22, 2008 | Comments: (0)

For server power measurement, there's only one shortcut

TAGS: Green benchmarks

On this Earth Day, let us examine the criteria that IT brings to its purchases and arrange to make power efficiency a top priority. All it takes is the will, a little homework, and the embrace of the delusion that there's any fit and fair way to compare the power consumption of two similar pieces of equipment.

[ Discover the techniques used by InfoWorld's 2008 Green 15 winners to make their IT more energy-efficient. ]

Last December 27, SPEC announced the availability of its SPECpower benchmark. One would think that having the heaviest of the benchmarking heavyweights pour concrete on that most slippery of metrics would give us something to go by. InfoWorld has been waiting for its copy of SPECpower, which SPEC acknowledges is a first step, since January, and I have a guess as to the reason we're still waiting. Perhaps some members of SPEC, which is primarily a consortium of vendors, have encountered the same issue that I have: Power measurement is only 5 percent process. It is 110 percent policy. The total exceeding 100 is appropriate, because like the predator/moron mortgages in which all American taxpayers are now unwilling investors, power benchmarking creates a debt that can never be satisfied. Neither SPEC nor anyone else can ever call the book on fairness policy for power benchmarking closed.

It shows wisdom on SPEC's part that it refers to SPECpower as a first step. But I think that SPEC started development of SPECpower with the wrong objective in mind, that being to derive a result (a figure of merit in SPEC's words) that tries to pour cement over that elusive marketing metric I hold in lowest esteem: performance per watt.

I am pleased that SPEC has made an effort to quantify "a performance." It is essential to have a meaningful constant for accomplishment so that a formula containing watts, which is a fair and concrete measure of effort invested, approaches an expression of efficiency. SPEC's formula counts the number of cycles through a Java server workload over a period of time, while the power draw during the same period is charted. The figure of merit, ssj_ops/watt, really isn't bad as distilled metrics go. SPEC doesn't deal in squishy numbers; there's no "13 SPECfoo_marks." So ssj_ops/watt is pretty good, until you try to use it to compare two servers.

Aside from fairness, there are some technical shortcomings in SPECpower, the granddaddy of which is reproducibility. If InfoWorld's Test Center attempted to validate the SPECpower rating published by a vendor, which is the sort of thing the Test Center likes to do, there is no chance that we'd derive a matching result. If our findings put the vendor in a better light than its own, they'd be overjoyed. If, however, we showed after much diligence that the vendor's published SPECpower results appear to be overstated, every vendor would seek to tear apart our testing methodology and policies, and the list I've compiled of foreseeable vendor objections is daunting. Our Test Center will do a better job of leveling the playing field across vendors than vendors can (and want to) do themselves, because we can replace environmental variables that will vary across vendor testing facilities with absolutes.

Those variables are doozies. SPEC requires disclosure of everything from system configuration to compiler flags in published results, but the impact of variations in compiler flags and clock speed of memory baffles buyers. Fortunately, we can make sense of these and translate them into buying advice. But with power, some variables that will appear to be satisfied through disclosure will actually be mercurial.

The example I'll offer is temperature. SPEC requires the use of a temperature probe. You can establish a policy of pegging all tests at an ambient temperature of, say, 76 degrees, but your 76 isn't the same as mine. Try it yourself. Take a simple infrared thermometer and walk around your datacenter, however grand or humble it is. Grab a baseline by pointing the thermometer at a piece of cardboard held at arm's length to get a basic ambient temperature. Then aim the thermometer at various surfaces, of varying materials and at varying heights, and varying distances from airflow. Compare the temperature of a server's top plate inside a closed rack to that of a similar server inside an open rack. Does a server that shares a rack with a storage array measure the same temperature as the server alone?

Temperature affects every server in a different way. The design of a server's cooling system and its programmed responses to high temperatures say much, if not everything, about a server's quality of design. For example, the cooling system in a cheaply made server will have one purpose, that being to suck air from the front of the chassis, and possibly the sides, and blast it out the back. The fans themselves, being electrified copper coils, generate heat, and being kept spinning at several thousand revolutions per minute in open, probably particle-laden air, makes them subject to failure. Most servers don't care how much heat it makes or where it goes, be it into the air or indirectly into the intake of the server above it, influencing its efficiency. But you have to worry about heat because you pay to move that heat outside the building, accounting for a large percentage of your operating cost. Perhaps server efficiency has to take into account its contribution to the duty cycle of the compressors in your air conditioning. Try to measure that.

My point is, don't expect easy answers to power efficiency. InfoWorld's Test Center is taking on power testing and considering SPECpower as part of that plan. In the meantime, I can tell you the secret to avoiding the homework on this one, and it is my constant advice: Count the number of active server power supplies in your shop and commit to reducing that number over time. That'll do nicely for now.

Posted by Tom Yager on April 22, 2008 11:52 AM

July 25, 2007 | Comments: (0)

In search of energy benchmarks

TAGS: AMD v Intel, Green benchmarks

Efficiency has rocketed to the top in server purchasing criteria, but benchmarking has not left the launch pad

I've been working on InfoWorld's power and cooling benchmark suite (I call them "Greenmark," but I haven't officially cleared that name's use) for several months. Why, you might wonder, haven't you seen any published Greenmark results in InfoWorld? Check out the comments thread following Ted Samson's Sustainable IT blog entry reporting Neal Nelson's tests showing that AMD is markedly superior to Intel in server power efficiency. It's remarkable how much emotion the commenters to Ted's blog invest in their arguments against Neal Nelson's findings. Some of these people skip technical objections and go straight after Nelson's reputation, alleging that AMD had purchased the positive outcome.

This seems an extreme reaction until you think about the impact that a headline like "AMD servers consume up to 44 percent less power than those based on Intel" can have. Energy-efficiency test reports will steer billions of dollars in IT spending and stock market investment over the next several years. One such headline this, run at an opportune time, can swing several points of market share over a period of several months, and if picked up by an analyst, a definitive finding such as Nelson's could have an immediate and substantial impact on a vendor's share price. I know that when InfoWorld's Test Center runs its first set of Greenmark results, there's a fair chance the vendor that finds itself in second place will launch a concerted effort to tear the tests, the reviewer, and the magazine apart. For that matter, the vendor itself might not need to lift a finger. Self-interest among shareholders and lesser investors in the chips and systems deemed less efficient can motivate a response more energetic and far-reaching than any that the vendor's marketing department could orchestrate. There is that much money at stake. The furor over Nelson's findings proves my point.

There's also a good deal of ass-covering there. Energy efficiency has rocketed to the top spot in server purchasing criteria. Management expects that everyone with input into server buying decisions has learned the ins and outs of power conservation, but there are some lazy IT folk who just fudge the rationale that justify major system purchases. Something like the Neal Nelson report can end up on the IT slacker's desk with a Post-It reading, "Server criteria review meeting today @ 10:00. Bring your energy-efficiency research notes."

The remarks in the Sustainable IT blog's comments thread barely hint at the firestorms that energy benchmarks will trigger once they become commonplace. It took real courage for Neal Nelson to take the first arrows along this trail. As for challenges to Nelson's objectivity and professionalism, I'll weigh in with the caveat that I won't be looking at Nelson's energy tests until I've finished developing my own. Neal Nelson has long experience and solid credentials in the performance testing business. It's been my experience that Nelson's organization creates fair and relevant tests, documents the process, makes itself accountable for the results, practices full disclosure, and responds to challenges to methodology and findings. A tester that provably meets those standards (and so very few do) earns the benefit of the doubt where integrity is concerned, and only test results published by independent testing organizations with a commitment to those standards can be taken seriously by investors of any stripe. Challenges to the tester's objectivity that aren't backed by direct proof of bad faith have to be ignored by those who develop tests and who use test results as criteria for purchases. That makes room for the kind of discussion that leads to better tests and to the wiser application of their results.

Posted by Tom Yager on July 25, 2007 03:00 AM