Enterprise Mac | Tom Yager » November 2006

November 17, 2006 | Comments: (0)

Errata and clarification: Xserve heat, noise and power

TAGS: Design and engineering, Servers and storage

This errata to my Xserve review is not a result of any feedback from Apple Computer.

Yesterday, working with a new reference workstation from AMD gave me a reason to put a 2U Woodcrest (3 GHz Core 2 Duo Xeon) white box server back in my rack. When I benchmarked the Intel and AMD machines against each other, the AMD absolutely blew the wings off the Xeon box. The margin was way too wide to credit to AMD's engineering ingenuity. I uncovered a serious flaw in that Woodcrest 2U server that invalidates the testing I've done on it to date. That server was the baseline for the noise, heat and power tests I ran on Xserve. I have re-run the tests from scratch and my findings are below.

For those interested, I detail the problems I found with the Woodcrest 2U server and the fixes I applied and the end of this post.

After the fixes, the 2U Woodcrest server's power utilization characteristics were a much nearer match for Xserve's. The white box's power floor rose to around 280 watts, within reasonable reach of Xserve's 300 watt floor. The white box's power ceiling rose to around 400 watts, right in line with Xserve.

Noise was a tougher nut to crack than power because the chassis are so different. I originally gave Xserve a non-scientific statistical handicap for being a 1U server: A 1U server requires more and smaller fans spinning at higher RPMs than a 2U server with identical components. And of course, 1U offers the unique benefit of doubling rack density. Compared to the white box's original measured 56 dBA, Xserve's 65 dBA was reasonably classified as "loud." Remeasuring the white box after the fixes were applied, its highest sound pressure level rose to 61 dBA. A 5 dBA difference is easily perceptible, but I would judge Xserve's noise level to fit within the 1U/2U handicap.

I decided to take an entirely different tack with heat. I hypothesized that Xserve's heat, not getting blown far from its chassis by the strong exhaust fans common to other designs, was pooling at the back, causing the steel at the rear of the machine to get quite hot (118 degrees F was my highest external reading, and I expect it's hotter inside). I surmised that the hot case metal warms thermal sensors that consequently make the fans spin faster. That would explain why Xserve's fan speeds rise in a sort of "runaway" pattern. It takes a lot of extra oomph for even a big gang of little fans to cover the distance from the drive bays to the back panel, and it's not as if there's nothing in the way. By the time the forced air makes it to the back of the case, it's reduced to a puff, and the case metal turns into a radiator.

It was a lot easier to test this hypothesis than it was to come up with. I clamped a small desk fan to the rear left corner of Xserve and positioned it to blow air horizontally across the rear panel. It was a kludge and I know that a more elegant solution would produce more dramatic results, but I proved my theory. My measure of effectiveness was Xserve's fan RPM. Within about 20-30 seconds of turning on the external fan, Xserve's internal fans began slowing down. The fans still spun up to react to increases in demand, but the fan speed curve fit the rise and fall of compute demand more tightly.

Incidentally, Xserve took a good couple of minutes to react to turning off the external fan. Perhaps it takes a while for the metal to heat up again.

I'm intentionally leaving out a lot of details, like testing first with lowered room temperature and nosing Xserve directly into the chilled air path, and turning on a powerful ceiling fan. None of these worked as consistently as the breeze across Xserve's backside. I'm sure that admins will write me to say that anyone worth his or her salt makes sure that there is a brisk airflow behind machines, or that they use closed racks with strong and constant forced air. Anyone running Xserve in an enterprise data center or high-performance computing cluster doesn't need to give Xserve any special consideration.

My advice reaches out to people running Xserve in places that are designed for habitability, not for the optimal operation of server equipment. I think that a small business, a shop with widely-distributed server stacks or individuals working with Xserve installed in workstations (referring to furniture) should know that the kind of rear airflow I described--blowing across, not into or away from the back of the system--seems to help cool and quiet Xserve during periods of heavy demand.

I will integrate abridged text from these new findings into my original review. I apologize to Apple and to readers for these errors.

The noise, temperature and power readings that I had taken from a white box Woodcrest (3 GHz Core 2 Duo Xeon) rack server to use as a baseline were flawed. The white box system came to me with the BIOS set to its factory defaults, the BMC (baseboard management controller) set to its defaults, and Windows Server 2003 already installed. I formatted the drive and reinstalled Windows, as I always do, and that went without incident. On returning the Woodcrest server to my rack, I followed my habit of checking the motherboard vendor's Web site for updates. There were two, one each for the BIOS and the BMC, along with a new version of the system's management console.

I discovered that firmware defaults that were previously not exposed for user adjustment were so conservative that the machine operated at, practically speaking, a desktop power profile. The yellow/red border temperature thresholds were set much lower than Apple's, so the white box machine spent much of its time throttled back to prevent overheating. Likewise, the CPUs were configured to enter a HALT state whenever possible. As a result, the Woodcrest white box originally ran cooler and drew less power than Xserve and was considerably quieter.

After the firmware updates gave me control of performance settings, I elected not to try to finesse individual parameters to make the white box eat and heat like an Xserve. Windows' opaque manipulation of power settings complicated matters. So I disabled the white box's power management entirely, and swung its temperature thresholds to the maximum. As described in the text above, this redefined the baseline against which Xserve was compared and consequently changed my analysis.

Posted by Tom Yager on November 17, 2006 02:25 PM

November 14, 2006 | Comments: (0)

Tell-all author Amit Singh dishes about Apple's fling with Linux

TAGS: Politics, strategy and culture, Software

Mmmmm...crow.

It's time for the Linux Anti-defamation League to get its gloat on. My new favorite book, "Mac OS X Internals, A Systems Approach" by Amit Singh, says this on page 23:

Apple and OSF* began a project to port Linux to run on various Power Macintosh platforms, with Linux hosted on top of OSF's Mach implementation. The project led to a core system called osfmk. The overall system was known as MkLinux. The first version of MkLinux was based on Linux 1.3. It was released as MkLinux DR1 in early 1996. Subsequent releases moved to Linux 2.0 and beyond. One of the releases was incorporated into Apple's reference release.
[...]
MkLinux used a single-server approach: The monolithic Linux kernel ran as a single Mach task. Mac OS X uses a kernel base derived from osfmk and includes many MkLinux enhancements.

When asked to explain why I had to find out about this by reading it in some damn book, Apple made the following statement: "I can't believe you're throwing this in my face ten years after the fact. We agreed never to talk about our pasts. Look, Linux was really getting around back then. Everybody was doing it, and I was rebelling against my whole Happy Mac image. I was like, I'm a grown up now, I can make my own choices, and I demand the freedom to be a brooding Mac, a wasted Mac, and yeah, a shallow Mac. When I got a crack at Linux, I jumped on the pile. I figured it would boost my rep. I used Linux for a while and threw it out when it got all full of attitude. It did leave some stuff in my loft, and before you ask, I have no idea where it is. Now, mister true confessions, is there something you'd like to tell me about you and Windows for Workgroups?"

I am man enough to admit that I was wrong. It turns out that Linux has one thing to recommend it after all.

Mmmmm...penguin.

*For those still in their awkward years, OSF was the Open Software Foundation. OSF started life as the more confrontationally-named FAF, an organization formed in response to the old AT&T's new usurious license fees for System V UNIX. Now that the new AT&T (The Blob) is here, I remind everyone that those who don't learn from history are doomed to repeat it.

Posted by Tom Yager on November 14, 2006 02:02 PM

November 09, 2006 | Comments: (0)

Apple Xserve: The final review

TAGS: Design and engineering, Servers and storage, Software

As usual, Apple knows something that its competitors don't, and after three weeks with Apple's new Xserve and OS X Server Tiger 10.4.8, I know it, too. Apple is taking a road that pundits will likely insist will lead Apple nowhere: It is doing a server appliance play, but not of a flavor that the market's seen before. While trends, or rather, the analysts who proclaim them, are pointing to the triumph of software as a service, outsourced applications, consulting, node-locked operating systems and other pay-as-you-go approaches, Apple is piloting a rocket-powered sled in the opposite direction.

Apple is going to sell complete server platforms that buyers purchase, operate themselves and actually own. Seriously. The customer pays the advertised price for an Xserve (starting at $2,999) and gets a server loaded and pre-configured with a server software suite (PDF) that alone meets the needs of the majority of Intel x86 rack server buyers. There are no subscriptions, no priority update service fees, and no client, device, mailbox or CPU licenses. None of the services is grayed out pending your purchase of an unlock key. Xserve has no try-and-buy, no time bombs and no trip wires telling you that you need to upgrade from Express this or that to Professional this or that. Xserve never phones home to beg for Apple's permission to use the server software already loaded on your system. And if Apple played the slick pricing games that its competitors do, Xserve's advertised price would be $2,000, with a one-item selection menu on the Buy Now page that reads "OS X Server, unlimited users (+$999)."

Xserve is an Intel x86 Xeon computer, and on its own, it represents exemplary hardware engineering. You'll find my detailed review of Xserve hardware in a previously-posted part of my review. To summarize, in designing Xserve, Apple had more in mind than making a standard Intel x86 server (with the standard defined as "able to run Windows," to which Xserve will stoop if so commanded). Instead, Apple engineered its quad-core Core microarchitecture Xeon server--which it could have purchased off the rack for $0 in R & D--to be an entry-priced server that meets the requirements of mid-level server buyers. In durability, serviceability, manageability and availability, Xserve more readily finds rivals among UNIX RISC servers than commodity Intel x86 systems.

Xserve is the new flag-bearer for the total Apple server platform, which encompasses Xserve, OS X Server, the Xserve RAID storage array, the Xsan SAN file system and the WebObjects large-scale server application framework used for iTunes and most of apple.com. While Xserve the computer more than holds its own among PC servers in its price class, it is the one-price combination of Xserve and OS X Server that dusts the competition.

Xserve strikes the perfect chord with everyone from the server neophytes and Windows refugees who want plug and play to the UNIX graybeards allergic to proprietary system software, equipment or development tools. Freed from the never-ending spending of Windows and the do-it-yourself shipbuilding of Linux, every single buyer of Xserve will end up doing more with Apple's server than they had in mind when they bought it. Apple makes it safe and simple to reach beyond the narrow purposes for which individual Windows and Linux servers are typically deployed. Xserve can run its full range of services--which include Web, database, J2EE, e-mail, anti-virus, IM, Windows and UNIX file/'print sharing, VPN, gateway, proxy, and firewall.

Xserve is uniquely easy to deploy and manage on its own or as part of an established heterogeneous network of systems, and yet it is not dumbed-down in the least. It carries the traditional benefits of Apple's "invented here" design, particularly the rock solid stability that OS X derives from being targeted to a limited number of controlled configurations (note that OS X Server runs on client Macs as well). And yet Xserve is an entirely standards-based, open design, but with none of the prefab, seen one seen 'em all engineering that typifies Intel x86 servers.

An appliance for some, a wide-open UNIX platform for others
Unlike other Intel OEMs' boxes, Xserve ships from Apple as a complete server platform, not as a computer. The difference? A server platform is a whole--hardware, OS, standardized services, GUI management, dev tools, server application frameworks, documentation and much more--that emerges from its shipping carton with functionality that fully satisfies the needs of the majority of buyers. When I plugged Xserve in for the first time, it did what I expected: It gave me my choice of services including, but not limited to, Web, database, J2EE, e-mail, blog, IM, Windows and UNIX file/'print sharing, gateway, proxy and firewall. It took about three minutes, with no reboot, to put my services on the air, and they were all immediately configured, reconfigurable and reporting to Apple's Server Manager and Server Monitor GUI consoles.

Linux users are accustomed to wading through several pages of checkboxes to select from among the bulging sack of software--some maintained, some not--burned onto the install DVDs. OS X Server doesn't subject users to that. If you ask for e-mail services, OS X Server sets up the open-source SMTP, Web mail, spam filter, virus checking and list management that Apple selected, validated and, in some cases fixed and enhanced (always giving its changes back to the projects). Apple isn't reticent to count and credit the projects are part of OS X Server. However, by default, projects and versions are abstracted by the install and management interfaces. That abstraction allows administrators to treat each major network service category as an integrated solution rather than a stack of pieces. Server Manager, the multi-server administrative console bundled with OS X Server and available as a free download for running on client systems, presents a unified interface that eliminates the need to configure each mail service component individually and deal with instabilities that arise from conflicting configurations. Apple brings this consistent, centralized approach to the Server Monitor console that handles Xserve hardware monitoring, reporting and notification.

Now, if you're nauseated by the notion of using a novice-friendly server appliance, Apple feels you. OS X Server's cozy GUI desktop and management tools co-exist with--not replace--the OS X Server platform's nuts and bolts UNIX-ness. Xserve will boot to a UNIX VGA text console, to the server's serial port or headless (no human interface devices attached). Xserve can be administered and configured entirely from the command line and script code. And nearly all of the non-user-facing pieces of OS X Server are available as Apple-supported open source published as Apple's Darwin project. Apple's development tools and documentation are free, and like OS X Server, the dev tools have consistent and well-integrated graphical interfaces that you can shove out of the way if you feel like roughing it. Is Apple's Aqua interface too dressy for you? Start X Window Xserve is a whole platform, but it is an infinitely malleable whole that you'll find familiar and not at all confining if you're a veteran of UNIX or Linux. I have yet to find a well-maintained open source project that doesn't list OS X or Darwin (the open source OS on which OS X is built) among its explicit build targets. Two massive open source repositories, Fink and Darwinports, track and package OS X/Darwin compatible projects. As a UNIX box, Xserve not only fast, familiar and standards-based, it is addictive.

Xserve from the outside
Xserve is a 1U, 1.75-inch tall rack server. Its chassis is made mostly of very stiff aluminum, with steel used where extra strength is needed (like the rack rails). Its cooling system is simple and effective: Two huge rectangular, grille-less intake ports, which Apple calls "chunnels," are set between the three removable drive bays. The remainder of Xserve's front panel is sparse. The interactive controls consist of a power button, a chassis/drive bay lock and a system identify button that lights an LED on the back panel to make it easier for someone working at the back of a large cluster of Xserves. The system ID light can also be activated remotely from Apple's server management GUI. The front panel buttons serve as stand-ins for a keyboard when Xserve is running headless. Patterns of button presses can, among other things, force Xserve to boot from the optical drive or from a network image.

LED indicators at the front panel are arranged and colored to make their purposes obvious: Main power, hard drive power and activity and the blue LED array that reports the CPU utilization for each of Xserve's four cores. The Ethernet link lights tipped me off to a small but potentially frustrating anomaly: Xserve's Ethernet ports are flipped. This becomes important when you have to configure lights-out management. For lights-out, channel 1 is the right Ethernet socket, and channel 2 is on the left when looking at the system from the rear. As a workaround, Apple recommends that you install Xserve upside down (no it doesn't).

The front panel also has the opening for the slot-loading optical drive, which can optionally be a Superdrive dual-layer DVD burner. OS X still lacks packet writing and DVD-RAM support, so you can't mount a disc as a read/write volume. Still, many users, myself included, find the built-in burner to be a blessing. Lastly, the front panel has the 400 Mbps FireWire socket for which Xserve administrators have 100 uses. It lets you use Target Disk Mode to mount external Macs' local storage as drive volumes, and Xserve can be a Target Disk Mode target for another Mac as well. FireWire also works as a secure, self-configuring point to point TCP/IP link, and of course, it can be used with FireWire video and audio devices and with external FireWire storage. My pet use for Xserve's FireWire is running Apple Remote Desktop. For most uses, this is as fast as a locally-connected keyboard and display.

Xserve's rear panel is dominated by Apple's power supplies. If you opt for redundant supplies, you'll have two AC sockets to fill. However, Xserve does not sound an alarm if you set the second supply up as a cold spare by not plugging it in. By default, the second power supply constantly splits the load with the first. There's no switching delay at fail-over, and in the event that an unexpected power burden threatens to drive the voltage on the critical +5V rail low, the supplies will compensate. Each slim, lightweight power supply has its own fan that is monitored and speed-controlled by Xserve's System Management Controller. When you remove one power supply, an internal "doggy door" swings down to close the hole.

The rest of Xserve's backside is a continuous grille. A strip across the bottom provides access to a DB9 serial port, a mini-DVI display connector, USB 2.0 ports, 800 Mbps FireWire and two gigabit Ethernet ports. As I mentioned, Xserve will boot to a local serial console. Apple has not yet equipped its lights-out management with the serial-over-LAN feature that would redirect that serial console traffic to an arbitrary network destination. The serial port is also used for uninterruptible power supply (UPS) units that notify connected systems of their power state. You can have the serial console or the UPS hooked to the DB9 connector, but not both. Fortunately, most standalone UPSes now come equipped with USB connections.

There are only two USB ports behind Xserve. This proved to be one too few in my testing, and I wished that there were a USB port on the front panel for human interface devices. The 800 Mbps FireWire port is conceivably useful for external storage, although there aren't many devices that use it. The fast FireWire port's value in my eyes lies in Target Disk Mode, Remote Desktop and dedicated server-to-server TCP/IP links. If you prefer, an 800-to-400 adapter cable will turn the high-speed port into a standard 400 Mbps FireWire port.

The mini-DVI connector uses a pigtail to connect to either an analog or digital display. Both pigtails are included, and the mini-DVI port, though tiny, is very sturdy and grips the connector well. The weight of the VGA or full-sized DVI plug at the other end poses no problem. Xserve's standard on-board GPU (graphics processing unit) is an ATI Radeon X1300 with 64 MB of dedicated video RAM. That's pretty sweet by server standards. It supports Quartz Extreme, Apple's accelerated graphics, and driving OS X's GUI at a graphical console is a pleasure. An upgraded card with 256 MB of video RAM is available, but this occupies one of the system's PCI Express expansion slots and generally heats things up. Don't get it if you don't absolutely need it.

Lifting Xserve's lid
Apple, long known for fancy zoned, heat-piped cooling designs, went for simplicity with Xserve. A wide bank of seven fans, each with dual independent rotors, stand in straight-line formation behind the drive bays. Except for the power supply fans, this long bank of fans produces the only airflow through Xserve. Thermal sensors are scattered around the system board, busses, memory sockets, CPUs and everywhere a temperature is worth taking. Instead, all of the inflow fans, the front-facing rotors, spin up and down in sync. The rear-facing outflow fans run at another synchronized speed.

A thin plastic shield covers the CPUs and creates a couple of broad airflow paths. The shield is flimsy and the screws holding it in place could be used for eyeglass frames. They're vibration-proofed by miniscule rubber washers that you will lose if you don't know they're there (but now you do). It's hard to wedge the shield back in; the right edge of the shield catches on a wire bundle. Finesse it, don't force it, and I advise using a gently magnetized sharp-pointed Philips screwdriver to remove and replace the shield.

Beneath the shield is where the action is. The standout is a pair of tall, all-copper heatsinks. PC enthusiasts and system builders treat these like diamonds, and if you can find them at all, they are seriously expensive. They're the only way to do what Apple's done, and that's run a pair of 3 GHz, dual-core Core microarchitecture Xeon CPUs without high-RPM dedicated fans. Apple's approach works: Under maximum stress, the CPUs are among the coolest components on the system board according to Apple's sensors. The heat sinks are attached to the system board with two large screws. Remove these and the Xeon chips in their ZIF (zero insertion force) sockets are yours for the swapping. Any desire you have to upgrade your Xserve CPUs (there is, at this writing, nothing better than the 3 GHz Xeon) must be tempered with the knowledge that Apple doesn't promise to support user-upgraded Xserves. However, if Apple had made its CPUs inaccessible, I'd have slammed the whole machine for messing with owners' rights.

Xserve has eight sockets for fully buffered DIMMs (dual in-line memory modules). The system I'm evaluating is stuffed with 8 GB, using 2 GB modules. There are Intel x86 servers with sixteen DIMM sockets. Why not sixteen sockets? I think that Apple made a design choice to conserve that real estate for a second power supply and a more serviceable interior layout. Apple had the advantage of entering the market when 2 GB memory modules were more readily available. For a long time, the only realistic way to squeeze 16 GB of RAM into an Intel x86 server was to install sixteen 1 GB modules. Xserve's RAM is readily user-upgradable. Apple sells memory modules, and it offers guidance on choosing modules from third parties.

The other noteworthy feature inside Xserve is the monolithic backplane into which removable drive modules plug. Each port in the backplane supports either a Serial ATA (SATA) or Serially Attached SCSI (SAS) drive. The Apple-proprietary drive trays and rear connectors are identical regardless of interface type. SATA is plenty fast enough for local storage, especially with OS X Server's software striping and mirroring. But sometimes an application calls for a 15,000 RPM SCSI device. Put SAS speed where you need it, and SATA capacity everywhere else. It's worth noting that Xserve's Apple Drive Modules are not compatible with Mac Pro, and Apple does not sell empty trays for Xserve. You can, however, upgrade a drive in an Apple Drive Module quite easily. Apple doesn't acknowledge this, but does claim that it optimizes drive firmware for server operation. Apple's hard drive mark-up is reasonable, so it's probably not worth the bother to put a new drive in an old tray. Like the CPUs, the fact that Apple didn't explicitly block you from using raw drives points to Apple's commitment to an open, standardized approach.

Xserve's environmental impact
Computers need to coexist with humans, and no vendor is more attuned to that than Apple. Its clients are renowned for their silent running, and Mac Pro, which has the same Core microarchitecture Xeon CPUs as Xserve, practically whispers. However, Apple's first crack at an Intel x86 server isn't everything that Intel's "performance per watt" campaign might lead prospective Xserve buyers to expect. Xserve is not distinctly frugal in its power consumption. It generates as much heat as I'd expect from a two-socket Netburst Xeon server. And Xserve is noisy. There's just no getting around that.

Measuring power consumption at minimum and maximum combined CPU and local hard drive load yielded measurements of 300 and 400 watts, respectively. The power supplies are capable of satisfying markedly higher power demand, creating a few hundred watts of headroom in my tested scenario. Power levels did not rise noticeably when two power supplies are installed, even though both supplies run constantly. However, Xserve exposes no user-definable power profiles; the Energy Saver preferences pane does not have a reduced speed setting. Xserve eats whatever it wants to. That's typical among Intel x86 rack servers, but I had hoped for the ability to set the system's maximum power level for those times when efficiency trumps performance. Xserve's automatic power management is fairly aggressive, though. CPU core voltage drops quickly when demand falls. Apple's administrative interface does not report the CPUs' clock speed, but I assume that it shifts as core voltage does.

At one foot from the front panel, Xserve's sound pressure level registered a fairly steady 60 dBA, and around back, 65 dBA. This is not an acoustic breakthrough when compared to the PowerPC-driven Xserve G5 or to two-socket Netburst Xeon servers. EPA guidelines set 80 dBA as the danger threshold for workers without hearing protection, so a single Xserve's fan noise falls well within the safe range. Even so, it highlights the importance of protecting your hearing in the server room. If you can't have a conversation without shouting, your environment is dangerously loud.

The new Xserve does not share a key quality with Xserve G5: The fan noise is not decreased by disabling one of the CPUs. Xserve G5 gets markedly quieter, but then, the PowerPC 970 CPU has more on-board peripherals. Shutting down a socket took bus drivers with it, while Intel's x86 busses are all external.

Xserve's exterior thermal profile strikes me as mostly average for a modern Intel x86 1U rack server. At the back panel, I measured peak temperatures of 118 degrees F, while the front panel ran only a few degrees above room temperature. The system's forced-air cooling is very effective. I was troubled by one focused hot spot of 108 degrees on the top cover, at least ten degrees hotter than the average for that surface in a room at exactly 78 degrees. High radiated heat is potentially hazardous for the equipment in the space above the heated device. It's harder for air conditioning to remove, and it's usually symptomatic of inadequate heat sinking. That may be something that Apple will address in a future design. However, the toaster on Xserve's system board, Intel's Core microarchitecture north bridge (front-side bus controller) chip, isn't something that Apple can easily chill. It runs as much as 60 degrees Fahrenheit hotter than the CPUs. In fact, it's hotter by a wide margin than any other component according to Apple's sensor data. Apple can convert this from radiated heat to forced-air heat, but its burden on power and cooling will remain.

Thank you and goodnight
I have referred throughout this story to the plethora of sensors inside Xserve. One of the delights of OS X Server is the simple, yet deep management and monitoring tools that come with it. Sensors and tools are not new to Apple. What is new is that sensor reports and management tools now operate whether the system's power is on or not, and whether you're on the same subnet as the server's Ethernet ports or not. The magic is lights-out management, long a feature demanded by Intel x86 server buyers but absent in Apple servers. Now, Apple has equipped Xserve with a standards-based baseboard management controller that operates continuously as long as AC power is available. The BMC draws negligible power when Xserve is off, making it very UPS-friendly, and you can reboot, power down and power up Xserve from anywhere. These operations previously required that OS X boot successfully, but an OS that won't boot is a pretty common reason for an administrator's Blackberry to buzz.

Xserve's lights-out management is powered by Intel, and it implements version 2.0 of the IPMI (intelligent platform management interface) specification. That doesn't mean much right now, because even though IPMI is a published specification, that spec is so impossibly convoluted that the only effective IPMI consoles are those created by OEMs, and these are closely guarded intellectual property. An open source project, openipmi, aims to change that, and remote management hardware maker Cyclades (now part of Avocent) has announced its plan to open its IPMI software development kit. In the meantime, Apple's only IPMI-equipped software is the Server Monitor GUI. I was not able to use Server Monitor to target Xserve remotely, and pointing Server Monitor at Xserve's lights-out IP address returned nothing (although that address did respond to pings). I have no doubt that this will evolve quickly, but Xserve users may want to keep polite pressure on Apple to make sure that on-board monitoring and logging facilities are accessible from local and remote text connections as well as the command line.

An objective reviewer's job is to find fault, and I've done my job. But the sum of Xserve's flaws is overwhelmed by the system's unique leading-edge, user and administrator-centric engineering. Xserve is far better than the commodity server that the Intel x86 market expects. But what really blasts Apple's competition is OS X Server. The present Tiger (10.4) release is more than a match for much more expensive commercial Linux, and far more capable out of the box than Windows 2003 Server. Early next year, OS X Server Leopard (10.5) will transform Apple's already industry-leading Xserve, including the model reviewed here, into an unimaginably feature-rich native 64-bit server platform. And guess what? When you buy it, you're done paying for it, and all of the services you have to buy, build or rent with Windows, Linux or pay-as-you-go service outsourcing, are installed on every Xserve's boot drive. Call me old fashioned, but I prefer Xserve's buy once, run forever approach.

Posted by Tom Yager on November 9, 2006 10:16 PM

November 07, 2006 | Comments: (0)

Xserve review Part 3: Power, cooling, heat and noise

TAGS: Creative Pro, Servers and storage

(Apple requested that I drop the "Xeon" from Xserve's product name. I don't mind. Since the Xserve G5 I was using has gone back, the distinction is moot for me. Xserve it is.)

This is a project that called for my removal from my office. I'm at the VMworld conference in Los Angeles, connected to Xserve via Remote Desktop, and finally able to write instead of test.

The consensus among commenters is that I should skip the rest of these notes and go straight to the review. I promised this bit on power, cooling and noise next, so I'll fulfill my promise and post the review after this.

Compared to a two socket, dual-core Netburst Xeon, Xserve eats like a bird, but there's no mistaking its power draw for a desktop's. Xserve has a relatively smooth soft start: It doesn't spike its power consumption excessively at power-up. Some servers pull in gallons of extra juice when first powered on to wind up all of the fan motors and disk spindles, and to initialize peripheral devices. A rack of Xserves won't overwhelm your circuit breakers when you fire them up simultaneously.

I measured the range of Xserve's operating power utilization, with varying levels of compute and peripheral load, at 295 to 400 watts. It doesn't have a wide intermediate range; it tended to hover at the top or the bottom. Watching power levels in real-time with Apple's Server Monitor, I saw only three distinct stages of power adjustment reflected in CPU wattage and core voltage. Apple's interface doesn't report clock speed. As it is, I'm not surprised that total power consumption, measured at the AC outlet, doesn't step up and down in smaller increments. Power utilization was nearly identical whether one or two power supplies were used.

As Apple explained it to me, Xserve's power management (which is not user-adjustable) favors performance over efficiency. When it appears to be in a quiescent state, Xserve is actually hard at work waiting for work to do. The advantage is that the system reacts with minimal latency to sudden changes in demand, whereas a server in a power-reduced state might have to wake from that state before it can meet that demand. The disadvantage is that the nominal floor for power draw and heat dissipation is set high, and OS X provides no easy means by which the administrator can lower it.

Xserve apparently has the option of being put to sleep, but the evaluation system wouldn't do it. A possibly related anomaly was that the processor preferences pane would not allow me to switch off CPU cores that had been subjected to\ heavy (100%) load. It permitted this when the system was less loaded, allowing me to knock the machine down to a single core. However, disabling cores, or disabling the second CPU entirely, had only a marginal effect on temperature and power.

Xserve contains a total of nine fans, seven side by side in an array and one each for the (optional) dual power supplies. Each of the fans in the main bank of seven has two independent rotors (and so, motors). The double rotors create a push/pull airflow that takes up relatively little real estate. All fans are monitored individually by Apple's System Management Controller, but at present, all of the fans change speed at the same time. All of intake rotors are assigned one RPM value, and the output rotors are assigned another.

The fans are monitored individually. This brought to my mind the possibility that Apple might update Xserve to reduce noise by adjusting fan speeds according to need. For example, Xserve's front-side bus controller runs markedly hotter than any other system board component, so giving it special attention might allow the other fans spin more slowly. On the one hand, it would make Xserve quieter, but on the other, it would add complexity to a cooling system that is currently simple and predictable at the cost of noise.

It appears that fourteen fans keep this server right on the brink of thermal balance for the configuration I'm running. The Xserve review unit arrived loaded with 3 GHz CPUs, two power supplies, a PCI Express dual gigabit Ethernet card, a PCI Express Fibre Channel host bus adapter, an 80 GB SATA boot drive, two 750 GB secondary SATA drives and 8 GB of FBDIMM memory. That's not worst-case for heat, but it's close enough. When I drove all four CPU cores at 100 percent, the seven fastest fans ramped up smoothly to about 9000 RPM. At that speed, Xserve seemed to take any opportunity to slow the fans. There are short delays in SPECjbb, the burn-in benchmark I chose, and as soon as Xserve hit the leading edge of that delay, the fan speed dropped quickly, but still smoothly.

OS X Server Tiger includes Server Monitor, a detailed dashboard of Xserve's sensors. The system is exceedingly self-aware, with temperature sensors planted everywhere there's a temperature worth measuring. Those numbers are fun to watch, but I was only concerned with measuring external temperature. In a 78 degree (F) space with unimpeded airflow fore and aft, the front of the chassis stayed cool. At maximum load, the rear panel hit a high temp of 118 degrees. The bottom and side surfaces were barely warm, but the top of the chassis averaged about 95 degrees except for a well-delineated hotspot that registered 108. I pity the system above an Xserve in a rack that isn't set up properly.

Finally, the noise numbers were about what my ear led me to expect: Xserve is loud. With seven fans turning at 9000 RPM and seven more turning at 5000 RPM, the sound pressure one foot from the front of the server was 60 dBA. One foot from the rear panel, the sound pressure was 65 dBA, placing Xserve among the loudest 1U rack servers I've used when under heavy load.

Posted by Tom Yager on November 7, 2006 12:02 AM