Last month, we discussed the fact that VMware was relaxing its benchmark publication policy... just a little bit. But it was enough to spark members of SWsoft to start thinking about a comprehensive virtualization benchmarking solution that the entire industry could use to assess different aspects of server virtualization technologies.

According to the Virtuozzo Blog site, their current thinking about benchmarking includes:

• The benchmark will be focused on server virtualization. When we have [a] desktop product, we'll focus on a benchmark for desktop technologies, but for now, it's just servers.


• The benchmark will be comprehensive and will measure several aspects of the technology.


• SWsoft will work with all interested parties - virtualization technologies vendors, hardware and software vendors and, of course, benchmark vendors - to ensure that the benchmark is accepted by the entire industry.

Their current thinking on the metrics of the benchmark include:

• Isolation - We'll determine how strong and complete the isolation between virtual environments is. Hereafter, by "virtual environment" I mean not just Virtuozzo VE but rather a system partition created by a virtualization technology.


• Security - We'll perform several attacks on the target system, trying to measure how robust the system is.


• Manageability - We'll perform typical management operations – such as provisioning, starting and stopping of virtual environments and applications – and measure how long does it take to do those as well as how easy or confusing it is for an admin to do them.


• Performance - We'll run a series of well-known performance tests, measuring performance of a single environment as well as performance of multiple concurrently running environments.


• Scalability - We'll measure how well a technology can use the resources provided by the hardware.


• Density - We'll measure how many concurrent environments it is possible to run with acceptable performance.


• Resource management - We'll determine if resource management capabilities provided by a virtualization technology are adequate for the test scenarios.

Kudos for someone in the industry for taking the torch and running with it. There needs to be some form of standard benchmarking that people can look to when trying to decide which virtualization platform they should go with.

If you are interested in helping to formulate these benchmarking metrics, send them your thoughts and ideas. They are looking forward to comments and participation from the industry.

Posted by David Marshall on July 17, 2006 03:33 PM