Skip to page content or Skip to Accesskey List.


Main Page Content

Of Servers And Benchmarks

Rated 3.89 (Ratings: 0)

Want more?


Seb Potter

Member info

User since: 22 Apr 1999

Articles written: 19

One of things that seems extremely popular in today's competitive server market seems to be the publishing of "independent" benchmarking tests for every new system. The epitomy of these publicity campaigns, and part of an ongoing battle that pits the open source community against their arch rivals, is the series of Mindcraft benchmarks that compare NT/IIS performance against that of Linux/Apache.

I'm not a Linux evangelist, nor do I kneel at the church of Microsoft. I believe in using the right solution for the right set of circumstances, so I'd like to explain why a benchmark can actually be extremely misleading.

The first thing to know is that tests such as this are mainly sponsored by a company who wants the tests to "find" a certain fact: namely that their product is better than the competition. However, there's a great difference between testing something to find what the results are, and finding results that prove what you want to prove.

Server benchmarks are, unfortunately for consumers, a great example of using a test to find specific results. It's extremely easy to say that a server is capable of serving 100 million static HTML pages in 24 hours, but look a little further at the results.

Firstly, very, very few sites with a large amount of content use static pages now. The outdated method of writing and updating every page separately has been replaced by dynamic technologies such as ASP and Cold Fusion. These scripting languages have extremely different server loads, and different requirements from serving static pages because of the pre-processing that needs to take place on the server.

It is also extremely important to understand the circumstances of the test, and how the hardware used can be weighted towards performance gains for certain software. Taking a recently used example, the Mindcraft tests used a RAID 0 disk array for the server. RAID 0 systems show significant performance differences depending on the operating system, however, they are not known for stability in situations that require it (such as a web server that needs continuous uptime). Most system administrators prefer to use RAID 5, which displays different characteristics under the tested systems, but is much more stable in real-world applications. The choice of disk array, although not realistic, could heavily weight a benchmark test in favour of a specific system.

The various ways in which benchmarks can be subtly manipulated to produce certain results are numerous, and the practice of using tests in this way are neither limited to certain companies, nor to the software industry. As long as marketing departments are willing to pay for publicity, the onus will remain with the consumer to try to make sense of the information that we are fed. In situations where your choice of hardware is mission-critical, it will always pay to get a second opinion.

Seb is a Jedi Master in the art of creating sites and keeping servers running. This often means hitting them repeatedly with forces that defy rational explanation, though he prefers to descibe it as "administration". When he's not practising his percussive skills on E450s and AS400s, he can be found masquerading as the senior developer for some widely varied clients. It's still not certain whether or not the meanings of CMS, CRM, and B2B have penetrated the alcoholic fog enveloping his brain, but he makes convincing noises to customers about XML, XSLT, Python, J2EE, PHP, Perl, C++, and OpenGL.

Seb has been in the web game pretty much since it began, and still has fond memories of the time when a web could be swept aside with a duster and spam was pork luncheon meat. Despite being the developer of one of the first commerce sites in Europe, he has yet to make any real money.

Being English, Seb doesn't like SOAP, but instead has recently discovered something called ZOPE. Zope is a platform that runs Plone which he thinks is the coolest thing since high-performance, real-time 3D APIs, which he often writes small games in.

Seb lives in the best little city in the world, and used to commute 5 hours a day on British trains. He is subsequently immune to all forms of torture techniques.

The access keys for this page are: ALT (Control on a Mac) plus: is an all-volunteer resource for web developers made up of a discussion list, a browser archive, and member-submitted articles. This article is the property of its author, please do not redistribute or use elsewhere without checking with the author.