Last month, I began a series of articles describing database application benchmarking. In the first article, I told you about different ways that you can construct your own database application benchmark. However, you don’t have to reinvent the wheel. The Transaction Processing Council (www.tpc.org) has already created a large number of database benchmarks that are extremely useful and informative.
I also described last month how the TPC provides several different types of benchmark tests. For example the TPC-C and TPC-E benchmarks are extremely useful for measuring transaction throughput. On the other hand, the TPC – H benchmark is useful for measuring business intelligence workloads.
Today, I would like to give you a primer on how to read the benchmark reports that are published by the major database and hardware vendors. You never know when a vendor will publish a new benchmark. There’s no set schedule for them to publish their test findings. Of course, you can always look for new advertisements from many of the vendors. But that’s very imprecise. I prefer to find out if there are new results on my own and so I typically start at http://tpc.org/information/results.asp. There, I’ll check to see if my favorite hardware or database vendors have published any new test results.
When you go to the TPC results listing webpage, you’ll find that you can check TPC benchmark results by performance, by price/performance, and by watts/performance. For example, you could check to see who were the top 10 winners of the TPC – E benchmark strictly on performance. When you click on the test that you wish to see the results for, you are taken to a top 10 list in table format. From there, you can check each specific test and look at all the details of those tests. Let’s say that you use IBM hardware, you could check to see if IBM was on the list and see how their test results ranked.
As it happens, IBM is currently number-two on the list. The test was conducted in August 2011 and you can also see that the auditor of the test was Mr. Doug Johnson. It’s important to note, that all TPC tests are officially audited. At the very bottom of that page, you’ll see three important hyperlinks. They are: executive summary, full disclosure report, and supporting files. Usually, I like to read only the executive summary.
The executive summary shows you all kinds of interesting things. For example, IBM built a 18.4 TB database for the specific TPC test. They used RAID level 10 on the transaction level log. They used RAID level 5 on the database files. You can also see the full configuration of the hardware, including the type and number of processors and course, along with the type and number of hard disks and SSDs.
One of the most useful aspects of the TPC benchmark executive summary and full disclosure report are the TPC pricing information pages. On these pages, you can see exactly what price the hardware vendor has set for each component of the system. If you’re not currently getting quantity discounts from your hardware vendor, you can use these pricing sheets to figure out what is a reasonable discount for your organization to get from the major vendors! Think you’re paying too much for a hard disk? Then check the TPC pricing report to see him much they typically cost below the MSRP.