The best database benchmarks are those that accurately and reliably reflect the applications and configuration of your own database infrastructure. On the other hand, the amount of work that goes into extracting your own transactional workload can be immense. An easier route is to learn and run your own TPC benchmarks, use one of the free tools to run the benchmark, and then extrapolate the TPC test results for your environments.
In light of the past several articles in this column about the TPC benchmarks, you’re probably wondering how you can do your own TPC benchmark test. First, is this caveat: A “true” TPC benchmark must go through a rigorous and expensive auditing process. The auditing process was put in place to make sure that database vendors do not make misleading or false performance claims, as well as to maintain the credibility of the TPC benchmarks. Only by releasing audited results can a test truly by called a TPC benchmark. Audited TPC benchmarks are quite costly and take a lot of time to set up and conduct. Most vendors who undergo the benchmarks either have full time staff dedicated to the proper testing process or hire certified consultants from the TPC-approved list of auditors.
You probably don’t want to go to all of that trouble, and neither do I, since I’m not publishing press releases or running public ad campaigns based on the results. So when I say “run your own TPC benchmark,” what I really mean is running a “TPC-like” benchmark which contains all of the activities of a regular TPC benchmark, but without the auditing.
Now that the nomenclature housekeeping is out of the way, let’s talk more about running your own TPC database benchmarks. The good news is that the TPC benchmarks are public industry standards, whose specifications are distributed by the Transaction Processing Council at no charge to the public. And, in even better news, there are a few free high-quality tools available to run TPC benchmarks for your own internal consumption. (There are also some excellent commercial tools available in the marketplace. But they are beyond the scope of this discussion).
So over the next couple columns, I’ll be taking you through the steps to run your own internal TPC benchmark and, in the process, demonstrating a few of my favorite tools. The tools that I’m currently planning to describe include:
HammerDB: I really like HammerDB for the TPC-C and TPC-H test. It can be found here. It’s also good for lots of database platforms like SQL Server, Oracle, MySQL and Riak.
Dell DVD Store: Another really good one is put out by Dell to show total IOPs, latency, and throughput. It also supports a number of database platforms, SQL Server, Oracle, PostgreSQL and MySQL.
TPC Tests: Some of the benchmarks of the Transaction Processing Council are available as tests from their own website. I’ll discuss these in more detail as well.
There may be some other tools out there that I haven’t examined yet. Are you running your own TPC benchmarks? Are you using a tool that’s not on my list? How did you do it? Since I’ll be writing these articles over the next couple months, I still have time to add more tools to the list if you recommend some good ones. I’d love to hear back from you!
Enjoy!
-Kev
Follow me on Twitter and Google.