How to advance TPC benchmarks with dependability aspects



Transactional systems are the core of the information systems of most organizations. Although there is general acknowledgement that failures in these systems often entail significant impact both on the proceeds and reputation of companies, the benchmarks developed and managed by the Transaction Processing Performance Council (TPC) still maintain their focus on reporting bare performance. Each TPC benchmark has to pass a list of dependability-related tests (to verify ACID properties), but not all benchmarks require measuring their performances. While TPC-E measures the recovery time of some system failures, TPC-H and TPC-C only require functional correctness of such recovery. Consequently, systems used in TPC benchmarks are tuned mostly for performance. In this paper we argue that nowadays systems should be tuned for a more comprehensive suite of dependability tests, and that a dependability metric should be part of TPC benchmark publications. The paper discusses WHY and HOW this can be achieved. Two approaches are introduced and discussed: augmenting each TPC benchmark in a customized way, by extending each specification individually; and pursuing a more unified approach, defining a generic specification that could be adjoined to any TPC benchmark.


Industry standard benchmarks, ACID properties, Durability, Dependability.


Second TPC technology conference on Performance evaluation, measurement and characterization of complex systems 2010

Cited by

Year 2013 : 1 citations

 Alessandro Gustavo Farias Fior, "Under Pressure Benchmark: A Large-Scale Availability Benchmark For Distributed Databases", Msc Thesis, Universidade Federal Do Paraná, Curitiba, Brazil, 2013.

Year 2011 : 1 citations

 1. Yantao Li, Charles Levine, “Extending TPC-E to Measure Availability in Database Systems”, Third TPC Technology Conference on Performance Evaluation and Benchmarking, TPCTC 2011, Seattle, WA, USA, August 29, 2011.