Why and how to benchmark XML databases

Benchmarks belong to the very standard repertory of tools deployed in datab ase development. Assessing the capabilities of a system, analyzing actual a nd potential bottlenecks, and, naturally, comparing the pros and cons of di fferent systems architectures have become indispensable tasks as databases management systems grow in complexity and capacity. In the course of the de velopment of XML databases the need for a benchmark framework has become mo re and more evident: a great many different ways to store XML data have bee n suggested in the past, each with its genuine advantages, disadvantages an d consequences that propagate through the layers of a complex database syst em and need to be carefully considered. The different storage schemes rende r the query characteristics of the data variably different, However, no con clusive methodology for assessing these differences is available to date. In this paper, we outline desiderata for a benchmark for XML databases draw ing from our own experience of developing an XML repository, involvement in the definition of the standard query language, and experience with standar d benchmarks for relational databases.