Performance Evaluation in Database Research: Principles and Experience

A significant part of today database research focuses on improving performance of a specific system. Quantitative experiments are the best way to validate such results. However, performing experiments is not always easy. Besides the complexity of the system under test, designing an experiment, chosing the right environment and parameter values, analyzing the data which is gathered, and reporting it to a third party in an expressive and intelligible way is hard. In this tutorial, we present a general roadmap to the above steps, based on classical measure taking theory, as well as our own experience. The tutorial is primarily aimed at MS and PhD students seeking to improve their experiment practices, but more senior attendants may also find it interesting. The tutorial will also devote a short time (~15 minutes) to tips and tricks on how to organize and present code that performs experiments, so that an outsider can repeat them.