SIGMOD 2008 was the first database conference that offered to test submitters' programs against their data to verify the experiments published. This paper discusses the rationale for this effort, the community's reaction, our experiences, and advice for future similar efforts.

Database Architectures

Manolescu, I., Afanasiev, L., Arion, A., Dittrich, J., Manegold, S., Polyzotis, N., … Shasha, D. (2008). The Repeatability Experiment of SIGMOD 2008. SIGMOD Record, 37(1), 39–45.