Towards Dynamic Catalogues

The International LOFAR Telescope is designed to carry out unique science in the spatial, spectral, polarisation and temporal domains. The Transients Key Science Project aims to study all transient and variable sources detected by LOFAR. One of its products will be an up-to-date catalogue of all sources detected by LOFAR, i.e. a spectral light-curve database, with real-time capabilities, and expected to grow gradually with 50−100 TB/yr. The response time to transient and variable events depends strongly on the query execution plans of the algorithms that search the LOFAR light-curve database for previous (non-)detections in the spatial, spectral, polarisation and temporal domains. Here we show how the Transients Key Science Project of LOFAR approaches these challenges by using column-stores, sharded databases and implementing the new array query language SciQL (pronounced as ’cycle’).