A pre-research on GWAC massive catalog data storage and processing system

GWAC (Ground Wide Angle Camera) poses huge challenges in large-scale catalogue storage and real-time processing of quick search of transients among wide field-of-view time-series data. Firstly, this paper proposes the concept of using databases’ capabilities of fast data processing and parallelism, which will improve system performance and availability through the integration of data storage and computing platform. To understand the feasibility of Column-store MonetDB in vast catalogue management, we carry out a variety of pilot experiments of key technologies. We conduct TPC-H benchmark, data loading benchmark and optimization, and key algorithm testing of astronomical source association, all compared with the traditional row store database. Then, we use MonetDB to realize cross-match Zone algorithm. UDF function is developed for customizable data loading. Tests results show t MonetDB database has a remarkable performance in large amounts of data management and is efficient in real-time data process, thus has the ability to deal with 2.5T catalog data. In the end we propose a wide field of view massive time serial observation data processing solution using the in-memory column store database MonetDB. The experimental results show that the feasibility of the scheme. The design plan of MonetDB-based massive catalogue data processing solution, is an efficient astronomical database solution that combines data processing and data management.