In this paper we describe VectorH: a new SQL-on-Hadoop system built on top of the fast Vectorwise analytical database system. VectorH achieves fault tolerance and scalable data storage by relying on HDFS, extending the state-of-the-art in SQL-on-Hadoop systems by instrumenting the HDFS block replication policy to ensure local reads under most circumstances. VectorH integrates with YARN for workload management, achieving a high degree of elasticity . Even though HDFS is an append-only filesystem, and it supports ordered table storage, VectorH can accommodate trickle updates through Positional Delta Trees (PDTs), a differential update structure that can be queried efficiently. We describe the main technical extensions to single-server Vectorwise that turned it into a Hadoop-based MPP system, in terms of workload management, parallel query optimization and execution, HDFS storage, transaction processing and Spark integration. In the evaluation section we compare VectorH with HAWQ, Impala, SparkSQL and Hive, showing orders of magnitude better performance than these competitors.
Additional Metadata
Keywords data storage, query optimization, parallel query execution, cluster computing, hadoop
THEME Information (theme 2)
Stakeholder Actian Corp., Amsterdam, Netherlands
Project Actian CWI Research Grant
Conference ACM SIGMOD International Conference on Management of Data
Grant This work was funded by the CWI PPS samenwerking; grant id pps/05050504 - Actian CWI Research Grant
Citation
Switakowski, M, Costea, A, Ionescu, A, Raducanu, B, Bârca, C, Sompolski, J, … Boncz, P.A. (2016). VectorH: taking SQL-on-Hadoop to the next level.

Additional Files
24383B.pdf Author Manuscript , 844kb