VectorH: taking SQL-on-Hadoop to the next level
Presented at the ACM SIGMOD International Conference on Management of Data, San Francisco
In this paper we describe VectorH: a new SQL-on-Hadoop system built on top of the fast Vectorwise analytical database system. VectorH achieves fault tolerance and scalable data storage by relying on HDFS, extending the state-of-the-art in SQL-on-Hadoop systems by instrumenting the HDFS block replication policy to ensure local reads under most circumstances. VectorH integrates with YARN for workload management, achieving a high degree of elasticity . Even though HDFS is an append-only filesystem, and it supports ordered table storage, VectorH can accommodate trickle updates through Positional Delta Trees (PDTs), a differential update structure that can be queried efficiently. We describe the main technical extensions to single-server Vectorwise that turned it into a Hadoop-based MPP system, in terms of workload management, parallel query optimization and execution, HDFS storage, transaction processing and Spark integration. In the evaluation section we compare VectorH with HAWQ, Impala, SparkSQL and Hive, showing orders of magnitude better performance than these competitors.
|Keywords||data storage, query optimization, parallel query execution, cluster computing, hadoop|
|THEME||Information (theme 2)|
|Stakeholder||Actian Corp., Amsterdam, Netherlands|
|Project||Actian CWI Research Grant|
|Conference||ACM SIGMOD International Conference on Management of Data|
|Grant||This work was funded by the CWI PPS samenwerking; grant id pps/05050504 - Actian CWI Research Grant|
Switakowski, M, Costea, A, Ionescu, A, Raducanu, B, Bârca, C, Sompolski, J, … Boncz, P.A. (2016). VectorH: taking SQL-on-Hadoop to the next level.
|24383B.pdf Author Manuscript , 844kb|