Spark SQL is one of the main component of the Apache Spark Framework. It is mainly used for structured data processing. It provides various Application Programmable Interfaces (APIs) in Python, Java, Scala, and R. Spark SQL integrates relational data processing with the functional programming API of Spark.
It provides a programming abstraction called Dataframe and can also act as a distributed querying engine (querying on different nodes of a cluster). It supports querying using either the Hive Query Language (HQL) or SQL.
Want to become master in Big Data Hadoop? Check out this Big Data & Spark Training in Toronto.
Check out this insightful video on Spark Tutorial For Beginners
If you are familiar with the Relational Database Management System (RDBMS) concepts and its tools, then you can say that Spark SQL is just an extension of the relational data processing. Big Data can be processed using Spark SQL, which is difficult to implement on a traditional database system.
Why Did Spark SQL Come into the Picture?
Before Spark SQL, there was Apache Hive, which was used for structured data processing. Apache Hive was originally developed to run on Apache Spark. But it had certain limitations like:
- Hive deploys MapReduce Algorithm for ad-hoc querying. We know that MapReduce Algorithm lags in performance when it comes to medium-sized datasets.
- During the execution of a workflow, if the processing fails, Hive does not have the capability to resume from the point where it failed.
- Apache Hive does not support real-time data processing; it uses batch processing instead. It collects the data and processes it in bulk later.
Having outlined all these drawbacks of Hive, it is clear that there was a scope for improvement, which is why Spark SQL came into picture.
Read about Apache Spark from Big Data & Spark Training and be master as an Apache Spark Specialist.
Understanding Spark SQL
Spark SQL provides faster execution than Apache Hive. It uses in-memory computation where the time required to move data in and out of a disk is lesser when compared to Hive.
- Spark SQL supports real-time data processing. This data is mainly generated from system servers, messaging applications, etc.
- It does not faces any migration difficulty, i.e., you can migrate or import anything which is written in Hive, without any difficulty. Whatever metastore you have used for Apache Hive can be used for Spark SQL also.
- Querying in Spark SQL is easier when compared to Apache Hive. Spark SQL queries are similar to the traditional RDBMS queries.
Now, let us understand about the architecture of Spark SQL. The architecture of Spark SQL consists of three layers:
- Language API: This layer consists of APIs supported by Python, Java, Scala, and R. Spark SQL is compatible with all those programming languages.
- Schema RDD: RDD (resilient distributed dataset) is a special data structure with which Spark core is equipped. As Spark SQL works on schemas, tables, and records, we can use Schema RDD as a temporary table. Schema RDDs are also known as Dataframes.
- Data Sources: Spark SQL can process data from various sources. Data sources for Spark SQL can be JSON files, Hive tables, Parquet files, and Cassandra database.
Features of Spark SQL
Let’s take a stroll into the aspects which make Spark SQL so popular in data processing.
- Integrated: One can mix SQL queries with Spark programs easily. Structured data can be queried inside Spark programs using either SQL or a Dataframe API. Running SQL queries alongside analytic algorithms is easy because of this tight integration.
- Hive Compatibility: Hive queries can be executed in Spark SQL without any changes to be done.
- Unified Data Access: Loading and querying data from various sources is possible.
- Standard Connectivity: Spark SQL can connect to Java and Oracle using the JDBC (Java Database Connectivity) and ODBC (Oracle Database Connectivity) APIs.
- Performance and Scalability: To make queries agile alongside computing hundreds of nodes using the Spark engine, Spark SQL incorporates a code generator, a cost-based optimizer and a columnar storage. This provides complete mid-query fault tolerance.
Want to grasp a detailed knowledge on Apache Spark? Read this extensive Spark Tutorial!
Spark SQL Libraries
Data Source API: This is used to read and write structured/unstructured data into Spark SQL. In Spark SQL, we can fetch the data from multiple sources.
Dataframe API: Dataframe API converts the fetched data into tabular columns which can further be used for SQL operations. These tables are equivalent to relational databases in SQL.
SQL Interpreter and Optimizer: Interpreters and Optimizers are used to optimize the queries written both in Spark SQL and Dataframes. They are used to run SQL queries faster than their RDD counterparts.
SQL Service: SQL service is used to fetch the interpreted and optimized data.
Querying Using Spark SQL
In Spark SQL, Datasets comprise distributed computation that is converted into a sequence of RDDs which is called DAG (Directed Acyclic Graph). Here is a simple example for querying using Spark SQL.
Adding Schema to RDDs Programmatically
We can add a schema to an existing RDD programmatically. It can be done by importing the respective APIs, applying transformations, etc.
In the below example, a text file is being stored onto an RDD. And then, we are adding a schema for that RDD.
If you have any query related to Spark and Hadoop, kindly refer our Big Data Hadoop & Spark Community.
The file employees.txt has the following data:
Prepare yourself for the Top Apache Spark Interview Questions And Answers Now!
Caching Tables In-memory
Here, we are creating a temporary view in memory for the above text file. Temporary views can be accessed faster as they are stored in memory. Since, the queries are being performed on a cached table, we will get the desired results in no time.
Caching is mainly used for faster execution.
So, this brings us to the end of this section. We will learn more about Dataframes in the coming Section.
Intellipaat provides the most comprehensive Big Data & Spark Course in Sydney to fast-track your career!