Spark SQL is one of the main components of the Apache Spark framework. It is mainly used for structured data processing. It provides various Application Programming Interfaces (APIs) in Python, Java, Scala, and R. Spark SQL integrates relational data processing with the functional programming API of Spark.
Spark SQL provides a programming abstraction called DataFrame and can also act as a distributed query engine (querying on different nodes of a cluster). It supports querying either with Hive Query Language (HiveQL) or with SQL.
Check out this insightful video on Spark Tutorial for Beginners:
Learn for free ! Subscribe to our youtube Channel.
If you are familiar with the Relational Database Management System (RDBMS) and its tools, then you can say that Spark SQL is just an extension of relational data processing. Big Data can be processed using Spark SQL, which is difficult to implement on a traditional database system.
Want to master end-to-end Big Data Hadoop? Check out this Big Data and Spark Training in Toronto!
Why did Spark SQL come into the picture?
Before Spark SQL, there was Apache Hive, which was used for structured data processing. Apache Hive was originally developed to run on Apache Spark, but it had certain limitations as follows:
- Hive deploys MapReduce algorithms for ad-hoc querying. We know that MapReduce algorithms lag in performance when it comes to medium-sized datasets.
- During the execution of a workflow, if the processing fails, Hive does not have the capability to resume from the point where it failed.
- Apache Hive does not support real-time data processing; it uses batch processing instead. It collects the data and processes it in bulk later.
Having outlined all these drawbacks of Hive, it is clear that there was a scope for improvement, which is why Spark SQL came into the picture.
Learn about Apache Spark from this Big Data and Spark Training and be a master!
Understanding Spark SQL
Spark SQL provides faster execution than Apache Hive. It uses in-memory computation where the time required to move data in and out of a disk is lesser when compared to Hive.
- Spark SQL supports real-time data processing. This data is mainly generated from system servers, messaging applications, etc.
- It does not face any migration difficulty, i.e., we can migrate or import anything which is written in Hive, without any difficulty. Whatever metastore we have used for Apache Hive can be used for Spark SQL as well.
- Querying in Spark SQL is easier when compared to Apache Hive. Spark SQL queries are similar to traditional RDBMS queries.
Now, let us understand the architecture of Spark SQL.
Architecture of Spark SQL
The architecture of Spark SQL consists of three layers as explained below:
- Language API: This layer consists of APIs supported by Python, Java, Scala, and R. Spark SQL is compatible with all these programming languages.
- SchemaRDD: An RDD (Resilient Distributed Dataset) is a special data structure with which Spark Core is equipped. As Spark SQL works on schemas, tables, and records, we can use a SchemaRDD as a temporary table. SchemaRDDs are also known as DataFrames.
- Data Sources: Spark SQL can process data from various sources. Data sources for Spark SQL can be JSON files, Hive tables, Parquet files, and Cassandra database.
Features of Spark SQL
Let’s take a stroll into the aspects that make Spark SQL so popular in data processing.
- Easy to Integrate: One can mix SQL queries with Spark programs easily. Structured data can be queried inside Spark programs using either SQL or a Dataframe API. Running SQL queries alongside analytic algorithms is easy because of this tight integration.
- Compatibility with Hive: Hive queries can be executed in Spark SQL as they are.
- Unified Data Access: Loading and querying data from various sources is possible.
- Standard Connectivity: Spark SQL can connect to Java and Oracle using JDBC (Java Database Connectivity) and ODBC (Oracle Database Connectivity) APIs.
- Performance and Scalability: To make queries agile, alongside computing hundreds of nodes using the Spark engine, Spark SQL incorporates a code generator, a cost-based optimizer, and columnar storage. This provides complete mid-query fault tolerance.
Grasp detailed knowledge of Apache Spark by going through this extensive Spark Tutorial!
Spark SQL Libraries
- Data Source API: This is used to read/write structured and unstructured data from/to Spark SQL. In Spark SQL, we can fetch the data from multiple sources.
- DataFrame API: DataFrame API converts the fetched data into tabular columns that can further be used for SQL operations. These tables are equivalent to relational databases in SQL.
- SQL Interpreter and Optimizer: Interpreters and Optimizers are used to optimize the queries written both in Spark SQL and DataFrames. They are used to run SQL queries faster than their RDD counterparts.
- SQL Service: SQL service is used to fetch the interpreted and optimized data.
Querying Using Spark SQL
In Spark SQL, Datasets comprise distributed computation that is converted into a sequence of RDDs called DAG (Directed Acyclic Graph). Here is a simple example for querying using Spark SQL.
If you have any query related to Spark and Hadoop, kindly refer to our Big Data Hadoop and Spark Community!
Adding a Schema to RDDs Programmatically
We can add a schema to an existing RDD programmatically. It can be done by importing the respective API, applying transformations, etc.
In the below example, a text file is being stored into an RDD. And then, we are adding a schema to that RDD.
The file employees.txt has the following data:
Prepare yourself for top jobs by learning from these Apache Spark Interview Questions and Answers!
Caching Tables In-memory
Here, we are creating a temporary view in memory for the above text file. Temporary views can be accessed faster as they are stored in memory. Since queries are performed on a cached table, we will get the desired results in no time.
Caching is mainly done for faster execution.
So, this brings us to the end of this section. We will learn about PySpark in the coming section.
Intellipaat provides the most comprehensive Big Data and Spark Course in Sydney to fast-track your career!