• Articles
  • Tutorials
  • Interview Questions

Spark Components - Explained

Introduction to Spark Components

The following image gives you a clear picture of the different Spark components.

Apache Spark Core

Apache Spark Core consists of a general execution engine for the Spark platform which is built as per the requirement. It provides in-built memory computing and references datasets stored in external storage systems.

Check out this insightful video on Spark Tutorial for Beginners:

Video Thumbnail

Spark allows developers to write code quickly with the help of a rich set of operators. While it takes a lot of lines of code in other programming languages, it takes fewer lines when written in Spark Scala. The following program will help you understand the way programming is done with Spark:

sparkContext.textFile("hdfs://...")
            .flatMap(line => line.split(" "))
            .map(word => (word, 1)).reduceByKey(_ + _)
            .saveAsTextFile("hdfs://...")

Get certified from the top Big Data and Spark Course in New York now!

Spark SQL

Spark SQL is a component on top of Spark Core that introduces a new set of data abstraction called SchemaRDD. SchemaRDD provides support for both structured and semi-structured data.

Below is the example of a Hive compatible query:

// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
sqlContext.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src")
// Queries are expressed in HiveQL
sqlContext.sql("FROM src SELECT key, value").collect().foreach(println)

Spark Streaming

It is one of the Apache Spark components, and it allows Spark to process real-time streaming data. It provides an API to manipulate data streams that match with the RDD API. It allows programmers to understand the project and switch through the applications that manipulate the data and give the outcome in real time. Similar to Spark Core, Spark Streaming strives to make the system fault-tolerant and scalable.
RDD API Example:
In this example, you will use a few transformations that are implemented to build a dataset of (String, Int) pairs called counts and then save it to a file.

text_file = sc.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) 
             .map(lambda word: (word, 1)) 
             .reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")

MLlib (Machine Learning Library)

Apache Spark is equipped with a rich library known as MLlib. This library contains a wide array of Machine Learning algorithms, classification, clustering, and collaboration filters, etc. It also includes a few low-level primitives. All these functionalities help Spark scale out across a cluster.

Fast-track your career by applying for this Big Data and Spark Training Course!

Prediction with Logistic Regression

In this example, you will take dataset values in terms of labels and feature vectors. You will learn to predict the labels from feature vectors using the method of logistic regression with Python:

# Every record of this DataFrame contains the label and
# features represented by a vector.
df = sqlContext.createDataFrame(data, ["label", "features"])
# Set parameters for the algorithm.
# Here, you need to limit the number of iterations to 10.
lr = LogisticRegression(maxIter=10)
# Fit the model to the data.
model = lr.fit(df)
# Given a dataset, predict each point's label, and show the results.
model.transform(df).show()

Prepare yourself for interviews with these Spark Interview Questions and Answers and step into a lucrative career!

GraphX

Spark also comes with a library to manipulate graphs and perform computations, which is called GraphX. Just like Spark Streaming and Spark SQL, GraphX also extends Spark RDD API, which creates a directed graph. It also contains numerous operators in order to manipulate the graphs, along with graph algorithms.
Consider the following example to model users and products as a bipartite graph:

class VertexProperty()
case class UserProperty(val name: String) extends VertexProperty
case class ProductProperty(val name: String, val price: Double) extends VertexProperty
// The graph might then have the type:
var graph: Graph[VertexProperty, String] = null

Intellipaat provides the most comprehensive Big Data and Spark Course in Bangalore to fast-track your career!

Course Schedule

Name Date Details
Big Data Course 19 Oct 2024(Sat-Sun) Weekend Batch
View Details
Big Data Course 26 Oct 2024(Sat-Sun) Weekend Batch
View Details
Big Data Course 02 Nov 2024(Sat-Sun) Weekend Batch
View Details

About the Author

Technical Reseach Analyst - Data Engineering

Abhijit is a Technical Research Analyst specializing in Deep Learning. He holds a degree in Computer Science with a focus on Data Science. Being proficient in Python, Scala, C++, Dart, and R, he is passionate about new-age technologies. Abhijit crafts insightful analyses and impactful content, bridging the gap between cutting-edge research and practical applications.