Apache Spark Tutorial - Learn Apache Spark from Scratch

What is Apache Spark?

Apache Spark is a lightning-fast cluster computing framework designed for real-time processing. Spark is an open-source project from Apache Software Foundation. Spark overcomes the limitations of Hadoop MapReduce, and it extends the MapReduce model to be efficiently used for data processing.

Spark is a market leader for big data processing. It is widely used across organizations in many ways. It has surpassed Hadoop by running 100 times faster in memory and 10 times faster on disks.

Overview of Big Data and Hadoop

The quantity of data is growing exponentially for many reasons these days. Our day-to-day activities in various sources generate lots of data. So, the term ‘Big Data‘ is used to denote a collection of large and complex datasets that is difficult to store and process using the available database management tools or traditional data processing applications.

Apache Hadoop was developed to enhance the usage of big data and solve the major issues related to it. Formally, Google invented a new methodology of processing data popularly known as MapReduce. Later, Doug Cutting and Mike Cafarella, inspired by the white paper of the MapReduce framework, developed Hadoop to apply MapReduce concepts to an open-source software framework that supported the Nutch search engine project.

Data can be categorized as Big Data based on various factors. The main concept common in all these factors is the amount of data. Let us understand the characteristics of Big Data which we have broken down into 5 Vs:

1. Velocity

Velocity refers to the speed at which data arrives. Every day, a huge amount of data is generated, stored, and analyzed.

2. Volume

Volume refers to the huge amount of data generated from credit cards, social media, IoT devices, smart home gadgets, videos, etc.

3. Variety

Variety refers to the different types of data. Data is mainly categorized into structured and unstructured data.

4. Veracity

Veracity refers to the quality of the data. It depends on the reliability and accuracy of the content. We should not store loads of data if the content is not reliable or accurate.

5. Value

Value is the most important part of Big Data. Having a vast amount of data is useless until we extract something meaningful from it.

Although Hadoop made a grasp on the market, there were some limitations. Hadoop is used to process data in various batches, therefore, real-time data streaming is not possible with Hadoop.

Apache Spark, unlike Hadoop clusters, allows real-time Data Analytics using Spark Streaming. For this reason, Apache Spark has quite a fast market growth these days. The median salary of a Data Scientist who uses Apache Spark is around US$100,000. Isn’t that crazy?

Considering the original case study, Hadoop was designed with much simpler storage infrastructure facilities. Let us discuss Apache Spark further in this Spark tutorial.

Check out this insightful video on Apache Spark Tutorial for Beginners:

Let’s first understand how data can be categorized as Big Data.

Before Spark, there was MapReduce that was used as a processing framework. Then, Spark got initiated as one of the research projects in 2009 at UC Berkeley AMPLab. It was later open-sourced in 2010. The major intention behind this project was to create a cluster management framework that supports various computing systems based on clusters. After its release in the market, Spark grew and moved to Apache Software Foundation in 2013. Now, most organizations across the world have incorporated Apache Spark for empowering their Big Data applications.

Check out this insightful video on ‘Apache Spark Tutorial for Beginners:

Most technology-based companies across the globe have moved toward Apache Spark. They were quick enough to understand the real value possessed by Sparks such as Machine Learning and interactive querying. Industry leaders such as Amazon, Huawei, and IBM have already adopted Apache Spark. The firms that were initially based on Hadoop, such as Hortonworks, Cloudera, and MapR, have also moved to Apache Spark.

Big Data Hadoop professionals surely need to learn Apache Spark since it is the next most important technology in Hadoop data processing. Moreover, even ETL professionals, SQL professionals, and Project Managers can gain immensely if they master Apache Spark. Finally, Data Scientists also need to gain in-depth knowledge of Spark to excel in their careers.

Spark can be extensively deployed in Machine Learning scenarios. Data Scientists are expected to work in the Machine Learning domain, and hence, they are the right candidates for Apache Spark training. Those who have an intrinsic desire to learn the latest emerging technologies can also learn Spark through this Apache Spark tutorial.

Today, there is widespread deployment of Big Data tools. With each passing day, the requirements of enterprises increase, and therefore, there is a need for a faster and more efficient form of data processing. Most streaming data is in an unstructured format, coming in thick and fast continuously. Here, in this Apache Spark tutorial, we look at how Spark is used successfully in different industries.

Banking

Spark is being increasingly adopted by the banking sector. It is mainly used here for financial fraud detection with the help of Spark ML. Banks use Spark to handle credit risk assessment, customer segmentation, and advertising. Apache Spark is also used to analyze social media profiles, forum discussions, customer support chats, and emails. This way of analyzing data helps organizations make better business decisions.

E-commerce

Spark is widely used in the e-commerce industry. Spark Machine Learning, along with streaming, can be used for real-time data clustering. Businesses can share their findings with other data sources to provide better recommendations to their customers. Recommendation systems are mostly used in the e-commerce industry to show new trends.

Healthcare

Apache Spark is a powerful computation engine to perform advanced analytics on patient records. It helps keep track of patients’ health records easily. The healthcare industry uses Spark to deploy services to get insights such as patient feedback, and hospital services, and to keep track of medical data.

Media

Many gaming companies use Apache Spark for finding patterns from their real-time in-game events. With this, they can derive further business opportunities by customizing things such as adjusting the complexity level of the game automatically according to players’ performance, etc. Some media companies, like Yahoo, use Apache Spark for targeted marketing, customizing news pages based on readers’ interests, and so on. They use tools such as Machine Learning algorithms for identifying the readers’ interests category. Eventually, they categorize such news stories in various sections and keep the reader updated on a timely basis.

Travel

Many people land up with travel planners to make their vacation a perfect one, and these travel companies depend on Apache Spark for offering various travel packages. TripAdvisor is one such company that uses Apache Spark to compare different travel packages from different providers. It scans hundreds of websites to find the best and most reasonable hotel prices, trip packages, etc.

Apache Spark has the following features:

• Polyglot – Spark code can be written in Python, R, Java, and Scala. There are shells provided for Scala and Python. These can be accessed from the installed directory.
• Speed – Spark can process large data that is 100 times faster than Hadoop MapReduce. This is possible because of controlled partitioning. Spark can manage data in partitions which help parallelize distributed data processing without excess network traffic.
• Multiple Formats – Spark supports multiple data sources. This makes access easier with the help of the Data Source API.
• Lazy Evaluation – Spark can delay the evaluation unless it’s absolutely necessary. This contributes majorly to its high speed.
• Real-Time Computation – Spark can compute in real time. Its latency is low as it can compute in memory. Spark is highly scalable with users running clusters with thousands of nodes.
• Hadoop Integration – Spark can be integrated with Hadoop. It helps all the Big Data Engineers who would have started their career with Hadoop.
• Machine Learning – Spark has a Machine Learning component called MLib. It comes in handy for processing big data. You don’t need to use different tools for processing and machine learning. There are numerous opportunities under Machine Learning.

Our Apache Spark tutorial won’t be complete without talking about the interesting use cases of Apache Spark. Let’s now look at a few use cases of Apache Spark.

Finding a Spark at Yahoo!

Yahoo! has over 1 billion monthly users. Therefore, it has to manage its data arriving at a fast rate on a huge scale. It uses a Hadoop cluster with more than 40,000 nodes to process data. So, it wanted a lightning-fast computing framework for data processing. Hence, Yahoo! adopted Apache Spark to solve its problem.

How Apache Spark Enhanced Data Science at Yahoo!

Although Spark is a quite fast computing engine, it is in demand for many other reasons as follows:

• It works with various programming languages.
• It has efficient in-memory processing.
• It can be deployed over Hadoop through YARN.

Yahoo! checked Spark over Hadoop using a project, which was intended to explore the power of Spark and Hadoop together. The project was implemented using Spark’s Scala API, which gets executed much faster through Spark, whereas Hadoop took more time for the same process.

Although Spark’s speed and efficiency are impressive, Yahoo! isn’t removing its Hadoop architecture. They need both; Spark will be preferred for real-time streaming and Hadoop will be used for batch processing. The most interesting fact here is that both can be used together through YARN.

Apache Spark at eBay

An American multinational e-commerce corporation, eBay creates a huge amount of data every day. eBay has lots of existing users, and it adds a huge number of new members every day. Except for sellers and buyers, the most important asset for eBay is data. eBay directly connects buyers and sellers. So, a lightning-fast engine is required to handle huge volumes of this real-time streaming data.

Apache Spark is mainly used to redefine better customer experience and overall performance at eBay. Apache Spark and Hadoop YARN combine the powerful functionalities of both. Hadoop’s thousands of nodes can be leveraged with Spark through YARN.

Hopefully, this tutorial gave you an insightful introduction to Apache Spark. Further, Spark Hadoop and Spark Scala are interlinked in this tutorial, and they are compared at various fronts. We will be learning Spark in detail in the coming sections of this Apache Spark tutorial.

Both Hadoop and Spark are open-source projects from Apache Software Foundation, and they are the flagship products used for Big Data Analytics. The key difference between MapReduce and Spark is their approach toward data processing. Spark can perform in-memory processing, while Hadoop MapReduce has to read from/write to a disk. Let us understand some major differences between Apache Spark and Hadoop in the next section of this Apache Spark tutorial.

These are the major differences between Apache Spark and Hadoop. But, what if we use Apache Spark with Hadoop? When we use both technologies together, it provides more powerful cluster computing with batch processing and real-time processing.

Spark is designed for the enhancement of the Hadoop stack. Spark can perform read/write data operations with HDFS, HBase, or Amazon S3. Hadoop users can use Apache Spark to enhance the computational capabilities of their Hadoop MapReduce system.

Apache Spark can be used with Hadoop or Hadoop YARN together. It can be deployed on Hadoop in three ways: Standalone, YARN, and SIMR.

Standalone Deployment

Spark provides a simple standalone deployment mode. This allows Spark to allocate all resources or a subset of resources in a Hadoop cluster. We can also run Spark in parallel with Hadoop MapReduce. Spark jobs can be deployed easily using HDFS data. Spark’s simple architecture makes it a preferred choice for Hadoop users.

Hadoop YARN Deployment

Apache Spark contains some configuration files for the Hadoop cluster. These config files can easily read/write to HDFS and YARN Resource Manager. We can easily run Spark on YARN without any pre-installation.

Spark in MapReduce (SIMR)

We can easily deploy Spark on MapReduce clusters as well. It will help us start experimenting with Spark to explore more.

Let us discuss some benefits of leveraging Hadoop and Spark together in the next section of this Apache Spark tutorial.

Most people think of Spark as a replacement for Hadoop, but instead of replacing Hadoop, we can consider Spark as a binding technology for Hadoop. However, Spark can run separately from Hadoop, where it can run on a standalone cluster. Meanwhile, Spark used on top of Hadoop can leverage its storage and cluster management.

Though Spark does not provide its own storage system, it can take advantage of Hadoop for that. Through this, we can make a powerful production environment using Hadoop capabilities. Spark can also use YARN Resource Manager for easy resource management. Spark can easily handle task scheduling across a cluster.

Apache Spark can use the disaster recovery capabilities of Hadoop as well. We can leverage Hadoop with Spark to receive better cluster administration and data management. Spark together with Hadoop provides better data security.

Spark Machine Learning provides capabilities that are not properly utilized in Hadoop MapReduce. Using a fast computation engine like Spark, these Machine Learning algorithms can now execute faster since they can be executed in memory. In MapReduce programs, on the other hand, the data gets moved in and out of the disks between different stages of the processing pipeline.

Next, in this Spark tutorial, we will check out some market leaders who have implemented Spark and Hadoop together.

Spark and Hadoop together make a powerful combination to handle Big Data Analytics. The following organizations are using Spark on Hadoop MapReduce and YARN.

Let us finally get into our main section of this Apache Spark tutorial, where we will be discussing ‘What is Apache Spark?’