bing
Flat 10% & upto 50% off + Free additional Courses. Hurry up!
×
UPTO
50%
OFF!

Downloading Spark and Getting Started

Step 1: Verifying the procedure for Java Installation

Java installation is one of the mandatory stuff in installing Spark application. Follow the following command to verify the JAVA version is been installed in your system.

$java -version

If Java is already installed on your system, you get to see the following output which is as follows:

java version "1.7.0_71"

Java(TM) SE Runtime Environment (build 1.7.0_71-b13)

Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode)

If in case you do not have Java already installed on your system, then Install Java before proceeding to the next step.

Step 2: Verifying the Scala installation

Scala language is used to implement Spark. So verify the Scala installation by using following command.

$scala -version

If Scala application is already installed on your system, you get to see the following response on the screen as shown below:

Scala code runner version 2.11.6 — Copyright 2002-2013, LAMP/EPFL

In case if you don’t have Scala already installed on your system, then proceed to next step need to be followed for Scala installation.

Step 3: Downloading Scala

Download the latest version of Scala. We are currently using scala-2.11.6 version. After downloading, you will be able to find the Scala tar file in the download folder.

Step 4: Installing Scala

Follow the below given steps for the installation of Scala.

Extract the Scala tar file

Type the following command for extracting the Scala tar file from the download version.

$ tar xvf scala-2.11.6.tgz

Move Scala software files by using the following commands for moving the Scala software files, to its respective directory (/usr/local/scala).

$ su –

Password:

# cd /home/Hadoop/Downloads/

# mv scala-2.11.6 /usr/local/scala

# exit

Set PATH for Scala

Use the following below command for setting up the PATH for Scala.

$ export PATH = $PATH:/usr/local/scala/bin

Verifying Scala Installation

After installation, it is safer to verify it again. Use the below command for verifying Scala installation.

$scala -version

If Scala is already installed on your system, you get the following response −

Scala code runner version 2.11.6 — Copyright 2002-2013, LAMP/EPFL

Step 5: Downloading Apache Spark

Download the latest version of Spark by visiting the following link Download Spark. As we are using spark-1.3.1-bin-hadoop2.6 version. After downloading it, you will find the Spark tar file in the download folder.

Step 6: Installing Spark

Follow the below steps given below for installing Spark.

Extracting Spark tar

Consider the following command for extracting the spark tar file as the requirement raised.

$ tar xvf spark-1.3.1-bin-hadoop2.6.tgz

Moving Spark software files

The following below commands are used for moving the Spark software files to respective directory (/usr/local/spark).

$ su –

Password:

# cd /home/Hadoop/Downloads/

# mv spark-1.3.1-bin-hadoop2.6 /usr/local/spark

# exit

Setting up the environment for Spark

Add the following below line to ~/.bashrc file which means adding the location, where the spark software files are located to the PATH variable type.

export PATH = $PATH:/usr/local/spark/bin

Use the following below command for sourcing the ~/.bashrc file.

$ source ~/.bashrc

Step 7: Verifying the Spark Installation application in your system :

Write the following command for opening Spark shell application version.

$spark-shell

If spark is installed successfully then you will be getting the following output.

Spark assembly has been built with Hive, including Datanucleus jars on classpath

Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties

15/06/04 15:25:22 INFO SecurityManager: Changing view acls to: hadoop

15/06/04 15:25:22 INFO SecurityManager: Changing modify acls to: hadoop

15/06/04 15:25:22 INFO SecurityManager: SecurityManager: authentication disabled;

ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)

15/06/04 15:25:22 INFO HttpServer: Starting HTTP Server

15/06/04 15:25:23 INFO Utils: Successfully started service naming ‘HTTP class server’ on port 43292.

Welcome to the Spark World

____              __

/ __/__  ___ _____/ /__

_\ \/ _ \/ _ `/ __/  ‘_/

/___/ .__/\_,_/_/ /_/\_\   version 1.4.0

/_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_71)

Type in expressions to have them evaluated as requirement is raised:

Spark context will be available as sc

scala>

Initializing Spark in Python

from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local").setAppName("My App")

sc = SparkContext(conf = conf)

Initializing Spark in Scala

import org.apache.spark.SparkConf

import org.apache.spark.SparkContext

import org.apache.spark.SparkContext._

val conf = new SparkConf().setMaster("local").setAppName("My App")

val sc = new SparkContext(conf)

Initializing Spark in Java

import org.apache.spark.SparkConf;

import org.apache.spark.api.java.JavaSparkContext;

SparkConf conf = new SparkConf().setMaster("local").setAppName("My App");

JavaSparkContext sc = new JavaSparkContext(conf);

These examples show the minimal way to initialize a SparkContext, where you pass two parameters:

  • A cluster URL, namely local in these examples, which tells Spark how to connect to a cluster. local is a special value that runs Spark on one thread on the local machine, without connecting to a cluster.
  • An application name, namely My App in these examples. This will identify your application on the cluster manager’s UI if you connect to a cluster.

"0 Responses on Downloading Spark and Getting Started"

100% Secure Payments. All major credit & debit cards accepted Or Pay by Paypal.
top

Sales Offer

  • To avail this offer, enroll before 31st March 2017.
  • This offer cannot be combined with any other offer.
  • This offer is valid on selected courses only.
  • Please use coupon codes mentioned below to avail the offer
offer-june

Sign Up or Login to view the Free Downloading Spark and Getting Started.