0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

n Spark Standalone mode, there are master and worker nodes.

Here are few questions:

  1. Does 2 worker instance mean one worker node with 2 worker processes?
  2. Does every worker instance hold an executor for specific application (which manages storage, task) or one worker node holds one executor?
  3. Is there a flow chart explain how spark runtime, such as word count?

1 Answer

0 votes
by (31.4k points)
edited by

Spark Standalone mode runs on a master-slave architecture..

Answering your 3 questions:

1. Does 2 worker instance mean one worker node with 2 worker processes?

In general, the term used for worker instance is called slave as it's a process to execute spark tasks/jobs. Suggested mapping for node(a physical or virtual machine) and worker is:

1 Node = 1 Worker process

2. Does every worker instance hold an executor for specific application (which manages storage, task) or one worker node holds one executor?

Yes,if the worker node has sufficient CPU, Memory and Storage, it may hold multiple executors (processes) .

Check this worker node image:

image

In a worker node the number of executors at a given point of time entirely depends upon the work load on the cluster and capability of the node to run the no of executors.

3. Is there a flow chart explain how spark runtime?

Keeping in mind the execution from Spark prospective over any resource manager for a program, below is the flow chart where joining of two rdds is done and some reduce operations are performed and then finally filter() is called:

image

If you want to know more about Spark, then do check out this awesome video tutorial:

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...