Does Spark need Hadoop?

Question

Can anyone tell me whether Spark needs Hadoop?

Answer 1

Spark and Hadoop are non-mutually exclusive software frameworks. For instance, Spark can optionally use HDFS (Hadoop Distributed File System) as a storage implement, but Spark does not require Hadoop MapReduce for its processing needs as it already has a computational engine. So, in short, Hadoop is not essential to implement Spark. However, make a note that in the real world, where data is being generated and accumulated in real-time and in huge amounts, it is imperative to use a storage system that can house data across various nodes in a distributed cluster. So, Spark coupled with HDFS is the most efficient way to implement Big Data.

Do you want to learn about big data? Enroll in Intellipaat's big data Hadoop course and learn from the professionals.