Intellipaat Back

Explore Courses Blog Tutorials Interview Questions

Spark: subtract two DataFrames

0 votes
4 views
in Big Data Hadoop & Spark by (11.4k points)

In Spark version 1.2.0 one could use subtract with 2 SchemRDDs to end up with only the different content from the first one

val onlyNewData = todaySchemaRDD.subtract(yesterdaySchemaRDD)


onlyNewData contains the rows in todaySchemRDD that do not exist in yesterdaySchemaRDD.

How can this be achieved with DataFrames in Spark version 1.3.0?

1 Answer

0 votes
by (32.3k points)
edited by

In order to achieve your goal, subtract() is available for Python Spark's dataframe.

df1.subtract(df2)

If you want to know more about Spark, then do check out this awesome video tutorial:

 

Related questions

0 votes
1 answer
Apache Spark RDD filter into two RDDs
asked Jul 29, 2019 in Big Data Hadoop & Spark by Aarav (11.4k points)
0 votes
1 answer
Join two ordinary RDDs with/without Spark SQL
asked Jul 18, 2019 in Big Data Hadoop & Spark by Aarav (11.4k points)

31k questions

32.8k answers

501 comments

693 users

Browse Categories

© COPYRIGHT 2011-2024 INTELLIPAAT.COM. ALL RIGHTS RESERVED.

...