0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I am trying to take my input data:

A    B       C
--------------
4    blah    2
2            3
56   foo     3


And add a column to the end based on whether B is empty or not:

A    B       C     D
--------------------
4    blah    2     1
2            3     0
56   foo     3     1


I can do this easily by registering the input dataframe as a temp table, then typing up a SQL query.

But I'd really like to know how to do this with just Scala methods and not having to type out a SQL query within Scala.

I've tried .withColumn, but I can't get that to do what I want.

1 Answer

0 votes
by (32.3k points)

I tried withColumn approach and it worked. I would suggest you to go through my code given below:

image

image

In above code I added the (100, null, 5) row for testing the isNull case.

image

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...