0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I have a DataFrame with Timestamp column, which i need to convert as Date format.

Is there any Spark SQL functions available for this?

1 Answer

0 votes
by (25.6k points)

You can just simply cast the column to date by following the code given below:

Scala:

import org.apache.spark.sql.types.DateType

val newDF = df.withColumn("dateColumn", df("timestampColumn").cast(DateType))

Pyspark:

df = df.withColumn('dateColumn', df['timestampColumn'].cast('date'))

Note:This solution uses functions available as part of the Spark SQL package, but it doesn't use the SQL language, instead it uses the robust DataFrame API, with SQL-like functions. It doesn’t use less reliable strings with actual SQL queries.

...