0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)
I am using Spark 1.3.1 (PySpark) and I have generated a table using a SQL query. I now have an object that is a DataFrame. I want to export this DataFrame object (I have called it "table") to a csv file so I can manipulate it and plot the columns. How do I export the DataFrame "table" to a csv file?

1 Answer

+1 vote
by (32.5k points)

If data frame fits in a driver memory and you want to save to local files system you can use toPandas method and convert Spark DataFrame to local Pandas DataFrame and then simply use to_csv:


Otherwise simply use spark-csv:

In Spark 2.0+ you can use csv data source directly:


Spark 1.4+


Spark 1.3

df.save('mycsv.csv', 'com.intelli.spark.csv')

Related questions

Welcome to Intellipaat Community. Get your technical queries answered by top developers !