Explore Courses Blog Tutorials Interview Questions
0 votes
in Big Data Hadoop & Spark by (11.4k points)
Is it possible to save DataFrame in spark directly to Hive.

I have tried with converting DataFrame to Rdd and then saving as text file and then loading in hive. But i am wondering if i can directly save dataframe to hive

1 Answer

0 votes
by (32.3k points)
edited by

One way is to create an in-memory temporary table and store them in hive table using sqlContext.

Lets set a data frame as myDf. You can create one temporary table using:


Then, simply use a hive statement to create table and from your temp table dump the data into it:

sqlContext.sql("create table mytable as select * from mytempTable");

If you want to know more about Spark, then do check out this awesome video tutorial:

Browse Categories