0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)
Is it possible to save DataFrame in spark directly to Hive.

I have tried with converting DataFrame to Rdd and then saving as text file and then loading in hive. But i am wondering if i can directly save dataframe to hive

1 Answer

0 votes
by (31.4k points)
edited by

One way is to create an in-memory temporary table and store them in hive table using sqlContext.

Lets set a data frame as myDf. You can create one temporary table using:

myDf.createOrReplaceTempView("mytempTable") 

Then, simply use a hive statement to create table and from your temp table dump the data into it:

sqlContext.sql("create table mytable as select * from mytempTable");

If you want to know more about Spark, then do check out this awesome video tutorial:

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...