Intellipaat Back

Explore Courses Blog Tutorials Interview Questions
0 votes
2 views
in Big Data Hadoop & Spark by (11.4k points)

I have the following Scala value:

val values: List[Iterable[Any]] = Traces().evaluate(features).toList


and I want to convert it to a DataFrame.

When I try the following:

sqlContext.createDataFrame(values)


I got this error:

error: overloaded method value createDataFrame with alternatives:

[A <: Product](data: Seq[A])(implicit evidence$2: reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame
[A <: Product](rdd: org.apache.spark.rdd.RDD[A])(implicit evidence$1: reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame
cannot be applied to (List[Iterable[Any]])
          sqlContext.createDataFrame(values)

 

1 Answer

0 votes
by (32.3k points)

You just need to first convert List[Iterable[Any]] to List[Row] and then put rows in RDD and prepare schema for the spark data frame.

To convert List[Iterable[Any]] to List[Row], we can say

val rows = values.map{x => Row(x:_*)}

and then having schema like schema, we can make RDD

val rdd = sparkContext.makeRDD[RDD](rows)

and finally create a spark data frame

val df = sqlContext.createDataFrame(rdd, schema)

31k questions

32.8k answers

501 comments

693 users

Browse Categories

...