Back

Explore Courses Blog Tutorials Interview Questions
0 votes
2 views
in Big Data Hadoop & Spark by (19k points)
In Spark version : 2.1

For example:

In pyspark, I create a list

test_list = [['Hello', 'world'], ['I', 'am', 'fine']]

then how to create a dataframe form the test_list, where the dataframe's type is like below:

DataFrame[words: array<string>]

1 Answer

0 votes
by (33.1k points)

To create a dataframe from the list in Spark SQL.

from pyspark.sql.types import *

cSchema = StructType([StructField("WordList", ArrayType(StringType()))])

# notice extra square brackets around each element of list 

test_list = [['Hello', 'world']], [['I', 'am', 'fine']]

df = spark.createDataFrame(test_list,schema=cSchema) 

Hope this answer helps you!

Browse Categories

...