Explore Courses Blog Tutorials Interview Questions
0 votes
in Big Data Hadoop & Spark by (120 points)

I am new to PySpark. I have a csv file with hyphen in column names. I could successfully read the file into a dataframe. However while writing the df to orc file I get an error like below-

java.lang.IllegalArgumentException: Missing required char ':' at 'struct

When I renamed the columns by removing hyphen, I could write the dataframe to orc. But I need the column names to have hyphen because I want to append this orc to an existing orc which has hyphen in column names.

Could someone please help me with this? Any help would be greatly appreciated!!!

Please log in or register to answer this question.

Browse Categories