I am new to PySpark. I have a csv file with hyphen in column names. I could successfully read the file into a dataframe. However while writing the df to orc file I get an error like below-
java.lang.IllegalArgumentException: Missing required char ':' at 'struct
When I renamed the columns by removing hyphen, I could write the dataframe to orc. But I need the column names to have hyphen because I want to append this orc to an existing orc which has hyphen in column names.
Could someone please help me with this? Any help would be greatly appreciated!!!
Back