How to write duplicate columns as header in csv file using java and spark

Question

I have to insert duplicate columns which empty 

Header : , , , xxx, , , YYY, , ,

dataSet.coalesce(1).write().mode(ignore).format(csv).options(header= true,\u0000=false).save(targetPath);

while writing into csv file am getting a issue following below 

Caused by: org.apache.spark.sql.AnalysisException: Found duplicate column(s) when inserting into file:/C:/Users/xxx/summaryReport/20190901: 

at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtils.scala:85)

Answer 1

Hi Hussain

You should assign a new name to each duplicate column. A unique name for every column would also help you to perform data preprocessing on a specific column.

I hope this answer will help you!

Comments

Hi Anurag,
Thanks for the suggestions !!!

Actual requirement is

Columns | A | B | C | D | E | F | G | H |
columns c and G should have headers XXX and YYY.
Columns A,B,D, E,F,H should be empty

I have fixed this . following below

created dataset with the above structure
while writing ds.write.options(header,false) header set as false .. it is resolves the issue