Replacing multiple “” from polybase string

Question

I have dataframe in which one of the columns is Error. Most of the cases the column has no value, but in some case it do get a value.

When I use to_csv on it that column is like

Country,Etl_Batch,Input_Date,Input_CampaignID,Tags,TargetGroupID,CampaignType,Duration,LeadTime,Notes,IsMultiChannel,IsRecurrence,Status,Error,Api_Executed_Datetime

RO,1511293247,2019-07-02,4177,,89,No Control,1,0,,False,True,Successful,,2019-07-16 15:26:00.696304

RO,1511293247,2019-07-02,4178,,232,Test/Control,3,0,,False,False,Successful,"Exception caught at HTTPHelper postXMLHTTPSRequest. Http response: <?xml version=""1.0"" encoding=""UTF-8""?><error><code>UMS-105</code><description>'activities' is not specified or invalid</description><severity>ERROR</severity></error>
Error message: The remote server returned an error: (400) Bad Request. ",2019-07-16 15:26:00.696304

RO,1511293247,2019-07-02,4179,,-1,Test/Control,3,0,,False,False,Successful,,2019-07-16 15:26:00.696304

When I create an external table using this data I get a polybase error.

As it has got text like

"Exception caught at HTTPHelper postXMLHTTPSRequest. Http response: <?xml version=""1.0"" encoding=""UTF-8""?><error><code>UMS-105</code><description>'activities' is not specified or invalid</description><severity>ERROR</severity></error>
Error message: The remote server returned an error: (400) Bad Request. "

This is a single string but since it has got "" (2 double quotes in between), it causes an error.

I have to remove '""' (2 double quotes) from within the string in python. How do I do that?

Answer 1

Use this below line of code:

def app(row):

    return str(row['Error']).replace("\"\""," ")

 

df['Error'] = df.apply(lambda x: app(x),axis = 1)