Azure Data Factory specify custom output filename when copying to Blob Storage

Question

I'm currently using ADF to copy files from an SFTP server to Blob Storage on a scheduled basis.

The filename structure is AAAAAA_BBBBBB_CCCCCC.txt.

Is it possible to rename the file before copying to Blob Storage so that I end up with a folder-like structure like below?

AAAAAA/BBBBBB/CCCCCC.txt

Answer 1

• It is possible with Azure Data Factory V2. We can make use of the “lookup activity” to get all the filenames of our source. For more clarification regarding “Lookup activity” in Azure Data Factory, refer to this documentation.

https://docs.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity

• Then we need to chain a “ForEach” activity which contains a copy activity, to iterate source file names. Both source and destination data set of copy activity have parameters for file name and folder path. For more clarification regarding “ForEach” activity in Azure Data Factory, refer to this documentation.

https://docs.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity

• For generating the destination folder path and file name based on your source file names, We could make use of “split and replace” functions. For more clarification regarding “split” and “replace” functions in Azure Data Factory, refer to this documentation.

https://docs.microsoft.com/en-us/azure/logic-apps/workflow-definition-language-functions-reference#split

https://docs.microsoft.com/en-us/azure/logic-apps/workflow-definition-language-functions-reference#replace