I'd like to set up an Azure Data Factory pipeline which performs a move (i.e. copy, verify, delete) operation rather than just a copy operation between Blob Storage and a Data Lake Store. I cannot seem to find any detail on how to do this.
Azure Data Factory does not have a built-in option for moving files but we can make use of copy operation instead.
Create a new data factory
After deployment of data factory, click on Author and monitor
Now select copy data
Mention the source data store as the blob storage and import the database which you want to copy, then create a linked service for it.
Mention the destination datastore as the Data Lake Store and then create a linked service for it.
Finally, run the pipeline to copy data from one Azure SQL Data Warehouse to another Azure SQL Data Warehouse.
While creating the linked service for Data lake store, you are required to enter the service principal id and key. These are the following steps to get those credentials:
Go to Azure dashboard
Click on Azure active directory
Click on App registrations
Create a new web application, and the application ID which gets generated is the service principal ID
Go to Certificates and secrets, create a new client secret, and then a password is generated which is the service principal key
Go to Azure Data Lake Store and give all access to the application which is created for generating the service principal key
Certification in Digital Marketing |
Certification in Big Data Analytics |
Certification in Cloud & Devops |
Certification in Full Stack Web Development
Big Data and Data Science Master’s Course |
Data Science Architect Master’s Program |
Cloud and DevOps Architect Master’s Course |
Artificial Intelligence Engineer Master’s Course |
Microsoft Azure Certification Master’s Training
Data Science Course Online |
Big Data Hadoop Certification Training |
Machine Learning Course Online |
Artificial Intelligence Course and Training |
Salesforce Certification Training: Administrator and App Builder |
Tableau Training and Certification Course |
Python Data Science Course & Training |
Ethical Hacking Course |
Business Analyst Course |
Courses in Cyber Security
© COPYRIGHT 2011-2021 INTELLIPAAT.COM. ALL RIGHTS RESERVED.
Download Salary Trends Now !
Learn how professionals like you got up to 100% Salary Hike.