In this Azure Data Factory Project, you are supposed to automate the transformation of the real-time video list from the YouTube channel. You will be storing multiple files at the dynamic location of Azure Data Lake Store and the same needs to transformed and copied to any data store. The list of the channels should be displayed on PowerBI dynamically.
Project 1: Fetch the list of videos from the attached dataset of YouTube channel with the highest views and likes to promote advertisements on the channel which has maximum traffic.
Topics: Azure Data Factory, Azure Data Lake, Triggers, SQL, Power BI
- Creating Azure Data Factory
- Creating Pipelines
- Creating a trigger that runs a pipeline on a schedule
- Transforming data using SQL
- Connecting Azure Data Lake to Power BI
Project 2: Working with Azure Data Factory, Data Lake and Azure SQL
Problem Statement: You are working as an Azure Architect for Zendrix Corp. This company is a service-based company and has its major revenue from the sales it makes for its subscription-based service.
The company needs to continuously monitor its lead flow from different countries. This helps them in strategizing how much they need to invest in Ad-Marketing for a particular country, this, in turn, helps them to achieve their desired sales targets.
Currently, the company has to manually synchronize data from their live SQL database to their BI tool, for checking the lead flow from different countries.
The company wants an automated solution, using which they will be able to see a live dashboard of the lead count. You as an Architect have suggested the following things:
- Use of Power BI Heat maps
- Use of Azure SQL instead of On-Premise SQL
- Use of Data Factory to automate the data lifecycle from SQL to the BI tool.
Help them achieve the above goals.
Project 3: Identify the videos that get maximum traffic in selected YouTube channels
Problem Statement: Getting the real-time list of maximum traffic fetching videos from YouTube channels to promote advertisements in the same channels (traffic should be considered on a weekly basis)
Description: There is a company ‘XYZ Pvt. Ltd’ that promotes advertisements in the maximum traffic generating YouTube channels (on a weekly basis) to drive profits. To maximize profitability, the marketing team that manages the posting of advertisements requires an interface using which they can get a real-time list of YouTube channels for promoting advertisements and monitoring the analytics of traffic on those channels.
Objective: As an Azure Data Factory specialist, you are supposed to automate the transformation of the real-time video list from YouTube channels on a weekly basis. This will help the marketing team promote advertisements on the right YouTube videos on targeted channels.
Note: The traffic can be analyzed on various parameters like the number of views, and likes or comments on a particular day. You can get these publicly available data from the YouTube API.
Case Study 1: Non-Relational Data Stores
Problem Statement: Knowledge check of non-relational databases: Categories and where to use them
Topics: NoSQL or Non-Relational Database, Azure Data Lake Storage and its key components.
- Scenarios where you can use NoSQL or Non-Relational Database.
- categories of Non-Relational or No SQL databases with relevant Azure services.
- Azure Data Lake Storage and its key components.
Case Study 2: Non-Relational Data Stores
Problem Statement: Copy data from Azure Blob Storage to Azure Data Lake Storage Gen2; Create an Azure Cosmos DB account and Demonstrate adding and removing regions from your Database account; Strategies for Partitioning data; Semantics of consistency levels in Cosmos DB
Topics: Azure Cosmos DB, Azure Data Factory, Blob Storage, Strategies for Partitioning Data, Semantics of consistency levels in Cosmos DB
- Azure Blob Storage
- Azure Data Lake Storage Gen2
- Azure Cosmos DB
- Partitioning data
- Consistency levels
Case Study 3: Relational Data Stores
Problem Statement: Knowledge check of Relational databases: Deployment models in Azure SQL; Create an elastic pool, Azure SQL Security Capabilities; Import Data From Blob Storage to Azure Synapse Analytics by Using PolyBase
Topics: Azure SQL, PolyBase, Azure Synapse Analytics
- Deployment models in Azure SQL
- Elastic Pool
- Azure Synapse Analytics
Case Study 4: Azure Batch, Azure Data Factory
Problem Statement: Working of Azure Batch; Flow Process of Data Factory; Types of Integration Runtime in Azure Data Factory; Transform data using Mapping data flows
Topics: Azure Batch, Data Factory, Integration Runtime, Mapping Data Flows
- Working of Azure Batch
- Integration Runtime in Azure Data Factory
- Transform data using Mapping data flows
Case Study 5: Azure Data Bricks, Azure Stream Analytics
Problem Statement: ETL Operation by using Azure Databricks; Working of Stream Analytics; Stream Analytics Windowing Functions
Topics: Azure Data Bricks, Azure Stream Analytics, Windowing Functions
- ETL operation by using Azure Databricks
- Working of Stream Analytics
- Windowing Functions
Case Study 6: Monitoring & Security
Problem Statement: Create, View, and Manage Metric alerts using Azure Monitor; Azure SQL Database Auditing
Topics: Azure Monitor, Alerts in Azure, Azure Security Logging & Auditing
- Azure Monitor
- Azure SQL Database Auditing