Azure Data Factory : Implement SCD Type 1

Course Overview

In this project, you will learn how to implement one of the most common concept in real world projects i.e. Slowly Changing Dimension Type 1, using Azure Data Factory. Pre-requisites: Azure subscription Azure Data Factory knowledge (Basic) Following are the tasks covered in this project: Task 1: Understand Slowly Changing Dimension (SCD) Type 1 In this task, we will try to understand the concept of Slowly Changing Dimension and its different types, but will focus on Type 1 using a simple example. Task 2: Create Azure services like Azure Data Factory, Azure SQL Database In this task, we are going to create the azure services like azure data factory and azure sql database which are going to be used in later tasks. Azure sql database is going to contain the staging and dimension table whereas azure data factory is going to be used to create the data pipeline Task 3: Create Staging and Dimension Table in Azure SQL Database In this task, we will create the staging and dimension table in azure sql database. Also, we will insert some dummy records in staging table Task 4: Create a ADF pipeline to implement SCD Type 1 (Insert Logic) In this task, we are going to create the pipeline in azure data factory and implement the logic to insert new records which exists in staging table but doesnt exist in dimension. This is one scenario/use case of SCD Type 1. Task 5: Create a ADF pipeline to implement SCD Type 1 (Update Logic) In this task, we are going to create the pipeline in azure data factory and implement the logic to update records which exists in staging table as well as in dimension. This is another use case/scenario of SCD Type 1 Task 6: Demo of ADF pipeline This is final task in which we will run the pipeline to see whether it satisfies both the use case/scenario of SCD Type 1 All the Best !!