Using Azure DevOps
for Azure Data Factory
Paul Andrew | Principal Consultant & Solution Architect
@MrPaulAndrew In/MrPaulAndrew MrPaulAndrew.com
https://github.com/mrpaulandrew
{Event/Location}-{Month}-{Year}
Azure Data Factory
A Quick Overview
What is Azure Data Factory?
Extract Transform Load
What is Azure Data Factory?
Extract Transform Load
Data Factory Components
Extract Transform Load
1 Linked Services
2 Data Sets
3 Activities
4 Pipelines
5 Triggers
Data Factory DevOps – CI/CD
Data Factory Continuous Integration
Debug
Git branch Service
Developer
save
merge
master publish
Deployed
Developer Service
Dev Prod
Developer
Data Factory Continuous Delivery
Debug
Git branch Service
save
merge
master publish
Deployed
Developer Service
Dev Prod
Data Factory Continuous Delivery
Option 1 – Single Data Factory Service Debug
Git branch Service
save
merge
master publish
Deployed
Developer Service
Dev Prod
Option 2 – ARM Templates for Multiple Data Factory Services
Debug
Test Service Live Service
branch Service
Git
save
merge
master ARM
Template
Developer
Dev Test Prod
Data Factory Publish
Git
master publish
Deployed
Developer Service
Debug Dev
Data Factory Publish
Git
master
adf_publish Deployed
Developer Service
Debug Dev
ARMTemplateForFactory.json
Data Factory Continuous Delivery Release-20190101
Master
Azure DevOps
Feature 101
Build Artifacts Feature 203
Releases Scoped Variables
Test
Azure Resource Group Deployment ARMTemplateForFactory.json
▪ Override Template Parameters
-factoryName "$(DataFactory.Name)"
Release
Approver
Production
Azure Resource Group Deployment
Data Factory Continuous Delivery
1 Linked Services
Azure DevOps
2 Data Sets
Build Artifacts
3 Activities
Releases Scoped Variables 4 Pipelines
Test 5 Triggers
Azure Resource Group Deployment ARMTemplateForFactory.json
▪ Override Template Parameters
-factoryName "$(DataFactory.Name)"
Release
Approver
Production
Azure Resource Group Deployment
Data Factory Continuous Delivery
1 Linked Services
Azure DevOps
Build Artifacts
Releases Scoped Variables
Test
Azure Resource Group Deployment ARMTemplateForFactory.json
▪ Override Template Parameters
-factoryName "$(DataFactory.Name)"
Release
Approver
Production
Azure Resource Group Deployment
Data Factory Continuous Delivery
1 Linked Services
Azure DevOps
2 Data Sets
Build Artifacts
3 Activities
Releases Scoped Variables 4 Pipelines
Test Set-AzDataFactoryV2Pipeline
5 Triggers
Set-AzDataFactoryV2Dataset
Set-AzDataFactoryV2LinkedService
Azure Resource Group Deployment ARMTemplateForFactory.json
▪ Override Template Parameters params.json
linkedservices.json
-factoryName "$(DataFactory.Name)"
Release
Approver pipelines & activites.json
Production
Set-AzDataFactoryV2IntegrationRuntime triggers.json
Set-AzDataFactoryV2Trigger
datasets.json
Azure Resource Group Deployment
Data Factory Continuous Delivery - Bonus
Option 3 1 Linked Services
Azure DevOps
2 Data Sets
Build Artifacts
3 Activities
Releases Scoped Variables 4 Pipelines
Test 5 Triggers
Run PowerShell params.json
linkedservices.json
Set-AzDataFactoryV2Pipeline pipelines & activites.json
Set-AzDataFactoryV2Dataset datasets.json
Set-AzDataFactoryV2LinkedService triggers.json
Release
Approver
Production
Run PowerShell params.json
Set-AzDataFactoryV2IntegrationRuntime
Set-AzDataFactoryV2Trigger
Data Factory DevOps Summary
What is your Which source How many What deployment What built artefacts are
code branching control tool to environments method do we we going to use?...
strategy? use? do we want? want to use? OR
How much control do
Feature you want?
ARMTemplate
ForFactory.json
Release
linkedservices.json
pipelines & activites.json
Feature datasets.json
triggers.json
Master
Thank you for listening…
Paul Andrew
Blog: mrpaulandrew.com
Email: paul@mrpaulandrew.com
Twitter: @mrpaulandrew
LinkedIn: In/mrpaulandrew
GitHub: github.com/mrpaulandrew
