Automating a multi region deployment with Azure Devops

For a recent project we’ve invested a lot of time into Azure Devops, and in the most part found it a very useful toolset for deploying our code to both Azure and AWS.

When we started on this process, YAML pipelines weren’t available for our source code provider – this meant everything had to be setup manually 🙁

However, recently this has changed 🙂 This post will run through a few ways you can optimize your release process and automate the whole thing.

First a bit of background and then some actual code examples.

Why YAML?

Setting up your pipelines via the UI is a really good way to quickly prototype things, however what if you need to change these pipelines to mimic deployment features alongside code features. Yaml allows you to keep the pipeline definition in the same codebase as the actual features. You deploy branch XXX and that can be configured differently to branch YYY.

Another benefit, the changes are then visible in your pull requests so validating changes is a lot easier.

Async Jobs

A big optimization we gained was to release to different regions in parallel. Yaml makes this very easy by using Jobs – each job can run on an agent and hence push to multiple regions in parallel.

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml

Yaml file templates

If you have common functionality you want to duplicate, e.g. ‘Deploy to Eu-West-1’, templates are a good way to split your functionality. They allow you to group logical functionality you want to run multiple times.

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Azure Devops rest API

All of your build/releases can be triggered via the UI portal, however if you want to automate that process I’d suggest looking into the rest API. Via this you can trigger, monitor and administer builds, releases and a whole load more.

We use powershell to orchestrate the process.

https://docs.microsoft.com/en-us/rest/api/azure/devops/build/builds/queue?view=azure-devops-rest-5.1

Variables, and variable groups

I have to confess, this syntax feels slightly cumbersome, but it’s very possible to reference variables passed into a specific pipeline along with global variables from groups you setup in the Library section of the portal.

Now, some examples

The root YAML file:

pr: none
trigger: none

variables:
- group: 'DataDog' # reference Variable groups if needed
- name : 'system.debug'
  value: true
- name : 'DynamicParameter' # these can be calculated off other variable values
  value: "name-$(EnvironmentName)-$(ColourName)"
- name: 'WebsiteFolder'
  value: 'Website/FolderName'

#- name: "EnvironmentName" # see the rest api example below for how to pass in variables
#  value: "Set externally"
#- name: "ColourName"
#  value: "Set externally"
#- name: "AwsCredentials"
#  value: "Set externally"

jobs:
- job: Build
  pool:
    vmImage: 'windows-2019' # vmImages: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent
  steps:  
 
  - task: NuGetToolInstaller@0
    displayName: 'Use NuGet 4.4.1'
    inputs:
      versionSpec: 4.4.1    

  - task: NuGetCommand@2 # if using secure artifacts, you can download them into a dotnetcore project this way
    displayName: 'NuGet restore'
    inputs:
      restoreSolution: 'Website/###.sln'
      feedsToUse: config
      nugetConfigPath: Website/nuget.config

  - task: Npm@1
    displayName: 'NPM install'
    inputs:
      workingDir: '$(WebsiteFolder)'     
      verbose: false

  - task: Npm@1
    displayName: 'NPM build scss'
    inputs:
      workingDir: '$(WebsiteFolder)'     
      command: custom
      verbose: false
      customCommand: 'run scss-build'

  - task: DotNetCoreCLI@2
    displayName: 'dotnet publish'
    inputs:
      command: publish
      publishWebProjects: false
      projects: '$(WebsiteFolder)/Website.csproj'
      arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)\Website'
      zipAfterPublish: false  

  - task: PublishPipelineArtifact@0 # in order to share the common build with multiple releases you need to publish the artifact
    inputs:
      artifactName: "Website"
      targetPath: '$(Build.ArtifactStagingDirectory)'

- job: ReleaseEU
  pool:
    vmImage: 'windows-2019'
  dependsOn: Build # these will only start when the 'Build' task above starts
  steps:
  - template: TaskGroups/DeployToRegion.yaml # this
    parameters:
      AwsCredentials: '$(AwsCredentials)'
      RegionName: 'eu-west-1'      
      EnvironmentName: '$(EnvironmentName)'
      ColourName: '$(ColourName)'
      DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group      

- job: ReleaseRegionN # Will run in parallel with ReleaseEU if you have enough build agents
  pool:
    vmImage: 'windows-2019'
  dependsOn: Build
  steps:
  - template: TaskGroups/DeployToRegion.yaml # this template file is shown below
    parameters:
      AwsCredentials: '$(AwsCredentials)'
      RegionName: 'ANother region'      
      EnvironmentName: '$(EnvironmentName)'
      ColourName: '$(ColourName)'
      DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group

pr: none

trigger: none

variables:

- group: 'DataDog' # reference Variable groups if needed

- name : 'system.debug'

value: true

- name : 'DynamicParameter' # these can be calculated off other variable values

value: "name-$(EnvironmentName)-$(ColourName)"

- name: 'WebsiteFolder'

value: 'Website/FolderName'

#- name: "EnvironmentName" # see the rest api example below for how to pass in variables

# value: "Set externally"

#- name: "ColourName"

# value: "Set externally"

#- name: "AwsCredentials"

# value: "Set externally"

jobs:

- job: Build

pool:

vmImage: 'windows-2019' # vmImages: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent

steps:

- task: NuGetToolInstaller@0

displayName: 'Use NuGet 4.4.1'

inputs:

versionSpec: 4.4.1

- task: NuGetCommand@2 # if using secure artifacts, you can download them into a dotnetcore project this way

displayName: 'NuGet restore'

inputs:

restoreSolution: 'Website/###.sln'

feedsToUse: config

nugetConfigPath: Website/nuget.config

- task: Npm@1

displayName: 'NPM install'

inputs:

workingDir: '$(WebsiteFolder)'

verbose: false

- task: Npm@1

displayName: 'NPM build scss'

inputs:

workingDir: '$(WebsiteFolder)'

command: custom

verbose: false

customCommand: 'run scss-build'

- task: DotNetCoreCLI@2

displayName: 'dotnet publish'

inputs:

command: publish

publishWebProjects: false

projects: '$(WebsiteFolder)/Website.csproj'

arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)\Website'

zipAfterPublish: false

- task: PublishPipelineArtifact@0 # in order to share the common build with multiple releases you need to publish the artifact

inputs:

artifactName: "Website"

targetPath: '$(Build.ArtifactStagingDirectory)'

- job: ReleaseEU

pool:

vmImage: 'windows-2019'

dependsOn: Build # these will only start when the 'Build' task above starts

steps:

- template: TaskGroups/DeployToRegion.yaml # this

parameters:

AwsCredentials: '$(AwsCredentials)'

RegionName: 'eu-west-1'

EnvironmentName: '$(EnvironmentName)'

ColourName: '$(ColourName)'

DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group

- job: ReleaseRegionN # Will run in parallel with ReleaseEU if you have enough build agents

pool:

vmImage: 'windows-2019'

dependsOn: Build

steps:

- template: TaskGroups/DeployToRegion.yaml # this template file is shown below

parameters:

AwsCredentials: '$(AwsCredentials)'

RegionName: 'ANother region'

EnvironmentName: '$(EnvironmentName)'

ColourName: '$(ColourName)'

DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group

The ‘DeployToRegion’ template:

parameters:
  AwsCredentials: ''
  RegionName: ''  
  EnvironmentName: ''
  ColourName: ''
  DatadogApiKey: ''  

steps:
- task: DownloadPipelineArtifact@1 # you can download artifacts from other builds if needed
  inputs:
      buildType: 'specific'
      project: 'Project Name'
      pipeline: '##'
      buildVersionToDownload: 'latest'
      artifactName: 'Devops'
      targetPath: '$(System.ArtifactsDirectory)/Devops'

- task: DownloadPipelineArtifact@1 # or download from the current one
  inputs:
      buildType: 'current'
      artifactName: 'Website'
      targetPath: '$(System.ArtifactsDirectory)'

- template: DeployToElasticBeanstalk.yaml # and can chain templates if needed
  parameters:
      AwsCredentials: '${{ parameters.AwsCredentials }}'
      RegionName: '${{ parameters.RegionName }}'      
      EnvironmentName: '${{ parameters.EnvironmentName }}'
      ColourName: '${{ parameters.ColourName }}'
      DatadogApiKey: '${{ parameters.DatadogApiKey }}'

parameters:

AwsCredentials: ''

RegionName: ''

EnvironmentName: ''

ColourName: ''

DatadogApiKey: ''

steps:

- task: DownloadPipelineArtifact@1 # you can download artifacts from other builds if needed

inputs:

buildType: 'specific'

project: 'Project Name'

pipeline: '##'

buildVersionToDownload: 'latest'

artifactName: 'Devops'

targetPath: '$(System.ArtifactsDirectory)/Devops'

- task: DownloadPipelineArtifact@1 # or download from the current one

inputs:

buildType: 'current'

artifactName: 'Website'

targetPath: '$(System.ArtifactsDirectory)'

- template: DeployToElasticBeanstalk.yaml # and can chain templates if needed

parameters:

AwsCredentials: '${{ parameters.AwsCredentials }}'

RegionName: '${{ parameters.RegionName }}'

EnvironmentName: '${{ parameters.EnvironmentName }}'

ColourName: '${{ parameters.ColourName }}'

DatadogApiKey: '${{ parameters.DatadogApiKey }}'

And finally some powershell to fire it all off:

### Example usage: .\TriggerBuild.ps1 -branch "release/release-006" -isReleaseCandidate $false -additionalReleaseParameters @{ "EnvironmentName" = "qa"; "ColourName" = "blue"; }

param (
    [Parameter(Mandatory = $true)][string]$branch,   
    [boolean]$isReleaseCandidate = $false,
    [HashTable]$additionalReleaseParameters = @{ }
)

$ErrorActionPreference = "Stop"

$authToken = Get-DevOpsAuthToken # see https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops for how to get a token
$accountName = "AzureDevopsAccountName" 
$projectName = "AzureDevopsProjectName"

$buildDefinitionIds = @(27) # the build pipeline id

Write-Host "Building with settings:"
Write-Host "Branch: '$branch'"
Write-Host "Tag as 'release-candidate' and retain build: $isReleaseCandidate"
Write-Host "Build definition IDs: $buildDefinitionIds"
Write-Host "Additional parameters: $($additionalReleaseParameters | ConvertTo-Json) "
Write-Host ""

$releaseIds = @()

$result = @{
    Success = $false;    
}

foreach ($definitionId in $buildDefinitionIds)
{
    $deploymentParams = @{
        "definition" = @{
            "id" = $definitionId;
        }
        "sourceBranch" = $branch;
    }

    if ($additionalReleaseParameters.GetEnumerator().length -gt 0)
    {
        $deploymentParams.parameters = $additionalReleaseParameters | ConvertTo-Json
    }

    $content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds?api-version=4.1" `
        -ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method POST -Body ($deploymentParams | ConvertTo-Json)).Content | ConvertFrom-Json

    $releaseIds += $content.id  

    Write-Host "Build $($content.id) queued: https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$($content.id)" -ForegroundColor Yellow
}

$aBuildFailed = $false

foreach ($releaseId in $releaseIds)
{
    $status = ""

    while ($status -ne "completed")
    {
        try
        {
            $content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -Headers (Get-DevOpsHeaders -AuthToken $authToken)).Content | ConvertFrom-Json
        }
        catch
        {
            Write-Host "  Error calling DevopsAPI. If this happens several times check the url: https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -ForegroundColor red
        }

        $status = $content.status

        Write-Host " Build id $releaseId has status: $status"

        if ($content.result -eq "failed" -or $content.result -eq "canceled")
        {
            $aBuildFailed = $true

            Write-Host "Build $releaseId failed - check https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$releaseId for details" -ForegroundColor Red
        }
        elseif ($content.result -eq "completed")
        {
            Write-Host "Build $releaseId completed successfully" -ForegroundColor Green
        }

        Start-Sleep -s 5
    }

    if ($isReleaseCandidate -eq $true)
    {
        Write-Host " Adding RC tags: release-candidate"
        $tags = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId/tags/release-candidate?api-version=4.1" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PUT).Content | ConvertFrom-Json

        Write-Host " Adding retain build to $releaseId"
        $updates = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$($releaseId)?api-version=4.1" -ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PATCH -Body (@{"retainedByRelease" = $true } | ConvertTo-Json)).Content | ConvertFrom-Json
    }
}

$result.Success = !$aBuildFailed

return $result

100

### Example usage: .\TriggerBuild.ps1 -branch "release/release-006" -isReleaseCandidate $false -additionalReleaseParameters @{ "EnvironmentName" = "qa"; "ColourName" = "blue"; }

param (

[Parameter(Mandatory = $true)][string]$branch,

[boolean]$isReleaseCandidate = $false,

[HashTable]$additionalReleaseParameters = @{ }

)

$ErrorActionPreference = "Stop"

$authToken = Get-DevOpsAuthToken # see https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops for how to get a token

$accountName = "AzureDevopsAccountName"

$projectName = "AzureDevopsProjectName"

$buildDefinitionIds = @(27) # the build pipeline id

Write-Host "Building with settings:"

Write-Host "Branch: '$branch'"

Write-Host "Tag as 'release-candidate' and retain build: $isReleaseCandidate"

Write-Host "Build definition IDs: $buildDefinitionIds"

Write-Host "Additional parameters: $($additionalReleaseParameters | ConvertTo-Json) "

Write-Host ""

$releaseIds = @()

$result = @{

Success = $false;

}

foreach ($definitionId in $buildDefinitionIds)

{

$deploymentParams = @{

"definition" = @{

"id" = $definitionId;

}

"sourceBranch" = $branch;

}

if ($additionalReleaseParameters.GetEnumerator().length -gt 0)

{

$deploymentParams.parameters = $additionalReleaseParameters | ConvertTo-Json

}

$content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds?api-version=4.1" `

-ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method POST -Body ($deploymentParams | ConvertTo-Json)).Content | ConvertFrom-Json

$releaseIds += $content.id

Write-Host "Build $($content.id) queued: https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$($content.id)" -ForegroundColor Yellow

}

$aBuildFailed = $false

foreach ($releaseId in $releaseIds)

{

$status = ""

while ($status -ne "completed")

{

try

{

$content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -Headers (Get-DevOpsHeaders -AuthToken $authToken)).Content | ConvertFrom-Json

}

catch

{

Write-Host " Error calling DevopsAPI. If this happens several times check the url: https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -ForegroundColor red

}

$status = $content.status

Write-Host " Build id $releaseId has status: $status"

if ($content.result -eq "failed" -or $content.result -eq "canceled")

{

$aBuildFailed = $true

Write-Host "Build $releaseId failed - check https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$releaseId for details" -ForegroundColor Red

}

elseif ($content.result -eq "completed")

{

Write-Host "Build $releaseId completed successfully" -ForegroundColor Green

}

Start-Sleep -s 5

}

if ($isReleaseCandidate -eq $true)

{

Write-Host " Adding RC tags: release-candidate"

$tags = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId/tags/release-candidate?api-version=4.1" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PUT).Content | ConvertFrom-Json

Write-Host " Adding retain build to $releaseId"

$updates = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$($releaseId)?api-version=4.1" -ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PATCH -Body (@{"retainedByRelease" = $true } | ConvertTo-Json)).Content | ConvertFrom-Json

}

$result.Success = !$aBuildFailed

return $result

Happy deploying 🙂

One thought on “Automating a multi region deployment with Azure Devops”

boro says:

October 22, 2019 at 8:22 am

If you run into issues with the nuget restore, consider replacing the step: ‘NuGet restore’ with:

– task: DotNetCoreCLI@2
displayName: ‘dotnet restore’
inputs:
command: restore
projects: ‘**/*.csproj’
feedsToUse: config
nugetConfigPath: ‘Website/nuget.config’

blog.boro2g .co.uk

Some ideas about coding, dev and all things online.

One thought on “Automating a multi region deployment with Azure Devops”

Leave a Reply Cancel reply