For a recent project we’ve invested a lot of time into Azure Devops, and in the most part found it a very useful toolset for deploying our code to both Azure and AWS.
When we started on this process, YAML pipelines weren’t available for our source code provider – this meant everything had to be setup manually 🙁
However, recently this has changed 🙂 This post will run through a few ways you can optimize your release process and automate the whole thing.
First a bit of background and then some actual code examples.
Why YAML?
Setting up your pipelines via the UI is a really good way to quickly prototype things, however what if you need to change these pipelines to mimic deployment features alongside code features. Yaml allows you to keep the pipeline definition in the same codebase as the actual features. You deploy branch XXX and that can be configured differently to branch YYY.
Another benefit, the changes are then visible in your pull requests so validating changes is a lot easier.
Async Jobs
A big optimization we gained was to release to different regions in parallel. Yaml makes this very easy by using Jobs – each job can run on an agent and hence push to multiple regions in parallel.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Yaml file templates
If you have common functionality you want to duplicate, e.g. ‘Deploy to Eu-West-1’, templates are a good way to split your functionality. They allow you to group logical functionality you want to run multiple times.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
Azure Devops rest API
All of your build/releases can be triggered via the UI portal, however if you want to automate that process I’d suggest looking into the rest API. Via this you can trigger, monitor and administer builds, releases and a whole load more.
We use powershell to orchestrate the process.
https://docs.microsoft.com/en-us/rest/api/azure/devops/build/builds/queue?view=azure-devops-rest-5.1
Variables, and variable groups
I have to confess, this syntax feels slightly cumbersome, but it’s very possible to reference variables passed into a specific pipeline along with global variables from groups you setup in the Library section of the portal.
Now, some examples
The root YAML file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
pr: none trigger: none variables: - group: 'DataDog' # reference Variable groups if needed - name : 'system.debug' value: true - name : 'DynamicParameter' # these can be calculated off other variable values value: "name-$(EnvironmentName)-$(ColourName)" - name: 'WebsiteFolder' value: 'Website/FolderName' #- name: "EnvironmentName" # see the rest api example below for how to pass in variables # value: "Set externally" #- name: "ColourName" # value: "Set externally" #- name: "AwsCredentials" # value: "Set externally" jobs: - job: Build pool: vmImage: 'windows-2019' # vmImages: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent steps: - task: NuGetToolInstaller@0 displayName: 'Use NuGet 4.4.1' inputs: versionSpec: 4.4.1 - task: NuGetCommand@2 # if using secure artifacts, you can download them into a dotnetcore project this way displayName: 'NuGet restore' inputs: restoreSolution: 'Website/###.sln' feedsToUse: config nugetConfigPath: Website/nuget.config - task: Npm@1 displayName: 'NPM install' inputs: workingDir: '$(WebsiteFolder)' verbose: false - task: Npm@1 displayName: 'NPM build scss' inputs: workingDir: '$(WebsiteFolder)' command: custom verbose: false customCommand: 'run scss-build' - task: DotNetCoreCLI@2 displayName: 'dotnet publish' inputs: command: publish publishWebProjects: false projects: '$(WebsiteFolder)/Website.csproj' arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)\Website' zipAfterPublish: false - task: PublishPipelineArtifact@0 # in order to share the common build with multiple releases you need to publish the artifact inputs: artifactName: "Website" targetPath: '$(Build.ArtifactStagingDirectory)' - job: ReleaseEU pool: vmImage: 'windows-2019' dependsOn: Build # these will only start when the 'Build' task above starts steps: - template: TaskGroups/DeployToRegion.yaml # this parameters: AwsCredentials: '$(AwsCredentials)' RegionName: 'eu-west-1' EnvironmentName: '$(EnvironmentName)' ColourName: '$(ColourName)' DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group - job: ReleaseRegionN # Will run in parallel with ReleaseEU if you have enough build agents pool: vmImage: 'windows-2019' dependsOn: Build steps: - template: TaskGroups/DeployToRegion.yaml # this template file is shown below parameters: AwsCredentials: '$(AwsCredentials)' RegionName: 'ANother region' EnvironmentName: '$(EnvironmentName)' ColourName: '$(ColourName)' DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group |
The ‘DeployToRegion’ template:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
parameters: AwsCredentials: '' RegionName: '' EnvironmentName: '' ColourName: '' DatadogApiKey: '' steps: - task: DownloadPipelineArtifact@1 # you can download artifacts from other builds if needed inputs: buildType: 'specific' project: 'Project Name' pipeline: '##' buildVersionToDownload: 'latest' artifactName: 'Devops' targetPath: '$(System.ArtifactsDirectory)/Devops' - task: DownloadPipelineArtifact@1 # or download from the current one inputs: buildType: 'current' artifactName: 'Website' targetPath: '$(System.ArtifactsDirectory)' - template: DeployToElasticBeanstalk.yaml # and can chain templates if needed parameters: AwsCredentials: '${{ parameters.AwsCredentials }}' RegionName: '${{ parameters.RegionName }}' EnvironmentName: '${{ parameters.EnvironmentName }}' ColourName: '${{ parameters.ColourName }}' DatadogApiKey: '${{ parameters.DatadogApiKey }}' |
And finally some powershell to fire it all off:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
### Example usage: .\TriggerBuild.ps1 -branch "release/release-006" -isReleaseCandidate $false -additionalReleaseParameters @{ "EnvironmentName" = "qa"; "ColourName" = "blue"; } param ( [Parameter(Mandatory = $true)][string]$branch, [boolean]$isReleaseCandidate = $false, [HashTable]$additionalReleaseParameters = @{ } ) $ErrorActionPreference = "Stop" $authToken = Get-DevOpsAuthToken # see https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops for how to get a token $accountName = "AzureDevopsAccountName" $projectName = "AzureDevopsProjectName" $buildDefinitionIds = @(27) # the build pipeline id Write-Host "Building with settings:" Write-Host "Branch: '$branch'" Write-Host "Tag as 'release-candidate' and retain build: $isReleaseCandidate" Write-Host "Build definition IDs: $buildDefinitionIds" Write-Host "Additional parameters: $($additionalReleaseParameters | ConvertTo-Json) " Write-Host "" $releaseIds = @() $result = @{ Success = $false; } foreach ($definitionId in $buildDefinitionIds) { $deploymentParams = @{ "definition" = @{ "id" = $definitionId; } "sourceBranch" = $branch; } if ($additionalReleaseParameters.GetEnumerator().length -gt 0) { $deploymentParams.parameters = $additionalReleaseParameters | ConvertTo-Json } $content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds?api-version=4.1" ` -ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method POST -Body ($deploymentParams | ConvertTo-Json)).Content | ConvertFrom-Json $releaseIds += $content.id Write-Host "Build $($content.id) queued: https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$($content.id)" -ForegroundColor Yellow } $aBuildFailed = $false foreach ($releaseId in $releaseIds) { $status = "" while ($status -ne "completed") { try { $content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -Headers (Get-DevOpsHeaders -AuthToken $authToken)).Content | ConvertFrom-Json } catch { Write-Host " Error calling DevopsAPI. If this happens several times check the url: https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -ForegroundColor red } $status = $content.status Write-Host " Build id $releaseId has status: $status" if ($content.result -eq "failed" -or $content.result -eq "canceled") { $aBuildFailed = $true Write-Host "Build $releaseId failed - check https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$releaseId for details" -ForegroundColor Red } elseif ($content.result -eq "completed") { Write-Host "Build $releaseId completed successfully" -ForegroundColor Green } Start-Sleep -s 5 } if ($isReleaseCandidate -eq $true) { Write-Host " Adding RC tags: release-candidate" $tags = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId/tags/release-candidate?api-version=4.1" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PUT).Content | ConvertFrom-Json Write-Host " Adding retain build to $releaseId" $updates = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$($releaseId)?api-version=4.1" -ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PATCH -Body (@{"retainedByRelease" = $true } | ConvertTo-Json)).Content | ConvertFrom-Json } } $result.Success = !$aBuildFailed return $result |
Happy deploying 🙂