When we started on this process, YAML pipelines weren’t available for our source code provider – this meant everything had to be setup manually

However, recently this has changed This post will run through a few ways you can optimize your release process and automate the whole thing.

First a bit of background and then some actual code examples.

Why YAML?

Setting up your pipelines via the UI is a really good way to quickly prototype things, however what if you need to change these pipelines to mimic deployment features alongside code features. Yaml allows you to keep the pipeline definition in the same codebase as the actual features. You deploy branch XXX and that can be configured differently to branch YYY.

Another benefit, the changes are then visible in your pull requests so validating changes is a lot easier.

Async Jobs

A big optimization we gained was to release to different regions in parallel. Yaml makes this very easy by using Jobs – each job can run on an agent and hence push to multiple regions in parallel.

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml

Yaml file templates

If you have common functionality you want to duplicate, e.g. ‘Deploy to Eu-West-1’, templates are a good way to split your functionality. They allow you to group logical functionality you want to run multiple times.

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Azure Devops rest API

All of your build/releases can be triggered via the UI portal, however if you want to automate that process I’d suggest looking into the rest API. Via this you can trigger, monitor and administer builds, releases and a whole load more.

We use powershell to orchestrate the process.

https://docs.microsoft.com/en-us/rest/api/azure/devops/build/builds/queue?view=azure-devops-rest-5.1

Variables, and variable groups

I have to confess, this syntax feels slightly cumbersome, but it’s very possible to reference variables passed into a specific pipeline along with global variables from groups you setup in the Library section of the portal.

Now, some examples

The root YAML file:

pr: none
trigger: none

variables:
- group: 'DataDog' # reference Variable groups if needed
- name : 'system.debug'
  value: true
- name : 'DynamicParameter' # these can be calculated off other variable values
  value: "name-$(EnvironmentName)-$(ColourName)"
- name: 'WebsiteFolder'
  value: 'Website/FolderName'

#- name: "EnvironmentName" # see the rest api example below for how to pass in variables
#  value: "Set externally"
#- name: "ColourName"
#  value: "Set externally"
#- name: "AwsCredentials"
#  value: "Set externally"

jobs:
- job: Build
  pool:
    vmImage: 'windows-2019' # vmImages: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent
  steps:  
 
  - task: NuGetToolInstaller@0
    displayName: 'Use NuGet 4.4.1'
    inputs:
      versionSpec: 4.4.1    

  - task: NuGetCommand@2 # if using secure artifacts, you can download them into a dotnetcore project this way
    displayName: 'NuGet restore'
    inputs:
      restoreSolution: 'Website/###.sln'
      feedsToUse: config
      nugetConfigPath: Website/nuget.config

  - task: Npm@1
    displayName: 'NPM install'
    inputs:
      workingDir: '$(WebsiteFolder)'     
      verbose: false

  - task: Npm@1
    displayName: 'NPM build scss'
    inputs:
      workingDir: '$(WebsiteFolder)'     
      command: custom
      verbose: false
      customCommand: 'run scss-build'

  - task: DotNetCoreCLI@2
    displayName: 'dotnet publish'
    inputs:
      command: publish
      publishWebProjects: false
      projects: '$(WebsiteFolder)/Website.csproj'
      arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)\Website'
      zipAfterPublish: false  

  - task: PublishPipelineArtifact@0 # in order to share the common build with multiple releases you need to publish the artifact
    inputs:
      artifactName: "Website"
      targetPath: '$(Build.ArtifactStagingDirectory)'

- job: ReleaseEU
  pool:
    vmImage: 'windows-2019'
  dependsOn: Build # these will only start when the 'Build' task above starts
  steps:
  - template: TaskGroups/DeployToRegion.yaml # this
    parameters:
      AwsCredentials: '$(AwsCredentials)'
      RegionName: 'eu-west-1'      
      EnvironmentName: '$(EnvironmentName)'
      ColourName: '$(ColourName)'
      DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group      

- job: ReleaseRegionN # Will run in parallel with ReleaseEU if you have enough build agents
  pool:
    vmImage: 'windows-2019'
  dependsOn: Build
  steps:
  - template: TaskGroups/DeployToRegion.yaml # this template file is shown below
    parameters:
      AwsCredentials: '$(AwsCredentials)'
      RegionName: 'ANother region'      
      EnvironmentName: '$(EnvironmentName)'
      ColourName: '$(ColourName)'
      DatadogApiKey: '$(DatadogApiKey)' # referenced from a variable group

The ‘DeployToRegion’ template:

parameters:
  AwsCredentials: ''
  RegionName: ''  
  EnvironmentName: ''
  ColourName: ''
  DatadogApiKey: ''  

steps:
- task: DownloadPipelineArtifact@1 # you can download artifacts from other builds if needed
  inputs:
      buildType: 'specific'
      project: 'Project Name'
      pipeline: '##'
      buildVersionToDownload: 'latest'
      artifactName: 'Devops'
      targetPath: '$(System.ArtifactsDirectory)/Devops'

- task: DownloadPipelineArtifact@1 # or download from the current one
  inputs:
      buildType: 'current'
      artifactName: 'Website'
      targetPath: '$(System.ArtifactsDirectory)'

- template: DeployToElasticBeanstalk.yaml # and can chain templates if needed
  parameters:
      AwsCredentials: '${{ parameters.AwsCredentials }}'
      RegionName: '${{ parameters.RegionName }}'      
      EnvironmentName: '${{ parameters.EnvironmentName }}'
      ColourName: '${{ parameters.ColourName }}'
      DatadogApiKey: '${{ parameters.DatadogApiKey }}'

And finally some powershell to fire it all off:

### Example usage: .\TriggerBuild.ps1 -branch "release/release-006" -isReleaseCandidate $false -additionalReleaseParameters @{ "EnvironmentName" = "qa"; "ColourName" = "blue"; }

param (
    [Parameter(Mandatory = $true)][string]$branch,   
    [boolean]$isReleaseCandidate = $false,
    [HashTable]$additionalReleaseParameters = @{ }
)

$ErrorActionPreference = "Stop"

$authToken = Get-DevOpsAuthToken # see https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops for how to get a token
$accountName = "AzureDevopsAccountName" 
$projectName = "AzureDevopsProjectName"

$buildDefinitionIds = @(27) # the build pipeline id

Write-Host "Building with settings:"
Write-Host "Branch: '$branch'"
Write-Host "Tag as 'release-candidate' and retain build: $isReleaseCandidate"
Write-Host "Build definition IDs: $buildDefinitionIds"
Write-Host "Additional parameters: $($additionalReleaseParameters | ConvertTo-Json) "
Write-Host ""

$releaseIds = @()

$result = @{
    Success = $false;    
}

foreach ($definitionId in $buildDefinitionIds)
{
    $deploymentParams = @{
        "definition" = @{
            "id" = $definitionId;
        }
        "sourceBranch" = $branch;
    }

    if ($additionalReleaseParameters.GetEnumerator().length -gt 0)
    {
        $deploymentParams.parameters = $additionalReleaseParameters | ConvertTo-Json
    }

    $content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds?api-version=4.1" `
        -ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method POST -Body ($deploymentParams | ConvertTo-Json)).Content | ConvertFrom-Json

    $releaseIds += $content.id  

    Write-Host "Build $($content.id) queued: https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$($content.id)" -ForegroundColor Yellow
}

$aBuildFailed = $false

foreach ($releaseId in $releaseIds)
{
    $status = ""

    while ($status -ne "completed")
    {
        try
        {
            $content = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -Headers (Get-DevOpsHeaders -AuthToken $authToken)).Content | ConvertFrom-Json
        }
        catch
        {
            Write-Host "  Error calling DevopsAPI. If this happens several times check the url: https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId" -ForegroundColor red
        }

        $status = $content.status

        Write-Host " Build id $releaseId has status: $status"

        if ($content.result -eq "failed" -or $content.result -eq "canceled")
        {
            $aBuildFailed = $true

            Write-Host "Build $releaseId failed - check https://dev.azure.com/$accountName/$projectName/_build/results?buildId=$releaseId for details" -ForegroundColor Red
        }
        elseif ($content.result -eq "completed")
        {
            Write-Host "Build $releaseId completed successfully" -ForegroundColor Green
        }

        Start-Sleep -s 5
    }

    if ($isReleaseCandidate -eq $true)
    {
        Write-Host " Adding RC tags: release-candidate"
        $tags = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$releaseId/tags/release-candidate?api-version=4.1" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PUT).Content | ConvertFrom-Json

        Write-Host " Adding retain build to $releaseId"
        $updates = (Invoke-WebRequest -uri "https://dev.azure.com/$accountName/$projectName/_apis/build/builds/$($releaseId)?api-version=4.1" -ContentType "application/json" -Headers (Get-DevOpsHeaders -AuthToken $authToken) -Method PATCH -Body (@{"retainedByRelease" = $true } | ConvertTo-Json)).Content | ConvertFrom-Json
    }
}

$result.Success = !$aBuildFailed

return $result

Happy deploying

The post Automating a multi region deployment with Azure Devops appeared first on blog.boro2g .co.uk.

Certain Sitecore operations push a lot of data to disk, things like: logging, media caches, icons, installation history and lots more. As part of your deployment I’d suggest you setup the following changes:

• Move the data folder to another location
  • This can be patched in with the sc.variable dataFolder
• Move the media cache
  • This can be patched in with setting Media.CacheFolder
  • Remember to also update the cleanup agents which look to delete old file based media entries. These are referenced in the CleanupAgent.
• Move the temp folder
  • Now, this isn’t quite so simple. The reason for noting this item is installing update files creates large __UpgradeHistory folders. Also, things like icons are cached in the /temp folder. For this reason /temp needs to live as /temp under the site root. With some help from support they concluded that rather than a virtual directory, a symlink was the best approach. For details on setting one up via powershell, see http://stackoverflow.com/questions/894430/powershell-hard-and-soft-links. As part of our startup scripts we included calling the script to shift things around.
  • The clue a folder is a symlink is the icon will change ala:
    

The post Sitecore Azure web-role disk space appeared first on blog.boro2g .co.uk.

If you want to push Sitecore counters through to this you are in luck. Make sure the counters are installed on the box – for web-roles this can be done via a startup task.

Then update your ‘diagnostics.wadcfg’ config to include the counters you want – make sure you include the (*)!

<!-- updated as per https://sdn.sitecore.net/Articles/Administration/Sitecore%20Performance/Optimizing%20Sitecore%206%20and%20later/Optimizing%20Performance%20in%20Sitecore.aspx -->
		<PerformanceCounterConfiguration counterSpecifier="\Sitecore.Caching(*)\CacheClearings" sampleRate="PT1M" />
		<PerformanceCounterConfiguration counterSpecifier="\Sitecore.Caching(*)\CacheHits" sampleRate="PT1M" />
		<PerformanceCounterConfiguration counterSpecifier="\Sitecore.Caching(*)\CacheMisses" sampleRate="PT1M" />

		<PerformanceCounterConfiguration counterSpecifier="\Sitecore.Data(*)\Data.ItemsAccessed" sampleRate="PT1M" />
		<PerformanceCounterConfiguration counterSpecifier="\Sitecore.Data(*)\Data.PhysicalReads" sampleRate="PT1M" />
		<PerformanceCounterConfiguration counterSpecifier="\Sitecore.Data(*)\Data.PhysicalWrites" sampleRate="PT1M" />

		<PerformanceCounterConfiguration counterSpecifier="\Sitecore.System(*)\Exceptions.ExceptionsThrown" sampleRate="PT1M" />

This caught me out for a while, I was missing the (*).

There is a handy cmd you can run to see all the counters available:

typeperf -q -o "C:\Temp\counters.txt"

The post Azure WAD Diagnostics – Sitecore counters appeared first on blog.boro2g .co.uk.

There are quite a few tools for doing this, even SQL management studio can help. The problem we faced was the source db had several filegroups setup, SQL Azure doesn’t support this.

To get around it we first created the DB through the portal, then scripted in the db structure – follow the steps in this link: http://blog.sqlauthority.com/2010/06/04/sql-server-generate-database-script-for-sql-azure/

The second step was then the data. We used the Azure MW tool (https://sqlazuremw.codeplex.com/) . When selecting what to migrate, select ‘data only’. This can take a while to run, several hours in our case, but at least got the data upto the cloud.

The post Migrating a sitecore DB to PAAS SQL Azure appeared first on blog.boro2g .co.uk.

In your solution you should end up with a file: diagnostics.wadcfg under the roles folder. One issue we ran into occurred when we changed the name of the counterpart project, the diagnostics file then sat in the wrong place. Ensure the structure is as follows:

- WebProject
-- files for your site
- CloudProject
-- Roles (default folder name)
--- WebProject (folder named the same as your webproject)
---- diagnostics.wadcfg

You can verify this by checking the Azure project ccproj file. You should see:

<ItemGroup>
    <ProjectReference Include="..\WebProject\WebProject.csproj">
      <Name>CloudProject</Name>
      ...      
      <RoleName>CloudProject</RoleName>   
      ...   
    </ProjectReference>
  </ItemGroup>
  <ItemGroup>
    <Folder Include="WebProjectContent\" />    
  </ItemGroup>
  <ItemGroup>
    <Content Include="WebProjectContent\diagnostics.wadcfg">
      <SubType>Content</SubType>
    </Content>
  </ItemGroup>

When the site fires up you should then see blob storage contain a folder called: wad-control-container. If you have a look in there it should contain a file that mimics the content of your diagnostics.wadcfg file.

I found it was useful to clear out this folder before testing new deployments.

The post Azure diagnostics not writing to blob storage appeared first on blog.boro2g .co.uk.

As part of setting these up I thought I’d share a few tips / gotchas that caught me out when running powershell from the cmd tasks.

My solution setup was:

• Site root
  • StartupScripts
    • Script.cmd
    • Script.ps1

So then I’d reference in the task:

 <Task commandLine=”StartupScripts\script.cmd” executionContext=”elevated” taskType=”simple” />

Nothing rocket science so far! So, why didn’t the script work? I could jump on the box and run the cmd and it would be fine.

How to debug the process?
I found the most useful way was to add markers from the cmd and the ps1. The cmd file looked like:

md c:\test
echo "Running" > "c:\test\log.txt"
powershell -command "Set-ExecutionPolicy Unrestricted" 2>> "c:\test\log.txt"
powershell .\startupScripts\script.ps1 %~dp0 2>> "c:\test\log.txt"
EXIT /B 0

Note, the .\startupScripts part of the ps1 path is v important!

Then the powershell:

$tempFolder = "c:\test"
$logFile = "ps_log.txt"
$siterootFolder = "\sitesroot\0\"
If (!(Test-Path "e:\"))
{
    $siterootFolder = "f:" + $siterootFolder
}
else
{
    $siterootFolder = "e:" + $siterootFolder
}

function Write-Log
{
    param( [string] $text )
    Write-Host $text;	
    $path = [System.IO.Path]::Combine("$tempFolder\$logFile")

    $fs = New-Object IO.FileStream($path, [System.IO.FileMode]::Append, [System.IO.FileAccess]::Write, [IO.FileShare]::Read)
    $sw = New-Object System.IO.StreamWriter($fs)

    $sw.WriteLine($text)
    $sw.Dispose()
    $fs.Dispose()
}

Write-Log "Output folder is $siterootFolder"

Note, if you try to write to log.txt you will get process locked exceptions as the cmd holds locks on the file.
There are all sorts of techniques for writing to a file, this example uses a StreamWriter. Hit up google for different examples.

The post Debugging Azure web-role startup tasks appeared first on blog.boro2g .co.uk.