Automating a multi region deployment with Azure Devops

For a recent project we’ve invested a lot of time into Azure Devops, and in the most part found it a very useful toolset for deploying our code to both Azure and AWS.

When we started on this process, YAML pipelines weren’t available for our source code provider – this meant everything had to be setup manually 🙁

However, recently this has changed 🙂 This post will run through a few ways you can optimize your release process and automate the whole thing.

First a bit of background and then some actual code examples.

Why YAML?

Setting up your pipelines via the UI is a really good way to quickly prototype things, however what if you need to change these pipelines to mimic deployment features alongside code features. Yaml allows you to keep the pipeline definition in the same codebase as the actual features. You deploy branch XXX and that can be configured differently to branch YYY.

Another benefit, the changes are then visible in your pull requests so validating changes is a lot easier.

Async Jobs

A big optimization we gained was to release to different regions in parallel. Yaml makes this very easy by using Jobs – each job can run on an agent and hence push to multiple regions in parallel.

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml

Yaml file templates

If you have common functionality you want to duplicate, e.g. ‘Deploy to Eu-West-1’, templates are a good way to split your functionality. They allow you to group logical functionality you want to run multiple times.

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Azure Devops rest API

All of your build/releases can be triggered via the UI portal, however if you want to automate that process I’d suggest looking into the rest API. Via this you can trigger, monitor and administer builds, releases and a whole load more.

We use powershell to orchestrate the process.

https://docs.microsoft.com/en-us/rest/api/azure/devops/build/builds/queue?view=azure-devops-rest-5.1

Variables, and variable groups

I have to confess, this syntax feels slightly cumbersome, but it’s very possible to reference variables passed into a specific pipeline along with global variables from groups you setup in the Library section of the portal.

Now, some examples

The root YAML file:

The ‘DeployToRegion’ template:

And finally some powershell to fire it all off:

Happy deploying 🙂

Sitecore Azure web-role disk space

If you need to deploy your app to Azure cloud-services you’ll notice the installation drive will yo-yo between E: and F: each time you deploy. These drives are always created with rather limited capacity – roughly 1.5GB.

Certain Sitecore operations push a lot of data to disk, things like: logging, media caches, icons, installation history and lots more. As part of your deployment I’d suggest you setup the following changes:

  • Move the data folder to another location
    • This can be patched in with the sc.variable dataFolder
  • Move the media cache
    • This can be patched in with setting Media.CacheFolder
    • Remember to also update the cleanup agents which look to delete old file based media entries. These are referenced in the CleanupAgent.
  • Move the temp folder
    • Now, this isn’t quite so simple. The reason for noting this item is installing update files creates large __UpgradeHistory folders. Also, things like icons are cached in the /temp folder. For this reason /temp needs to live as /temp under the site root. With some help from support they concluded that rather than a virtual directory, a symlink was the best approach. For details on setting one up via powershell, see http://stackoverflow.com/questions/894430/powershell-hard-and-soft-links. As part of our startup scripts we included calling the script to shift things around.
    • The clue a folder is a symlink is the icon will change ala:
      symlink

Azure WAD Diagnostics – Sitecore counters

Azure allows you to pipe loads of stats and diagnostic information to blob and table storage. Google for “WAD Azure” and you’ll find a lot more information.
Its great for logging things like the event queue, performance counters and loads more over time.

If you want to push Sitecore counters through to this you are in luck. Make sure the counters are installed on the box – for web-roles this can be done via a startup task.

Then update your ‘diagnostics.wadcfg’ config to include the counters you want – make sure you include the (*)!

This caught me out for a while, I was missing the (*).

There is a handy cmd you can run to see all the counters available:

Migrating a sitecore DB to PAAS SQL Azure

Moving to the cloud offers many new opportunities for how you store and handle your data. One challenge we ran into was how to migrate a large Sitecore web db into a PAAS sql azure db.

There are quite a few tools for doing this, even SQL management studio can help. The problem we faced was the source db had several filegroups setup, SQL Azure doesn’t support this.

To get around it we first created the DB through the portal, then scripted in the db structure – follow the steps in this link: http://blog.sqlauthority.com/2010/06/04/sql-server-generate-database-script-for-sql-azure/

The second step was then the data. We used the Azure MW tool (https://sqlazuremw.codeplex.com/) . When selecting what to migrate, select ‘data only’. This can take a while to run, several hours in our case, but at least got the data upto the cloud.

Azure diagnostics not writing to blob storage

Azure offers some very useful out the box tools for pooling diagnostic information into blob and table storage. http://azure.microsoft.com/en-gb/documentation/articles/cloud-services-dotnet-diagnostics/ contains lots of information on this.

In your solution you should end up with a file: diagnostics.wadcfg under the roles folder. One issue we ran into occurred when we changed the name of the counterpart project, the diagnostics file then sat in the wrong place. Ensure the structure is as follows:

You can verify this by checking the Azure project ccproj file. You should see:

When the site fires up you should then see blob storage contain a folder called: wad-control-container. If you have a look in there it should contain a file that mimics the content of your diagnostics.wadcfg file.

I found it was useful to clear out this folder before testing new deployments.

Debugging Azure web-role startup tasks

One feature Azure offers for getting your  boxes configured is the notion of startup tasks – I won’t go into too much detail here as there is lots available online e.g. https://msdn.microsoft.com/en-us/library/azure/hh180155.aspx

As part of setting these up I thought I’d share a few tips / gotchas that caught me out when running powershell from the cmd tasks.

My solution setup was:

  • Site root
    • StartupScripts
      • Script.cmd
      • Script.ps1

So then I’d reference in the task:

 <Task commandLine=”StartupScripts\script.cmd” executionContext=”elevated” taskType=”simple” />

Nothing rocket science so far! So, why didn’t the script work? I could jump on the box and run the cmd and it would be fine.

How to debug the process?
I found the most useful way was to add markers from the cmd and the ps1. The cmd file looked like:

Note, the .\startupScripts part of the ps1 path is v important!

Then the powershell:

Note, if you try to write to log.txt you will get process locked exceptions as the cmd holds locks on the file.
There are all sorts of techniques for writing to a file, this example uses a StreamWriter. Hit up google for different examples.