I love containers. I've said before that I think they're the future. Just as hardly anyone installs on tin any more since we're so comfortable with Virtualization, I think that in a few years time hardly anyone will deploy VMs - we'll all be on containers. However, container orchestration is still a challenge. Do you choose Kubernetes or Swarm or DCOS? (For my money I think Kubernetes is the way to go). But that means managing a cluster of nodes (VMs). What if you just want to deploy a single container in a useful manner?

You can do that now using Azure Container Instances (ACI). You can also host a container in an Azure Web App for Containers. The Web App for Containers is what I'll use for this post - mostly because I already know how to do A/B testing with Azure Web Apps, so once the container is running then you get all the same paradigms as you would for "conventional" web apps - like slots, app settings in the portal etc.

In this post I'll cover publishing a .NET Core container to Azure Web Apps using VSTS with an ARM template. However, since the hosting technology is "container" you can host whatever application you want - could be Java/TomCat or node.js or python or whatever. The A/B Testing principles will still apply.

You can grab the code for this demo from this Github repo.

Overview of the Moving Parts

There are a couple of moving parts for this demo:

• The source code. This is just a File->New Project .NET Core 2.0 web app. I've added a couple lines of code and Application Insights for monitoring - but other than that there's really nothing there. The focus of this post is about how to A/B test, not how to make an app!
• Application Insights - this is how you can monitor the web app to make sure
• An Azure Container Registry (ACR). This is a private container repository. You can use whatever repo you want, including DockerHub.
• A VSTS Build. I'll show you how to set up a build in VSTS to build the container image and publish it to the ACR.
• An ARM template. This is the definition of the resources necessary for running the container in Azure Web App for Containers. It includes a staging slot and Application Insights.
• A VSTS Release. I'll show you how to create a release that will spin up the web app and deploy the container. Then we'll set up Traffic Manager to (invisibly) divert a percentage of traffic from the prod slot to the staging slot - this is the basis for A/B testing and the culmination of the all the other steps.

The Important Source Bits

Let's take a look at some of the important code files. Firstly, the Startup.cs file:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
	var aiKey = Environment.GetEnvironmentVariable("AIKey");
	if (aiKey != null)
	{
		TelemetryConfiguration.Active.InstrumentationKey = aiKey;
	}
	...

Notes:

• Line 3: I read the environment variable "AIKey" to get the Application Insights key
• Lines 4 - 7: If there is a key, then I set the key for the Application Insights config

The point here is that for configuration, I want to get values from the environment. This would include database connection strings etc. Getting them from the environment lets me specify them in the Web App appSettings so that I don't have to know the values at build time - only at release time.

Let's look at the Dockerfile:

FROM microsoft/aspnetcore:2.0
ARG source
ENV AIKey="11111111-2222-3333-4444-555555555555"

WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "DockerWebApp.dll"]

Notes:

• Line 3 was the only thing I added to make the AIKey configurable as an environment variable

Finally, let's look at the ARM template that defines the resources. The file is too large to paste here and it's Json so it's not easy to read. You can have a look at the file yourself. The key points are:

• a HostingPlan with kind = "linux" and properties.reserved = true. This creates a Linux app plan.
• a Site with properties.siteConfig.appSettings that specify the DOCKER_REGISTRY_SERVER_URL, DOCKER_REGISTRY_SERVER_USERNAME, DOCKER_REGISTRY_SERVER_PASSWORD (using the listCredentials function) and AIKey. We do not specify the DOCKER_CUSTOM_IMAGE_NAME for reasons that will be explained later. Any other environment variables (like connection strings) could be specified here.
• a staging slot (called blue) that has the same settings as the production slot
• a slotconfignames resource that locks the DOCKER_CUSTOM_IMAGE_NAME so that the value is slot-sticky
• an Application Insights resource - the key for this resource is referenced by the appSettings section for the site and the slot

Building (and Publishing) Your Container

At this point we can look at the build. The build needs to compile, test and publish the .NET Core app and build a container. It then needs to publish the container to the ACR (or whatever registry you created).

Create a new build using the ASP.NET Core Web App template and then edit the steps to look as follows:

Make sure you point the Source settings to the repo (you can create a new one and import mine from the Github URL if you want to try this yourself). I then changed the queue to Hosted Linux Preview and changed the name to something appropriate.

On the Options page I set the build number format to 1.0$(rev:.r) which gives a 1.0.0, 1.0.1, 1.0.2 etc. format for my build number.

Click on the Build task and set the arguments to --configuration $(BuildConfiguration) /p:Version=$(Build.BuildNumber). This versions the assemblies to match the build number.

Click on the Publish task and set Publish Web Apps, change the arguments to --configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory) /p:Version=$(Build.BuildNumber) and unselect Zip files. This puts the output files into the artifact staging directory and doesn't zip them. It also publishes with a version number matching the build number:

I deleted the Test task since I don't have tests in this simple project - but you can of course add testing in before publishing.

I then deleted the Publish build artifact tasks since this build won't be publishing an artifact - it will be pushing a container image to my ACR.

In order to build the docker container image, I first need to put the Dockerfile at the correct location relative to the output. So I add a Copy Files task in and configure it to copy the Dockerfile to the artifact staging directory:

Now I add a 2 Docker tasks: the has a Build an image Action. I set the ACR by selecting the settings from the dropdowns. I set the path to the Dockerfile and specify "DockerWebApp" as the source build argument (the Publish task will have places the compiled site and content into this folder in the artifact staging directory). I set Qualify the Image name to correctly tag the container with the ACR prefix and I include the Latest tag in the build (so the current build is always the Latest).

The 2nd Docker task has Action set to Publish an Image. I set the ACR like the Docker Build task. I also change the image name to $(Build.Repository.Name):$(Build.BuildNumber) instead of $(Build.Repository.Name):$(Build.BuildId) and I set the Latest tag.

Now I can run the build. Lots of green! I can also see the image in my ACR in the Azure portal:

Woot! We now have a container image that we can host somewhere.

Releasing the Container

Now that we have a container and an infrastructure template, we can define a release. Here's what the release looks like:

There are 2 incoming artifacts: the build and the Git repo. The build doesn't actually have any artifacts itself - I just set the build as the trigger mechanism. I specify a "master" artifact filter so that only master builds trigger this release. The Git repo is referenced for the deployment scripts (in this case just the ARM template). I started with an empty template and then changed the Release number format to $(Build.BuildNumber)-$(rev:r) in the Options page.

There are 3 environments: Azure blue, Azure prod and blue failed. These are all "production" environments - you could have Dev and Staging environments prior to these environments. However, I want to A/B test in production, so I'm just showing the "production environment" here.

Let's look at the Azure blue environment:

There are 3 tasks: Azure Resource Group Deployment (to deploy the ARM template), an Azure CLI command to deploy the correct container, and a Traffic Manager Route Traffic task. The Azure Resource Group Deployment task specifies the path to the ARM template and parameters files as well as the Azure endpoint for the subscription I want to deploy to. I specify a variable called $(RGName) for the resource group name and then override the parameters for the template using $(SiteName) for the name of the web app in Azure, $(ImageName) for the name of the container image, $(ACR) for the name of my ACR and $(ACRResourceGroup) for the name of the resource group that contains my ACR. Once this task has run, I will have the following resources in the resource group:

Let's take a quick look at the app settings for the site:

At this point the site (and slot) are provisioned, but they still won't have a container running. For that, we need to specify which container (and tag) to deploy. The reason I can't do this in the ARM template is because I want to update the staging slot and leave the prod slot on whatever container tag it is on currently. Let's imagine I specified "latest" for the prod slot - then when we run the template, the prod slot will update, which I don't want. Let's say we specify latest for the blue slot - then the blue slot will update - but what version do we specify in the template for the prod slot? We don't know that ahead of time. So to work around these issues, I don't specify the container tag in the template - I use an Azure CLI command to update it after the template has deployed (or updated) all the other infrastructure and settings.

To deploy a container, we add an Azure CLI task and specify a simple inline script:

Here I select version 1.* of the task (version 0.* doesn't let you specify an inline script). I set the script to sleep for 30 seconds - if I don't do this, then the site is still updating or something and the operation succeeds, but doesn't work - it doesn't actually update the image tag. I suspect that the update is async and so if you don't wait for it to complete, then issuing another tag change succeeds but is ignored. It's not pretty, but that's the only workaround I've found. After pausing for a bit, we invoke the "az webapp config container set" command, specifying the site name, slot name, resource group name and image name. (If you're running this phase on a Linux agent, use $1, $2 etc. for the args - if you're running this on a Windows agent then specify the args using %1, %2 etc. in the script). Then I pass the arguments in - using $(SiteName), blue, $(RGName) and $(ImageName):$(Build.BuildNumber) for the respective arguments.

If you now navigate to the site in the Azure portal and click on the Docker Container tab on the slot, you'll see the container settings:

You can see that the image name and version are specified.

The final task in this environment (a Route Traffic task from my Build and Release extension pack) adds a traffic manager rule - we divert 20% of traffic to the blue slot (unobtrusively to the client):

There's a snag to this method: the first time you deploy, there's nothing (yet) in the prod slot. This is just a first time condition - so after you run this environment for the very first time, navigate to the Azure portal and click on the site. Then swap the blue and prod slots. Now the prod slot is running the container and the blue slot is empty. Repeat the deployment and now both slots have the latest version of the container.

Let's go back to the release and look at the Azure prod environment. This has a pre-approval set so that someone has to approve the deployment to this environment. The Azure blue has a post-deployment approver so that someone can sign off on the release - approved if the experiment works, rejected if it's not. The Azure prod environment triggers when the Azure blue environment is successful - all it has to do is swap the slots and reset the traffic router to reroute 100% of traffic to the prod slot:

So what do we do if the experiment fails? We can reject the Azure blue environment (so that the Azure prod environment doesn't run). We then manually run the blue fail environment - this just resets the traffic to route 100% back to prod. It does not swap the slots:

Don't forget to specify values for the variables we used:

Running a Test

So imagine I have container version 1.0.43 in both slots. Now I make a change and commit and push - this triggers a build (I enabled CI) and we get version 1.0.45 (1.0.44 had an intermittent build failure). This triggers the release (I enabled CD) and now the blue slot has version 1.0.45 and 20% of traffic from the prod slot is going to the blue slot.

Let's navigate to the blue and prod slots and see them side-by-side:

The traffic routing is "sticky" to the user - so if a user navigates to the prod slot and gets diverted to the blue slot, then all requests to the site from the user go to the blue slot. Try opening some incognito windows and hitting the prod site - you'll get the blue content 20% of the time! You can also force the routing using a query parameter - just tack "?x-ms-routing-name=blue" onto the end of any request and you'll end up on the blue slot:

Now you wait for users to generate traffic (or in my case, I totally fake client traffic - BWAHAHA!). So how do we know if the experiment is successful? We use Application Insights.

Application Insights Analytics

Let's click on the Azure portal and go to the App Insights resource for our web app. We can see some telemetry, showing traffic to the site:

But how do we know which requests went to the blue slot and how many went to the prod slot? It turns out that's actually pretty simple. Click the Analytics button to launch App Insights Analytics. We enter a simple query:

We can clearly see that there is more usage of the 1.0.45 contact page. Yes, this metric is bogus - the point is to show you that you can slice by "Application_Version" and so you actually have metrics to determine if your new version (1.0.45) is better or worse that 1.0.43. Maybe it's more traffic to a page. Maybe it's more sales. Maybe it's less exceptions or better response time - all of these metrics can be sliced by Application_Version.

Conclusion

Deploying containers to Azure Web Apps for Containers is a great experience. Once you have the container running, you can use any Web App paradigm - such as Traffic Routing or appSettings - so it's easy if you've ever done any Web App deployments before.

There are a couple of key practices that are critical to A/B testing: telemetry and unobtrusive routing.

Telemetry is absolutely critical to A/B testing: if you can't decide if A is better (or worse) than B, then there's no point deploying both versions. Application Insights is a great tool for telemetry in general - but especially with A/B testing, since you can put some data and science behind your hypotheses. You don't have to use AppInsights - but you do have to have some monitoring tool or framework in order to even contemplate A/B testing.

The other key is how you release the A and B sites or apps. Having traffic manager seamlessly divert customer traffic is an excellent way to do this since the customer is none the wiser about which version they are seeing - so they don't have to change their URLs or anything obscure. You could also use LaunchDarkly or some other feature flag mechanism - as long as your users don't have to change their usual way of accessing your app. This will give you "real" data. If users have to go to a beta site, they could change their behavior subconsciously. Maybe that isn't a big deal - but at least prefer "seamless" routing between A and B sites before you explicitly tell users to navigate to a new site altogether.

Happy testing!

Recently I was working with a customer that was struggling with test environments. Their environments are complex and take many weeks to provision and configure - so they are generally kept around even though some of them are not frequently used. Besides a laborious, error-prone manual install and configuration process that usually takes over 10 business days, the team has to maintain all the clones of this environment. This means that at least two senior team members are required just to maintain existing dev and test environments as well as create new ones.

Using Azure ARM templates and VSTS Release Management with Deployment Groups, we were able to show how we could spin up the entire environment in just under two hours. That's a 50x improvement in lead time! And it's more consistent since the entire process is automated and the scripts are all source controlled so there's auditability. This is a huge win for the team. Not only can they spin up an environment in a fraction of the time they are used to, they can now decommission environments that are not frequently used (some environments were only used twice a year). That means they have less maintenance to worry about. When they need an environment for a short time, they spin it up, used it and then throw it away. They've also disseminated "tribal knowledge" from a few team members' heads to a button click - meaning anyone can create a new environment now.

This was my first time working with a larger scale provisioning and configuration project that uses Deployment Groups - and this post documents some of the lessons that we learned along the way.

A Brief Glossary

Before we jump into the tips, I need to get some definitions out of the way. In VSTS, a Release Definition is made up of multiple Environments. Typically you see DEV, STAGE and PROD but you can have multiple "environments" that target the same set of machines.

The above VSTS release has three "environments":

• Infrastructure Provision
• Runs an ARM template to provision VMs, VNets, Storage and any other infrastructure required for the environment
• Infrastructure Config
• Configure the OS of each machine, DNS and any other "low-level" settings
• App Install
• Install and configure the application(s)

This separation also allows you to run the "Infrastructure Provision" environment and then set it to a manual trigger and just trigger the config environment - particularly useful when you're developing the pipeline, since you can skip environments that end up being no-ops but take a couple minutes to pass through.

Within an Environment, you can have 1..n phases. You specify tasks inside a phase - these are the smallest unit of work in the release.

In the above image, there are several phases within the "Infrastructure Config" environment. Each phase (in this case) is running a single task, but you can run as many tasks as you need for that particular phase.

There are three types of phases: agentless, agent-based or deployment-group based. You can think of agentless phases as phases that are executed on VSTS. Agent-based phases are executed on agent(s) in a build or release queue. Deployment Group phases are executed on all agents (with optional tag matching) within the specified Deployment Group. The agent for agent-based or deployment-group based is the same agent under the hood - the difference is that deployment group agents are only referenced through the Deployment Group while build/release agents are accessed through queues. You'd typically use agent queues for build servers or for "proxy servers" in releases (where the tasks are executing on the proxy but acting on other machines). Deployment Groups are used when you don't know the machines ahead of time - like when you're spinning up a set of machines in the cloud on demand. They also allow you to target multiple machines at the same time.

The VSTS Deployment agent joins a machine (this can be any machine anywhere that can connect to VSTS) to a Deployment Group. The agent is cross-platform (runs on DotNET Core) so it can run on practically any machine anywhere. It connects out to VSTS meaning you don't need to open incoming firewall ports at all. The agent runs on the machine and so any scripts you write can execute locally - which simplifies configuration dramatically. Executing remote instructions is typically much harder to do - you have to think about your connection and security and so on. Executing locally is much easier.

TL;DR - The Top 10 Tips and Tricks

Here are my top 10 tips and tricks:

1. Spin Up Azure VMs with the VSTS Deployment Agent Extension
1. This allows you to configure everything else locally on each machine
2. Use Tagging for Parallelization and Specialization
1. Tagging the VSTS agent allows you to repeat the same actions on many machines in parallel and/or distinguish machines for unique actions
3. Use Phases to Start New Sessions
1. Each phase in an Environment gets a new session, which is useful in a number of scenarios
4. Update Your PowerShell PackageProviders and Install DSC Modules
1. If you're using DSC, install the modules in a separate step to ensure that they are available when you run DSC scripts. You may need to update your Package Providers for this to work
5. Install Azure PowerShell and use the Azure PowerShell task
1. If you're going to be doing any scripting to Azure, you can quickly install Azure PowerShell so that you can use the Azure PowerShell task
6. Use PowerShell DSC for OS Configuration
1. Configuring Windows Features, firewalls and so on is best done with PowerShell DSC
7. Use Plain PowerShell for Application Install and Config
1. Expressing application state can be challenging - so use "plain" PowerShell for application install and config
8. Attaching Data Disks in Azure VMs
1. If you add data disks in your ARM template, you still need to mount them in the OS of the VM
9. Configuring DNS on Azure VNets
1. If you create an Active Directory Domain Controller or DNS, you'll need to do some other actions on the VNet too
10. Wait on machines when they reboot
1. If you reboot a machine and don't pause, the subsequent deployment steps fail because the agent goes offline.

In the next section I'll dig into each tip.

Tip 1: Spin Up Azure VMs with the VSTS Deployment Agent Extension

You can install the VSTS Deployment Agent (or just "the agent" for the remainder of this post) on any machine using a simple script. The script downloads the agent binary and configures it to connect the agent to your VSTS account and to the specified Deployment Group. However, if you're spinning up machines by using an ARM template, you can also install the agent via the VSTS extension. In order to do this you need a Personal Access Token (or PAT), the name of the VSTS account, the name of the Deployment Group and optionally some tags to tag the agent with. Tags will be important when you're distinguishing between machines in the same Deployment Group later on. You'll need to create the Deployment Group in VSTS before you run this step.

Here's a snippet of an ARM template that adds the extension to the Deployment Group:

{
    "name": "[parameters('settings').vms[copyIndex()].name]",
    "type": "Microsoft.Compute/virtualMachines",
    "location": "[resourceGroup().location]",
    "apiVersion": "2017-03-30",
    "dependsOn": [
      ...
    ],
    "properties": {
      "hardwareProfile": {
        "vmSize": "[parameters('settings').vms[copyIndex()].size]"
      },
      "osProfile": {
        "computerName": "[parameters('settings').vms[copyIndex()].name]",
        "adminUsername": "[parameters('adminUsername')]",
        "adminPassword": "[parameters('adminPassword')]"
      },
      "storageProfile": {
        "imageReference": "[parameters('settings').vms[copyIndex()].imageReference]",
        "osDisk": {
          "createOption": "FromImage"
        },
        "dataDisks": [
            {
                "lun": 0,
                "name": "[concat(parameters('settings').vms[copyIndex()].name,'-datadisk1')]",
                "createOption": "Attach",
                "managedDisk": {
                    "id": "[resourceId('Microsoft.Compute/disks/', concat(parameters('settings').vms[copyIndex()].name,'-datadisk1'))]"
                }
            }
        ]
      },
      "networkProfile": {
        "networkInterfaces": [
          {
            "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(parameters('settings').vms[copyIndex()].name, if(equals(parameters('settings').vms[copyIndex()].name, 'JumpBox'), '-nicpub', '-nic')))]"
          }
        ]
      }
    },
    "resources": [
      {
        "name": "[concat(parameters('settings').vms[copyIndex()].name, '/TeamServicesAgent')]",
        "type": "Microsoft.Compute/virtualMachines/extensions",
        "location": "[resourceGroup().location]",
        "apiVersion": "2015-06-15",
        "dependsOn": [
          "[resourceId('Microsoft.Compute/virtualMachines/', concat(parameters('settings').vms[copyIndex()].name))]"
        ],
        "properties": {
          "publisher": "Microsoft.VisualStudio.Services",
          "type": "TeamServicesAgent",
          "typeHandlerVersion": "1.0",
          "autoUpgradeMinorVersion": true,
          "settings": {
            "VSTSAccountName": "[parameters('vstsAccount')]",
            "TeamProject": "[parameters('vstsTeamProject')]",
            "DeploymentGroup": "[parameters('vstsDeploymentGroup')]",
            "Tags": "[parameters('settings').vms[copyIndex()].tags]"
          },
          "protectedSettings": {
            "PATToken": "[parameters('vstsPat')]"
          }
        }
      },
      ...

Notes:

• The extension is defined in the highlighted lines
• The "settings" section of the extension is where you specify the VSTS account name, team project name, deployment group name and comma-separated list of tags for the agent. You also need to supply a PAT that has access to join machines to the Deployment Group
• You can also specify a "Name" property if you want the agent name to be custom. By default it will be machineName-DG (so if the machine name is WebServer, the agent will be named WebServer-DG.

Now you have a set of VMs that are bare-boned but have the VSTS agent installed. They are now ready for anything you want to throw at them - and you don't need to worry about ports or firewalls or anything like that.

There are some useful extensions and patterns for configuring VMs such as Custom Script or Join Domain. The problem with these scripts is that the link to the script has to be either in a public place or in a blob store somewhere, or they assume existing infrastructure. This can complicate deployment. Either you need to publish your scripts publically or you have to deal with uploading scripts and generating SAS tokens. So I recommend just installing the VSTS agent and let it do everything else that you need to do - especially since the agent will download artifacts (like scripts and build binaries) as a first step in any deployment phase.

Tip 2: Use Tagging for Parallelization and Specialization

Tags are really important for Deployment Groups. They let you identify machines or groups of machines within a Deployment Group. Let's say you have a load balanced application with two webservers and a SQL server. You'd probably want identical configuration for the webservers and a completely different configuration for the SQL server. In this case, tag two machines with WEBSERVER and the other machine with SQL. Then you'll define the tasks in the phase  - when the phase runs, it executes all the tasks on all the machines that match the filter - for example, you can target all WEBSERVER machines with a script to configure IIS. These will execute in parallel (you can configure it to work serially if you want to) and so you'll only specify the tasks a single time in the definition and you'll speed up the deployment.

Be careful though: multiple tags use AND (not OR) logic. This means if you want to do something like join a domain on machines with WEBSERVER and SQL, you would think you could specify WEBSERVER, SQL as the tag filter in the phase tag filter. But since the tags are joined with an AND, you'll see the phase won't match any machines. So you'd have to add a NODE tag (or something similar) and apply it to both webservers and SQL machine and then target NODE for things you want to do on all the machines.

The above image shows the tag filtering on the Phase settings. Note too the parallelization settings.

Tip 3: Use Phases to Start New Sessions

At my customer we were using Windows 2012 R2 machines. However, we wanted to use PowerShell DSC for configuring the VMs and you need Windows Management Framework 5.0 to get DSC. So we executed a PowerShell task to upgrade the PowerShell to 5.x:

if ($PSVersionTable.PSVersion.Major -lt 5) {
    $powershell5Url = "https://go.microsoft.com/fwlink/?linkid=839516"
    wget -Uri $powershell5Url -OutFile "wmf51.msu"
    Start-Process .\wmf51.msu -ArgumentList '/quiet' -Wait
}

Notes:

• Line 1: This script checks the major version of the current PowerShell
• Lines 2,3: If it's less than 5, then the script downloads PowerShell 5.1 (the path to the installer can be update to whichever PowerShell version you need)
• Line 4: The installer is invoked with the quiet parameter

However, if we then called a task right after the update task, we'd still get the old PowerShell since all tasks within a phase are executed in the same session. We just added another phase with the same Deployment Group settings - the second phase started a new session and we got the upgraded PowerShell.

This doesn't work for environment variables though. When you set machine environment variables, you have to restart the agent. The VSTS team are working on providing a task to do this, but for now you have to reboot the machine. We'll cover how to do this in Tip 10.

Tip 4: Update Your PowerShell PackageProviders and Install DSC Modules

You really should be using PowerShell DSC to configure Windows. The notation is succinct and fairly easy to read and Windows plays nicely with DSC. However, if you're using custom modules (like xNetworking) you have to ensure that the modules are installed. You can pre-install all the modules so that your scripts can assume the modules are already installed. To install modules you'll need to update your Package Providers. Here's how to do it:

Import-Module PackageManagement
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force

You'll need to start a new Phase in order to pick up the new packages. Then you'll be able to install modules:

Install-Module -Name xActiveDirectory -Force
Install-Module -Name xNetworking -Force
Install-Module -Name xStorage -Force
Install-Module -Name xDSCDomainjoin -Force
Install-Module -Name xComputerManagement -Force

Not all machines need all the modules, but this step is so quick I found it easier to just enumerate and install all the modules anyway. That way I know that any machine could run any DSC script I throw at it.

Tip 5: Install Azure PowerShell

If you're going to do anything against Azure from the VMs (in our case we were downloading binaries from a blob store) then you'll want to use the Azure PowerShell task. This task provides an authenticated context (via a preconfigured endpoint) so you don't have to worry about adding passwords or anything to your script. However, for it to work, you'll need to install Azure PowerShell. Again this must be a separate phase so that subsequent phases can make use of the Azure cmdlets. To do this simply add a PowerShell task and run this line of script: Install-Module AzureRM -AllowClobber -Force

Tip 6: Use PowerShell DSC for OS Configuration

OS configuration can easily be specified by describing the state: is IIS installed or not? Which other OS roles are installed? So DSC is the perfect tool for this kind of work. You can use a single DSC script to configure a group of machines (or nodes, in DSC) but since we have the VSTS agent you can simply write your scripts for each machine using "node localhost". DSC script are also (usually) idempotent - so they work no matter what state the environment is in when the script executes. No messy if statements to check various conditions - DSC does it for you.

When you're doing DSC, you should first check if there is a "native" resource for your action - for example, configuring Windows Features uses the WindowsFeature resource. However, there are some custom actions you may want to perform. There are tons of extensions out there - we used xActiveDirectory to configure an Active Directory Domain Controller settings, for example.

There are times when you'll want to do some custom work that there simply is no custom module for. In that case, you'll need to use the Script resource. The script resource is composed of three parts: GetScript, TestScript and SetScript. GetScript is optional and should return the current state as an object if specified. TestScript should return a boolean - true for "the state is correct" or false for "the state is not correct". If TestScript returns a false, then the SetScript is invoked. Here's an example Script we wrote to configure SMB on a machine according to Security requirements:

Script SMBConfig
{
	GetScript = { @{ Result = Get-SmbServerConfiguration } }
	TestScript =
	{
		$config = Get-SmbServerConfiguration
		$needConfig = $config.EnableSMB2Protocol -and (-not ($config.EnableSMB1Protocol))
		if ($needConfig) {
				Write-Host "SMB settings are not correct." 
		}
		$needConfig
	}
	SetScript =
	{
		Write-Host "Configuring SMB settings" 
		Set-SmbServerConfiguration -EnableSMB1Protocol $false -Force
		Set-SmbServerConfiguration -EnableSMB2Protocol $true -Force
	}
}

Notes:

• Line 1: Specify the type of resource (Script) and a unique name
• Line 3: The GetScript returns an hash table with a Result property that describes the current state - in this case, the SMB settings on the machine
• Line 4: The start of the TestScript
• Line 6: Query the SMB settings
• Line 7: Determine if we need to configure anything or not - this is a check on the SMBProtocol states
• Lines 8-10: Write a message if we do need to set state
• Line 11: return the bool: true if the state is correct, false otherwise
• Lines 16-17: correct the state of the machine - in this case, set protocols accordingly

Tip 7: Use Plain PowerShell for Application Install and Config

Expressing application state and configuration as a DSC script can be challenging. I once wrote some DSC that could install SQL. However, I ended up using a Script resource - and the TestScript just checked to see if a SQL service was running. This check isn't enough to determine if SQL features are installed according to some config.

Instead of writing long Script resources, I just revert to "plain" PowerShell for app install and configuration. This is especially true for more complicated apps. Just make sure your script are idempotent - that is that they can run and succeed every time. For example, if you're installing a service, you may want to first check to see if the service exists before running the installer (otherwise the installer may fail since the service already exists). This allows you to re-run scripts again if other scripts fail.

Tip 8: Attaching Data Disks in Azure VMs

If you're creating data disks for your VMs, then you usually specify the size and type of disk in the ARM template. But even if you add a disk, you need to attach it in the OS. To do this, I used the xStorage DSC extension. This requires a disk number. When we started, the data disk was always disk 2. Later, we added Azure Disk Encryption - but this added another disk and so our disk numbers were off. We ended up needing to add some logic to determine the data disk number and pass that in as a parameter to the DSC configuration:

Configuration DiskConfig
{
	param
	(
		[Parameter(Mandatory)]
		[string]$dataDiskId
	)

	# import DSC Resources 
	Import-DscResource -ModuleName PSDscResources
	Import-DscResource -ModuleName xStorage

	Node localhost
	{
        LocalConfigurationManager
		{
			ActionAfterReboot = 'ContinueConfiguration'
			ConfigurationMode = 'ApplyOnly'
			RebootNodeIfNeeded = $true
		}

        xWaitforDisk DataDisk
        {
            DiskId = $dataDiskId
            RetryIntervalSec = 60
            RetryCount = 3
        }

        xDisk FVolume
        {
            DiskId = $dataDiskId
            DriveLetter = 'F'
            FSLabel = 'Data'
            DependsOn = "[xWaitforDisk]DataDisk"
        }
    }
}

# work out what disk number the data disk is on
$dataDisks = Get-Disk -FriendlyName "Microsoft Virtual Disk" -ErrorAction SilentlyContinue
if ($dataDisk -eq $null) {
	$dataDisk = Get-Disk -FriendlyName "Microsoft Storage Space Device" -ErrorAction SilentlyContinue
}
# filter to GPT partitions
$diskNumber = 2
Get-Disk | Out-Host -Verbose
$dataDisk = Get-Disk | ? { $_.PartitionStyle -eq "RAW" -or $_.PartitionStyle -eq "GPT" }
if ($dataDisk -eq $null) {
	Write-Host "Cannot find any data disks"
} else {
	if ($dataDisk.GetType().Name -eq "Object[]") {
		Write-Host "Multiple data disks"
		$diskNumber = $dataDisk[0].Number
	} else {
		Write-Host "Found single data disk"
		$diskNumber = $dataDisk.Number
	}
}
Write-Host "Using $diskNumber for data disk mounting"

DiskConfig -ConfigurationData .\ConfigurationData.psd1 -dataDiskId "$($diskNumber)"
Start-DscConfiguration -Wait -Force -Path .\DiskConfig -Verbose 

Notes:

• Lines 3-7: We specify that the script requires a dataDiskId parameter
• Lines 11,12: Import the modules we need
• Lines 23-28: Wait for disk with number $dataDiskId to be available (usually it was immediately anyway)
• Lines 30-36: Mount the disk and assign drive letter F with a label of Data
• Lines 41-43: Get potential data disks
• Lines 46-59: Calculate the data disk number, defaulting to 2
• Lines 62,63: Compile the DSC and the invoke the configuration manager to "make it so"

Tip 9: Configuring DNS on Azure VNets

In our example, we needed a Domain Controller to be on one of the machines. We were able to configure the domain controller using DSC. However, I couldn't get the other machines to join the domain since they could never find the controller. Eventually I realized the problem was a DNS problem. So we added the DNS role to the domain controller VM. We also added a private static IP address for the domain controller so that we could configure the VNet accordingly. Here's a snippet of the DSC script for this:

WindowsFeature DNS
{
        Ensure = "Present"
        Name = "DNS"
        DependsOn = "[xADDomain]ADDomain"
}

xDnsServerAddress DnsServerAddress
{
        Address        = '10.10.0.4', '127.0.0.1'
        InterfaceAlias = 'Ethernet 2'
        AddressFamily  = 'IPv4'
        DependsOn = "[WindowsFeature]DNS"
}

Notes:

• Lines 1-6: Configure the DNS feature
• Lines 8-14: Configure the network DNS NIC using the static private IP 10.10.0.4

Now we needed to configure the DNS on the Azure VNet to use the domain controller IP address. We used this script:

param($rgName, $vnetName, $dnsAddress)
$vnet = Get-AzureRmVirtualNetwork -ResourceGroupName $rgName -Name $vnetName
if ($vnet.DhcpOptions.DnsServers[0] -ne $dnsAddress) {
    $vnet.DhcpOptions.DnsServers = @($dnsAddress)
    Set-AzureRmVirtualNetwork -VirtualNetwork $vnet
}

Notes:

• Line 2: Get the VNet using the resource group name and VNet name
• Line 3: Check if the DNS setting of the VNet is correct
• Lines 4,5: If it's not, then set it to the internal IP address of the DNS server

This script needs to run as an Azure PowerShell script task so that it's already logged in to an Azure context (the equivalent of running Login-AzureRMAccount -ServicePrincipal). It's sweet that you don't have to provide any credentials in the script!

Now that we've set the DNS on the VNet, we have to reboot every machine on the VNet (otherwise they won't pick up the change). That brings us to the final tip.

Tip 10: Wait on Machines When They Reboot

You can easily reboot a machine by running this (plain) PowerShell: Restart-Machine -ComputerName localhost -Force. This is so simple that you can do it as an inline PowerShell task:

Rebooting the machine is easy: it's waiting for it to start up again that's more challenging. If you have a task right after the reboot task, the deployment fails since the agent goes offline. So you have to build in a wait. The simplest method is to add an agentless phase and add a Delay task:

However, you can be slightly more intelligent if you poll the machine states using some Azure PowerShell:

param (
    [string]$ResourceGroupName,
    [string[]]$VMNames = @(),
    $TimeoutMinutes = 2,
    $DelaySeconds = 30
)

Write-Host "Delay for $DelaySeconds seconds."
Start-Sleep -Seconds $DelaySeconds

if($VMNames.Count -eq 0)
{
    $VMNames = (Get-AzureRmVm -ResourceGroupName $ResourceGroupName).Name
    Write-Host "Getting VM names."
}

$seconds = 10
$desiredStatus = "PowerState/running"

foreach($vmName in $VMNames)
{
    $timer = [Diagnostics.Stopwatch]::StartNew()
    Write-Host "Getting statuses of VMs."
    $statuses = (Get-AzureRmVm -ResourceGroupName $ResourceGroupName -VMName $vmName -Status).Statuses
    $status = $statuses | Where-Object { $_.Code -eq $desiredStatus }
    while($status -eq $null -and ($timer.Elapsed.TotalMinutes -lt $TimeoutMinutes))
    {
        Write-Verbose "Retrying in $($seconds) seconds."
        Start-Sleep -Seconds $seconds
        $statuses = (Get-AzureRmVm -ResourceGroupName $ResourceGroupName -VMName $vmName -Status).Statuses
        $status = $statuses | Where-Object { $_.Code -eq $desiredStatus }
    }

    if($timer.Elapsed.TotalMinutes -ge $TimeoutMinutes)
    {
        Write-Error "VM restart exceeded timeout."
    }
    else
    {
        Write-Host "VM name $($vmName) has current status of $($status.DisplayStatus)."
    }
}

Notes:

• The script requires a resource group name, an optional array of machine names (otherwise it will poll all the VMs in the resource group), a delay (defaulted to 30 seconds) and a timeout (defaulted to 2 minutes)
• The script will delay for a small period (to give the machines time to start rebooting) and then poll them until they're all running or the timeout is reached.

This script has to run in an Azure PowerShell task in an agent phase from either the Hosted agent or a private agent - you can't run it in a Deployment Group phase since those machines are rebooting!

Conclusion

Deployment Groups are very powerful - they allow you to dynamically target multiple machines and execute configuration in a local context. This makes complex environment provisioning and configuration much easier to manage. However, it's always good to know limitations, gotchas and practical tips when designing a complex deployment workflow. Hopefully these tips and tricks make your life a bit easier.

Happy deploying!

If you've ever had to create a complex ARM template, you'll know it can be a royal pain. You've probably been tempted to split out your giant template into smaller templates that you can link to, only to discover that you can only link to a sub-template if the sub-template is accessible via some public URI. Almost all of the examples in the Template Quickstart repo that have links simply refer to the public Github URI of the linked template. But what if you want to refer to a private repo of templates?

Using Blob Containers

The solution is to use blob containers. You upload the templates to a private container in an Azure Storage Account and then create a SAS token for the container. Then you create the full file URI using the container URI and the SAS token. Sounds simple, right? Fortunately with VSTS Release Management, it actually is easy.

As an example, let's look at this template that is used to create a VNet and some subnets. First we'll look at the VNet template (the linked template) and then how to refer to it from a parent template.

The Child Template

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "vnetName": {
      "type": "string"
    },
    "vnetPrefix": {
      "type": "string"
    },
    "subnets": {
      "type": "object"
    }
  },
  "variables": {
  },
  "resources": [
    {
      "name": "[parameters('vnetName')]",
      "type": "Microsoft.Network/virtualNetworks",
      "location": "[resourceGroup().location]",
      "apiVersion": "2016-03-30",
      "dependsOn": [],
      "tags": {
        "displayName": "vnet"
      },
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[parameters('vnetPrefix')]"
          ]
        }
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/virtualNetworks/subnets",
      "tags": {
        "displayName": "Subnets"
      },
      "copy": {
        "name": "iterator",
        "count": "[length(parameters('subnets').settings)]"
      },
      "name": "[concat(parameters('vnetName'), '/', parameters('subnets').settings[copyIndex()].name)]",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[parameters('vnetName')]"
      ],
      "properties": {
        "addressPrefix": "[parameters('subnets').settings[copyIndex()].prefix]"
      }
    }
  ],
  "outputs": {
  }
}

Notes:

• There are 3 parameters: the VNet name and prefix (strings) and then an object that contains the subnet settings
• The first resource is the VNet itself - nothing complicated there
• The second resource uses copy to create 0 or more instances. In this case, we're looping over the subnets.settings array and creating a subnet for each element in that array, using copyIndex() as the index as we loop

There's really nothing special here - using a copy is slightly more advanced, and the subnets parameter is a complex object. Otherwise, this is plain ol' ARM json.

The Parent Template

The parent template has two things that are different from "normal" templates: it needs two parameters (containerUri and containerSasToken) that let it refer to the linked (child) template and it invokes the template by specifying a "Microsoft.Resources/deployments" resource type. Let's look at an example:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "containerUri": {
      "type": "string"
    },
    "containerSasToken": {
      "type": "string"
    }
  },
  "variables": {},
  "resources": [
    {
      "apiVersion": "2017-05-10",
      "name": "linkedTemplate",
      "type": "Microsoft.Resources/deployments",
      "properties": {
        "mode": "incremental",
        "templateLink": {
          "uri": "[concat(parameters('containerUri'), '/Resources/vNet.json', parameters('containerSasToken'))]",
          "contentVersion": "1.0.0.0"
        },
        "parameters": {
          "vnetName": { "value": "testVNet" },
          "vnetPrefix": { "value": "10.0.0.0/16" },
          "subnets": {
            "value": {
              "settings": [
                {
                  "name": "subnet1",
                  "prefix": "10.0.0.0/24"
                },
                {
                  "name": "subnet2",
                  "prefix": "10.0.1.0/24"
                }
              ]
            }
          }
        }
      }
    }
  ],
  "outputs": {}
}

Notes:

• There are two parameters that pertain to the linked template: the containerUri and the SAS token
• In the resources, there is a "Microsoft.Resources/deployment" resource - this is how we invoke the child template
• In the templateLink, the URI is constructed by concatenating the containerUri, the path to the child template within the container, and the SAS token
• Parameters are passed inline - note that even simple parameters look like JSON objects (see vNetName and vnetPrefix)

Initially I tried to make the subnets object an array: but this blew up on the serialization. So I made an object called "settings" that is an array. So the subnets value property is an object called "settings" that is an array. You can look back at the child template to see how I dereference the object to get the values: to get the name of a subnet, I use "parameters('subnet').settings[index].name" (where index is 0 or 1 or whatever). The copy uses the length() method to get the number of elements in the array and then I can use copyIndex() to get the current index within the copy.

Of course the parent template can contain other resources - I just kept this example really simple to allow us to zoom in on the linking bits.

Source Structure

Here's a look at how I laid out the files in the Azure Resource Group project:

You can see how the vNet.json (the child template) is inside a folder called "Resources". I use that as the relative path when constructing the URI to the child template.

The Release Definition

Now all the hard work is done! To get this into a release, we just create a storage account in Azure (that we can copy the templates to) and we're good to go.

Now create a new release definition. Add the repo containing the templates as a release artifact. Then in your environment, drop two tasks: Azure File Copy and Azure Resource Group Deployment. We configure the Azure File Copy task to copy all our files to the storage account into a container called templates. We also need to give the task two variable names: one for the containerUri and one for the SAS token:

Once this task has executed, the templates will be available in the (private) container with the same folder structure as we have in Visual Studio.

On the next task, we can select the parent template as the template to invoke. We can pass in any parameters that are needed - at the very least, we need the containerUri and SAS token, so we pass in the variables from the previous task using $() notation:

Now we can run the release and voila - we'll have a vNet with two subnets.

Conclusion

Refactoring templates into linked templates is good practice - it's DRY (don't repeat yourself) and can make maintenance of complicated templates a lot easier. Using VSTS Release Management and a storage container, we can quickly, easily and securely make linked templates available and it all just works ™.

Happy deploying!

I recently worked with a customer that had some containers running Python code. The code was written by data scientists and recently a dev had been hired to help the team get some real process in place. As I was helping them with their CI/CD pipeline (which used a Dockerfile to build their image, publish to Azure Container Registry and then spun up the containers in Azure Container Instances), I noted that there were no unit tests. Fortunately the team was receptive to adding tests and I didn't have to have a long discussion proving that they absolutely need to be unit testing.

Python Testing with PyTest

I've done a bit of Python programming, but am by no means an expert. However, after a few minutes of searching I realized there are a ton of good Python frameworks out there. One I came across is PyTest (which is cool since it's very pythonic). This great post (Python Testing 101: PyTest) from Andy Knight even had a Github repo with some code and tests. So when I got back to the hotel, I forked the repo and was able to quickly spin up a build definition in VSTS that runs the tests - with code coverage!

Continuous Testing with VSTS

At a high level, here are the steps that your build has to perform:

• Clone the repo
• Install PyTest and other packages required
• Run PyTest, instructing it to output the results to (JUnit) XML and produce (Cobertura) coverage reports
• Publish the test results
• (Optional) Fix the styling of the HTML coverage reports
• Publish the coverage reports

Why the "fix styles" step? When we create the HTML coverage reports (so that you can see which lines of code are covered and which are not) we publish them to VSTS. However, for security reasons VSTS blocks the css styling when viewing these reports in the Build Summary page. Fortunately if you inline the styles, you get much prettier reports - so we use a node.js package to do this for us.

Fortunately I've already done this and published the VSTS build definition JSON file in my forked repo. Here's how you can import the code, import the CI definition and run it so you can see this in action yourself.

Import the Repo from Github

The source code is in a Github repo - no problem! We'll import into VSTS and then we can mess with it. We can even fork the Github repo, then import it - that way we can sumbit pull requests on Github for changes we make. In this case, I'll just import the repo directly without forking.

Log in to VSTS and navigate to the Code hub. Then click on the repo button in the toolbar and click Import repository.

Enter the URL and click Import.

Now we have the code in VSTS! Remember, this is just Git, so this is just another remote (that happens to be in VSTS).

Import the Build Definition

Now we can import the build definition. First, navigate to the Github repo and clone it or download the PythonTesting-CI.json file. Then open VSTS and navigate to a team project and click on Build and Release (in the Blue toolbar at the top) to navigate to the build and release hub. Click on Builds (in the grey secondary toolbar) and click the "+Import" button.

In the import dialog, browse to the json file you downloaded previously and click Import.

You'll then see the build definition - there are a couple things that need to be fixed, but the steps are all there.

Note how the Agent queue is specifying "Hosted Linux Preview" - yep, this is running on a Linux VM. Now you don't have to do this, since Python will run on Windows, but I like the Linux agent - and it's fast too! Rename the definition if you want to.

Now we'll fix the "Get sources" section. Click on "Get sources" to tell the build where to get the sources from. Make sure the "This account" tile is selected and then set the repository to "python-testing-101" or whatever you named your repo when you imported. You can optionally set Tag sources and other settings.

One more addition: click on the Triggers tab and enable the CI trigger:

Now you can click Save and queue to queue a new build! While it's running, let's look at the tasks.

1. Install Packages: this Bash task uses curl to get the pip install script, then install pip and finally install pytest and pytest-cov packages. If you're using a private agent you may not have to install pip, but pip doesn't come out of the box on the Hosted Linux agent so that's why I install it.
2. Run Tests: invoke python -m pytest (which will run any tests in the current folder or subfolders), passing --junitxml=testresults.xml and the pycov args to create both an XML and HTML report
3. Install fix-styles package: this just runs "npm install" in the root directory to install this node.js package (you can see this if you look at the package.json file)
4. Fix styles: we run the "fix" script from the package.json file, which just invokes the fix-styles.js file to inline the styling into the HTML coverage reports
5. Publish Test Results: we publish the XML file, which is just a JUnit test result XML file. Note how under Control Options, this task is set to run "Even if a previous task has failed, unless the build was cancelled". This ensures that the publish step works even when tests fail (otherwise we won't get test results when tests fail).
6. Publish Code Coverage: This task published the XML (Cobertura) coverage report as well as the (now style-inlined) HTML reports

Really simple! Let's navigate to the build run (click on the build number that you just queued) and - oh dear, the tests failed!

Seems there is a bug in the code. Take a moment to see how great the test result section is - even though there are failing tests. Then click on Tests to see the failing tests:

All 4 failing tests have "subtract" in them - easy to guess that we have a problem in the subtract method! If we click on a test we can also see the stack trace and the failed assertions from the test failure. Click on the "Bug" button above the test to log a bug with tons of detail!

Just look at that bug: with a single button click we have exception details, stack traces and links to the failing build. Sweet!

Now let's fix the bug: click on the Code hub and navigate to example-py-pytest/com/automationpanda/example and click on the calc_func.py file. Yep, there's a problem in the subtract method:

Click on the Edit button and change that pesky + to a much better -. Note, this isn't what you'd usually do - you'd normally create a branch from the Bug, pull the repo, fix the bug and push. Then you'd submit a PR. For the sake of this blog, I'm just fixing the code in the code editor.

Click the Commit button to save the change. In "Work items to link" find the Bug that we created earlier and select it. Then click Commit.

The commit will trigger a new build! Click on Build and you'll see a build is already running. Click on the build number to open the build.

This time it's a success! Click on the build number to see the report - this time, we see all the tests are passing and we have 100% coverage - nice!

If you click on "Code Coverage*" just below the header, you'll see the (prettified) HTML reports. Normally you won't have 100% coverage and you'll want to see which methods have coverage and which don't - you would do so by browsing your files here and noting which lines are covered or not by the color highlighting:

Also note that we can see that this build is related to the Bug (under Associated work items). It's almost like we're professional developers…

Conclusion

Just because there is "Visual Studio" in the name, it doesn't mean that VSTS can't do Python - and do it really, really well! You get detailed test logging, continuous integration, code coverage reports and details - and for very little effort. If you're not testing your Python code - just do it ™ with VSTS!

This week at MVP summit, I showed some of my colleagues a trick that I use to manage identity hell. I have several accounts that I use to access VSTS and the Azure Portal: my own Microsoft Account (MSA), several org accounts and customer org accounts. Sometimes I want to open a release from my 10th Magnitude VSTS account so that I can grab some tasks to put into CustomerX VSTS release. The problem is that if I open the 10M account in a browser, and then open a new browser, I have to sign out of the 10M account and sign in with the CustomerX account and then the windows break… identity hell.

At first I used to open InPrivate or Incognito windows. That gave me the ability to get to 4 different profiles: IE and IE InPrivate, Chrome and Chrome Incognito. But then my incognito windows don't have cached identities or history or anything that I like to have in my browser. Hacky - very hacky.

Solution: Chrome People

About 2 years ago I stumbled onto Chrome People (or Profiles). This really simple "trick" has been fantastic and I almost never open Incognito anymore. In the upper right of the Chrome chrome (ahem) there is a little text that tells you what your current "person" is:

Click that text to open the People hub:

Here you can see that I have 5 People: ColinMSA, 10M, AdminNWC and NWC and another customer profile. To switch profiles, I just click on the name. To add a person, just click "Manage people".

I can easily add a new person from this view - and I can assign an icon to the person.

When you create a new person, Chrome creates a shortcut to that person's browser on the desktop. I end up clicking on that and adding it to my taskbar:

If I want to open up the Azure Portal or VSTS using my MSA, I click the ColinMSA icon and I'm there. If I need to open my customer VSTS or Portal, I just click that icon. Each window is isolated and my identities don't leak. Very neat, very clean. Under the hood, the shortcuts just add a small arg to the Chrome.exe launcher: --profile-directory="Profile 1". The first profile is Default, the second is Profile 1, the third Profile 2 and so on.

Final Thoughts

You can also do something similar in FireFox, but I like Chrome. This simple trick helps me sort out my identity hell and I can quickly switch to different identity contexts without having to sign in and out all the time. For my MSA I sign into my Google account, but I don't do that for the other browsers. All in all it's a great way to manage multiple identities.

Happy browsing!

I've been working on some pretty complicated infrastructure deployment pipelines using my release management tool of choice (of course): VSTS Release Management. In this particular scenario, we're deploying a set of VMs to a region. We then want to deploy exactly the same setup but in a different region. Conceptually, this is like duplicating infrastructure between different datacenters.

Here's what the DEV environment in a release could look like:

If we're duplicating this to 5 regions, we'd need to clone the environment another 4 times. However, that would mean that any updates to any tasks would need to be duplicated over all 5 regions. It's easy to forget to update or to fat-finger a copy - isn't there a better way to maintain sets of tasks? I'm glad you asked…

DRY - Don't Repeat Yourself

DRY (Don't Repeat Yourself) is a common coding practice - any time you find yourself copying code, you should extract it into a function so that you only have to maintain that logic in a single place. We can do the same thing in a release (or build) using Task Groups. Task Groups are like functions that you can call from releases (or builds) from many places - but maintain in a single place. Just like functions, they have parameters that you can set when you "call" them. Click the selector (checkmark icon to the right of each task) to select all the tasks you want to group, right-click and select "Create task group":

A popup asks for the name of the Task Group and bubbles up all the parameters that are used in the tasks within the group. You can update the defaults and descriptions and click Create (helpful hint: make variables for all the values so that the variable becomes the default rather than a hard-coded value - this will make it easier to re-use the Task Group when you clone environments later):

So far, so good:

However, there's a snag: looking at the parameters section, you'll notice that we don't have any parameter for the Azure Service Endpoint. Let's open the tasks and update the value in the dropdown to $(AzureSubscription):

Now you can see that the parameter is bubble up and surfaced as a parameter on the Task Group - it even has the dropdown with the Service Endpoints. Nice!

Consuming the Task Group

Open up the release again. You'll see that you now have a new parameter on the Task Group: the AzureSubscription. We'll select the DEV sub from the dropdown.

Also note how the phase is now a single "task" (which is just a call to the Task Group). Under the hood, when the release is created, Release Management deletes the task group and replaces it with the tasks from the Task Group - so any values that are likely to change or be calculated on the fly should be variables.

Let's now clone the DEV environment to UAT-WESTUS and to UAT-EASTUS.

If we edit the UAT-WESTUS, we can edit the service endpoint (and any other parameters) that we need to for this environment:

Excellent! Now we can update the Task Group in a single place even if we're using it in dozens of environments. Of course you'd need to update the other parameter values to have environment-specific values (Scopes) in the Variables section.

Conclusion

Task Groups are a great way to keep your releases (or builds) DRY - even allowing you to parameterize the Azure Service Endpoint so that you can duplicate infrastructure across different subscriptions or regions in Azure.

Happy deploying!

There are a lot of ALM MVPs that advocate the "One Team Project to Rule Them All" when it comes to Visual Studio Team Services (VSTS) and Team Foundation Server (TFS). I've been recommending it for a long time to any customer I work with. My recommendation was based mostly on experience - I've experienced far too much pain when organizations have multiple Team Projects, or even worse, multiple Team Project Collections.

While on a flight to New Jersey I watched a fantastic talk by Allan Kelley titled Continuous Delivery and Conway's Law. I've heard about Conway's Law before and know that it is applied to systems design. A corollary to Conway's Law, referred to as Inverse Conway Maneuver, is to structure your organization intentionally to promote a desired system architecture. This has a lot of appeal to me with regards to DevOps - since DevOps is not a tool or a product, but a culture: a way of thinking.

With these thoughts in mind, as I was watching Kelley's talk I had an epiphany: you can perform an Inverse Conway Maneuver by the way you structure your VSTS account or TFS install!

What is Conway's Law?

In April 1968, Mel Conway published a paper called "How Do Committees Invent?" The central thesis of this paper is this: "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." In other words, the design of your  organizational and team structures will impose itself on your system designs. At least, if they are out of sync, you will experience friction. The Inverse Conway Maneuver recognizes Conway's Law and makes it intentional: use organizational and team structure to promote desired systems design. For example, distributed teams tend to develop more modular products, while centralized teams tend to develop monoliths.

Historical side-note: Conway's paper was rejected by Harvard Business Review in 1967 since Conway had "failed to prove his thesis". Ironically, a team of MIT and Harvard Business School researchers published a paper in 2015 which found "strong evidence" to support the hypothesis.

How does this apply to VSTS and TFS? I'll explain, but it's awkward having to type "VSTS and TFS". For the remainder of this blog I'll just write VSTS - but the same principles apply to TFS: the VSTS account is like a TFS Team Project Collection. If we equate VSTS account to Team Project Collection, then the rest of the hierarchy (Team Project, Team etc.) is exactly equivalent. In short, when I say VSTS account I also mean TFS Team Project Collection.

One Objective: Deliver Value to End Users

In days gone by, IT was a service center to Business. Today, most organizations are IT companies - irrespective of the industry they operate in. Successful businesses are those that embrace the idea that IT is a business enabler and differentiator, not just a cost center. There should be very little (if any) division between "business" and "IT" - there is one team with one goal: deliver value to customers. Interestingly the definition of DevOps, according to Donovan Brown (Principal DevOps Manager at Microsoft), is "the union of people, process and products to enable continuous delivery of value to our endusers" (emphases mine).

One Objective means everyone is aligned to the overall goal of the business. If you look at two of the Principles behind the Agile Manifesto:

• Business people and developers must work together daily throughout the project.
• The best architectures, requirements, and designs emerge from self-organizing teams.

you'll see a common theme: aligning everyone to the One Objective. The point I'm making is that there needs to be a "one team" culture that permeates the organization. DevOps is cultural before it's about tools and products. But putting the thinking into practice is no easy task. Fortunately, having the correct VSTS structures supports an Inverse Conway Maneuver.

One Team Project

So how do you use VSTS for an Inverse Conway Maneuver? You have a single VSTS account with a single Team Project.

Having all the work in a single Team Project allows you to view work at a "portfolio" (or organizational) level - that is, across the entire organization. This is (currently) impossible to do with multiple VSTS accounts and very difficult with multiple Team Project Collections. Even viewing portfolio level information with  multiple Team Projects can be difficult. Work item queries are scoped to Team Projects by default; widgets, dashboards, builds, releases, package feeds, test plans - these all live at Team Project level. If you have multiple Team Projects you've probably experienced a fragmented view of work across the organization. Interestingly, that's probably not only from a VSTS point of view, but this structure (by Conway's Law) is probably responsible for silos within the organization.

Hence the recommendation for a single "Team Project to Rule Them All." Not only will this allow anyone to see work at a portfolio level, but this allows teams to share source repositories, build definitions, release definitions, reports and package feeds. It's a technical structure that encourages the One Objective.

Teams

I can hear you already: "But I have 500 developers/analysts/testers/DBAs/Ops managers (let's say engineers, shall we?) - how can I possibly organize them under a single team project?" That's where Teams come in. Teams allow organizations to organize work into manageable sets. When you're an engineer and you want to deliver value, you probably only need a general idea of the One Objective, rather than having to know the minutia of every bit of work across the entire organization. Having your team's work in a separate ring-fenced area allows you to focus on what you need day-to-day. You can go up to the portfolio level when you need a wider context - but you probably don't need that every day. Leadership will more likely spend most of their time looking at work at the portfolio level rather than all the way down to the minutia of the team-level work.

So how should you organize your teams? Again, Conway's Law is going to have enormous impact here. Do you have a 3-tier application? Then you might be tempted to create a DBA Team, a Service Team and a UI Team. Perhaps create a Mobile team and a Data Analytics Team too. Surely that's reasonable, right?

The answer, to quote Consultese (the dialect of the consultant) is: It Depends. Perhaps that is the way to go since that is how your application is architected. But that could be boxing you in: horizontally composed teams violate the Agile principle of cross-functional teams. A better approach is to have your teams composed around functional area or module. Where possible, they should be loosely coupled. Again by Conway's Law this will start reflecting in your app architecture - and you'll start seeing your applications become loosely coupled services. Have you ever wondered why micro-services are so popular today? Could it be the Agile movement started to break huge monolithic organizations into small, loosely-coupled, cross-functional and self-organizing teams, and now we're starting to see that reflected in our architecture? Inverse Conway Maneuvers at work.

In short, create Teams around functional areas and use Area Paths to denote ownership of that work. If an Epic/Feature/Story belongs to a team, put it in the Area Path for that team and it appears on their backlogs. Another tip is that your Area Paths should be durable (long-lived) while your work items should not: work items should have a definite start and end date. Don't make an Epic for "Security" since that's not likely to end at a specific date. Rather, have an Area Path for Security and place work items in that area path.

Organizational Dimensions in VSTS

There are four axes that most organizations use to organize work: functional area, iteration, release and team. Unfortunately, VSTS only really gives us two: Area and Iteration. While Release Management in VSTS is brilliant, there isn't yet a first-class citizen for the concept of a Release. And while you can create a custom Team Field in TFS and slice teams on that field, you can't do so in VSTS, so you have to munge Team and Area Path together. In my experience it's best not to fight these limits: use Area Path to denote Team, use iterations to time-box, and if you really need a Release concept, add a custom field.

Areas and Work Item States

Organizations will still need inter-team communication, but this should be happening far less frequently that intra-team communication. That's why we optimize for intra-team communication. It's also why co-locating a team wherever possible is so important. If you do this, then by Conway's Law you are more likely to end up with modules that are stable, resilient, independent and optimized.

We've already established that vertical Teams are tied to Area Paths. Each Team "owns" a root area path, typically with the same name as the Team. This is the area path for the team's backlog. The team can then create sub-areas if they need do (leave this up to the team - they're self-organizing after all). Kanban boards can be customized at the team level, so each team can decide on whatever columns and swim-lanes they want in order to optimize their day-to-day work. Again, leave this up to the team rather than dictating from the organizational level.

Work Item states can't be customized at Team level - only at the Team Project level. If you only have a single Team Project, that means every team inherits the same work item states. This is actually a good thing: a good design paradigm is to have standard communication protocols, and to have services have good contracts or interfaces, without dictating what the internals of the service should look like. This is reflected by the common "language" of work item state, but let's teams decide how to manage work internally via customized Kanban boards. Let Conway's Law work for you!

Iterations

While teams should have independent backlogs and areas, they should synchronize on cadence. That is, it's best to share iterations. This means that teams are independent during a sprint, but co-ordinate at the end of the sprint. This enforces the loose coupling: teams are going to have dependencies and you still want teams to communicate - you just want to streamline that communication. Sharing iterations and synchronizing on that heartbeat is good for the Teams as well as the software they're delivering.

Enterprise Alignment vs Team Autonomy

The VSTS team have a single Team Project Collection for their work. They speak about Enterprise Alignment vs Team Autonomy. I heard a great illustration the other day: the Enterprise is like a tanker - it takes a while to turn. Agile Teams are like canoes - they can turn easily. However, try to get 400 canoes pointed in the same direction! As you work to self-organizing teams, keep them on the One Objective so that they're pointed in the same direction. Again, that's why I like the One Team Project concept: the Team Project is the One Direction, while Teams still get autonomy in their Team Areas for daily work.

Organizing Source Code, Builds, Releases, Test Plans and Feeds

If you have a single Team Project, then you'll have a challenge: all repositories, builds, releases, test plans and package feeds are in a single place. Builds have the concept of Build Folders, so you can organize builds by folders if you need to. However, repos, releases, test plans and feeds don't have folders. That means you'll need a good naming strategy and make use of Favorites to manage the noise. In my opinion this is a small price to pay for the benefits of One Team Project.

Security

Often I come across organizations that want to set up restrictions on who can see what. In general: don't do this! Why do you care if Team A can see Team B's backlog? In fact it should be encouraged! Find out what other teams are working on so that you can better manage dependencies and eliminate double work. Same principle with Source Code: why do you care if Team C and see Team D's repos?

There are of course exceptions: if you have external contractors, you may want to restrict visibility for them. In VSTS, deny overrides allow, so in general, leave permissions as "Not Set" and then explicitly Deny groups when you need to. The Deny should be the exception rather than the rule - if not, you're probably doing something wrong.

Of course you want to make sure you Branch Policies (with Pull Requests) in your source code and approval gates in your Releases. This ensures that the teams are aware of code changes and code deployments. Don't source control secrets - store them in Release Management or Azure Key Vault. And manage by exception: every action in VSTS is logged in the Activity Log, so you can always work out who did what after the fact. Trust your teams!

Conclusion

Don't fight Conway's Law: make it work for you! Slim down to a single VSTS Account with a single Team Project and move all your Teams into that single Team Project. Give Teams the ability to customize their sub-areas, backlogs, boards and so on: this gives a good balance of Enterprise Alignment and Team Autonomy.

Here is a brief summary of how you should structure your VSTS account:

• A single VSTS Account (or TFS Team Project Collection)
• A single Team Project
• Multiple Teams, all owning their root Area Path
• Shared Iteration Paths
• Use naming conventions/favorites for Repos, Releases, Test Plans and Feeds
• Use folders for organizing Build Definitions
• Enforce Branch Policies in your Repos and use Approval Gates in Release Management
• Have simple permissions with minimal DENY (prefer NOT SET and ALLOW)

Happy delivering!

Visual Studio Team Services (VSTS) is a cloud platform. That means it's publicly accessible from anywhere - at least, by default. However, Enterprises that are moving from TFS to VSTS may want to ensure that VSTS is only accessed from a corporate network or some white-list of IPs.

To enable conditional access to VSTS, you'll have to have an Azure Active Directory (AAD) backed VSTS account. The conditional access is configured on AAD, not on VSTS itself, since that's where the authentication is performed. For instructions on how to do this, read this MSDN article.

Audit Access

However, this may be a bit heavy-handed. Perhaps you just want to audit access that isn't from a white-list of IPs, instead of blocking access totally. If you're an administrator, you may have come across the Usage page in VSTS. To get there, navigate to the landing page for your VSTS account, click the gear icon and select Usage:

This page will show you all your access to VSTS. To see the IPs, you have to add the "IP Address" column in the column options:

Nice, but what about other users? To get that you have to use the VSTS REST API.

Dumping an Exception Report with PowerShell

There is an (undocumented) endpoint for accessing user access. It's https://<your account>.visualstudio.com/_apis/Utilization/UsageSummary with a whole string of query parameters. And since it's a REST call, you'll need to be authenticated so you'll have to supply a header with a base-64 encoded Personal Access Token (PAT).

Using PowerShell, you can make a call to the endpoint and then filter results where the IP is not in the white-list of IPs. Fortunately for you, I've made a gist for you, which you can access here. When you call the script, just pass in your account name, your PAT, the start and end date and the white-list of IPs. Any access from an IP not in this list is dumped to a CSV.

Limitations

There are some limitations to the API:

1. You'll need to be a Project Collection or Account admin to make this call (since there's no documentation, I'm guessing here).
2. You can only go back 28 days, so if you need this as an official exception report you'll have to schedule the run.

Conclusion

VSTS knows the client IP for any access. Using the API, you can dump a list of access events that are not from a white-list of IPs.

Happy auditing!

Most development requires working against some kind of database. Some teams choose to use Object Relational Mappers (ORMs) like Entity Framework. I think that should be the preferred method of dealing with databases (especially code-first), but there are times when you just have to work with a database schema.

Recently I had to demo ReadyRoll in VSTS. I have to be honest that I don’t like the paradigm of ReadyRoll – migration-based history seems like a mess compared to model-based history (which is the approach that SSDT takes). That’s a subject for another post (some day) or a discussion over beers. However, there was one thing that I really liked – the ability to preview database changes in a build. The ReadyRoll extension on the VSTS marketplace allows you to do just that.

So I stole the idea and made a task that allows you to see SSDT schema changes from build to build.

Using the Task

Let’s consider the scenario: you have an SSDT project in source control and you’re building the dacpac in a Team Build. What the task does is allow you to see what’s changed from one build to the next. Here’s what you need to do:

1. Install Colin’s ALM Corner Build Tasks Extension from the VSTS Marketplace
2. Edit the build definition and go to Options. Make sure “Allow Scripts to Access OAuth Token” is checked, since the task requires this. (If you forget this, you’ll see 403 errors in the task log).
3. Make sure that the dacpac you want to compare is being published to a build drop.
4. Add a “DacPac Schema Compare” task

That’s it! Here’s what the task looks like:

Enter the following fields:

1. The name of the drop that your dacpac file is going to be published to. The task will look up the last successful build and download the drop in order to get the last dacpac as the source to compare.
2. The name of the dacpac (without the extension). This is typically the name of the SSDT project you’re building.
3. The path to the compiled dacpac for this build – this is the target dacpac path and is typically the bin folder of the SSDT project.

Now run your build. Once the build completes, you’ll see a couple new sections in the Build Summary:

The first section shows the schema changes, while the second shows a SQL-CMD file so you can see what would be generated by SqlPackage.exe.

Now you can preview schema changes of your SSDT projects between builds! As usual, let me know here, on Twitter or on Github if you have issues with the task.

Happy building!

Yes, I’ve been playing with Docker again – no big surprise there. This time I decided to take a look at scaling an application that’s in a Docker container. Scaling and load balancing are concepts you have to get your head around in a microservices architecture!

Another consideration when load balancing is of course shared memory. Redis is a popular mechanism for that (and since we’re talking Docker I should mention that there’s a Docker image for Redis) – but for this POC I decided to keep the code very simple so that I could see what happens on the networking layer. So I created a very simple .NET Core ASP.NET Web API project and added a single MVC page that could show me the name of the host machine. I then looked at a couple of load balancing options and started hacking until I could successfully (and easily) load balance three Docker container instances of the service.

The Code

The code is stupid simple – for this POC I’m interested in configuring the load balancer more than anything, so that’s ok. Here’s the controller that we’ll be hitting:

namespace NginXService.Controllers
{
    public class HomeController : Controller
    {
        // GET: /<controller>/
        public IActionResult Index()
        {
            // platform agnostic call
            ViewData["Hostname"] = Environment.GetEnvironmentVariable("COMPUTERNAME") ??
                Environment.GetEnvironmentVariable("HOSTNAME");

            return View();
        }
    }
}

Getting the hostname is a bit tricky for a cross-platform app, since *nix systems and windows use different environment variables to store the hostname. Hence the ?? code.

Here’s the View:

@{
    <h1>Hello World!</h1>
    <br/>

    <h3>Info</h3>
    <p><b>HostName:</b> @ViewData["Hostname"]</p>
    <p><b>Time:</b> @string.Format("{0:yyyy-MM-dd HH:mm:ss}", DateTime.Now)</p>
}

I had to change the Startup file to add the MVC route. I just changed the app.UseMvc() line in the Configure() method to this:

app.UseMvc(routes =>
{
    routes.MapRoute(
        name: "default",
        template: "{controller=Home}/{action=Index}/{id?}");
});

Finally, here’s the Dockerfile for the container that will be hosting the site:

FROM microsoft/dotnet:1.0.0-core

# Set the Working Directory
WORKDIR /app

# Configure the listening port
ARG APP_PORT=5000
ENV ASPNETCORE_URLS http://*:$APP_PORT
EXPOSE $APP_PORT

# Copy the app
COPY . /app

# Start the app
ENTRYPOINT dotnet NginXService.dll

Pretty simple so far.

Proxy Wars: HAProxy vs nginx

After doing some research it seemed to me that the serious contenders for load balancing Docker containers boiled down to HAProxy and nginx (with corresponding Docker images here and here). In the end I decided to go with nginx for two reasons: firstly, nginx can be used as a reverse proxy, but it can also serve static content, while HAProxy is just a proxy. Secondly, the nginx website is a lot cooler – seemed to me that nginx was more modern than HAProxy (#justsaying). There’s probably as much religious debate about which is better as there is about git rebase vs git merge. Anyway, I picked nginx.

Configuring nginx

I quickly pulled the image for nginx (docker pull nginx) and then set about figuring out how to configure it to load balance three other containers. I used a Docker volume to keep the config outside the container – that way I could tweak the config without having to rebuild the image. Also, since I was hoping to spin up numerous containers, I turned to docker-compose. Let’s first look at the nginx configuration:

worker_processes 1;

events { worker_connections 1024; }

http {

    sendfile on;

    # List of application servers
    upstream app_servers {

        server app1:5000;
        server app2:5000;
        server app3:5000;

    }

    # Configuration for the server
    server {

        # Running port
        listen [::]:5100;
        listen 5100;

        # Proxying the connections
        location / {

            proxy_pass         http://app_servers;
            proxy_redirect     off;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;

        }
    }
}

This is really a bare-bones config for nginx. You can do a lot in the config. This config does a round-robin load balancing, but you can also configure least_connected, provide weighting for each server and more. For the POC, there are a couple of important bits:

• Lines 10-16: this is the list of servers that nginx is going to be load balancing. I’ve used aliases (app1, app2 and app3, all on port 5000) which we’ll configure through docker-compose shortly.
• Lines 22-23: the nginx server itself will listen on port 5100.
• Line 26, 28: we’re passing all traffic on to the configured servers.

I’ve saved this config to a file called nginx.conf and put it into the same folder as the Dockerfile.

Configuring the Cluster

To configure the whole cluster (nginx plus three instances of the app container) I use the following docker-compose yml file:

version: '2'

services:
  app1:
    image: colin/nginxservice:latest
  app2:
    image: colin/nginxservice:latest
  app3:
    image: colin/nginxservice:latest

  nginx:
    image: nginx
    links:
     - app1:app1
     - app2:app2
     - app3:app3
    ports:
     - "5100:5100"
    volumes:
     - ./nginx.conf:/etc/nginx/nginx.conf

That’s 20 lines of code to configure a cluster – pretty sweet! Let’s take a quick look at the file:

• Lines 4-9: Spin up three containers using the image containing the app (that I built separately, since I couldn’t figure out how to build and use the same image multiple times in a docker-compose file).
• Line 12: Spin up a container based on the stock nginx image.
• Lines 13-16: Here’s the interesting bit: we tell docker to create links between the nginx container and the other containers, aliasing them with the same names. Docker creates internal networking (so it’s not exposed publically) between the containers. This is very cool – the nginx container can reference app1, app2 and app3 (as we did in the nginx config file) and docker takes care of figuring out the IP addresses on the internal network.
• Line 18: map port 5100 on the nginx container to an exposed port 5100 on the host (remember we configured nginx to listen on the internal 5100 port).
• Line 20: map the nginx.conf file on the host to /etc/nginx/nginx.conf within the container.

Now we can simply run docker-compose up to run the cluster!

You can see how docker-compose pulls the logs into a single stream and even color-codes them!

The one thing I couldn’t figure out was how to do a docker build on an image and use that image in another container within the docker-compose file. I could just have three build directives, but that felt a bit strange to me since I wanted to supply build args for the image. So I ended up doing the docker build to create the app image and then just using the image in the docker-compose file.

Let’s hit the index page and then refresh a couple times:

You can see in the site (the hostname) as well as in the logs how the containers are round-robining:

Conclusion

Load balancing containers with nginx is fairly easy to accomplish. Of course the app servers don’t need to be running .NET apps – nginx doesn’t really care, since it’s just directing traffic. However, I was pleased that I could get this working so painlessly.

Happy load balancing!

Release Management (RM) is awesome – mostly because it works off the amazing cross platform build engine. Also, now that pricing is announced, we know that it won’t cost an arm and a leg!

When I work with customers to adopt RM, I see two kinds of deployments: repeatable and ad-hoc. RM does a great job at repeatable automation – that is, it is great at doing the same thing over and over. But what about deployments that are different in some way every time? Teams love the traceability that RM provides – not just the automation logs, but also the approvals. It would be great if you could track ad-hoc releases using RM.

The Problem

The problem is that RM doesn’t have a great way to handle deployments that are slightly different every time. Let’s take a very typical example: ad-hoc SQL scripts. Imagine that you routinely perform data manipulation on a production database using SQL scripts. How do you audit what script was run and by whom? “We can use RM!” I hear you cry. Yes you can – but there are some challenges.

Ah-hoc means (in this context) different every time. That means that the script you’re running is bound to change every time you run the release. Also, depending on how dynamic you want to go, even the target servers could change – sometimes you’re executing against server A, sometimes against server B, sometimes both. “Just make the script name or server name a variable that you can change at queue time,” I hear you say. Unfortunately, unlike builds, you can’t specify parameter values at queue time. You could create a release in draft and then edit the variables for that run of the release, but this isn’t a great experience since you’re bound to forget things – and you’ll have to do this every time you start a release.

A Reasonable Solution

I was at a customer who were trying to convert to RM from a home-grown deployment tool. Besides “repeatable” deployments their tool was handling several hundred ad-hoc deployments a month, so they had to decide whether or not to keep the ad-hoc deployments in the home-grown tool or migrate to RM. So I mailed the Champs List – a mailing list direct to other MVPs and the VSTS Product Group in Microsoft (being an MVP has to have some benefits, right?) – and asked them what they do for ad-hoc deployments. It turns out that they use ad-hoc deployments to turn feature switches on and off, and they run their ad-hoc deployments with RM – and while I didn’t get a lot of detail, I did get some ideas.

I see three primary challenges for ad-hoc release definitions:

1. What to execute

• Where does the Release get the script to execute? You could create a network share and put a script in there called “adhoc.sql” and get the release to execute that script every time it runs. Tracking changes is then a challenge – but we’re developers and already know how to track changes, right? Yes: source control. So source control the script – that way every time the release runs, it gets the latest version of the script and runs that. Now you can track what executed and who changed the script. And you can even perform code-review prior to starting the release – bonus!

• Is there an echo here? Well no – it’s just that if the release is executing the same script every time, there’s a danger that it could well – execute the same script. That means you have to either write your script defensively – that is, in such a manner that it is idempotent (has the same result no matter how many times you run it) or you have to keep a record of whether or not the script has been run before, say using a table in a DB or an Azure table or something. I favor idempotent scripts, since I think it’s a good habit to be in when you’re automating stuff anyway – so for SQL that means doing “if this record exists, skip the following steps” kind of logic or using MERGE INTO etc.

• Are you executing against the same server every time or do the targets need to be more dynamic? There are a couple of solutions here – you could have a text doc that has a list of servers, and the release definition reads in the file and then loops, executing the script against the target servers one by one. This is dynamic, but dangerous – what if you put in a server that you don’t mean to? Or you could create an environment per server (if you have a small set of servers this is ok) and then set each environment to manual deployment (i.e. no trigger). Then when you’re ready to execute, you create the release, which just sits there until you explicitly tell it which environment (server) to deploy to.

Recommended Steps

While it’s not trivial to set up an ad-hoc deployment pipeline in RM, I think it’s feasible. Here’s what I’m going to start recommending to my customers:

1. Create a Git repository with a well-defined script (or root script)
2. Create a Release that has a single artifact link – to the Git repo you just set up, on the master branch
3. Create an environment per target server. In that environment, you’re conceptually just executing the root script (this could be more complicated depending on what you do for ad-hoc deployments). All the server credentials etc. are configured here so you don’t have to do them every time. You can also configure approvals if they’re required for ad-hoc scripts. Here’s an example where a release definition is targeting (potentially) ServerA, ServerB and/or ServerC. This is only necessary if you have a fixed set of target servers and you don’t always know which server you’re going to target:
2. Here I’ve got an example of copying a file (the root script, which is in a Git artifact link) to the target server and then executing the script using the WinRM SQL task. These tasks are cloned to each server – of course the server name (and possibly credentials) are different for each environment – but you only have to set this up once.
4. Configure each environment to have a manual trigger (under deployment conditions). This allows you to select which server (or environment) you’re deploying to for each instance of the release:
5. Enable a branch policy on the master branch so that you have to create a Pull Request (PR) to get changes into master. This forces developers to branch the repo, modify the script and then commit and create a PR. At that point you can do code review on the changes before queuing up the release.
6. When you’ve completed code review on the PR, you then create a release. Since all the environments are set to manual trigger, you now can go and manually select which environment you want to deploy to:
2. Here you can see how the status on each environment is “Not Deployed”. You can now use the deploy button to manually select a target. You can of course repeat this if you’re targeting multiple servers for this release.

Conclusion

With a little effort, you can set up an ad-hoc release pipeline. This gives you the advantages of automation (since the steps and credentials etc. are already set up) as well as auditability and accountability (since you can track changes to the scripts as well as any approvals). How do you, dear reader, handle ad-hoc deployments? Sound off in the comments!

Happy releasing!

I’ve posted previously about deploying web applications using Team Build and Release Management (see Config Per Environment vs Tokenization in Release Management and WebDeploy, Configs and Web Release Management). However, reviewing those posts recently at customers I’ve been working with, I’ve realized that these posts are a little outdated, you need pieces of both to form a full picture and the scripts that I wrote for those posts are now encapsulated in Tasks in my marketplace extension. So in this post I’m going to do a complete end-to-end walkthrough of deploying web applications using Team Build and Release Management. I’ll be including handling configs – arguably the hardest part of the whole process.

Overview

Let’s start with an overview of the process. I like to think of three distinct “areas” – source control, build and release. Conceptually you have tokenized config in source control (more on how to do this coming up). Then you have a build that takes in the source control and produces a WebDeploy package – a single tokenized package that is potentially deployable to multiple environments. The build should not have to know about anything environment specific (such as the correct connection string for Staging or Production, for example). Then release takes in the package and (conceptually) performs two steps: inject environment values for the tokens, and then deploy using WebDeploy. Here’s a graphic (which I’ll refer to as the Flow Diagram) of the process:

I’ve got some details in this diagram that we’ll cover shortly, so don’t worry about the details for now. The point is to clearly separate responsibilities (especially for build and release): build compiles source code, runs tests and performs static code analysis and so on. It shouldn’t have to know anything about environments – the output of the build is a single package with “holes” for environment values (tokens). Release will take in this single package and deploy it to each environment, plugging environment values into the holes as it goes. This guarantees that the bits you test in Dev and Staging are the same bits that get deployed to Prod!

Deep Dive: Configuration

Let’s get down into the weeds of configuration. If you’re going to produce a single package from build, then how should you handle config? Typically you have (at least) a connection string that you want to be different for each environment. Beyond that you probably have appSettings as well. If the build shouldn’t know about these values when it’s creating the package, then how do you manage config? Here are a some options:

1. Create a web.config for each environment in source control
   • Pros: All your configs are in source control in their entirety
   • Cons: Lots of duplications – and you have to copy the correct config to the correct environment; requires a re-build to change config values in release management
2. Create a config transform for each environment
   • Pros: Less duplication, and you have all the environment values in source control
   • Cons: Requires a project (or solution) config per environment, which can lead to config bloat; requires that you create a package per environment during build; requires a re-build to change config values in release management
3. Tokenize using a single transform and parameters.xml

• Pros: No duplication; enables a single package that can be deployed to multiple environments; no rebuild required to change config values in release management
• Cons: Environment values aren’t in source control (though they’re in Release Management); learning curve

Furthermore, if you’re targeting Azure, you can use the same techniques as targeting IIS, or you can use Azure Resource Manager (ARM) templates to manage your configuration. This offloads the config management to the template and you assume that the target Azure Web App is correctly configured at the time you run WebDeploy.

Here’s a decision tree to make this a bit easier to digest:

Let’s walk through it:

• If you’re deploying to Azure, and using ARM templates, just make sure that you configure the settings correctly in the template (I won’t cover how to do this in this post)
• If you’re deploying to IIS (or you’re deploying to Azure and don’t have ARM templates or just want to manage config in the same manner as you would for IIS), you should create a single publish profile using right-click->Publish (on the web application) called “Release”. This should target the release configuration and you should tokenize the connection strings in the wizard (details coming up)
• Next, if you have appSettings, you’ll have to create a parameters.xml file (details coming up)
• Commit to source control

For the remainder of this post I’m going to assume that you’re deploying to IIS (or to Azure and handling config outside of an ARM template).

Creating a Publish Profile

So what is this publish profile and why do you need it? The publish profile enables you to:

• provide a single transform (via the Web.release.config) that makes your config release-ready (removing the debug compilation property, for example)
• tokenize the connection strings

To create the profile, right-click your web application project and select “Publish…”. Then do the following:

• Select “Custom” to create a new custom profile. Name this “Release” (you can name this anything, but you’ll need to remember the name for the build later)
• On the Connection page, change the Publish method to “Web Deploy Package”. Type anything you want for the filename and leave the site name empty. Click Next.
• On the Settings page, select the configuration you want to compile. Typically this is Release – remember that the name of the configuration here is how the build will know which transform to apply. If you set this to Release, it will apply Web.Release.config – if you set it to Debug it will apply Web.Debug.Release. Typically you want to specify Release here since you’re aiming to get this site into Prod (otherwise why are you coding at all?) and you probably don’t want debug configuration in Prod!
• You’ll see a textbox for each connection string you have in your Web.config. You can either put a single token in or a tokenized connection string. In the example below, I’ve used a single token (“__AppDbContextStr__”) for the one connection string and a tokenized string (“Server=__DbServer__;Initial Catalog=__DbName__;User Name=__DbUser__;Password=__DbPassword__”) for the other (just so you can see the difference). I’m using double underscore pre- and post-fix for the tokens:
• Now click “Close” (don’t hit publish). When prompted to save the profile, select yes. This creates a Release.pubxml file in the Properties folder (the name of the file is the name of the profile you selected earlier):

Creating a parameters.xml File

The publish profile takes care of the connection strings – but you will have noticed that it doesn’t ask for values for appSettings (or any other configuration) anywhere. In order to tokenize anything in your web.config other than connection strings, you’ll need to create a parameters.xml file (yes, it has to be called that) in the root of your web application project. When the build runs, it will use this file to expose properties for you to tokenize (it doesn’t actually transform the config at build time).

Here’s a concrete example: in my web.config, I have the following snippet:

<appSettings>
  ...
  <add key="Environment" value="debug!" />
</appSettings>

There’s an appSetting key called “Environment” that has the value “debug!”. When I run or debug out of Visual Studio, this is the value that will be used. If I want this value to change on each environment I target for deployment, I need to add a parameters.xml file to the root of my web application with the following xml:

<?xml version="1.0" encoding="utf-8" ?>
<parameters>
  <parameter name="Environment" description="doesn't matter" defaultvalue="__Environment__" tags="">
    <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/appSettings/add[@key='Environment']/@value">
    </parameterentry>
  </parameter>
</parameters>

Lines 3-6 are repeated for each parameter I want to configure. Let’s take a deeper look:

• parameter name (line 3) – by convention it should be the name of the setting you’re tokenizing
• parameter description (line 3) – totally immaterial for this process, but you can use it if you need to. Same with tags.
• parameter defaultvalue (line 3) – this is the token you want injected – notice the double underscore again. Note that this can be a single token or a tokenized string (like the connection strings above)
• parameterentry match (line 4) – this is the xpath to the bit of config you want to replace. In this case, the xpath says in the “configuration” element, find the “appSetting” element, then find the “add” element with the key property = ‘Environment’ and replace the value parameter with the defaultvalue.

Here you can see the parameters.xml file in my project:

To test your transform, right-click and publish the project using the publish profile (for this you may want to specify a proper path for the Filename in the Connection page of the profile). After a successful publish, you’ll see 5 files. The important files are the zip file (where the bits are kept – this is all the binary and content files, no source files) and the SetParameters.xml file:

Opening the SetParameters.xml file, you’ll see the following:

<?xml version="1.0" encoding="utf-8"?>
<parameters>
  <setParameter name="IIS Web Application Name" value="Default Web Site/WebDeployMe_deploy" />
  <setParameter name="Environment" value="__Environment__" />
  <setParameter name="DefaultConnection-Web.config Connection String" value="__AppDbContextStr__" />
  <setParameter name="SomeConnection-Web.config Connection String" value="Server=__DbServer__;Initial Catalog=__DbName__;User Name=__DbUser__;Password=__DbPassword__" />
</parameters>

You’ll see the tokens for the appSetting (Environment, line 4) and the connection strings (lines 5 and 6). Note how the tokens live in the SetParameters.xml file, not in the web.config file! In fact, if you dive into the zip file and view the web.config file, you’ll see this:

<connectionStrings>
  <add name="DefaultConnection" connectionString="$(ReplacableToken_DefaultConnection-Web.config Connection String_0)" providerName="System.Data.SqlClient" />
  <add name="SomeConnection" connectionString="$(ReplacableToken_SomeConnection-Web.config Connection String_0)" providerName="System.Data.SqlClient" />
</connectionStrings>
<appSettings>
  ...
  <add key="Environment" value="debug!" />
</appSettings>

You can see that there are placeholders for the connection strings, but the appSetting is unchanged from what you have in your web.config! As long as your connection strings have placeholders and your appSettings are in the SetParameters.xml file, you’re good to go – don’t worry, WebDeploy will still inject the correct values for your appSettings at deploy time (using the xpath you supplied in the parameters.xml file).

Deep Dive: Build

You’re now ready to create the build definition. There are some additional build tasks which may be relevant – such as creating dacpacs from SQL Server Data Tools (SSDT) projects to manage database schema changes – that are beyond the scope of this post. As for the web application itself, I like to have builds do the following:

• Version assemblies to match the build number (optional, but recommended)
• Run unit tests, code analysis and other build verification tasks
• Create the WebDeploy package

To version the assemblies, you can use the VersionAssemblies task from my build tasks extension in the marketplace. You’ll need the ReplaceTokens task for the release later, so just install the extension even if you’re not versioning. To show the minimum setup required to get the release working, I’m skipping unit tests and code analysis – but this is only for brevity. I highly recommend that unit testing and code analysis become part of every build you have.

Once you’ve created a build definition:

• Click on the General tab and change the build number format to 1.0.0$(rev:.r). This makes the first build have the number 1.0.0.1, the second build 1.0.0.2 etc.
• Add a VersionAssemblies task as the first task. Set the Source Path to the folder that contains the projects you want to version (typically the root folder). Leave the rest defaulted.
• Leave the NuGet restore task as-is (you may need to edit the solution filter if you have multiple solutions in the repo)
• On the VS Build task, edit the MSBuild Arguments parameter to be /p:DeployOnBuild=true /p:PublishProfile=Release /p:PackageLocation=$(build.artifactstagingdirectory)
  • This tells MSBuild to publish the site using the profile called Release (or whatever name you used for the publish profile you created) and place the package in the build artifact staging directory
• Now you should put in all your code analysis and test tasks – I’m omitting them for brevity
• The final task should be to publish the artifact staging directory, which at this time contains the WebDeploy package for your site

Run the build. When complete, the build drop should contain the site zip and SetParameters.xml file (as well as some other files):

You now have a build that is potentially deployable to multiple environments.

Deep Dive: Release

In order for the release to work correctly, you’ll need to install some extensions from the Marketplace. If you’re targeting IIS, you need to install the IIS Web App Deployment Using WinRM Extension. For both IIS and Azure deployments, you’ll need the ReplaceTokens task from my custom build tasks extension.

There are a couple of ways you can push the WebDeploy package to IIS:

• Use the IIS Web App Deployment using WinRM task. This is fairly easy to use, but requires that you copy the zip and SetParameters files to the target server deploying.
• Use the cmd file that gets generated with the zip and SetParameters files to deploy remotely. This requires you to know the cmd parameters and to have the WebDeploy remote agent running on the target server.

I recommend the IIS task generally – unless for some or other reason you don’t want to open up WinRM.

So there’s some configuration required on the target IIS server:

• Install WebDeploy
• Install WebDeploy Remote Agent – for using the cmd. Note: if you install via Web Platform Installer you’ll need to go to Programs and Features and Modify the existing install, since the remote agent isn’t configured when installing WebDeploy via WPI
• Configure WinRM – for the IIS task. You can run “winrm quickconfig” to get the service started. If you need to deploy using certificates, then you’ll have to configure that too (I won’t cover that here)
• Firewall – remember to open ports for WinRM (5895 or 5986 for default WinRM HTTP or HTTPS respectively) or 8172 for the WebDeploy remote agent (again, this is the default port)
• Create a service account that has permissions to copy files and “do stuff” in IIS – I usually recommend that this user account be a local admin on the target server

Once you’ve done that, you can create the Release Definition. Create a new release and specify the build as the primary artifact source. For this example I’m using the IIS task to deploy (and create the site in IIS – this is optional). Here are the tasks you’ll need and their configs:

• Replace Tokens Task
  • Source Path – the path to the folder that contains the SetParameters.xml file within the drop
  • Target File Pattern – set to *.SetParameters.xml
• Windows Machine File Copy Task
  • Copy the drop folder (containing the SetParameters.xml and website zip files) to a temp folder on the target server. Use the credentials of the service account you created earlier. I recommend using variables for all the settings.
• (Optional) WinRM – IIS Web App Management Task
  • Use this task to create (or update) the site in IIS, including the physical path, host name, ports and app pool. If you have an existing site that you don’t want to mess with, then skip this task.
• WinRM – IIS Web App Deployment Task
  • This task takes the (local) path of the site zip file and the SetParameters.xml file and deploys to the target IIS site.
  • You can supply extra WebDeploy args if you like – there are some other interesting switches under the MSDeploy Additional Options section.

Finally, open the Environment variables and supply name/value pairs for the values you want injected. In the example I’ve been using, I’ve got a token for Environment and then I have a tokenized connection string with tokens for server, database, user and password. These are the variables that the Replace Tokens task uses to inject the real environment-specific values into the SetParameters file (in place of the tokens). When WebDeploy runs, it transforms the web.config in the zip file with the values that are in the SetParameters.xml file. Here you can see the variables:

You’ll notice that I also created variables for the Webserver name and admin credentials that the IIS and Copy Files tasks use.

You can of course do other things in the release – like run integration tests or UI tests. Again for brevity I’m skipping those tasks. Also remember to make the agent queue for the environment one that has an agent that can reach the target IIS server for that environment. For example I have an agent queue called “webdeploy” with an agent that can reach my IIS server:

I’m now ready to run the deployment. After creating a release, I can see that the tasks completed successfully! Of course the web.config is correct on the target server too.

Deploying to Azure

As I’ve noted previously if you’re deploying to Azure, you can put all the configuration into the ARM template (see an example here– note how the connection strings and Application Insights appSettings are configured on the web application resource). That means you don’t need the publish profile or parameters.xml file. You’ll follow exactly the same process for the build (just don’t specify the PublishProfile argument). The release is a bit easier too – you first deploy the resource group using the Azure Deployment: Create or Update Resource Group task like so:

You can see how I override the template parameters – that’s how you “inject” environment specific values.

Then you use a Deploy AzureRM Web App task (no need to copy files anywhere) to deploy the web app like so:

I specify the Azure Subscription – this is an Azure ARM service endpoint that I’ve preconfigured – and then the website name and optionally the deployment slot. Here I am deploying to the Dev slot – there are a couple extensions in the marketplace that allow you to swap slots (usually after you’ve smoke-tested the non-prod slot to warm it up and ensure it’s working correctly). This allows you to have zero downtime. The important bit here is the Package or Folder argument – this is where you’ll specify the path to the zip file.

Of course if you don’t have the configuration in an ARM template, then you can just skip the ARM deployment task and run the Deploy AzureRM Web App task. There is a parameter called SetParameters file (my contribution to this task!) that allows you to specify the SetParameters file. You’ll need to do a Replace Tokens task prior to this to make sure that environment specific values are injected.

For a complete walkthrough of deploying a Web App to Azure with an ARM template, look at this hands-on-lab.

Conclusion

Once you understand the pieces involved in building, packaging and deploying web applications, you can fairly easily manage configuration without duplicating yourself – including connection strings, appSettings and any other config – using a publish profile and a parameters.xml file. Then using marketplace extensions, you can build, version, test and package the site. Finally, in Release Management you can inject environment specific values for your tokens and WebDeploy to IIS or to Azure.

Happy deploying!

Last week I posted an end-to-end walkthrough about how to build and deploy web apps using Team Build and Release Management – including config management. The post certainly helps you if you’re on the .NET 4.x Framework – but what about deploying .NET Core apps?

The Build Once Principle

If you’ve ever read any of my blogs you’ll know I’m a proponent of the “build once” principle. That is, your build should be taking source code and (after testing and code analysis etc.) producing a single package that can be deployed to multiple environments. The biggest challenge with a “build once” approach is that it’s non-trivial to manage configuration. If you’re building a single package, how do you deploy it to multiple environments when the configuration is different on those environments? I present a solution in my walkthrough – use a publish profile and a parameters.xml file to tokenize the configuration file during build. Then replace the tokens with environment values at deploy time. I show you how to do that starting with the required source changes, how the build works and finally how to craft your release definition for token replacements and deployment.

AppSettings.json

However, .NET Core apps are a different kettle of fish. There is no web.config file (by default). If you File->New Project and create a .NET Core web app, you’ll get an appsettings.json file. This is the “new” web.config if you will. If you then go to the .NET Core documentation, you’ll see that you can create multiple configuration files using “magic” names like appsettings.dev.json and appsettings.prod.json (these are loaded up during Startup.cs). I understand the appeal of this approach, but to me it feels like having multiple web.config files which you replace at deployment time (like web.dev.config and web.prod.config). I’m not even talking about config transforms – just full config files that you keep in source control and (conceptually) overwrite during deployment. So you’re duplicating code – which is bad juju.

I got to thinking about how to handle configuration for .NET Core apps, and after mulling it over and having a good breakfast chat fellow MVP Scott Addie, I thought about tokenizing the appsettings.json file. If I could figure out a clean way to tokenize the file at build time, then I could use my existing ReplaceTokens task (part of my marketplace extension) during deploy time to fill in environment specific values. Unfortunately there’s no config transform for JSON files, so I decided to create a Tokenizer task that could read in a JSON file and then auto-replace values with tokens (based on the object hierarchy).

Tokenizer Task

To see this in action, I created a new .NET Core Web App in Visual Studio. I then added a custom config section. I ended up with an appsettings.json file that looks as follows:

{
  "ConnectionStrings": {
    "DefaultConnection": "Server=(localdb)\\mssqllocaldb;Database=aspnet-WebApplication1-26e8893e-d7c0-4fc6-8aab-29b59971d622;Trusted_Connection=True;MultipleActiveResultSets=true"
  },
  "Tricky": {
    "Gollum": "Smeagol",
    "Hobbit": "Frodo"
  },
  "Logging": {
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "Debug",
      "System": "Information",
      "Microsoft": "Information"
    }
  }
}

Looking at this config, I can see that I might want to change the ConnectionStrings.DefaultConnection as well as the Tricky.Gollum and Tricky.Hobbit settings (yes, I’m reading the Lord of the Rings – I’ve read it about once a year since I was 11). I may want to change Logging.LogLevel.Default too.

Since the file is JSON, I figured I could create a task that reads the file in and then walks the object hierarchy, replacing values with tokens as it goes. But I realized that you may not want to replace every value in the file, so the task would have to take an explicit include (for only replacing certain values) or exclude list (for replacing all but certain values).

I wanted the appsettings file to look like this once the tokenization had completed:

{
  "ConnectionStrings": {
    "DefaultConnection": "__ConnectionStrings.DefaultConnection__"
  },
  "Tricky": {
    "Gollum": "__Tricky.Gollum__",
    "Hobbit": "__Tricky.Hobbit__"
  },
  "Logging": {
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "__Logging.LogLevel.Default__",
      "System": "Information",
      "Microsoft": "Information"
    }
  }
}

You can see the tokens on the highlighted lines.

After coding for a while on the plane (#RoadWarrior) I was able to create a task for tokenizing a JSON file (perhaps in the future I’ll make more file types available – or I’ll get some Pull Requests!). Having recently added unit tests for my Node tasks, I was able to bang this task out rather quickly.

The Build Definition

With my shiny new Tokenize task, I was ready to see if I could get the app built and deployed. Here’s what my build definition looks like:

The build tasks perform the following operations:

1. Run dotnet with argument “restore” (restores the package dependencies)
2. Tokenize the appsettings.json file
3. At this point I should have Test, Code Annalysis etc. – I’ve omitted these quality tasks for brevity
4. Run dotnet with arguments “publish src/CoreWebDeployMe --configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)/Temp” (I’m publishing the folder that contains my .NET Core web app with the BuildConfiguration and placing the output in the Build.ArtifactStagingDirectory/Temp folder)
5. Zip the published folder (the zip task comes from this extension)
6. Remove the temp folder from the staging directory (since all the files I need are now in the zip)
7. Upload the zip as a build drop

The Tokenize task is configured as follows:

Let’s look at the arguments:

• Source Path – the path containing the file(s) I want to tokenize
• File Pattern – the mini-match pattern for the file(s) within the Source Path I want to tokenize
• Tokenize Type – I only have json for now
• IncludeFields – the list of properties in the json file that I want the Tokenizer to tokenize
• ExcludeFields – I could have used a list of properties I wanted to exclude from tokenization here instead of using the Include Fields property

Once the build completes, I now have a potentially deployable .NET Core web application with a tokenized appsettings file. I could have skipped the zip task and just uploaded the site unzipped, but uploading lots of little files takes longer than uploading a single larger file. Also, I was thinking about the deployment – downloading a single larger file (I guessed) was going to be faster than downloading a bunch of smaller files.

The Release

I was expecting to have to unzip the zip file, replace the tokens in the appsettings.json file and then re-zip the file before invoking WebDeploy to push the zip file to Azure. However, the AzureRM WebDeploy task recently got updated, and I noticed that what used to be “Package File” was now “Package File or Folder”. So the release turned out to be really simple:

1. Unzip the zip file to a temp folder using an inline PowerShell script (why is there no complementary Unzip task from the Trackyon extension?)
2. Run ReplaceTokens on the appsettings.json file in the temp folder
3. Run AzureRM WebDeploy using the temp folder as the source folder

Here’s how I configured the PowerShell task:

The script takes in the sourceFile (the zip file) as well as the target path (which I set to a temp folder in the drop folder):

param(
  $sourceFile,
  $targetPath)

Expand-Archive -Path $sourceFile -DestinationPath $targetPath -Force

My first attempt deployed the site – but the ReplaceTokens task didn’t replace any tokens. After digging a little I figured out why – the default regex pattern – __(\w+)__ – doesn’t work when the token name have periods in them. So I just updated the regex to __(\w+[\.\w+]*)__ (which reads “find double underscore, followed by a word, followed by a period and word repeated 0 or more times, ending with double underscore”.

That got me closer – one more change I had to make was replacing the period with underscore in the variable names on the environment:

Once the ReplaceTokens task was working, the Deploy task was child’s play:

I just made sure that the “Package or Folder” was set to the temp path where I unzipped the zip file in the first task. Of course at this point the appsettings.json now contains real environment-specific values instead of tokens, so WebDeploy can go and do its thing.

Conclusion

It is possible to apply the Build Once principle to .NET Core web applications, with a little help from my friends Tokenizer and ReplaceTokens in the build and release respectively. I think this approach is fairly clean – you get to avoid duplication in source code, build a single package and deploy to multiple environments. Of course my experimentation is available to your for free from the tasks in my marketplace extension! Sound off in the comments if you think this is useful (or horrible)…

Happy releasing!

Recently I received some criticism from a customer. Sometimes I find it difficult to process criticism – I justify or argue or dismiss. Some of that is my personality – I like to be right! Part of that is the fact that I strive for excellence, so when I’m told I missed the mark, it can feel like I’m being told I’m a failure. You, dear reader, probably strive for perfection – but let’s face facts: we’re not perfect. If you’re like me and you have a difficult time receiving criticism, then this post is for you – hopefully you can learn something from how I process.

I’m Not Perfect

This one’s tough. My natural instinct when receiving criticism is to justify. For example, the criticism might be, “You didn’t finish when you said you would.” My inclination is to retort: “Well, you weren’t clear enough on what you wanted,” or something like that. However, the most critical key to successfully processing criticism is to remain teachable– and that means acknowledging that I missed the mark. I have to tell myself not to argue, not to justify. I have to take a step back and see the situation from the other perspective.

I Have Blind Spots

That leads to the second critical principle – I have blind spots. No matter how much I stare in the mirror to gel my hair to perfection, I still can’t see what’s going on with that stubborn crown on the back of my head! Even if I’m prone to introspection and self-improvement, I’m going to miss stuff. About me. If I reject criticism outright, I’ll never get a chance to see into those blind spots. I have to let criticism be a catalyst to stepping back and honestly assessing what I said or did from someone else’s perspective. I can only improve if there’s something to work on – so I have to let criticism surface things that I can work on.

I Am Not Defined By a Moment

This is a big one for me – I can tend to take criticism hard, so it becomes overwhelming. I have to realize that even if I blow it, that moment (or engagement) doesn’t define me. I’m more than this one moment. I may have gotten this thing wrong, but I get a lot of things right too! Remembering previous moments where I got things right helps me process moments when I get things wrong.

I Can’t Always Win

Sometimes, no matter how hard I try, I can’t win. Someone is going to be disappointed in something I did or said. Most of the time I don’t set out to disappoint, but life happens. Expectations aren’t clear, or are just different, or communication fails. Things beyond my control happen. I have admit that I lost a round – as long as I get up and keep on going!

Learning is a Team Sport

Sometimes criticism is deserved. Sometimes it isn’t. And sometimes it’s hard to tell the difference. I make sure I surround myself with people that know and love me – that way, when I’m criticized I have a team I can go to. I like to make my team diverse – my colleagues of course, but also my friends and family. Even if the criticism is work-related, sometimes having a “personal” perspective can help process a “professional” issue. I also make sure I get someone who’s more experienced than me who can mentor me through a situation.

Often criticism has some meat and some bones. Take the meat, spit out the bones. My team helps me to sort the meat from the bones. They help me to keep things in perspective.

Make it Right

Finally, if it’s appropriate to do so, make it right. Sometimes I can take some criticism and just improve, learn and get better. Sometimes I may need to make things right. My team helps me figure out “action items” – things I can do to improve, but also things that I can do to make it right. This doesn’t always apply, but I like to look for things to do or say that will make things right. Although doing this without justifying myself is challenging for me!

Conclusion

Unless you’re particularly reclusive, you’ll get criticized at some point. Learning how to embrace and deal with criticism is an important skill to learn. If you use it as a chance to learn and improve, and surround yourself with people who can coach and encourage you, you can process criticism positively and become better!

Happy learning!

* Image by innoxiuss used under Creative Commons

(Read part 2 here)

I haven’t blogged for a long while – it’s been a busy few months!

One of the things I love about being a DevOps consultant is that I have to be technically proficient – I can’t help teams develop best practices if I don’t know the technology to at least a reasonable depth – but I also get to be a catalyst for change. I love the cultural dynamics of DevOps. After all, as my friend Donovan Brown says, “DevOps is the union of people, processes and tools…”. When you involve people, then you get to watch (or, in my case, influence) culture. And it fascinates me.

I only recently read Continuous Delivery by Jez Humble and David Farley. I was pleased at how much of their insights I’ve been advocating “by instinct” over my years of ALM and DevOps consulting. Reading their book sparked the thoughts that I’ll put into this two-part post.

This part will introduce the thought that DevOps and architecture are symbiotic – good architecture makes for good DevOps, and good DevOps drives good architecture. Ill look at Builds and Source Control in particular. In part 2, I’ll discuss infrastructure as code, database design, automated testing and monitoring and how they relate to DevOps and vice versa.

Tools, tools, tools

Over the past few months, the majority of my work has been to help teams implement Build/Release Pipelines. This seems inevitable to me given the state of DevOps in the market in general – most teams have made (or are in the process of making) a shift to agile, iterative frameworks for delivering their software. As they get faster, they need to release more frequently. And if they’ve got manual builds and deployments, the increasing frequency becomes a frustration because they can’t seem to deploy fast enough (or consistently enough). So teams are starting to want to automate their build/release flows.

It’s natural, therefore, to immediately look for a tool to help automation. And for a little help from your friends at Northwest Cadence to help you do it right!

Of course my tool of choice for build/release pipelines is Visual Studio Team Services (VSTS) or Team Foundation Server (TFS) for a number of reasons:

1. The build agent is cross platform (it’s built on .NET Core, so runs wherever .NET Core runs)
2. The build agent is also the release agent
3. The build agent can run tests 
4. The task-based system has a good Web-based UI, allowing authoring from wherever you have a browser
5. The logging is great – allowing fast debugging of build issues
6. Small custom logic can easily be handled with inline scripts
7. If you can script it, the agent can do it – whether it’s bat, ps1 or sh 
8. Extensions are fairly easy to create
9. There is a large and thriving marketplace for extensions

Good Architecture Means Easier DevOps

Inevitably implementing build automation impacts how you organize your source control. And implementing a release pipeline impacts how you test. And implementing continuous deployment impacts IT, since there’s suddenly a need to be able to spin up and configure and tear down environments on the fly. I love seeing this progression – but it’s often painful for the teams I’m working with. Why? Because teams start realizing that if their architecture was better, it would make other parts of the DevOps pipeline far easier to implement.

For example, if you start automating releases, pretty soon you start wanting to run automated tests since your tests start becoming the bottleneck to delivery. At this point, if you’ve used good architectural principles like interfaces and inversion of control, writing unit tests is far easier. If you haven’t, you have a far harder time writing tests.

Good architecture can make DevOps easier for you and your team. We’ve been told to do these things, and often we’ve found reasons not to do them (“I don’t have time to make an interface for everything!” or “I’ll refactor that class to make it more testable in the next sprint” etc. etc.). Hopefully I can show you how these architectural decisions, if done with DevOps in mind, will not only make your software better but help you to implement better DevOps, more easily!

The Love Triangle: Source Control, Branches and Builds

I really enjoy helping teams implement their first automated builds. Builds are so foundational to good DevOps – and builds tend to force teams to reevaluate their code layout (structure), dependencies and branching strategy.

Most of the time, the teams have their source code in some sort of source control system. Time and time again, the teams that have a good structure and simple branching strategies have a far easier time getting builds to work well.

Unfortunately, most repositories I look at are not very well structured. Or the branches represent environments so you see MAIN/DEV/PROD (which is horrible even though most teams don’t know why – read on if this is you). Or they have checked binaries into source control instead of using a package manager like NuGet. Or they have binary references instead of project references.

Anyway, as we get the build going, we start to uncover potential issues most teams don’t even know they have (like missing library references or conflicting package versions). After some work and restructuring, we manage to get a build to compile. Whoop!

Branching

After the initial elation and once the party dies down, we take a look at the branching strategy. “We need a branch for development, then we promote to QA, and once QA signs off we promote to PROD. So we need branches for each of these environments, right?” This is still a very pervasive mindset. However, DevOps – specifically release pipelines – should operate on a simple principle: build once, deploy many times. In other words, the bits that you deploy to DEV should be the same bits that you deploy to QA and then to PROD. Don’t just take my work for it: read Continuous Delivery – the authors emphasize this over and over again!. You can’t do that if you have to merge and build each time you want to promote code between environments. So how do you track what code is where and promote parallel development?

Builds and Versioning

I advocate for a master/feature branch strategy. That is, you have your stable code on master and then have multiple feature branches (1 to n at any one time) that developers work on. Development is done on the feature branch and then merged into master via Pull Request when it’s ready. At that point, a build is queued which versions the assemblies and tags the source with the version (which is typically the build number).

That’s how you keep track of what code is where – by versions and tags that your build keeps the keys for. That way, you can do hotfixes directly onto master even if you’ve already merged code that is in the pipeline and not yet in production. For example, say you have 1.0.0.6 in prod and you merge some code in for a new feature. The build kicks in and produces version 1.0.0.7 which gets automatically deployed to the DEV environment for integration testing. While that’s going on, you get a bug report from PROD. Oh no! We’ve already merged in code that isn’t yet in PROD, so how do we fix it on master?!?

It’s easy – we know that 1.0.0.6 is in PROD, so we branch the code using tag 1.0.0.6 (which the build tagged in the repo when it ran the 1.0.0.6 build). We then fix the issue in the branch build off of this branch. A new build – 1.0.0.8. We take a quick look at this and fast-track it through until it’s deployed and business can continue. In the meantime, we can abandon the 1.0.0.7 build that’s currently in the deployment pipeline. We merge the hotfix branch back to the master and do a new build – 1.0.0.9 that now has the hotfix as well as the new feature. No sweat.

“Hang on. Pull Requests? Feature branches? That sounds like Git.” Yes, it does. If you’re not on Git, then you’d better have a convincing reason not to be. You can do a lot of this with TFVC, but it’s just harder. So just get to Git. And as a side benefit, you’ll get a far richer code review experience (in the form of Pull Requests) so your quality is likely to improve. And merging is easier. And you can actually cherry-pick. I could go on and on, but there’s enough Git bigots out there that I don’t need to add my voice too. But get to Git. Last word. Just Do It.

Small Repos, Microservices and Package Management

So you’re on Git and you have a master/feature branch branching strategy. But you have multiple components or layers and you need them to live together for compilation, so you put them all into one repo, right? Wrong. You need to separate out your components and services into numerous small repos. Each repo should have Continuous Integration (CI) on it. This change forces teams to start decomposing their monolithic apps into shared libraries and microservices. “Wait – what? I need to get into microservices to get good DevOps?” I hear you yelling. Well, another good DevOps principle is releasing small amounts of change often. And if everything is mashed together in a giant repo, it’s hard to do that. So you need to split up your monoliths into smaller components that can be independently released. Yet again, good architecture (loose coupling, strict service boundaries) promotes good DevOps – or is it DevOps finally motivating you to Do It Right™ like you should have all along?

You’ve gone ahead and split out shared code and components. But now your builds don’t work because your common code (your internal libraries) are suddenly in different repos. Yes, you’re going to need some package management tool. Now, as a side benefit, teams can opt-in to changes in common libraries rather than being forced to update project references. This is a great example of how good DevOps influences good development practices! Even if you just use a file share as a NuGet source (you don’t necessarily need a full-blown package management system) you’re better off.

Conclusion

In this post, we’ve looked at how good source control structure, branching strategies, loosely coupled architectures and package management can make DevOps easier. Or perhaps how DevOps pushes you to improve all of these. As I mentioned, good architecture and DevOps are symbiotic, feeding off each other (for good or bad). So make sure it’s for good! Now go and read part 2 of this post.

Happy architecting!

In part 1 I introduced some thoughts as to how good architecture makes DevOps easier. And how good DevOps drives better architecture – a symbiotic relationship. I discussed how good source control structure, branching strategies, loosely coupled architectures and package management can make DevOps easier. In this post I’ll share some thoughts along the same lines for infrastructure as code, database design and management, monitoring and test automation.

Infrastructure as Code

Let’s say you get your builds under control, you’re versioning, you get your repos sorted and you get package management in place. Now you’re starting to produce lots and lots of builds. Unfortunately, a build doesn’t add value to anyone until it’s in production! So you’ll need to deploy it. But you’ll need infrastructure to deploy to. So you turn to IT and fill in the forms and wait 2 weeks for the servers…

Even if you can quickly spin up tin (or VMs more likely) or you’re deploying to PaaS, you still need to handle configuration. You need to configure your servers (if you’re on IaaS) or your services (on PaaS). You can’t afford to do this manually each time you need infrastructure. You’re going to need to be able to automatically spin up (and down) resources when you need them.

Spinning Up From Scratch

I was recently at a customer that were using AWS VMs for their DevTest infrastructure. We were evaluating if we could replicate their automated processes in Azure. The problem was they hadn’t scripted creation of their environments from scratch– they would manually configure a VM until it was “golden” and then use that as a base image for spinning up instances. Now that we were wanting to change their cloud host, they couldn’t do it easily because someone would have to spin up an Azure VM and manually configure it. If they had rather used the principle of scripting and automating configuration, we could have used the existing scripts to quickly spin up test machines on any platform or host. Sometimes you don’t know you need something until you actually need it – so do the right things early and you’ll be better off in the long run. Get into the habit of configuring via code rather than via UI.

Deploy Whole Components – Always

In Continuous Delivery, Humble and Farley argue that it’s easier to deploy your entire system each time than trying to figure out what the delta is and only deploy that. If you craft your scripts and deployments so that they are idempotent, then this shouldn’t be a problem. Try to prefer declarative scripting (such as PowerShell DSC) over imperative scripting (like pure PowerShell). Not only is it easier to “read” the configuration, but the system can check if a component is in the required state, and if it is, just “no-op”. Make sure your scripts work irrespective of the initial state of the system.

If you change a single class, should you just deploy that assembly? It’s far easier to deploy the entire component (be that a service or a web app) than trying to work out what changed and what need to be deployed. Tools can also help here – web deploy, for example, only deploys files that are different. You build the entire site and it calculates at deployment time what the differences are. Same with SDDT for database schema changes.

Of course, getting all the requirements and settings correct in a script in source control is going to mean that you need to cooperate and team up with the IT guys (and gals). You’re going to need to work together to make sure that you’re both happy with the resulting infrastructure. And that’s good for everyone.

Good Database Design and Management

Where does your logic live?

How does DevOps influence database design and management? I used to work for a company where the dev manager insisted that all our logic be in stored procedures. “If we need to make a change quickly,” he reasoned, “then we don’t need to recompile, we can just update the SP!” Needless to say, our code was virtually untestable, so we just deployed and hoped for the best. And spent a lot of time debugging and fighting fires. It wasn’t pretty.

Stored procedures are really hard to test reliably. And they’re hard to code and debug. So you’re better off leaving your database to store data. Placing logic into component or services lets you test the logic without having to spin up databases with test data – using mocks or fakes or doubles lets you abstract away where the data is stored and test the logic of your apps. And that makes DevOps a lot easier since you can test a lot more during the build phase. And the earlier you find issues (builds are “earlier” than releases) the lest it costs to fix them and the easier it is to fix them.

Managing Schema

What about schema? Even if you don’t have logic in your database in the form of stored procedures, you’re bound to change the schema at some stage. Don’t do it using manual scripts. Start using SQL Server Data Tools (SSDT) for managing your schema. Would you change the code directly on a webserver to implement new features? Of course not – you want to have source control and testing etc. So why don’t we treat databases the same way? Most teams seem happy to “just fix it on the server” and hope they can somehow replicate changes made on the DEV databases to QA and PROD. If that’s you – stop it! Get your schema into an SSDT project and turn off DDL-write permissions so that they only way to change a database schema is to change the project, commit and let the pipeline make the change.

The advantage of this approach is that you get a full history (in source control) of changes made to your schema. Also, sqlpackage calculates the diff at deployment time between your model and the target database and only updates what it needs to to make the database match the model. Idempotent and completely uncaring as to the start state of the database. Which means hassle free deployments.

Automated Testing

I’ve already touched on this topic – using interfaces and inversion of control makes your code testable, since you can easily mock out external dependencies. Each time you have code that interacts with an external system (be it a database or a web API) you should abstract it as an interface. Not only does this uncouple your development pace from the pace of the external system, it allows you to much more easily test your application by mocking/stubbing/faking the dependency. Teams that have well-architected code are more likely to test their code since the code is easier to test! And tested code produces fewer defects, which means more time delivering features rather than fighting fires. Once again, good architecture is going to ease your DevOps!

Once you’ve invested in unit testing, you’ll want to start doing some integration testing. This requires code to be deployed to environments so that it can actually hit the externals systems. If everything is a huge monolithic app, then as tests fail you won’t know why they failed. Smaller components will let you more easily isolate where issues occur, leading to faster mean time to detection (MTTD). And if you set up so that you can deploy components independently (since they’re loosely coupled, right!) then you can recover quickly, leading to faster mean time to recovery (MTTR).

You’ll want to have integration tests that operate “headlessly”. Prefer API calls and HTTP requests over UI tests since UI tests are notoriously hard to create correctly and tend to be fragile. However, if you do get to UI tests, then good architecture can make a big difference here too. Naming controls uniquely means UI test frameworks can find them more easily (and faster) so that UI testing is faster and more reliable. The point surfaces again that DevOps is making you think about how you structure even your UI!

Monitoring

Unfortunately, very few teams that I come across have really good monitoring in place. This is often the “forgotten half-breed” of the DevOps world – most teams get source code right, test right and deploy right – and then wash their hands. “Prod isn’t my responsibility – I’m a dev!” is a common culture. However, good monitoring means that you’re able to more rapidly diagnose issues, which is going to save you time and effort and keep you delivering value (debugging is not delivering value). So you’ll need to think about how to monitor your code, which is going to impact on your architecture.

Logging is just monitoring 1.0. What about utilization? How do you monitor how much resources your code is consuming? And how do you know when to spin up more resources for peak loads? Can you even do that – or do your web services require affinity? Ensuring that your code can run on 1 or 100 servers will make scaling a lot easier.

But beyond logging and performance monitoring, there’s a virtually untapped wealth of what I call “business monitoring” that very few (if any) teams seem to take advantage of. If you’re developing an e-commerce app, how can you monitor what product lines are selling well? And can you correlate user profiles to spending habits? The data is all there – if you can tap into it. Application Insights, coupled with analytics and PowerBI can empower a new level of insight that your business didn’t even know existed. DevOps (which included monitoring) will drive you to architect good “business monitoring” into your apps.

Build and Release Responsibilities

One more nugget that’s been invaluable for successful pipelines: know what builds do and what releases do. Builds should take source code (and packages) as inputs, run quality checks such as code analysis and unit testing, and produce packaged binaries as outputs. These packages should be “environment agnostic” – that is, they should not need to know about environments or be tied to environments. Similarly your builds should not need connections strings or anything like that since the testing that occurs during a build should be unit tests that are fast and have no external dependencies.

This means that you’ll have to have packages that have “holes” in them where environment values can later be injected. Or you may decide to use environment variables altogether and have no configuration files. However you do it, architecting configuration correctly (and fit for purpose, since there can be many correct ways) will make deployment far easier.

Releases need to know about environments. After all, they’re going to be deploying to the environments, or perhaps even spinning them up and configuring them. This is where your integration and functional tests should be running, since some infrastructure is required.

Conclusion

Good architecture makes good DevOps a lot easier to implement – and good DevOps feeds back into improving the architecture of your application as well as your processes. The latest trend of “shift left” means you need to be thinking about more than just solving a problem in code – you need to be thinking beyond just coding. Think about how the code you’re writing is going to be tested. And how it’s going to be configured on different environments. And how it’s going to be deployed. And how you’re going to spin up the infrastructure you need to run it. And how you’re going to monitor it.

The benefits, however, of this “early effort” will pay off many times over in the long run. You’ll be faster, leaner and meaner than ever. Happy DevOps architecting!

The other day I was doing a POC to run some Selenium tests in a Release. I came across some Selenium docker images that I thought would be perfect – you can spin up a Selenium grid (or hub) container and then join as many node containers as you want to (the node container is where the tests will actually run). The really cool thing about the node containers is that the container is configured with a browser (there are images for Chrome and Firefox) meaning you don’t have to install and configure a browser or manually run Selenium to join the grid. Just fire up a couple containers and you’re ready to test!

The source code for this post is on Github.

Here’s a diagram of the components:

The Tests

To code the tests, I use Selenium WebDriver. When it comes to instantiating a driver instance, I use the RemoteWebDriver class and pass in the Selenium Grid hub URL as well as the capabilities that I need for the test (including which browser to use) – see line 3:

private void Test(ICapabilities capabilities)
{
    var driver = new RemoteWebDriver(new Uri(HubUrl), capabilities);
    driver.Navigate().GoToUrl(BaseUrl);
    // other test steps here
}

[TestMethod]
public void HomePage()
{
    Test(DesiredCapabilities.Chrome());
    Test(DesiredCapabilities.Firefox());
}

Line 4 includes a setting that is specific to the test – in this case the first page to navigate to.

When running this test, we need to be able to pass the environment specific values for the HubUrl and BaseUrl into the invocation. That’s where we can use a runsettings file.

Test RunSettings

The runsettings file for this example is simple – it’s just XML and we’re just using the TestRunParameters element to set the properties:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
    <TestRunParameters>
        <Parameter name="BaseUrl" value="http://bing.com" />
        <Parameter name="HubUrl" value="http://localhost:4444/wd/hub" />
    </TestRunParameters>
</RunSettings>

You can of course add other settings to the runsettings file for the other environment specific values you need to run your tests. To test the setting in VS, make sure to go to Test->Test Settings->Select Test Settings File and browse to your runsettings file.

The Build

The build is really simple – in my case I just build the test project. Of course in the real world you’ll be building your application as well as the test assemblies. The key here is to ensure that you upload the test assemblies as well as the runsettings file to the drop (more on what’s in the runsettings file later). The runsettings file can be uploaded using two methods: either copy it using a Copy Files task into the artifact staging directory – or you can mark the file’s properties in the solution to “Copy Always” to ensure it’s copied to the bin folder when you compile. I’ve selected the latter option.

Here’s what the properties for the file look like in VS:

Here’s the build definition:

The Docker Host

If you don’t have a docker host, the fastest way to get one is to spin it up in Azure using the Azure CLI – especially since that will create the certificates to secure the docker connection for you! If you’ve got a docker host already, you can skip this section – but you will need to know where the certs are for your host for later steps.

Here are the steps you need to take to do that (I did this all in my Windows Bash terminal):

1. Install node and npm
2. Install the azure-cli using “npm install –g azure-cli”
3. Run “azure login” and log in to your Azure account
4. Don’t forget to set your subscription if you have more than one
5. Create an Azure Resource Group using “azure group create <name> <location>”
6. Run “azure vm image list –l westus –p Canonical” to get a list of the Ubuntu images. Select the Urn of the image you want to base the VM on and store it – it will be something like “Canonical:UbuntuServer:16.04-LTS:16.04.201702240”. I’ve saved the value into $urn for the next command.
7. Run the azure vm docker create command – something like this:

   azure vm docker create --data-disk-size 22 --vm-size "Standard_d1_v2" --image-urn $urn --admin-username vsts --admin-password $password --nic-name "cd-dockerhost-nic" --vnet-address-prefix "10.2.0.0/24" --vnet-name "cd-dockerhost-vnet" --vnet-subnet-address-prefix "10.2.0.0/24" --vnet-subnet-name "default" --public-ip-domain-name "cd-dockerhost"  --public-ip-name "cd-dockerhost-pip" --public-ip-allocationmethod "dynamic" --name "cd-dockerhost" --resource-group "cd-docker" --storage-account-name "cddockerstore" --location "westus" --os-type "Linux" --docker-cert-cn "cd-dockerhost.westus.cloudapp.azure.com"

Here’s the run from within my bash terminal:

Here’s the result in the Portal:

Once the docker host is created, you’ll be able to log in using the certs that were created. To test it, run the following command:

docker -H $dockerhost --tls info

I’ve included the commands in a fish script here.

The docker-compose.yml

The plan is to run multiple containers – one for the Selenium Grid hub and any number of containers for however many nodes we want to run tests in. We can call docker run for each container, or we can be smart and use docker-compose!

Here’s the docker-compose.yml file:

hub:
  image: selenium/hub
  ports:
    - "4444:4444"
  
chrome-node:
  image: selenium/node-chrome
  links:
    - hub

ff-node:
  image: selenium/node-firefox
  links:
    - hub

The Release

The release requires us to run docker commands to start a Selenium hub and then as many nodes as we need. You can install this extension from the marketplace to get docker tasks that you can use in build/release. Once the docker tasks get the containers running, we can run our tests, passing in the hub URL so that the Selenium tests hit the hub container, which will distribute the tests to the nodes based on the desired capabilities. Once the tests complete, we can optionally stop the containers.

Define the Docker Endpoint

In order to run commands against the docker host from within the release, we’ll need to configure a docker endpoint. Once you’ve installed the docker extension from the marketplace, navigate to your team project and click the gear icon and select Services. Then add a new Docker Host service, entering your certificates:

Docker VSTS Agent

We’re almost ready to create the release – but you need an agent that has the docker client installed so that it can run docker commands! The easiest way to do this – is to run the vsts agent docker image on your docker host. Here’s the command:

docker -H $dockerhost --tls run --env VSTS_ACCOUNT=$vstsAcc --env VSTS_TOKEN=$pat --env VSTS_POOL=docker -it microsoft/vsts-agent

I am connecting this agent to a queue called docker – so I had to create that queue in my VSTS project. I wanted a separate queue because I want to use the docker agent to run the docker commands and then use the hosted agent to run the tests – since the tests need to run on Windows. Of course I could have just created a Windows VM with the agent and the docker bits – that way I could run the release on the single agent.

The Release Definition

Create a new Release Definition and start from the empty template. Set the build to the build that contains your tests so that the tests become an artifact for the release. Conceptually, we want to spin up the Selenium containers for the test, run the tests and then (optionally) stop the containers. You also want to deploy your app, typically before you run your tests – I’ll skip the deployment steps for this post. You can do all three of these phases on a single agent – as long as the agent has docker (and docker-compose) installed and VS 2017 to run tests. Alternatively, you can do what I’m doing and create three separate phases – the docker commands run against a docker-enabled agent (the VSTS docker image that I we just got running) while the tests run off a Windows agent. Here’s what that looks like in a release:

Here are the steps to get the release configured:

1. Create a new Release Definition and rename the release by clicking the pencil icon next to the name
2. Rename “Environment 1” to “Test” or whatever you want to call the environment
3. Add a “Run on agent” phase (click the dropdown next to the “Add Tasks” button)
4. Set the queue for that phase to “docker” (or whatever queue you are using for your docker-enabled agents)
5. In this phase, add a “Docker-compose” task and configure it as follows:
   2. Change the action to “Run service images” (this ends up calling docker-compose up)
   3. Uncheck Build Images and check Run in Background
   4. Set the Docker Host Connection
6. In the next phase, add tasks to deploy your app (I’m skipping these tasks for this post)
7. Add a VSTest task and configure it as follows:
   2. I’m using V2 of the Test Agent task
   3. I update the Test Assemblies filter to find any assembly with UITest in the name
   4. I point the Settings File to the runsettings file
   5. I override the values for the HubUrl and BaseUrl using environment variables
   6. Click the ellipses button on the Test environment and configure the variables, using the name of your docker host for the HubUrl (note also how the port is the port from the docker-compose.yml file):
8. In the third (optional) phase, I use another Docker Compose task to run docker-compose down to shut down the containers
   2. This time set the Action to “Run a Docker Compose command” and enter “down” for the Command
   3. Again use the docker host connection

We can now queue and run the release!

My release is successful and I can see the tests in the Tests tab (don’t forget to change the Outcome filter to Passed – the grid defaults this to Failed):

Some Challenges

Docker-compose SSL failures

I could not get the docker-compose task to work using the VSTS agent docker image. I kept getting certificate errors like this:

SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

I did log an issue on the VSTS Docker Tasks repo, but I’m not sure if this is a bug in the extension or the VSTS docker agent. I was able to replicate this behavior locally by running docker-compose. What I found is that I can run docker-compose successfully if I explicitly pass in the ca.pem, cert.pem and key.pem files as command arguments – but if I specified them using environment variables, docker-compose failes with the SSL error. I was able to run docker commands successfully using the Docker tasks in the release – but that would mean running three commands (assuming I only want three containers) in the pre-test phase and another three in the post-test phase to stop each container. Here’s what that would look like:

You can use the following commands to run the containers and link them (manually doing what the docker-compose.yml file does):

run -d -P --name selenium-hub selenium/hub

run -d --link selenium-hub:hub selenium/node-chrome

run -d --link selenium-hub:hub selenium/node-firefox

To get the run for this post working, I just ran the docker-compose from my local machine (passing in the certs explicitly) and disabled the Docker Compose task in my release.

EDIT (3/9/2017): I figured out the issue I was having: when I created the docker host I wasn’t specifying a CN for the certificates. The default is *, which was causing my SSL issues. When I configured the CN correctly using

--docker-cert-cn” "cd-dockerhost.westus.cloudapp.azure.com", everything worked nicely.

Running Tests in the Hosted Agent

I also could not get the test task to run successfully using the hosted agent – but it did run successfully if I used a private windows agent. This is because at this time VS 2017 is not yet installed on the hosted agent. Running tests from the hosted agent will work just fine once VS 2017 is installed onto it.

Pros and Cons

This technique is quite elegant – but there are pros and cons.

Pros:

• Get lots of Selenium nodes registered to a Selenium hub to enable lots of parallel testing (refer to my previous blog on how to run tests in parallel in a grid)
• No config required – you can run tests on the nodes as-is

Cons:

• Only Chrome and Firefox tests supported, since there are only docker images for these browsers. Technically you could join any node you want to to the hub container if you wanted other browsers, but at that point you may as well configure the hub outside docker anyway.

Conclusion

I really like how easy it is to get a Selenium grid up and running using Docker. This should make testing fast – especially if you’re running tests in parallel. Once again VSTS makes advanced pipelines easy to tame!

Happy testing!

A good DevOps pipeline should utilize the principle of build once, deploy many times. In fact, I’d go so far as to say it’s essential for a good DevOps pipeline. That means that you have to have a way to manage your configuration in such a way that the package coming out of the build process is tokenized somehow so that when you release to different environments you can inject environment-specific values. Easier said that done – until now.

Doing it the Right but Hard Way

Currently I’ve recommended that you use WebDeploy to do this. You define a publish profile to handle connection string and a parameters.xml file to handle any other config you want to tokenize during build. This produces a WebDeploy zip file along with a (now tokenized) SetParameters.xml file. Then you use the ReplaceTokens task from my VSTS build/release task pack extension and inject the environment values into the SetParameters.xml file before invoking WebDeploy. This works, but it’s complicated. You can read a full end to end walkthrough in this post.

Doing it the Easy Way

A recent release to the Azure Web App deploy task in VSTS has just dramatically simplified the process! No need for parameters.xml or publish profiles at all.

Make sure your build is producing a WebDeploy zip file. You can read my end to end post on how to add the build arguments to the VS Build task – but now you don’t have to specify a publish profile. You also don’t need a parameters.xml in the solution. The resulting zip file will deploy (by default) with whatever values you have in the web.config at build time.

Here’s what I recommend:

/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"

You can now just paste that into the build task:

You can see the args (in the new build UI). This tells VS to create the WebDeploy zip and put it into the artifact staging directory. The Publish Artifact Drop task uploads anything that it’s the artifact staging directory (again, by default) – which at the time it runs should be the WebDeploy files.

The Release

Here’s where the magic comes in: drop in an Azure App Service Deploy task. Set it’s version to 3.*(preview). You’ll see a new section called “File Transforms & Variable Substitution Options”. Just enable the “XML Override substitution”.

That’s it! Except for defining the values we want to use for the said substitution. To do this, open the web.config and look at your app setting keys or connection string names. Create a variable that matches the name of the setting and enter a value. In my example, I have Azure B2C so I need a key called “ida:Tenant” so I just created a variable with that name and set the value for the DEV environment. I did the same for the other web.config variables:

Now you can run your release!

Checking the Web.Config Using Kudu

Once the release had completed, I wanted to check if the value had been set. I opened up the web app in the Azure portal, but there were no app settings defined there. I suppose that makes sense – the substitutions are made onto the web.config itself. So I just opened the Kudu console for the web app and cat’ed the web.config by typing “cat Web.config”. I could see that the environment values had been injected!

Conclusion

It’s finally become easy to manage web configs using the VSTS Azure Web App Deploy task. No more publish profiles, parameters.xml files, SetParameters.xml files or token replacement. It’s refreshingly clean and simple. Good job VSTS team!

I did note that there is also the possibility of injecting environment-specific values into a json file – so if you have .NET CORE apps, you can easily inject values at deploy time.

Happy releasing!

I have a build/release task pack in the marketplace. I’ve just added a new task that allows you to add tags to builds or releases in the pipeline, inspired by my friend and fellow MVP Rene van Osnabrugge’s excellent post.

Here are a couple of use cases for this task:

1. You want to trigger releases, but only for builds on a particular branch with a particular tag. This trigger only works if the build is tagged during the build. So you could add a TagBuild task to your build that is only run conditionally (for example for buildreason = Pull Request). Then if the condition is met, the tag is set on the build and the release will trigger in turn, but only for builds that have the tag set.
2. You want to tag a build from a release once a release gets to a certain environment. For example, you can add a TagBuild task and tag the primary build once all the integration tests have passed in the integration environment. That way you can see which builds have passed integration tests simply by querying the tags.

Of course you can use variables for the tags – so you could tag the build with the release(s) that have made it to prod by specifying $(Release.ReleaseNumber) as the tag value.

There are of course a ton of other use cases!

Tag Types

You can see the tag type matrix for the “tag type” (which can be set to Build or Release) in the docs.

Conclusion

Let me know if you have issues or feedback. Otherwise, happy taggin’!

DevOps is a journey that every team should at least have started by now. Most of the engagements I have been on in the last year or so have been in the build/release automation space. There are still several practices that I think teams must invest in to remain competitive – unit testing during builds and integration testing during releases are crucial foundations for more advanced DevOps, which I’ve blogged about (a lot) before. However, Application Performance Monitoring (APM) is also something that I believe is becoming more and more critical to successful DevOps teams. And one application of monitoring is hypothesis driven development.

Hypothesis Driven Development using App Service Slots

There are some prerequisites for hypothesis driven development: you need to have metrics that you can measure (I highly, highly recommend using Application Insights to gather the metrics) and you have to have a hypothesis that you can quickly test. Testing in production is the best way to do this – but how do you manage that?

If you’re deploying to Azure App Services, then it’s pretty simple: create a deployment slot on the Web App that you can deploy the “experimental” version of your code to and divert a small percentage of traffic from the real prod site to the experimental slot. Then monitor your metrics. If you’re happy, swap the slots, instantly promoting the experiment. If it does not work, then you’ve failed fast – and you can back out.

Sounds easy. But how do you do all of that in an automated pipeline? Well, you can already deploy to a slot using VSTS and you can already swap slots using OOB tasks. What’s missing is the ability to route a percentage of traffic to a slot.

Route Traffic Task

To quote Professor Farnsworth, “Good news everyone!” There is now a VSTS task in my extension pack that allows you to configure a percentage of traffic to a slot during your release – the Route Traffic task. To use it, just deploy the new version of the site to a slot and then drop in a Route Traffic task to route a percentage of traffic to the staging site. At this point, you can approve or reject the experiment – in both cases, take the traffic percentage down to 0 to the slot (so that 100% traffic goes to the production slot) ad then if the experiment is successful, swap the slots.

What He Said – In Pictures

To illustrate that, here’s an example. In this release I have DEV and QA environments (details left out for brevity), and then I’ve split prod into Prod-blue, blue-cleanup and Prod-success. There is a post-approval set on Prod-blue. For both success and failure of the experiment, approve the Prod-blue environment. At this stage, blue-cleanup automatically runs, turning the traffic routing to 0 for the experimental slot. Then Prod-success starts, but it has a pre-approval set that you can approve only if the experiment is successful: it swaps the slots.

Here is the entire release in one graphic:

In Prod-blue, the incoming build is deployed to the “blue” slot on the web app:

Next, the Route Traffic task routes a percentage of traffic to the blue slot (in this case, 23%):

If you now open the App Service in the Azure Portal, click on “Testing in Production” to view the traffic routing:

Now it’s time to monitor the two slots to check if the experiment is successful. Once you’ve determined the result, you can approve the Prod-blue environment, which automatically triggers the blue-cleanup environment, which updates the traffic routing to route 0% traffic to the blue slot (effectively removing the traffic route altogether).

Then the Prod-success environment is triggered with a manual pre-deployment configured – reject to end the experiment (if it failed) or approve to execute the swap slot task to make the experimental site production.

Whew! We were able to automate an experiment fairly easily using the Route Traffic task!

Conclusion

Using my new Route Traffic task, you can easily configure traffic routing into your pipeline to conduct true A/B testing. Happy hypothesizing!