I am a huge fan of Aurelia– and that was even when I was working with it in the beta days. I recently had to do some development to display d3 graphs, and needed a simple SPA app. Of course I decided to use Aurelia. During development, I was again blown away by how well thought out Aurelia is – and using some new (to me) tooling, the experience was super. In this post I’ll walk through the tools that I used as well as the build/release pipeline that I set up to host the site in Azure.

Tools

Here are the tools that I used:

1. aurelia-cli to create the project, scaffold and install components, build and run locally
2. VS Code for frontend editing, with a great Aurelia extension
3. Visual Studio 2017 for coding/running the API
4. TypeScript for the Aurelia code
5. Karma (with phantomJS) and Istanbul for frontend testing and coverage
6. .NET Core for the Aurelia host as well as for an API
7. Azure App Services to host the web app
8. VSTS for Git source control, build and release

The Demo App and the Challenges

To walk through the development process, I’m going to create a stupid-simple app. This isn’t a coding walkthrough per se – I want to focus on how to use the tooling to support your development process. However, I’ll demonstrate the challenges as well as the solutions, hopefully showing you how quickly you can get going and do what you do best – code!

The demo app will be an Aurelia app with just a REST call to an API. While it is a simple app, I’ll walk through a number of important development concepts:

1. Creating a new project
2. Configuring VS Code
3. Installing components
4. Building, bundling and running the app locally
5. Handling different configs for different environments
6. Automated build, test and deployment of the app

Creating the DotNet Projects

There are some prerequisites to getting started, so I installed all of these:

• nodejs
• npm
• dotnet core
• aurelia-cli
• VS Code
• VS 2017

Once I had the prereqs installed, I created a new empty folder (actually I cloned an empty Git repo – if you don’t clone a repo, remember to git init). Since I wanted to peg the dotnet version, I created a new file called global.json:

{
  "sdk": {
    "version": "1.0.4"
  }
}

Then I created a new dotnet webapi project to “host” the Aurelia app in a folder called frontend and another dotnet project to be the API in a folder called API:

The commands are:

mkdir frontend
cd frontend
dotnet new webapi
cd ..
mkdir API
cd API
dotnet new webapi

I then opened the API project in Visual Studio. Pressing save prompted me to create a solution file, which I did in the API folder. I also created an empty readme.txt file in the wwwroot folder (I’ll explain why when we get to the build) and changed the Launch URL in the project properties to “api/values”:

When I press F5 to debug, I see this:

Creating the Aurelia Project

I was now ready to create the Aurelia skeleton. The last time I used Aurelia, there was no such thing as the aurelia-cli– so it was a little bumpy getting started. I found using the cli and the project structure it creates for building/bundling made development smooth as butter. So I cd’d back to the frontend folder and ran the aurelia-cli command to create the Aurelia project: au new --here. The “--here” is important because it tells the aurelia-cli to create the project in this directory without creating another subdirectory. A wizard then walked me through some choices: here are my responses:

• Target platform: .NET Core
• Transpiler: TypeScript
• Template: With minimum minification
• CSS Processor: Less
• Unit testing: Yes
• Install dependencies: Yes

That created the Aurelia project for me and installed all of the nodejs packages that Aurelia requires. Once the install completed, I was able to run by typing “au run”:

Whoop! The skeleton is up, so it’s time to commit!

You can find the repo I used for this post here. There are various branches – start is the the start of the project up until now – in other words, the absolute bare skeleton of the project.

Configuring VS Code

Now that I have a project structure, I can start coding. I’ve already got Visual Studio for the API project, which I could use for the frontend editing, but I really like doing nodejs development in VS Code. So I open up the frontend folder in VS Code.

I’ve also installed some VS Code extensions:

1. VSCode Great Icons– makes the icons in the file explorer purdy (don’t forget to configure your preferences after you install the extension!)
2. TSLint– lints my TypeScript as I code
3. aurelia– palette commands and html intellisense

Configuring TSLint

There is already an empty tslint.json file in the root of the frontend project. Once you’ve installed the VS Code TSLint extension, you’ll see lint warnings in the status bar: though you have to first configure which rules you want to run. I usually start by extending the tslint:latest rules. Edit the tslint.json file to look like this:

{
  "extends": ["tslint:latest"],
  "rules": {
    
  }
}

Now you’ll see some warnings and green squigglies in the code:

I don’t care about the type of quotation marks (single or double) and I don’t care about alphabetically ordering my imports, so I override those rules:

{
  "extends": ["tslint:latest"],
  "rules": {
    "ordered-imports": [
      false
    ],
    "quotemark": [
      false
    ]
  }
}

Of course you can put whatever ruleset you want into this file – but making a coding standard for your team that’s enforced by a tool rather than in a wiki or word doc is a great practice! A helpful tip is that if you edit the json file in VS Code you get intellisense for the rules – and you can see the name of the rule in the warnings window.

Installing Components

Now we can use the aurelia cli (au) to install components. For example, I want to do some REST calls, so I want to install the fetch-client:

au install aurelia-fetch-client whatwg-fetch

This not only adds the package, but amends the aurelia.json manifest file (in the aurelia_project folder) so that the aurelia-fetch-client is bundled when the app is “compiled”. I also recommend installing whatwg-fetch which is a fetch polyfill. Let’s create a new class which is a wrapper for the fetch client:

import { autoinject } from 'aurelia-framework';
import { HttpClient } from 'aurelia-fetch-client';

const baseUrl = "http://localhost:1360/api";

@autoinject
export class ApiWrapper {
    public message = 'Hello World!';
    public values: string[];

    constructor(public client: HttpClient) {
		client.configure(config => {
			config
				.withBaseUrl(baseUrl)
				.withDefaults({
					headers: {
						Accept: 'application/json',
					},
				});
		});
	}
}

Note that (for now) we’re hard-coding the baseUrl. We’ll address config shortly.

We can now import in the ApiWrapper (via injection) and call the values method:

import { autoinject } from 'aurelia-framework';
import { ApiWrapper } from './api';

@autoinject
export class App {
  public message = 'Hello World!';
  public values: string[];

  constructor(public api: ApiWrapper) {
    this.initValues();
  }

  private async initValues() {
    try {
      this.values = await this.api.client.fetch("/values")
        .then((res) => res.json());
    } catch (ex) {
      console.error(ex);
    }
  }
}

Here’s the updated html for the app.html page:

Nothing too fancy – but shown here to be complete. I’m not going to make a full app here, since that’s not the goal of this post.

Finally, we need to enable CORS on the Web API (since it does not allow CORS by default). Add the Microsoft.AspNet.Cors package to the API project and then add the services.AddCors() and app.UseCors() lines (see this snippet):

public void ConfigureServices(IServiceCollection services)
{
  // Add framework services.
  services.AddMvc();
	services.AddCors();
}

// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
  loggerFactory.AddConsole(Configuration.GetSection("Logging"));
  loggerFactory.AddDebug();

  app.UseCors(p => p.AllowAnyOrigin().AllowAnyMethod());
  app.UseMvc();
}

Now we can get this when we run the project (using “au run”):

If you’re following along in the repo code, the changes are on the “Step1-AddFetch” branch.

Running Locally

Running locally is trivial. I end up with Visual Studio open and pressing F5 to run the backend API project – the frontend project is just as trivial. In VSCode, with the frontend folder open, just hit ctrl-shift-p to bring up the command palette and then type/select “au run --watch” to launch the frontend “build”. This transpiles the TypeScript to JavaScript, compiles Less (or SASS) to css, bundles and minifies all your html, compiled css and JavaScript into a single app-bundle.js in wwwroot\scripts. It also minifies and bundles Aurelia and its dependencies into vendor-bundle.js, using the settings from the aurelia.json file. It’s a lot of work, but Aurelia takes care of it all for you – just run “au run” to do all that stuff and launch a server. If you add the --watch parameter, the process watches your source files (html, Less, TypeScript) and automatically recompiles everything and refreshes the browser automagically using browsersync. It’s as smooth as butter!

Config Management

Attempt 1 – Using environment.ts

Let’s fix up the hard-coded base URL for the api class. Aurelia does have the concept of “environments” – you can see that by looking in the src\environment.ts file. You would be tempted to change the values of that file, but you’ll see that if you do, the contents get overwritten each time Aurelia compiles. Instead, open up the aurelia-project\environments folder, where you’ll see three environment files – dev, stage and prod.ts. To change environment, just enter “au run --env dev” to get the dev environment or “au run --env prod” to get the prod environment. (Unfortunately you can’t change the environment using VSCode command palette, so you have to run the run command from a console or from the VSCode terminal).

Let’s edit the environments to put the api base URL there instead of hard-coding it:

export default {
  apiBaseUrl: "http://localhost:64705/api",
  debug: true,
  testing: true,
};

Of course we add the apiBaseUrl property to the stage and prod files too!

With that change, we can simply import the environment and use the value of the property in the api.ts file:

import { autoinject } from 'aurelia-framework';
import { HttpClient } from 'aurelia-fetch-client';
import environment from './environment';

@autoinject
export class ApiWrapper {
    public message = 'Hello World!';
    public values: string[];

    constructor(public client: HttpClient) {
        client.configure(config => {
            config
                .withBaseUrl(environment.apiBaseUrl)
                .withDefaults({
                    headers: {
                        Accept: 'application/json',
                    },
                });
        });
    }
}

The important changes are on line 2 (reading in the environment settings) and line 13 (using the value). Now we can run for different environments. If you’re following along in the repo code, the changes are on the “Step2-EnvTsConfig” branch.

Attempt 2 – Using a Json File

There’s a problem with the above approach though – if we have secrets (like access tokens or keys) then we don’t want them checked into source control. Also, when we get to build/release, we want the same build to go to multiple environments – using environment.ts means we have to build once for each environment and then select the correct package for the corresponding environment – it’s nasty. Rather, we want to be able to configure the environment settings during a release. This puts secret information in the release tool instead of source control, which is much better, and allows a single build to be deployed to any number of environments.

Unfortunately, it’s not quite so simple (at first glance). The environment.ts file is bundled into app-bundle.js, so there’s no way to inject values at deploy time, unless you want to monkey with the bundle itself. It would be much better to take a leaf out of the .NET CORE playbook and set up a Json config file. Fortunately, there’s an Aurelia plugin that allows you to do just that! Conveniently, it’s called aurelia-configuration.

Run “au install aurelia-configuration” to install the module.

Now (by convention) the config module looks for a file called “config\config.json”. So in the src folder, add a new folder called config and add a new file into the config folder called config.json:

{
      "api": {
            "baseUri": "http://localhost:12487/api"
      }
}

We can then inject the AureliaConfiguration class into our classes and call the get() method to retrieve a variable value. Let’s change the api.ts file again:

import { autoinject } from 'aurelia-framework';
import { HttpClient } from 'aurelia-fetch-client';
import { AureliaConfiguration } from 'aurelia-configuration';

@autoinject
export class ApiWrapper {
    public message = 'Hello World!';
    public values: string[];

    constructor(public client: HttpClient, private aureliaConfig: AureliaConfiguration) {
        client.configure(config => {
            config
                .withBaseUrl(aureliaConfig.get("api.baseUri"))
                .withDefaults({
                    headers: {
                        Accept: 'application/json',
                    },
                });
        });
    }
}

Line 3 has us importing the type, line 10 has the constructor arg for the autoinjection and we get the value on line 13.

We also have to tell Aurelia to use the config plugin. Open main.ts and add the plugin code (line 8 below):

import {Aurelia} from 'aurelia-framework';
import environment from './environment';

export function configure(aurelia: Aurelia) {
  aurelia.use
    .standardConfiguration()
    .feature('resources')
    .plugin('aurelia-configuration');
  ...

There’s one more piece to this puzzle: the config.json file doesn’t get handled anywhere, so running the program won’t work. We need to tell the Aurelia bundler that it needs to add in the config.json file and publish it to the wwwroot folder. To do that, we can add in a copyFiles target onto the aurelia.json settings file:

{
  "name": "frontend",
  "type": "project:application",
  "platform": {
    ...
  },
  ...
  "build": {
    "targets": [
     ...
    ],
    "loader": {
      ...
    },
    "options": {
      ...
    },
    "bundles": [
      ...
    ],
    "copyFiles": {
      "src/config/*.json": "wwwroot/config"
    }
  }
}

At the bottom of the file, just after the build.bundles settings, we add the copyFiles target. The config.json file is now copied to the wwwroot/config folder when we build, ready to be read at run time! If you’re following along in the repo code, the changes are on the “Step3-JsonConfig” branch.

Testing

Authoring the Tests

Of course the API project would require tests – but doing .NET testing is fairly simple and there’s a ton of guidance on how to do that. I was more interested in testing the frontend (Aurelia) code with coverage results.

When I created the frontend project, Aurelia created a test stub project. If you open the test folder, there’s a simple test spec in unit\app.spec.ts:

import {App} from '../../src/app';

describe('the app', () => {
  it('says hello', () => {
    expect(new App().message).toBe('Hello World!');
  });
});

We’ve changed the App class, so this code won’t compile correctly. Now we need to pass an ApiWrapper to the App constructor. And if we want to construct an ApiWrapper, we need an AureliaConfiguration instance as well as an HttpClient instance. We’re going to want to mock the API calls that the frontend makes, so let’s stub out a mock implementation of HttpClient. I add a new class in src\test\unit\utils\mock-fetch.ts:

import { HttpClient } from 'aurelia-fetch-client';

export class HttpClientMock extends HttpClient {
}

We’ll flesh this class out shortly. For now, it’s enough to get an instance of HttpClient for the ApiWrapper constructor. What about the AureliaConfiguration instance? Fortunately, we can create (and even configure) one really easily:

let aureliaConfig = new AureliaConfiguration();
aureliaConfig.set("api.baseUri", "http://test");

We add the “api.BaseUri” key since that’s the value that the ApiWrapper reads from the configuration object. We can now flesh out the remainder of our test:

import {App} from '../../src/app';
import {ApiWrapper} from '../../src/api';
import {HttpClientMock} from './utils/mock-fetch';
import {AureliaConfiguration} from 'aurelia-configuration';

describe('the app', () => {
  it('says hello', async done => {
    // arrange
    let aureliaConfig = new AureliaConfiguration();
    aureliaConfig.set("api.baseUri", "http://test");

    const client = new HttpClientMock();
    client.setup({
      data: ["testValue1", "testValue2", "testValue3"],
      headers: {
        'Content-Type': "application/json",
      },
      url: "/values",
    });
    const api = new ApiWrapper(client, aureliaConfig);

    // act
    let sut: App;
    try {
      sut = new App(api);
    } catch (e) {
      console.error(e);
    }

    // assert
    setTimeout(() => {
      expect(sut.message).toBe('Hello World!');
      expect(sut.values.length).toBe(3);
      expect(sut.values).toContain("testValue1");
      expect(sut.values).toContain("testValue2");
      expect(sut.values).toContain("testValue3");
      done();
    }, 10);
  });
});

Notes:

• Lines 13-19: configure the mock fetch response (we’ll see the rest of the mock HttpClient class shortly)
• Line 20: instantiate a new ApiWrapper
• Lines 23-28: call the App constructor
• Lines 31-38: we wrap the asserts in a timeout since the App constructor calls an async method (perhaps there’s a better way to do this?)

Let’s finish off the test code by looking at the mock-fetch class:

import { HttpClient } from 'aurelia-fetch-client';
export class HttpClientMock extends HttpClient {
}

export interface IMethodConfig {
    url: string;
    method?: string;
    status?: number;
    statusText?: string;
    headers?: {};
    data?: {};
};

export class HttpClientMock extends HttpClient {
    private config: IMethodConfig[] = [];

    public setup(config: IMethodConfig) {
        this.config.push(config);
    }

    public async fetch(input: Request | string, init?: RequestInit) {
        let url: string;
        if (typeof input === "string") {
            url = input;
        } else {
            url = input.url;
        }

        // find the matching setup method
        let methodConfig: IMethodConfig;
        methodConfig = this.config.find(c => c.url === url);
        if (!methodConfig) {
            console.error(`---MockFetch: No such method setup: ${url}`);
            return Promise.reject(new Response(null,
                {
                    status: 404,
                    statusText: `---MockFetch: No such method setup: ${url}`,
                }));
        }

        // set up headers
        let responseInit: ResponseInit = {
            headers: methodConfig.headers || {},
            status: methodConfig.status || 200,
            statusText: methodConfig.statusText || "",
        };

        // get a unified request object
        let request: Request;
        if (Request.prototype.isPrototypeOf(input)) {
            request = ( input);
        } else {
            request = new Request(input, responseInit || {});
        }

        // create a response object
        let response: Response;
        const data = JSON.stringify(methodConfig.data);
        response = new Response(data, responseInit);

        // resolve or reject accordingly
        return response.status >= 200 && response.status < 300 ?
            Promise.resolve(response) : Promise.reject(response);
    }
}

I won’t go through the whole class, but essentially you configure a mapping of routes to responses so that when the mock object is called it can return predictable data.

With those changes in place, we can run the tests using “au test”. This launches Chrome and runs the test. The Aurelia project did the heavy lifting to configure paths for the test runner (Karma) so that the tests “just work”.

Going Headless and Adding Reports and Coverage

Now that we can run the tests in Chrome with results splashed to the console, we should consider how these tests would run in a build. Firstly, we want to produce a report file of some sort so that the build can save the results. We also want to add coverage. Finally, we want to run headless so that we can run this on an agent that doesn’t need access to a desktop to launch a browser!

We’ll need to add some development-time node packages to accomplish these changes:

yarn add karma-phantomjs-launcher karma-coverage karma-tfs gulp-replace --dev

With those package in place, we can change the karma.conf.js file to use phantomjs (a headless browser) instead of Chrome. We’re also going to add in the test result reporter, coverage reporter and a coverage remapper. The coverage will report coverage on the JavaScript files, but we would ideally want coverage on the TypeScript files – that’s what the coverage remapper will do for us.

Here’s the new karma.conf.js:

'use strict';
const path = require('path');
const project = require('./aurelia_project/aurelia.json');
const tsconfig = require('./tsconfig.json');

let testSrc = [
  { pattern: project.unitTestRunner.source, included: false },
  'test/aurelia-karma.js'
];

let output = project.platform.output;
let appSrc = project.build.bundles.map(x => path.join(output, x.name));
let entryIndex = appSrc.indexOf(path.join(output, project.build.loader.configTarget));
let entryBundle = appSrc.splice(entryIndex, 1)[0];
let files = [entryBundle].concat(testSrc).concat(appSrc);

module.exports = function(config) {
  config.set({
    basePath: '',
    frameworks: [project.testFramework.id],
    files: files,
    exclude: [],
    preprocessors: {
      [project.unitTestRunner.source]: [project.transpiler.id],
      'wwwroot/scripts/app-bundle.js': ['coverage']
    },
    typescriptPreprocessor: {
      typescript: require('typescript'),
      options: tsconfig.compilerOptions
    },
    reporters: ['progress', 'tfs', 'coverage', 'karma-remap-istanbul'],
    port: 9876,
    colors: true,
    logLevel: config.LOG_INFO,
    autoWatch: true,
    browsers: ['PhantomJS'],
    singleRun: false,
    // client.args must be a array of string.
    // Leave 'aurelia-root', project.paths.root in this order so we can find
    // the root of the aurelia project.
    client: {
      args: ['aurelia-root', project.paths.root]
    },

    phantomjsLauncher: {
      // Have phantomjs exit if a ResourceError is encountered (useful if karma exits without killing phantom)
      exitOnResourceError: true
    },

    coverageReporter: {
      dir: 'reports',
      reporters: [
        { type: 'json', subdir: 'coverage', file: 'coverage-final.json' },
      ]
    },

    remapIstanbulReporter: {
      src: 'reports/coverage/coverage-final.json',
      reports: {
        cobertura: 'reports/coverage/cobertura.xml',
        html: 'reports/coverage/html'
      }
    }
  });
};

Notes:

• Line 25: add a preprocessor to instrument the code that we’re going to execute
• Line 31: we add reporters to produce results files (tfs), coverage and remapping
• Lines 45-48: we configure a catch-all to close phantomjs if something fails
• Lines 50-55: we configure the coverage to output a Json coverage file
• Lines 57-63: we configure the remapper so that we get TypeScript coverage results

One gotcha I had that I couldn’t find a work-around for: the html files that are generated showing which lines of code were hit is generated with incorrect relative paths and the src folder (with detailed coverage) generated outside the html report folder. Eventually, I decided that a simple replace and file move was all I needed, so I modified the test.ts task in the aurelia-project\tasks folder:

// hack to fix the relative paths in the generated mapped html report
let fixPaths = done => {
  let repRoot = path.join(__dirname, '../../reports/');
  let repPaths = [
    path.join(repRoot, 'src/**/*.html'),
    path.join(repRoot, 'src/*.html'),
  ];
  return gulp.src(repPaths, { base: repRoot })
        .pipe(replace(/(..\/..\/..\/)(\w)/gi, '../coverage/html/$2'))
        .pipe(gulp.dest(path.join(repRoot)));
};

let unit;

if (CLIOptions.hasFlag('watch')) {
  unit = gulp.series(
    build,
    gulp.parallel(
      watch(build, onChange),
      karma,
      fixPaths
    )
  );
} else {
  unit = gulp.series(
    build,
    karma,
    fixPaths
  );
}

I add new tasks called “updateIndex” and “copySrc” that fix up the paths for me. Perhaps there’s a config setting for the remapper that will render this obsolete, but this was the best I could come up with.

Now when you run “au test” you get a result file and coverage results for the TypeScript code all in the html folder with the correct paths. If you’re following along in the repo code, these changes are on the master branch (this is the final state of the demo code).

Automated Build and Test

We now have all the pieces in place to do a build. The build is fairly straightforward once you work out how to invoke the Arelia cli. Starting with a .NET Core Web App template, here is the definition I ended up with:

Here are the task settings:

1. .NET Core Restore – use defaults
2. .NET Core Build
1. Change “Arguments” to --configuration $(BuildConfiguration) --version-suffix $(Build.BuildNumber)
2. The extra bit added is the version-suffix arg which produces binaries with the same version as the build number
3. npm install
1. Change “working folder” to frontend (this is the directory of the Aurelia project).\node_modules\aurelia-cli\bin\aurelia-cli.js test
4. Run command
1. Set “Tool” to node
2. Set “Arguments” to .\node_modules\aurelia-cli\bin\aurelia-cli.js test
3. Expand “Advanced” and set “Working folder” to frontend
4. This runs the tests and produces the test results and coverage results files
5. Run command
1. Set “Tool” to node
2. Set “Arguments” to .\node_modules\aurelia-cli\bin\aurelia-cli.js build --env prod
3. Expand “Advanced” and set “Working folder” to frontend
4. This does transpilation, minification and bundling so that we’re ready to deploy
6. Publish Test Results
1. Set “Test Result Format” to VSTest
2. Set “Test results files” to frontend/testresults/TEST*.xml
3. Set “Test run title” to Aurelia
7. Publish code coverage Results
1. Set “Code Coverage Tool” to Cobertura
2. Set “Summary File” to $(Build.SourcesDirectory)/frontend/reports/coverage/cobertura.xml
3. Set “Report Directory” to $(System.DefaultWorkingDirectory)/frontend/reports/coverage/html
8. .NET Core Publish
1. Make sure “Publish Web Projects” is checked – this is why I added a dummy readme file into the wwwroot folder of the API app, otherwise it’s not published as a web project
2. Set “Arguments” to --configuration $(BuildConfiguration) --output $(build.artifactstagingdirectory) --version-suffix $(Build.BuildNumber)
3. Make sure “Zip Published Projects” is checked
9. On the Options Tab
1. Set the build number format to 1.0.0$(rev:.r) to give the build number a 1.0.0.x format
2. Set the default agent queue to Hosted VS2017 (or you can select a private build agent with VS 2017 installed)

Now when I run the build, I get test and coverage results in the summary:

The coverage files are there if you click the Code Coverage results tab, but there’s a problem with the css.

The <link> elements are stripped out of the html pages when the iFrame for the coverage results shows – I’m working with the product team to find a workaround for this. If you download the results from the Summary page and unzip them, you get the correct rendering.

I can also see both web projects ready for deployment in the Artifacts tab:

We’re ready for a release!

The Release Definition

I won’t put the whole release to Azure here – the key point to remember is the configuration. We’ve done the work to move the configuration into the config.json file for this very reason.

Once you’ve set up an Azure endpoint, you can add in an “Azure App Services Deploy” task. Select the subscription and app service and then change the “Package or folder” from “$(System.DefaultWorkingDirectory)/**/*.zip” to “$(System.DefaultWorkingDirectory)/drop/frontend.zip” (or API.zip) to deploy the corresponding site. To handle the configuration, you simply add “wwwroot/config/config.json” to the “JSON variable substitution”.

Now we can define an environment variable for the substitution. Just add one with the full “JSON path” for the variable. In our case, we want “api.baseUri” to be the name and then put in whatever the corresponding environment value is:

We can repeat this for other variables if we need more.

Conclusion

I really love the Aurelia framework – and with the solid Aurelia cli, development is a really good experience. Add to that simple build and release management to Azure using VSTS, and you can get a complete site skeleton with full CI/CD in half a day. And that means you’re delivering better software, faster – always a good thing!

Happy Aurelia-ing!

In Part 1 I looked at how to develop multi-container apps using Kubernetes (k8s) - and more specifically, minikube, which is a full k8s environment that runs a single node on a VM on your laptop. In that post I walk through cloning this repo (be sure to look at the docker branch) which contains two containers: a DotNet Core API container and a frontend SPA (Aurelia) container (also hosted as static files in a DotNet Core app). I show how to build the containers locally and get them running in minikube, taking advantage of ConfigMaps to handle configuration.

In this post, I will show you how to take the local development into CI/CD and walk through creating an automated build/release pipeline using VSTS. We'll create an Azure Container Registry and Azure Container Services using k8s as the orchestration mechanism.

I do recommend watching Nigel Poulton's excellent Getting Started with Kubernetes PluralSight course and reading this post by Atul Malaviya from Microsoft. Nigel's course was an excellent primer into Kubernetes and Atul's post was helpful to see how VSTS and k8s interact - but neither course nor post quite covered a whole pipeline. How do you update your images in a CI/CD pipeline was a question not answered to my satisfaction. So after some experimentation, I am writing this post!

Creating a k8s Environment using Azure Container Services

You can run k8s on-premises, or in AWS or Google Cloud. However, I think Azure Container Services makes spinning up an k8s cluster really straightforward. However, the pipeline I walk through in this post is cloud-host agnostic - it will work against any k8s cluster. We'll also set up a private Container Registry in Azure, though once again, you can use any container registry you choose.

To spin up a k8s cluster you can use the portal, but the Azure CLI makes it a snap and you get to save the keys you'll need to connect, so I'll use that mechanism. I'll also use Bash for Windows with kubectl, but any platform running kubectl and the Azure CLI will do.

Here are the commands:

# set some variables
export RG="cd-k8s"
export clusterName="cdk8s"
export location="westus"
# create a folder for the cluster ssh-keys
mkdir cdk8s

# login and create a resource group
az login
az group create --location $location --name $RG

# create an ACS k8s cluster
az acs create --orchestrator-type=kubernetes --resource-group $RG --name=$ClusterName --dns-prefix=$ClusterName --generate-ssh-keys --ssh-key-value ~/cdk8s/id_rsa.pub --location $location --agent-vm-size Standard_DS1_v2 --agent-count 2

# create an Azure Container Registry
az acr create --resource-group $RG --name $ClusterName --location $location --sku Basic --admin-enabled

# configure kubectl
az acs kubernetes get-credentials --name $ClusterName --resource-group $RG --file ~/cdk8s/kubeconfig --ssh-key-file ~/cdk8s/id_rsa
export KUBECONFIG="~/cdk8s/kubeconfig"

# test connection
kubectl get nodes
NAME                    STATUS                     AGE       VERSION
k8s-agent-96607ff6-0    Ready                      17m       v1.6.6
k8s-agent-96607ff6-1    Ready                      17m       v1.6.6
k8s-master-96607ff6-0   Ready,SchedulingDisabled   17m       v1.6.6

Notes:

• Lines 2-4: create some variables
• Line 6: create a folder for the ssh-keys and kubeconfig
• Line 9: login to Azure (this prompts you to open a browser with the device login - if you don't have an Azure subscription create a free one now!)
• Line 10: create a resource group to house all the resources we're going to create
• Line 13: create a k8s cluster using the resource group we just created and the name we pass in; generate ssh-keys and place them in the specified folder; we want 2 agents (nodes) with the specified VM size
• Line 16: create an Azure Container registry in the same resource group with admin access enabled
• Line 19: get the credentials necessary to connect to the cluster using kubectl; use the supplied ssh-key and save the creds to the specified kubeconfig file
• Line 20: tell kubectl to use this config rather than the default config (which may have other k8s clusters or minikube config)
• Line 23: test that we can connect to the cluster
• Lines 24-27: we are indeed connecting successfully!

If you open a browser and navigate to the Azure portal and then open your resource group, you'll see how much stuff got created by the few preceding simple commands:

Don't worry - you'll not need to manage these resources yourself. Azure and the k8s cluster manage them for you!

Namespaces

Before we actually create the build and release for our container apps, let's consider the promotion model. Typically there's Dev->UAT->Prod or something similar. In the case of k8s, minikube is the local dev environment - and that's great since this is a full k8s cluster on your laptop - so you get to run your code locally including using k8s "meta-constructs" such as configMaps. So what about UAT and Prod? You could spin up separate clusters, but that could end up being expensive. You can also share the prod cluster resources by leveraging namespaces. Namespaces in k8s can be security boundaries, but they can also be isolation boundaries. I can deploy new versions of my app to a dev namespace - and even though that namespace shares the resources of the prod namespace, it's completely invisible, including its own IPs etc. Of course I shouldn't load test in this configuration since loading the dev namespace is going to potentially steal resources from prod apps. This is conceptually similar to deployment slots in Azure App Services - they can be used to test apps lightly before promoting to prod.

When you spin up a k8s cluster, besides kube-system and kube-public namespaces (which house the k8s pods) there is a "default" namespace. If you don't specify otherwise, any services, deploymens or pods you create will go to this namespace. However, let's create two additional namespaces: dev and prod. Here's the yml:

apiVersion: v1
kind: Namespace
metadata:
  name: dev
---
apiVersion: v1
kind: Namespace
metadata:
  name: prod

This file contains the definitions for both namespaces. Run the apply command to create the namespaces. Once completed, you can list all the namespaces in the cluster:

kubectl apply -f namespaces.yml
namespace "dev" created
namespace "prod" created

kubectl get namespaces
NAME          STATUS    AGE
default       Active    27m
dev           Active    20s
kube-public   Active    27m
kube-system   Active    27m
prod          Active    20s

Configuring the Container Registry Secret

One more piece of setup before we get to the code: when the k8s cluster is pulling images to run, we're going to want it to pull from the Container Registry we just created. Access to this registry is secured since this is a private registry. So we need to configure a registry secret that we can just reference in our deployment yml files. Here are the commands:

az acr credential show --name $ClusterName --output table
USERNAME    PASSWORD                          PASSWORD2
----------  --------------------------------  --------------------------------
cdk8s       some-long-key-1                   some-long-key-2

kubectl create secret docker-registry regsecret --docker-server=$ClusterName.azurecr.io --docker-username=$ClusterName --docker-password=<some-long-key-1> --docker-email=admin@azurecr.io
secret "regsecret" created

The first command uses az to get the keys for the admin user (the admin username is the same as the name of the Container registry: so I created cdk8s.azurecr.io and so the admin username is cdk8s). Pass in one of the keys (it doesn't really matter which one) as the password. The email address is not used, so this can be anything. We now have a registry secret called "regsecret" that we can refer to when deploying to the k8s cluster. K8s will use this secret to authenticate to the registry.

Configure VSTS Endpoints

We now have the k8s cluster and container registry configured. Let's add these endpoints to VSTS so that we can push containers to the registry during a build and perform commands against the k8s cluster during a release. The endpoints allow us to abstract away authentication so that we don't need to store credentials in our release definitions directly. You can also restrict who can view/consume the endpoints using endpoint roles.

Open VSTS and navigate to a Team Project (or just create a new one). Go to the team project and click the gear icon to navigate to the settings hub for that Team Project. Then click Services. Click "+ New Services" and create a new Docker Registry endpoint. Use the same credentials you used to create the registry secret in k8s using kubectl:

Next create a k8s endpoint. For the url, it will be https://$ClusterName.$location.cloudapp.azure.com (where clustername and location are the variables we used earlier to create the cluster). You'll need to copy the entire contents of the ~/cdk8s/kubeconfig file (or whatever you called it) that was output when you ran the az acs kubernetes get-credential command into the credentials textbox:

We now have two endpoints that we can use in the build/release definitions:

The Build

We can now create a build that compiles/tests our code, creates docker images and pushes the images to the Container Registry, tagging them appropriately. Click on Build & Release and then click on Builds to open the build hub. Create a new build definition. Select the ASP.NET Core template and click apply. Here are the settings we'll need:

• Tasks->Process: Set the name to something like k8s-demo-CI and select the "Hosted Linux Preview" queue
• Options: change the build number format to "1.0.0$(rev:.r)" so that the builds have a 1.0.0.x format
• Tasks->Get Sources: Select the Github repo, authorizing via OAuth or PAT. Select the AzureAureliaDemo and set the default branch to docker. You may have to fork the repo (or just import it into VSTS) if you're following along.
• Tasks->DotNet Restore - leave as-is
• Tasks->DotNet Build - add "--version-suffix $(Build.BuildNumber)" to the build arguments to match the assembly version to the build number
• Tasks->DotNet Test - disable this task since there are no DotNet tests in this solution (you can of course re-enable this task when you have tests)
• Tasks->Add an "npm" task. Set the working folder to "frontend" and make sure the command is "install"
• Tasks->Add a "Command line" task. Set the tool to "node", the arguments to "node_modules/aurelia-cli/bin/aurelia-cli.js test" and the working folder to "frontend". This will run Aurelia tests.
• Tasks->Add a "Publish test results" task. Set "Test Results files" to "test*.xml" and "Search Folder" to "$(Build.SourcesDirectory)/frontend/testresults". This publishes the Aurelia test results.
• Tasks->Add a "Publish code coverage" task. Set "Coverage Tool" to "Cobertura", "Summary File" to "$(Build.SourcesDirectory)/frontend/reports/coverage/cobertura.xml" and "Report Directory" to "$(Build.SourcesDirectory)/frontend/reports/coverage/html". This publishes the Aurelia test coverage results.
• Tasks->Add a "Command line" task. Set the tool to "node", the arguments to "node_modules/aurelia-cli/bin/aurelia-cli.js build --env prod" and the working folder to "frontend". This transpiles, processes and packs the Aurelia SPA app.
• Tasks->DotNet Publish. Change the Arguments to "-c $(BuildConfiguration) -o publish" and uncheck "Zip Published Projects"
• Tasks->Add a "Docker Compose" task. Set the "Container Registry Type" to "Azure Container Registry" and set your Azure subscription and container registry to the registry we created an endpoint for earlier. Set "Additional Docker Compose Files" to "docker-compose.vsts.yml", the action to "Build service images" and "Additional Image Tags" to "$(Build.BuildNumber)" so that the build number is used as the tag for the images.
• Clone the "Docker Compose" task. Rename it to "Push service images" and change the action to "Push service images". Check the "Include Latest Tag" checkbox.
• Tasks->Publish Artifact. Set both "Path to Publish" and "Artifact Name" to k8s. This publishes the k8s yml files so that they are available in the release.

The final list of tasks looks something like this:

You can now Save and Queue the build. When the build is complete, you'll see the test/coverage information in the summary.

You can also take a look at your container registry to see the newly pushed service images, tagged with the build number.

The Release

We can now configure a release that will create/update the services we need. For that we're going to need to manage configuration. Now we could just hard-code the configuration, but that could mean sensitive data (like passwords) would end up in source control. I prefer to tokenize any configuration and have Release Management keep the sensitive data outside of source control. VSTS Release Management allows you to create secrets for individual environments or releases or you can create them in reusable Variable Groups. You can also now easily integrate with Azure Key Vault.

To replace the tokens with environment-specific values, we're going to need a task that can do token substitution. Fortunately, I've got a (cross-platform) ReplaceTokens task in Colin's ALM Corner Build & Release Tasks extension on the VSTS Marketplace. Click on the link to navigate to the page and click install to install the extension onto your account.

From the build summary page, scroll down on the right hand side to the "Deployments" section and click the "Create release" link. You can also click on Releases and create a new definition from there. Start from an Empty template and select your team project and the build that you just completed as the source build. Check the "Continuous Deployment" checkbox to automatically trigger a release for every good build.

Rename the definition to "k8s" or something descriptive. On the "General" tab change the release number format to "$(Build.BuildNumber)-$(rev:r)" so that you can easily see the build number in the name of the release. Back on Environments, rename Environment 1 to "dev". Click on the "Run on Agent" link and make sure the Deployment queue is "Hosted Linux Preview". Add the following tasks:

• Replace Tokens
• Source Path: browse to the k8s folder
• Target File Pattern: "*-release.yml". This performs token replacement on any yml file with a name that ends in "-release." There's 3: back- and frontend service/deployment files and the frontend config file. This task finds the tokens in the file (with pre- and postfix __) and looks for variables with the same name. Each variable is replaced with its corresponding value. We'll create the variables shortly.
• Kubernetes Task 1 (apply frontend config)
• Set the k8s connection to the endpoint you created earlier. Also set the connection details for the Azure Container Registry. This applies to all the Kubernetes tasks. Set the Command to "apply", check the "Use Configuration Files" option and set the file to the k8s/app-demo-frontend-config-release.yml file using the file picker. Add "--namespace $(namespace)" to the arguments textbox.
• Kubernetes Task 2 (apply backend service/deployment definition)
• Set the same connection details for the k8s service and Azure Container Registry. This time, set "Secret Name" to "regsecret" (this is the name of the secret we created when setting up the k8s cluster, and is also the name we refer to for the imagePullSecret in the Deployment definitions). Check the "Force update secret" setting. This ensures that the secret value in k8s matches the key from Azure. You could also skip this option since we created the key manually.
• Set the Command to "apply", check the "Use Configuration Files" option and set the file to the k8s/app-demo-backend-release.yml file using the file picker. Add "--namespace $(namespace)" to the arguments textbox.
• Kubernetes Task 3 (apply frontend service/deployment definition)
• This is the same as the previous task except that the filename is k8s/app-demo-frontend-release.yml.
• Kubernetes Task 4 (update backend image)
• Set the same connection details for the k8s service and Azure Container Registry. No secret required here. Set the Command to "set" and specify Arguments as "image deployment/demo-backend-deployment backend=$(ContainerRegistry)/api:$(Build.BuildNumber) --record --namespace=$(namespace)".
• This updates the version (tag) of the container image to use. K8s will do a rolling update that brings new containers online and takes the old containers offline in such a manner that the service is still up throughout the bleed over.
• Kubernetes Task 5 (update the frontend image)
• Same as the previous task except the Arguments are "image deployment/demo-frontend-deployment frontend=$(ContainerRegistry)/frontend:$(Build.BuildNumber) --record --namespace=$(namespace)"
• Click on the "…" button on the "dev" card and click Configure Variables. Set the following values:
• BackendServicePort: 30081
• FrontendServicePort: 30080
• ContainerRegistry: <your container reg>.azurecr.io
• namespace: $(Release.EnvironmentName)
• AspNetCoreEnvironment: development
• baseUri: http://$(BackendServiceIP)/api
• BackendServiceIP: 10.0.0.1

This sets environment-specific values for all the variables in the yml files. The Replace Tokens task takes care of injecting into the files for us. Let's take a quick look at one of the tokenized files (tokenized lines are highlighted):

apiVersion: v1
kind: Service
metadata:
  name: demo-frontend-service
  labels:
    app: demo
spec:
  selector:
    app: demo
    tier: frontend
  ports:
    - protocol: TCP
      port: 80
      nodePort: __FrontendServicePort__
  type: LoadBalancer
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: demo-frontend-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: demo
        tier: frontend
    spec:
      containers:
        - name: frontend
          image: __ContainerRegistry__/frontend
          ports:
          - containerPort: 80
          env:
          - name: "ASPNETCORE_ENVIRONMENT"
            value: "__AspNetCoreEnvironment__"
          volumeMounts:
            - name: config-volume
              mountPath: /app/wwwroot/config/
          imagePullPolicy: Always
      volumes:
        - name: config-volume
          configMap:
            name: demo-app-frontend-config
      imagePullSecrets:
        - name: regsecret

A note on the value for BackendServiceIP: we use 10.0.0.1 as a temporary placeholder, since Azure will create an IP for this service when k8s spins up the backend service (you'll see a public IP address in the resource group in the Azure portal). We will have to run this once to create the services and then update this to the real IP address so that the frontend service works correctly. We also use $(Release.EnvironmentName) as the value for namespace - so "dev" (and later "prod") need to match the namespaces we created, including casing.

If the service/deployment and config don't change, then the first 3 k8s tasks are essentially no-ops. Only the "set" commands are actually going to do anything. But this is great - since the service/deployment and config files can be applied idempotently! They change when they have to and don't mess anything up when they don't change - perfect for repeatable releases!

Save the definition. Click "+ Release" to create a new release. Click on the release number (it will be something like 1.0.0.1-1) to open the release. Click on logs to see the logs.

Once the release has completed, you can see the deployment in the Kubernetes dashboard. To open the dashboard, execute the following command:

az acs kubernetes browse -n $ClusterName -g $RG --ssh-key-file ~/cdk8s/id_rsa

Proxy running on 127.0.0.1:8001/ui
Press CTRL+C to close the tunnel...
Starting to serve on 127.0.0.1:8001

The last argument is the path to the SSH key file that got generated when we created the cluster - adjust your path accordingly. You can now open a browser to http://localhost:8001/ui. Change the namespace dropdown to "dev" and click on Deployments. You should see 2 successful deployments - each showing 2 healthy pods. You can also see the images that are running in the deployments - note the build number as the tag!

To see the services, click on Services.

Now we have the IP address of the backend service, so we can update the variable in the release. We can then queue a new release - this time, the frontend configuration is updated with the correct IP address for the backend service (in this case 23.99.58.48). We can then browse to the frontend service IP address and see our service is now running!

Creating Prod

Now that we are sure that the dev environment is working, we can go back to the release and clone "dev" to "prod". Make sure you specify a post-approval on dev (or a pre-approval on prod) so that there's a checkpoint between the two environments.

We can then just change the node ports, AspNetCoreEnvironment and BackendServiceIP variables and we're good to go! Of course we need to deploy once to the prod namespace before we see the k8s/Azure assigned IP address for the prod backend and then re-run the release to update the config.

We could also remove the nodePort from the definitions altogether and let k8s decide on a node port - but if it's explicit then we know what port the service is going to run on within the cluster (not externally).

I did get irritated having to specify "--namespace" for each command - so irritated, in fact, that I've created a Pull Request in the vsts-tasks Github repo to expose namespace as an optional UI element!

End to End

Now that we have the dev and prod environments set up in a CI/CD pipeline, we can make a change to the code. I'll change the text below the version to "K8s demo" and commit the change. This triggers the build, creating a newer container image and running tests, which in turn triggers the release to dev. Now I can see the change in dev (which is on 1.0.0.3 or some newer version than 1.0.0.1), while prod is still on version 1.0.0.1.

Approve dev in Release Management and prod kicks off - and a few seconds later prod is now also on 1.0.0.3.

I've exported the json definitions for both the build and the release into this folder - you can attempt to import them (I'm not sure if that will work) but you can refer to them in any case.

Conclusion

k8s shows great promise as a solid container orchestration mechanism. The yml infrastructure-as-code is great to work with and easy to version control. The deployment mechanism means you can have very minimal (if any) downtime when deploying and having access to configMaps and secrets makes the entire process secure. Using the Azure CLI you can create a k8s cluster and Azure Container registry with a couple simple commands. The VSTS integration through the k8s tasks makes setting up CI/CD relatively easy - all in all it's a great development workflow. Throw in minikube as I described in Part 1 of this series, which gives you a full k8s cluster for local development on your laptop, and you have a great dev/CI/CD workflow.

Of course a CI/CD pipeline doesn't battle test the actual applications in production! I would love to hear your experiences running k8s in production - sound out in the comments if you have some experience of running apps in a k8s cluster in prod!

Happy k8sing!

If you've read my blog before, you'll probably know that I am huge fan of Docker and containers. When was the last time you installed software onto bare metal? Other than your laptop, chances are you haven't for a long time. Virtualization has transformed how we think about resources in the datacenter, greatly increasing the density and utilization of resources. The next evolution in density is containers - just what VMs are to physical servers, containers are to VMs. Soon, almost no-one will work against VMs anymore - we'll all be in containers. At least, that's the potential.

However, as cool as containers are for packaging up apps, there's still a lot of uncertainty about how to actually run containers in production. Creating a single container is a cool and satisfying experience for a developer, but how do you run a cluster and scale containers? How do you monitor your containers? How do you manage faults? This is where we enter the world of container orchestration.

This post will cover the local development experience with Kubernetes and minikube. Part 2 covers the CI/CD pipeline to a Kubernetes cluster in Azure.

Orchestrator Wars

There are three popular container orchestration systems - Mesos, Kubernetes and Docker Swarm. I don't want to go into a debate on which one you should go with (yet) - but they're all conceptually similar.  They all work off configuration as code for spinning up lots of containers across lots of nodes. Kubernetes does have a couple features that I think are killer for DevOps: ConfigMaps, Secrets and namespaces.

In short, namespaces allow you to segregate logical environments in the same cluster - the canonical example is a DEV namespace where you can run small copies of your PROD environment for testing. You could also use namespaces for different security contexts or multi-tenancy. ConfigMaps (and Secrets) allow you to store configuration outside of your containers - which means you can have the same image running in various contexts without having to bake environment-specific code into the images themselves.

Kubernetes Workflow and Pipeline

In this post, I want to look at how you would develop with Kubernetes in mind. We'll start by looking at the developer workflow and then move on to how the DevOps pipeline looks in the next post. Fortunately, having MiniKube (a one-node Kubernetes cluster that runs in a VM) means that you can develop against a fully features cluster on your laptop! That means you can take advantage of cluster features (like ConfigMaps) without having to be connected to a production cluster.

So what would the developer workflow look like? Something like this:

1. Develop code
2. Build image from Dockerfile or docker-compose files
3. Run service in MiniKube (which spins up containers from the images you just built)

It turns out that Visual Studio 2017 (and/or VS Code), Docker and MiniKube make this a really smooth experience.

Eventually you're going to move to the DevOps pipeline - starting with a build. The build will take the source files and Dockerfiles and build images and push them to a private container registry. Then you'll want to push configuration to a Kubernetes cluster to actually run/deploy the new images. It turns out that using Azure and VSTS makes this DevOps pipeline smooth as butter! That will be the subject of Part 2 - for now, we'll concentrate on the developer workflow.

Setting up the Developer Environment

I'm going to focus on a Windows setup, but the same setup would apply to Mac or Linux environments as well. To set up a local development environment, you need to install the following:

1. Docker
2. Kubectl
3. MiniKube

You can follow the links and run the installs. I had a bit of trouble with MiniKube on HyperV - by default, MiniKube start (the command that creates the MiniKube VM) just grabs the first HyperV virtual network it finds. I had a couple, and the one that MiniKube grabbed was an internal network, which caused MiniKube to fail. I created a new virtual network called minikube in the HyperV console and made sure it was an external network. I then used the following command to create the MiniKube VM:

c:
cd \
minikube start --vm-driver hyperv --hyperv-virtual-switch minikube

Note: I had to cd to c:\ - if I did not, MiniKube failed to create the VM.

My external network if connected to my WiFi. That means when I join a new network, my minikube VM gets a new IP. Instead of having to update the kubeconfig each time, I just added a hosts entry in my hosts file (c:\windows\system32\drivers\etc\hosts on Windows) using "<IP> kubernetes", where IP is the IP address of the minikube VM - obtained by running "minikube ip". To update the kubeconfig, run this command:

kubectl config set-cluster minikube --server=https://kubernetes:8443 --certificate-authority=c:/users/<user>/.minikube/ca.crt

where <user> is your username, so that the cert points to the ca.crt file generated into your .minikube directory.

Now if you join a new network, you just update the IP in the hosts file and your kubectl commands will still work. The certificate is generated for a hostname "kubernetes" so you have to use that name.

If everything is working, then you should get a neat response to "kubectl get nodes":

PS:\> kubectl get nodes
NAME       STATUS    AGE       VERSION
minikube   Ready     11m       v1.6.4

To open the Kubernetes UI, just enter "minikube dashboard" and a browser will launch:

Finally, to "re-use" the minikube docker context, run the following command:

& minikube docker-env | Invoke-Expression

Now you are sharing the minikube docker socket. Running "docker ps" will return a few running containers - these are the underlying Kubernetes system containers. It also means you can create images here that the minikube cluster can run.

You now have a 1-node cluster, ready for development!

Get Some Code

I recently blogged about Aurelia development with Azure and VSTS. Since I already had a couple of .NET Core sites, I thought I would see if I could get them running in a Kubernetes cluster. Clone this repo and checkout the docker branch. I've added some files to the repo to support both building the Docker images as well as specifying Kubernetes configuration. Let's take a look.

The docker-compose.yml file specifies a composite application made up of two images: api and frontend:

version: '2'

services:
  api:
    image: api
    build:
      context: ./API
      dockerfile: Dockerfile

  frontend:
    image: frontend
    build:
      context: ./frontend
      dockerfile: Dockerfile

The Dockerfile for each service is straightforward: start from the ASP.NET Core 1.1 image, copy the application files into the container, expose port 80 and run "dotnet app.dll" (frontend.dll and api.dll for each site respectively) as the entry point for each container:

FROM microsoft/aspnetcore:1.1
ARG source
WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "API.dll"]

To build the images, we need to dotnet restore, build and publish. Then we can build the images. Once we have images, we can configure a Kubernetes service to run the images in our minikube cluster.

Building the Images

The easiest way to get the images built is to use Visual Studio, set the docker-compose project as the startup project and run. That will build the images for you. But if you're not using Visual Studio, then you can build the images by running the following commands from the root of the repo:

cd API
dotnet restore
dotnet build
dotnet publish -o obj/Docker/publish
cd ../frontend
dotnet restore
dotnet build
dotnet publish -o obj/Docker/publish
cd ..
docker-compose -f docker-compose.yml build

Now if you run "docker images" you'll see the minikube containers as well as images for the frontend and the api:

Declaring the Services - Configuration as Code

We can now define the services that we want to run in the cluster. One of the things I love about Kubernetes is that it pushes you to declare the environment you want rather than running a script. This declarative model is far better than an imperative model, and we can see that with the rise of Chef, Puppet and PowerShell DSC. Kubernetes allows us to specify the services we want exposed as well as how to deploy them. We can define various Kubernetes objects using a simple yaml file. We're going to declare two services: an api service and a frontend service. Usually, the backend services won't be exposed outside the cluster, but since the demo code we're deploying is a single page app (SPA), we need to expose the api outside the cluster.

The services are rarely going to change - they specify what services are available in the cluster. However, the underlying containers (or in Kubernetes speak, pods) that make up the service will change. They'll change as they are updated and they'll change as we scale out and then back in. To manage the containers that "make up" the service, we use a construct known as a Deployment. Since the service and deployment are fairly tightly coupled, I've placed them into the same file, so that we have a frontend service/deployment file (k8s/app-demo-frontend-minikube.yml) and an api service/deployment file (k8s/app-demo-backend-minikube.yml). The service and deployment definitions could live separately too if you want. Let's take a look at the app-demo-backend.yml file:

apiVersion: v1
kind: Service
metadata:
  name: demo-backend-service
  labels:
    app: demo
spec:
  selector:
    app: demo
    tier: backend
  ports:
    - protocol: TCP
      port: 80
      nodePort: 30081
  type: NodePort
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: demo-backend-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: demo
        tier: backend
    spec:
      containers:
      - name: backend
        image: api
        ports:
        - containerPort: 80
        imagePullPolicy: Never

Notes:

• Lines 1 - 15 declare the service
• Line 4 specified the service name
• Line 8 - 10 specify the selector for this service. Any pod that has the labels app=demo and tier=frontend will be load balanced for this service. As requests come into the cluster that target this service, the service will know how to route the traffic to its underlying pods. This makes adding, removing or updating pods easy since all we have to do is modify the selector. The service will get a static IP, but the underlying pods will get dynamic IPs that will change as they move through their lifecycle. However, this is transparent to us, since we just target the service and all is good.
• Line 14 - we want this service exposed on port 30081 (mapping to port 80 on the pods, as specified in line 13)
• Line 15 - the type NodePort specifies that we want Kubernetes to give the service a port on the same IP as the cluster. For "real" clusters (in a cloud provider like Azure) we would change this to get an IP from the cloud host.
• Lines 17 - 34 declare the Deployment that will ensure that there are containers (pods) to do the work for the service. If a pod dies, the Deployment will automatically start a new one. This is the construct that ensures the service is up and running.
• Line 22 specifies that we want 2 instances of the container for this service at all times
• Lines 26 and 27 are important: they must match the selector labels from the service
• Line 30 specifies the name of the container within the pod (in this case we only have a single container in this pod anyway, which is generally what you want to do)
• Line 31 specifies the name of the image to run - this is the same name as we specified in the docker-compose file for the backend image
• Line 33 exposes port 80 on this container to the cluster
• Line 34 specifies that we never want Kubernetes to pull the image since we're going to build the images into the minikube docker context. In a production cluster, we'll want to specify other policies so that the cluster can get updated images from a container registry (we'll see that in Part 2).

The frontend definition for the frontend service is very similar - except there's also some "magic" for configuration. Let's take a quick look:

spec:
  containers:
    - name: frontend
      image: frontend
      ports:
      - containerPort: 80
      env:
      - name: "ASPNETCORE_ENVIRONMENT"
        value: "Production"
      volumeMounts:
        - name: config-volume
          mountPath: /app/wwwroot/config/
      imagePullPolicy: Never
  volumes:
    - name: config-volume
      configMap:
        name: demo-app-frontend-config

Notes:

• Line 30: name the container in the pod
• Line 31: specify the name of the image for this container - matching the name in the docker-compose file
• Lines 34 - 36: an example of how to specify environment variables for a service
• Lines 37 - 39: this is a reference to a volume mount (specified lower down) for mounting a config file, telling Kuberenetes where in the container file system to mount the file. In this case, Kubernetes will mount the volume with name "config-volume" to the path /app/wwwroot/config inside the container.
• Lines 41 - 44: this specifies a volume - in this case a configMap volume to use for the configuration (more on this just below). Here we tell Kubernetes to create a volume called config-volume (referred to by the container volumeMount) and to base the data for the volume off a configMap with the name demo-app-frontend-config

Handling Configuration

We now have a couple of container images and can start running them in minikube. However, before we start that, let's take a moment to think a little about configuration. If you've ever heard me speak or read my blog, you'll know that I am a huge proponent of "build once, deploy many times". This is a core principle of good DevOps. It's no different when you consider Kubernetes and containers. However, to achieve that you'll have to make sure you have a way to handle configuration outside of your compiled bits - hence mechanisms like configuration files. If you're deploying to IIS or Azure App Services, you can simply use the web.config (or for DotNet Core the appsettings.json file) and just specify different values for different environments. However, how do you do that with containers? The entire app is self-contained in the container image, so you can't have different versions of the config file - otherwise you'll need different versions of the container and you'll be violating the build once principle.

Fortunately, we can use volume mounts (a container concept) in conjunction with secrets and/or configMaps (a Kubernetes concept). In essence, we can specify configMaps (which are essentially key-value pairs) or secrets (which are masked or hidden key-value pairs) in Kubernetes and then just mount them via volume mounts into containers. This is really powerful, since the pod definition stays the same, but if we have a different configMap we get a different configuration! We'll see how this works when we deploy to a cloud cluster and use namespaces to separate dev and production environments.

The configMaps can also be specified using configuration as code. Here's the configuration for our configMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-app-frontend-config
  labels:
    app: demo
    tier: frontend
data:
  config.json: |
    {
      "api": {
        "baseUri": "http://kubernetes:30081/api"
      }
    }

Notes:

• Line 2: we specify that this is a configMap definition
• Line 4: the name we can refer to this map by
• Line 9: we're specifying this map using a "file format" - the name of the file is "config.json"
• Lines 10 - 14: the contents of the config file

Aside: Static Files Symlink Issue

I did have one issue when mounting the config file using configMaps: inside the container the volume mount to /app/www/config/config.json ends up being a symlink. I got the idea of using a configMap in the container from this excellent post by Anthony Chu, in which he mounts an application.json file that the Startup.cs file can consume. Apparently he didn't have any issues with the symlink in the Startup file. However, in the case of my demo frontend app, I am using a config file that is consumed by the SPA app - and that means, since it's on the client side, the config file needs to be served from the DotNet Core app, just like the html or js files. No problem - we've already got a UseStaticFiles call in Startup, so that should just serve the file, right? Unfortunately, it doesn't. At least, it only serves the first few bytes of the file.

I took a couple of days to figure this out - there's a conversation on Github you can read if you're interested. In short, the symlink length is not the length of the file, but the length of the path to the file. The StaticFiles middleware reads FileInfo.Length bytes when the file is requested, but since the length isn't the full length of the file, only the first few bytes were being returned. I was able to create a FileProvider that worked around the issue.

Running the Images in Kubernetes

To run the services we just created in minikube, we can just use kubectl to apply the configurations. Here's the list of commands (the highlighted lines):

PS:\> cd k8s
PS:\> kubectl apply -f .\app-demo-frontend-config.yml
configmap "demo-app-frontend-config" created

PS:\> kubectl apply -f .\app-demo-backend-minikube.yml
service "demo-backend-service" created
deployment "demo-backend-deployment" created

PS:\> kubectl apply -f .\app-demo-frontend-minikube.yml
service "demo-frontend-service" created
deployment "demo-frontend-deployment" created

And now we have some services! You can open the minikube dashboard by running "minikube dashboard" and check that the services are green:

And you can browse to the frontend service by navigating to http://kubernetes:30080:

The list (value1 and value2) are values coming back from the API service - so the frontend is able to reach the backend service in minikube successfully!

Updating the Containers or Containers

If you update your code, you're going to need to rebuild the container(s). If you update the config, you'll have to re-run the "kubectl apply" command to update the configMap. Then, since we don't need high-availability in dev, we can just delete the running pods and let the replication set restart them - this time with updated config and/or code. Of course in production we won't do this - I'll show you how to do rolling updates in the next post when we do CI/CD to a Kubernetes cluster.

For dev though, I get the pods, delete them all and then watch Kubernetes magically re-start the containers again (with new IDs) and voila - updated containers.

PS:> kubectl get pods
NAME                                       READY     STATUS    RESTARTS   AGE
demo-backend-deployment-951716883-fhf90    1/1       Running   0          28m
demo-backend-deployment-951716883-pw1r2    1/1       Running   0          28m
demo-frontend-deployment-477968527-bfzhv   1/1       Running   0          14s
demo-frontend-deployment-477968527-q4f9l   1/1       Running   0          24s

PS:> kubectl delete pods demo-backend-deployment-951716883-fhf90 demo
-backend-deployment-951716883-pw1r2 demo-frontend-deployment-477968527-bfzhv demo-frontend-deployment-477968527-q4f9l
pod "demo-backend-deployment-951716883-fhf90" deleted
pod "demo-backend-deployment-951716883-pw1r2" deleted
pod "demo-frontend-deployment-477968527-bfzhv" deleted
pod "demo-frontend-deployment-477968527-q4f9l" deleted

PS:> kubectl get pods
NAME                                       READY     STATUS    RESTARTS   AGE
demo-backend-deployment-951716883-4dsl4    1/1       Running   0          3m
demo-backend-deployment-951716883-n6z4f    1/1       Running   0          3m
demo-frontend-deployment-477968527-j2scj   1/1       Running   0          3m
demo-frontend-deployment-477968527-wh8x0   1/1       Running   0          3m

Note how the pods get updated IDs - since they're not the same pods! If we go to the frontend now, we'll see updated code.

Conclusion

I am really impressed with Kubernetes and how it encourages infrastructure as code. It's fairly easy to get a cluster running locally on your laptop using minikube, which means you can develop against a like-for-like environment that matched prod - which is always a good idea. You get to take advantage of secrets and configMaps, just like production containers will use. All in all this is a great way to do development, putting good practices into place right from the start of the development process.

Happy sailing! (Get it? Kubernetes = helmsman)

VSTS supports service hooks like Slack, AppVeyor, Bamboo and a host of other ALM tools. You can also create your own hooks using a simple WebHooks API. There's an example here. However, one thing that is missing from the sample is any kind of authentication.

Why care? Well, simply put - without authentication, anyone could trigger events to your event sink. That may or may not be a big deal, but I prefer to be secure by default.

Now there are a couple of ways you could do auth - you could use AAD or OpenConnect and get a token and use that for the WebHook subscription. That would probably work, but VSTS won't renew the token automatically (at least I don't think it will) so you'll have to update the webhook subscription manually every time the token expires.

The other way is to use Basic Auth. When you subscribe to a webhook in VSTS, you can pass a Basic username/password. The username and password are base64 encoded and added to a header for the requests. Assuming your using HTTPS (so that you don't get man-in-the-middle attacks) you can use this for a relatively safe authentication method. Once you extract the username/password from the header, you can validate them however you want.

In this post I'll cover how to create a Web Hook project that includes Basic Auth as well as logging to Application Insights. I'll also cover how to debug and test your service using Postman.

The source code for this stub project is here so you can just grab that if you want to get going.

Creating the Project

Initially I wanted to create the project in .NET Core. However, I wanted to use a NuGet package and the package unfortunately only supports .NET 4.x. So I'll just use that.

Open Visual Studio 2017 and click File->New Project and create a new ASP.NET Web Application. Select Web API from the project type dialog (oh how I love that you don't have to do this in ASP.NET Core) and ensure you have "No authentication" (we'll add Basic Auth shortly). This creates a new project and even includes Application Insights.

Adding Packages

We're going to add a few NuGet packages. Right-click the web project and add the following packages: Microsoft.AspNet.WebHooks.Receivers.VSTS (contains webhook handler abstract class and event payload classes) and Microsoft.ApplicationInsights.TraceListener (which will send Trace.WriteLines to AppInsights).

Once the packages are installed, you can (optionally) update all the packages. The project templates sometimes have older package versions, so I usually like to do this so that I'm on the latest NuGet package versions from the get go.

Adding a WebHook Handler

This is the class that will do the work. Add a new folder called "Handlers" and create a new class called "VSTSHookHandler".

using Microsoft.AspNet.WebHooks;
using Microsoft.AspNet.WebHooks.Payloads;
using System.Threading.Tasks;
using System.Diagnostics;

namespace vsts_webhook_with_auth.Handlers
{
	public class VSTSHookHandler : VstsWebHookHandlerBase
	{
		public override Task ExecuteAsync(WebHookHandlerContext context, WorkItemCreatedPayload payload)
		{
			Trace.WriteLine($"Event WorkItemCreated triggered for work item {payload.Resource.Id}");
			return base.ExecuteAsync(context, payload);
		}
	}
}

Notes:

• Line 8: We inherit from VstsWebHookHandlerBase - this base class has abstract methods for all the VSTS service hook events that we can listen for.
• Line 10: We override the async WorkItemCreated event - there are other events that you can override depending on what you need. Add as many overrides as you need. This method also gets the context and the payload for the event for us.
• Line 12: We are writing log entries to Trace - because we've added the AppInsights TraceListener, these end up in AppInsights where we can search for particular messages.
• Line 13: Here is where you will implement your logic to respond to the event. For this stub project, I just call the base method (which is essentially a no-op).

Adding a BasicAuthHandler

Add a new class to the Handlers folder called BasicAuthHandler. You can get the full class here, but we only need to see the SendAsync method for our discussion:

protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
	var credentials = ParseAuthorizationHeader(request);

	if (credentials != null && CredentialsAreValid(credentials))
	{
		var identity = new BasicAuthenticationIdentity(credentials.Name, credentials.Password);
		Thread.CurrentPrincipal = new GenericPrincipal(identity, null);
		return base.SendAsync(request, cancellationToken);
	}
	else
	{
		var response = request.CreateResponse(HttpStatusCode.Unauthorized, "Access denied");
		AddChallengeHeader(request, response);
		return Task.FromResult(response);
	}
}

Notes:

• Line 3: The ParseAuthorizationHeader() method extracts the username/password from the auth header
• Line 5: We check that there are credentials and that they "are valid" (in this case that they match hard-coded values we'll add to the web.config)
• Lines 7,8: We add the auth details to the CurrentPrincipal, which has the effect of marking the request as "authenticated"
• Line 9: We forward the request on to the remainder of the pipeline - which is the VSTSHookHandler class methods at this point
• Lines 13-15: We handle the unauthorized scenario

Configuration

We can now add the handlers into the message processing pipeline. Open the Global.asax.cs file and modify the Application_Start() method by adding in the highlighted line (and resolving the namespace):

protected void Application_Start()
{
	GlobalConfiguration.Configuration.MessageHandlers.Add(new BasicAuthenticationHandler());

	AreaRegistration.RegisterAllAreas();
	GlobalConfiguration.Configure(WebApiConfig.Register);
	FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
	RouteConfig.RegisterRoutes(RouteTable.Routes);
	BundleConfig.RegisterBundles(BundleTable.Bundles);
}

Our auth handler is now configured to trigger on requests.

Next, open up App_Start/WebApiConfig.cs and modify the Register() method with the highlighted line:

config.Routes.MapHttpRoute(
	name: "DefaultApi",
	routeTemplate: "api/{controller}/{id}",
	defaults: new { id = RouteParameter.Optional }
);

config.InitializeReceiveVstsWebHooks();

This registers the handler class to respond to VSTS events.

Finally, open the web.config file and add the following appSetting keys:

<appSettings>
	...
	<add key="WebHookUsername" value="vsts"/>
	<add key="WebHookPassword" value="P@ssw0rd"/>
	<add key="MS_WebHookReceiverSecret_VSTS" value="C8B7F962-2B5A-4973-81F3-8888D53CF86E"/>
  </appSettings>

Notes:

• The MS_WebHookReceiverSecret_VSTS is a "code" that the VSTS webhook requires to be in the query params for the service call. This can be anything as long as it is longer than 32 and less than 128 chars. You can also have multiple codes. This code lets you handle different projects firing the same events - you can tie the code to a project so that you can do project specific logic.
• WebHookUsername and WebHookPassword are hardcoded username/password for validation - you can ignore these if you need some other way to validate the values.

Configuring SSL in VS

To run the site using SSL from VS, you'll need to enable that in the project properties. Click on the web project node (so that it is selected in the Solution Explorer). Then press F4 to open the properties pane (this is different to right-click -> Properties). Change SSL Enabled to true. Make a note of the https URL.

You can now right-click the project and select Properties. Change the startup URL to the https URL so that you always get the SSL site.

Testing from PostMan

You can now run the site. You'll notice when you first run it that the cert is invalid (it's self-signed). In order to get Postman working to test the webhooks, I had to read these instructions. In the end, I just opened the https URL in IE and imported the cert to Trusted Root Certification Authorities. I then shut down Chrome and restarted and all was good.

I opened Postman and entered this url: https://localhost:44388/api/webhooks/incoming/vsts?code=C8B7F962-2B5A-4973-81F3-8888D53CF86E, changing the method to POST. I made sure that the code was the same value as the MS_WebHookReceiverSecret_VSTS key in my web.config. I then opened the docs page and grabbed the sample payload for the WorkItemCreated event and pasted this into the Body for the request. I updated the type to application/json (which adds a header).

Hitting Send returns a 401 - which is expected since we haven't provided a username/password. Nice!

Now let's test adding the auth. Go back to Postman and click on Authorization. Change the Type to "Basic Auth" and enter the username/password that you hard-coded into your web.config. Click "Update Request" to add the auth header:

Now press Send again. We get a 200!

Of course you can now set breakpoints and debug normally. If you open AppInsights Viewer in VS you'll see the traces - these will eventually make their way to AppInsights (remember to update your key when you create a real AppInsights resource in Azure).

Registering the Secure Hook in VSTS

You can now clean up the project a bit (remove the home controller etc.) and deploy the site to Azure (or your own IIS web server) using VSTS Build and Release Management (friends don't let friends right-click Publish) and you're ready to register the hook in VSTS. Open your VSTS Team Project and head to Configuration->Service Hooks. Add a new service hook. Enter the URL (including the code query param) for your service. Then just enter the same username/password you have in the website web.config and you're good to go! Hit test to make sure it works.

Conclusion

Securing your WebHooks from VSTS isn't all that hard - just add the BasicAuthHandler and configure the basic auth username/password in the WebHook subscription in VSTS. Now you can securely receive events from VSTS. I would really like to see the VSTS team update the NuGet packages to support .NET Core WebAPI, but the 4.x version is fine in the interim.

Happy hooking!

Azure SQL is a great service - you get your databases into the cloud without having to manage all that nasty server stuff. However, one of the problems with Azure SQL is that you have to authenticate using SQL authentication - a username and password. However, you can also authenticate via Azure Active Directory (AAD) tokens. This is analogous to integrated login using Windows Authentication - but instead of Active Directory, you're using AAD.

There are a number of advantages to AAD Authentication:

• You no longer have to share logins since users log in with their AAD credentials, so auditing is better
• You can manage access to databases using AAD groups
• You can enable "app" logins via Service Principals

In order to get this working, you need:

• To enable AAD authentication on the Azure SQL Server
• A Service Principal
• Add logins to the database granting whatever rights required to the service principal
• Add code to get an auth token for accessing the database
• If you're using Entity Framework (EF), create a new constructor for your DbContext

In this post I'll walk through creating a service principal, configuring the database for AAD auth, creating code for retrieving a token and configuring an EF DbContext for AAD auth.

Create a Service Principal

Azure lets you configure service principals - these are like service accounts on an Active Directory. The advantage to this is that you can configure access to resources for the service and not have to worry about users leaving the org (or domain) and having to change creds and so on. Service principals get keys that can be rotated for better security too. You'll need the service principal when you configure your app to connect to the database.

You can create a service principal using the portal or you can do it easily using:

# Azure CLI 2.0
az ad sp create-for-rbac --name CoolAppSP --password SomeStrongPassword

# PowerShell
# get the application we want a service principal for
$app = Get-AzureRmADApplication -DisplayNameStartWith MyDemoApp
New-AzureRmADServicePrincipal -ApplicationId $app.ApplicationId -DisplayName CoolAppSP -Password SomeStrongPassword

Of course you need to provide a proper strong password! Take a note of the servicePrincipalNames property - the one that looks like a GUID. We'll need this later.

Configuring AAD on the Database

In order to use AAD against the SQL Server, you'll need to configure an AAD admin (user or group) for the database. You can do this in the portal by browsing to the Azure SQL Server (not the database) and clicking "Active Directory Admin". In the page that appears, click "Set Admin" and assign a user or group as the AAD admin.

Once you've done that, you need to grant Azure AD users (or groups) permissions in the databases (not the server). To do that you have to connect to the database using an Azure AD account. Open Visual Studio or SQL Server Management Studio and connect to the database as the admin (or a member of the admin group) using "Active Directory Password Authentication" or "Azure Directory Integrated Authentication" from the Authentication dropdown:

If you don't see these options, then you'll need to update your SQL Management Studio or SSDT. If you're domain joined to the Azure Active Directory domain, you can use the integrated method - in my case my laptop isn't domain joined so I used the password method. For username and password, I used my Azure AD (org account) credentials. Once you're logged in and connected to the database, execute the following T-SQL:

CREATE USER [CoolAppSP] FROM EXTERNAL PROVIDER
EXEC sp_addrolemember 'db_owner', 'CoolAppSP'

Of course you'll use the name of the Service Principal you created earlier - the name for the Login is the same name as the service principal you created, or can be the email address of a specific user or group display name if you're granting access to specific AAD users or groups so that they can access the db directly. And of course the role doesn't have to be dbowner - it can be whatever role you need it to be.

Authenticating using the Service Principal

There are a couple of pieces we need in order to authenticate an application to the Azure SQL database using AAD credentials. The first is a token (it's an OAuth token) that identifies the service principal. Secondly, we need to construct a database connection that uses the token to authenticate to the server.

Retrieve a Token from AAD

To get a token, we'll need to call Azure AD and request one. For this, you'll need the Microsoft.IdentityModel.Clients.ActiveDirectory Nuget package.

Here's the code snippet I used to get a token from AAD:

public async Task<string> GetAccessTokenAsync(string clientId, string clientSecret, string authority, string resource, string scope)
{
	var authContext = new AuthenticationContext(authority, TokenCache.DefaultShared);
	var clientCred = new ClientCredential(clientId, clientSecret);
	var result = await authContext.AcquireTokenAsync(resource, clientCred);

	if (result == null)
	{
		throw new InvalidOperationException("Could not get token");
	}

	return result.AccessToken;
}

Notes:

• Line 1: We need some information in order to get the token: ClientId and ClientSecret are from the service principal. The authority, resource and scope will need to be passed in too (more on this later).
• Line 3: We're getting a token from the "authority" or tenant in Azure
• Line 4: We create a new client credential using the id and secret of the "client" (in this case, the service principal)
• Line 5: We get a token for this client onto the "resource"
• Line 7: We throw if we don't get a token back
• Line 12: If we do get a token, return it to the caller

The client id is the "application ID" of the service principal (the guid in the servicePrincipalNames property of the service principal). To get the secret, log in to the portal and click in the Active Directory blade. Click on "App Registration" and search for your service principal. Click on the service principal to open it. Click on Keys and create a key - make a note of the key so that you can add this to configurations. This key is the clientSecret that the GetAccessToken method needs.

For authority, you'll need to supply the URL to your Azure tenant. You can get this by running "az account show" (Azure CLI 2.0) or "Get-AzureRmSubscription" (PowerShell). Make a note of the tenantId of the subscription (it's a GUID). Once you have that, the authority is simply "https://login.windows.net/{tenantId}". The final piece of info required is the resource - for Azure SQL access, this is simply "https://database.windows.net/". The scope is just empty string - for databases, the security is configured per user (using the role assignments on the DB you configured earlier). The authentication is done using Azure AD via the token - the database is doing authorization. In other words, Azure lets an Azure AD user in when they present a valid token - the database defines what the user can do once they're in via roles.

Creating a SQL Connection

We've now got a way to get a token - so we can create a SQL Connection to the database. Here's a code snippet:

public async Task<SqlConnection> GetSqlConnectionAsync(string tenantId, string clientId, string clientSecret, string dbServer, string dbName)
{
	var authority = string.Format("https://login.windows.net/{0}", tenantId);
	var resource = "https://database.windows.net/";
	var scope = "";
	var token = await GetTokenAsync(clientId, clientSecret, authority, resource, scope);

	var builder = new SqlConnectionStringBuilder();
	builder["Data Source"] = $"{dbServer}.database.windows.net";
	builder["Initial Catalog"] = dbName;
	builder["Connect Timeout"] = 30;
	builder["Persist Security Info"] = false;
	builder["TrustServerCertificate"] = false;
	builder["Encrypt"] = true;
	builder["MultipleActiveResultSets"] = false;

	var con = new SqlConnection(builder.ToString());
	con.AccessToken = token;
	return con;
}

Notes:

• Line 1: All the info we need for the connection
• Lines 3 - 5: Prepare the info for the call to get the token
• Line 6: Get the access token
• Lines 8-15: Prepare the SQL connection string to the Azure SQL database - tweak the properties (like Connect Timeout) appropriately.
• Line 17: Create the connection
• Line 18: Inject the token into the connection object

You'd now be able to use the connection just like you would any SqlConnection object.

Entity Framework DataContext Changes

If you're using Entity Framework for data access, you'll notice there's no obvious way to use the SqlConnection object that's now configured to access the Azure SQL database. You'll need to create a constructor on your DbContext:

public class CoolAppDataContext : DbContext
{
	public CoolAppDataContext(SqlConnection con)
		: base(con, true)
	{
		Database.SetInitializer<CoolAppDataContext>(null);
	}

	public DbSet<Product> Products { get; set; }

	...
}

Notes:

• Line 3: A constructor that accepts a SqlConnection object
• Line 4: Call the base constructor method
• Line 5: Override the initializer for the context's Database object

Now you can use the above methods to construct a SqlConnection to an Azure SQL database using AAD credentials and pass it in to the DbContext - and you're good to go!

Conclusion

Configuring an application to use Azure AD credentials to connect to an Azure SQL database is straightforward once you have all the pieces in place. There's some configuration you need to ensure is in place, but once it's configured you can stop using SQL Authentication to access your cloud databases - and that's a win!

Happy connecting!

I love containers. I've said before that I think they're the future. Just as hardly anyone installs on tin any more since we're so comfortable with Virtualization, I think that in a few years time hardly anyone will deploy VMs - we'll all be on containers. However, container orchestration is still a challenge. Do you choose Kubernetes or Swarm or DCOS? (For my money I think Kubernetes is the way to go). But that means managing a cluster of nodes (VMs). What if you just want to deploy a single container in a useful manner?

You can do that now using Azure Container Instances (ACI). You can also host a container in an Azure Web App for Containers. The Web App for Containers is what I'll use for this post - mostly because I already know how to do A/B testing with Azure Web Apps, so once the container is running then you get all the same paradigms as you would for "conventional" web apps - like slots, app settings in the portal etc.

In this post I'll cover publishing a .NET Core container to Azure Web Apps using VSTS with an ARM template. However, since the hosting technology is "container" you can host whatever application you want - could be Java/TomCat or node.js or python or whatever. The A/B Testing principles will still apply.

You can grab the code for this demo from this Github repo.

Overview of the Moving Parts

There are a couple of moving parts for this demo:

• The source code. This is just a File->New Project .NET Core 2.0 web app. I've added a couple lines of code and Application Insights for monitoring - but other than that there's really nothing there. The focus of this post is about how to A/B test, not how to make an app!
• Application Insights - this is how you can monitor the web app to make sure
• An Azure Container Registry (ACR). This is a private container repository. You can use whatever repo you want, including DockerHub.
• A VSTS Build. I'll show you how to set up a build in VSTS to build the container image and publish it to the ACR.
• An ARM template. This is the definition of the resources necessary for running the container in Azure Web App for Containers. It includes a staging slot and Application Insights.
• A VSTS Release. I'll show you how to create a release that will spin up the web app and deploy the container. Then we'll set up Traffic Manager to (invisibly) divert a percentage of traffic from the prod slot to the staging slot - this is the basis for A/B testing and the culmination of the all the other steps.

The Important Source Bits

Let's take a look at some of the important code files. Firstly, the Startup.cs file:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
	var aiKey = Environment.GetEnvironmentVariable("AIKey");
	if (aiKey != null)
	{
		TelemetryConfiguration.Active.InstrumentationKey = aiKey;
	}
	...

Notes:

• Line 3: I read the environment variable "AIKey" to get the Application Insights key
• Lines 4 - 7: If there is a key, then I set the key for the Application Insights config

The point here is that for configuration, I want to get values from the environment. This would include database connection strings etc. Getting them from the environment lets me specify them in the Web App appSettings so that I don't have to know the values at build time - only at release time.

Let's look at the Dockerfile:

FROM microsoft/aspnetcore:2.0
ARG source
ENV AIKey="11111111-2222-3333-4444-555555555555"

WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "DockerWebApp.dll"]

Notes:

• Line 3 was the only thing I added to make the AIKey configurable as an environment variable

Finally, let's look at the ARM template that defines the resources. The file is too large to paste here and it's Json so it's not easy to read. You can have a look at the file yourself. The key points are:

• a HostingPlan with kind = "linux" and properties.reserved = true. This creates a Linux app plan.
• a Site with properties.siteConfig.appSettings that specify the DOCKER_REGISTRY_SERVER_URL, DOCKER_REGISTRY_SERVER_USERNAME, DOCKER_REGISTRY_SERVER_PASSWORD (using the listCredentials function) and AIKey. We do not specify the DOCKER_CUSTOM_IMAGE_NAME for reasons that will be explained later. Any other environment variables (like connection strings) could be specified here.
• a staging slot (called blue) that has the same settings as the production slot
• a slotconfignames resource that locks the DOCKER_CUSTOM_IMAGE_NAME so that the value is slot-sticky
• an Application Insights resource - the key for this resource is referenced by the appSettings section for the site and the slot

Building (and Publishing) Your Container

At this point we can look at the build. The build needs to compile, test and publish the .NET Core app and build a container. It then needs to publish the container to the ACR (or whatever registry you created).

Create a new build using the ASP.NET Core Web App template and then edit the steps to look as follows:

Make sure you point the Source settings to the repo (you can create a new one and import mine from the Github URL if you want to try this yourself). I then changed the queue to Hosted Linux Preview and changed the name to something appropriate.

On the Options page I set the build number format to 1.0$(rev:.r) which gives a 1.0.0, 1.0.1, 1.0.2 etc. format for my build number.

Click on the Build task and set the arguments to --configuration $(BuildConfiguration) /p:Version=$(Build.BuildNumber). This versions the assemblies to match the build number.

Click on the Publish task and set Publish Web Apps, change the arguments to --configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory) /p:Version=$(Build.BuildNumber) and unselect Zip files. This puts the output files into the artifact staging directory and doesn't zip them. It also publishes with a version number matching the build number:

I deleted the Test task since I don't have tests in this simple project - but you can of course add testing in before publishing.

I then deleted the Publish build artifact tasks since this build won't be publishing an artifact - it will be pushing a container image to my ACR.

In order to build the docker container image, I first need to put the Dockerfile at the correct location relative to the output. So I add a Copy Files task in and configure it to copy the Dockerfile to the artifact staging directory:

Now I add a 2 Docker tasks: the has a Build an image Action. I set the ACR by selecting the settings from the dropdowns. I set the path to the Dockerfile and specify "DockerWebApp" as the source build argument (the Publish task will have places the compiled site and content into this folder in the artifact staging directory). I set Qualify the Image name to correctly tag the container with the ACR prefix and I include the Latest tag in the build (so the current build is always the Latest).

The 2nd Docker task has Action set to Publish an Image. I set the ACR like the Docker Build task. I also change the image name to $(Build.Repository.Name):$(Build.BuildNumber) instead of $(Build.Repository.Name):$(Build.BuildId) and I set the Latest tag.

Now I can run the build. Lots of green! I can also see the image in my ACR in the Azure portal:

Woot! We now have a container image that we can host somewhere.

Releasing the Container

Now that we have a container and an infrastructure template, we can define a release. Here's what the release looks like:

There are 2 incoming artifacts: the build and the Git repo. The build doesn't actually have any artifacts itself - I just set the build as the trigger mechanism. I specify a "master" artifact filter so that only master builds trigger this release. The Git repo is referenced for the deployment scripts (in this case just the ARM template). I started with an empty template and then changed the Release number format to $(Build.BuildNumber)-$(rev:r) in the Options page.

There are 3 environments: Azure blue, Azure prod and blue failed. These are all "production" environments - you could have Dev and Staging environments prior to these environments. However, I want to A/B test in production, so I'm just showing the "production environment" here.

Let's look at the Azure blue environment:

There are 3 tasks: Azure Resource Group Deployment (to deploy the ARM template), an Azure CLI command to deploy the correct container, and a Traffic Manager Route Traffic task. The Azure Resource Group Deployment task specifies the path to the ARM template and parameters files as well as the Azure endpoint for the subscription I want to deploy to. I specify a variable called $(RGName) for the resource group name and then override the parameters for the template using $(SiteName) for the name of the web app in Azure, $(ImageName) for the name of the container image, $(ACR) for the name of my ACR and $(ACRResourceGroup) for the name of the resource group that contains my ACR. Once this task has run, I will have the following resources in the resource group:

Let's take a quick look at the app settings for the site:

At this point the site (and slot) are provisioned, but they still won't have a container running. For that, we need to specify which container (and tag) to deploy. The reason I can't do this in the ARM template is because I want to update the staging slot and leave the prod slot on whatever container tag it is on currently. Let's imagine I specified "latest" for the prod slot - then when we run the template, the prod slot will update, which I don't want. Let's say we specify latest for the blue slot - then the blue slot will update - but what version do we specify in the template for the prod slot? We don't know that ahead of time. So to work around these issues, I don't specify the container tag in the template - I use an Azure CLI command to update it after the template has deployed (or updated) all the other infrastructure and settings.

To deploy a container, we add an Azure CLI task and specify a simple inline script:

Here I select version 1.* of the task (version 0.* doesn't let you specify an inline script). I set the script to sleep for 30 seconds - if I don't do this, then the site is still updating or something and the operation succeeds, but doesn't work - it doesn't actually update the image tag. I suspect that the update is async and so if you don't wait for it to complete, then issuing another tag change succeeds but is ignored. It's not pretty, but that's the only workaround I've found. After pausing for a bit, we invoke the "az webapp config container set" command, specifying the site name, slot name, resource group name and image name. (If you're running this phase on a Linux agent, use $1, $2 etc. for the args - if you're running this on a Windows agent then specify the args using %1, %2 etc. in the script). Then I pass the arguments in - using $(SiteName), blue, $(RGName) and $(ImageName):$(Build.BuildNumber) for the respective arguments.

If you now navigate to the site in the Azure portal and click on the Docker Container tab on the slot, you'll see the container settings:

You can see that the image name and version are specified.

The final task in this environment (a Route Traffic task from my Build and Release extension pack) adds a traffic manager rule - we divert 20% of traffic to the blue slot (unobtrusively to the client):

There's a snag to this method: the first time you deploy, there's nothing (yet) in the prod slot. This is just a first time condition - so after you run this environment for the very first time, navigate to the Azure portal and click on the site. Then swap the blue and prod slots. Now the prod slot is running the container and the blue slot is empty. Repeat the deployment and now both slots have the latest version of the container.

Let's go back to the release and look at the Azure prod environment. This has a pre-approval set so that someone has to approve the deployment to this environment. The Azure blue has a post-deployment approver so that someone can sign off on the release - approved if the experiment works, rejected if it's not. The Azure prod environment triggers when the Azure blue environment is successful - all it has to do is swap the slots and reset the traffic router to reroute 100% of traffic to the prod slot:

So what do we do if the experiment fails? We can reject the Azure blue environment (so that the Azure prod environment doesn't run). We then manually run the blue fail environment - this just resets the traffic to route 100% back to prod. It does not swap the slots:

Don't forget to specify values for the variables we used:

Running a Test

So imagine I have container version 1.0.43 in both slots. Now I make a change and commit and push - this triggers a build (I enabled CI) and we get version 1.0.45 (1.0.44 had an intermittent build failure). This triggers the release (I enabled CD) and now the blue slot has version 1.0.45 and 20% of traffic from the prod slot is going to the blue slot.

Let's navigate to the blue and prod slots and see them side-by-side:

The traffic routing is "sticky" to the user - so if a user navigates to the prod slot and gets diverted to the blue slot, then all requests to the site from the user go to the blue slot. Try opening some incognito windows and hitting the prod site - you'll get the blue content 20% of the time! You can also force the routing using a query parameter - just tack "?x-ms-routing-name=blue" onto the end of any request and you'll end up on the blue slot:

Now you wait for users to generate traffic (or in my case, I totally fake client traffic - BWAHAHA!). So how do we know if the experiment is successful? We use Application Insights.

Application Insights Analytics

Let's click on the Azure portal and go to the App Insights resource for our web app. We can see some telemetry, showing traffic to the site:

But how do we know which requests went to the blue slot and how many went to the prod slot? It turns out that's actually pretty simple. Click the Analytics button to launch App Insights Analytics. We enter a simple query:

We can clearly see that there is more usage of the 1.0.45 contact page. Yes, this metric is bogus - the point is to show you that you can slice by "Application_Version" and so you actually have metrics to determine if your new version (1.0.45) is better or worse that 1.0.43. Maybe it's more traffic to a page. Maybe it's more sales. Maybe it's less exceptions or better response time - all of these metrics can be sliced by Application_Version.

Conclusion

Deploying containers to Azure Web Apps for Containers is a great experience. Once you have the container running, you can use any Web App paradigm - such as Traffic Routing or appSettings - so it's easy if you've ever done any Web App deployments before.

There are a couple of key practices that are critical to A/B testing: telemetry and unobtrusive routing.

Telemetry is absolutely critical to A/B testing: if you can't decide if A is better (or worse) than B, then there's no point deploying both versions. Application Insights is a great tool for telemetry in general - but especially with A/B testing, since you can put some data and science behind your hypotheses. You don't have to use AppInsights - but you do have to have some monitoring tool or framework in order to even contemplate A/B testing.

The other key is how you release the A and B sites or apps. Having traffic manager seamlessly divert customer traffic is an excellent way to do this since the customer is none the wiser about which version they are seeing - so they don't have to change their URLs or anything obscure. You could also use LaunchDarkly or some other feature flag mechanism - as long as your users don't have to change their usual way of accessing your app. This will give you "real" data. If users have to go to a beta site, they could change their behavior subconsciously. Maybe that isn't a big deal - but at least prefer "seamless" routing between A and B sites before you explicitly tell users to navigate to a new site altogether.

Happy testing!

Recently I was working with a customer that was struggling with test environments. Their environments are complex and take many weeks to provision and configure - so they are generally kept around even though some of them are not frequently used. Besides a laborious, error-prone manual install and configuration process that usually takes over 10 business days, the team has to maintain all the clones of this environment. This means that at least two senior team members are required just to maintain existing dev and test environments as well as create new ones.

Using Azure ARM templates and VSTS Release Management with Deployment Groups, we were able to show how we could spin up the entire environment in just under two hours. That's a 50x improvement in lead time! And it's more consistent since the entire process is automated and the scripts are all source controlled so there's auditability. This is a huge win for the team. Not only can they spin up an environment in a fraction of the time they are used to, they can now decommission environments that are not frequently used (some environments were only used twice a year). That means they have less maintenance to worry about. When they need an environment for a short time, they spin it up, used it and then throw it away. They've also disseminated "tribal knowledge" from a few team members' heads to a button click - meaning anyone can create a new environment now.

This was my first time working with a larger scale provisioning and configuration project that uses Deployment Groups - and this post documents some of the lessons that we learned along the way.

A Brief Glossary

Before we jump into the tips, I need to get some definitions out of the way. In VSTS, a Release Definition is made up of multiple Environments. Typically you see DEV, STAGE and PROD but you can have multiple "environments" that target the same set of machines.

The above VSTS release has three "environments":

• Infrastructure Provision
• Runs an ARM template to provision VMs, VNets, Storage and any other infrastructure required for the environment
• Infrastructure Config
• Configure the OS of each machine, DNS and any other "low-level" settings
• App Install
• Install and configure the application(s)

This separation also allows you to run the "Infrastructure Provision" environment and then set it to a manual trigger and just trigger the config environment - particularly useful when you're developing the pipeline, since you can skip environments that end up being no-ops but take a couple minutes to pass through.

Within an Environment, you can have 1..n phases. You specify tasks inside a phase - these are the smallest unit of work in the release.

In the above image, there are several phases within the "Infrastructure Config" environment. Each phase (in this case) is running a single task, but you can run as many tasks as you need for that particular phase.

There are three types of phases: agentless, agent-based or deployment-group based. You can think of agentless phases as phases that are executed on VSTS. Agent-based phases are executed on agent(s) in a build or release queue. Deployment Group phases are executed on all agents (with optional tag matching) within the specified Deployment Group. The agent for agent-based or deployment-group based is the same agent under the hood - the difference is that deployment group agents are only referenced through the Deployment Group while build/release agents are accessed through queues. You'd typically use agent queues for build servers or for "proxy servers" in releases (where the tasks are executing on the proxy but acting on other machines). Deployment Groups are used when you don't know the machines ahead of time - like when you're spinning up a set of machines in the cloud on demand. They also allow you to target multiple machines at the same time.

The VSTS Deployment agent joins a machine (this can be any machine anywhere that can connect to VSTS) to a Deployment Group. The agent is cross-platform (runs on DotNET Core) so it can run on practically any machine anywhere. It connects out to VSTS meaning you don't need to open incoming firewall ports at all. The agent runs on the machine and so any scripts you write can execute locally - which simplifies configuration dramatically. Executing remote instructions is typically much harder to do - you have to think about your connection and security and so on. Executing locally is much easier.

TL;DR - The Top 10 Tips and Tricks

Here are my top 10 tips and tricks:

1. Spin Up Azure VMs with the VSTS Deployment Agent Extension
1. This allows you to configure everything else locally on each machine
2. Use Tagging for Parallelization and Specialization
1. Tagging the VSTS agent allows you to repeat the same actions on many machines in parallel and/or distinguish machines for unique actions
3. Use Phases to Start New Sessions
1. Each phase in an Environment gets a new session, which is useful in a number of scenarios
4. Update Your PowerShell PackageProviders and Install DSC Modules
1. If you're using DSC, install the modules in a separate step to ensure that they are available when you run DSC scripts. You may need to update your Package Providers for this to work
5. Install Azure PowerShell and use the Azure PowerShell task
1. If you're going to be doing any scripting to Azure, you can quickly install Azure PowerShell so that you can use the Azure PowerShell task
6. Use PowerShell DSC for OS Configuration
1. Configuring Windows Features, firewalls and so on is best done with PowerShell DSC
7. Use Plain PowerShell for Application Install and Config
1. Expressing application state can be challenging - so use "plain" PowerShell for application install and config
8. Attaching Data Disks in Azure VMs
1. If you add data disks in your ARM template, you still need to mount them in the OS of the VM
9. Configuring DNS on Azure VNets
1. If you create an Active Directory Domain Controller or DNS, you'll need to do some other actions on the VNet too
10. Wait on machines when they reboot
1. If you reboot a machine and don't pause, the subsequent deployment steps fail because the agent goes offline.

In the next section I'll dig into each tip.

Tip 1: Spin Up Azure VMs with the VSTS Deployment Agent Extension

You can install the VSTS Deployment Agent (or just "the agent" for the remainder of this post) on any machine using a simple script. The script downloads the agent binary and configures it to connect the agent to your VSTS account and to the specified Deployment Group. However, if you're spinning up machines by using an ARM template, you can also install the agent via the VSTS extension. In order to do this you need a Personal Access Token (or PAT), the name of the VSTS account, the name of the Deployment Group and optionally some tags to tag the agent with. Tags will be important when you're distinguishing between machines in the same Deployment Group later on. You'll need to create the Deployment Group in VSTS before you run this step.

Here's a snippet of an ARM template that adds the extension to the Deployment Group:

{
    "name": "[parameters('settings').vms[copyIndex()].name]",
    "type": "Microsoft.Compute/virtualMachines",
    "location": "[resourceGroup().location]",
    "apiVersion": "2017-03-30",
    "dependsOn": [
      ...
    ],
    "properties": {
      "hardwareProfile": {
        "vmSize": "[parameters('settings').vms[copyIndex()].size]"
      },
      "osProfile": {
        "computerName": "[parameters('settings').vms[copyIndex()].name]",
        "adminUsername": "[parameters('adminUsername')]",
        "adminPassword": "[parameters('adminPassword')]"
      },
      "storageProfile": {
        "imageReference": "[parameters('settings').vms[copyIndex()].imageReference]",
        "osDisk": {
          "createOption": "FromImage"
        },
        "dataDisks": [
            {
                "lun": 0,
                "name": "[concat(parameters('settings').vms[copyIndex()].name,'-datadisk1')]",
                "createOption": "Attach",
                "managedDisk": {
                    "id": "[resourceId('Microsoft.Compute/disks/', concat(parameters('settings').vms[copyIndex()].name,'-datadisk1'))]"
                }
            }
        ]
      },
      "networkProfile": {
        "networkInterfaces": [
          {
            "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(parameters('settings').vms[copyIndex()].name, if(equals(parameters('settings').vms[copyIndex()].name, 'JumpBox'), '-nicpub', '-nic')))]"
          }
        ]
      }
    },
    "resources": [
      {
        "name": "[concat(parameters('settings').vms[copyIndex()].name, '/TeamServicesAgent')]",
        "type": "Microsoft.Compute/virtualMachines/extensions",
        "location": "[resourceGroup().location]",
        "apiVersion": "2015-06-15",
        "dependsOn": [
          "[resourceId('Microsoft.Compute/virtualMachines/', concat(parameters('settings').vms[copyIndex()].name))]"
        ],
        "properties": {
          "publisher": "Microsoft.VisualStudio.Services",
          "type": "TeamServicesAgent",
          "typeHandlerVersion": "1.0",
          "autoUpgradeMinorVersion": true,
          "settings": {
            "VSTSAccountName": "[parameters('vstsAccount')]",
            "TeamProject": "[parameters('vstsTeamProject')]",
            "DeploymentGroup": "[parameters('vstsDeploymentGroup')]",
            "Tags": "[parameters('settings').vms[copyIndex()].tags]"
          },
          "protectedSettings": {
            "PATToken": "[parameters('vstsPat')]"
          }
        }
      },
      ...

Notes:

• The extension is defined in the highlighted lines
• The "settings" section of the extension is where you specify the VSTS account name, team project name, deployment group name and comma-separated list of tags for the agent. You also need to supply a PAT that has access to join machines to the Deployment Group
• You can also specify a "Name" property if you want the agent name to be custom. By default it will be machineName-DG (so if the machine name is WebServer, the agent will be named WebServer-DG.

Now you have a set of VMs that are bare-boned but have the VSTS agent installed. They are now ready for anything you want to throw at them - and you don't need to worry about ports or firewalls or anything like that.

There are some useful extensions and patterns for configuring VMs such as Custom Script or Join Domain. The problem with these scripts is that the link to the script has to be either in a public place or in a blob store somewhere, or they assume existing infrastructure. This can complicate deployment. Either you need to publish your scripts publically or you have to deal with uploading scripts and generating SAS tokens. So I recommend just installing the VSTS agent and let it do everything else that you need to do - especially since the agent will download artifacts (like scripts and build binaries) as a first step in any deployment phase.

Tip 2: Use Tagging for Parallelization and Specialization

Tags are really important for Deployment Groups. They let you identify machines or groups of machines within a Deployment Group. Let's say you have a load balanced application with two webservers and a SQL server. You'd probably want identical configuration for the webservers and a completely different configuration for the SQL server. In this case, tag two machines with WEBSERVER and the other machine with SQL. Then you'll define the tasks in the phase  - when the phase runs, it executes all the tasks on all the machines that match the filter - for example, you can target all WEBSERVER machines with a script to configure IIS. These will execute in parallel (you can configure it to work serially if you want to) and so you'll only specify the tasks a single time in the definition and you'll speed up the deployment.

Be careful though: multiple tags use AND (not OR) logic. This means if you want to do something like join a domain on machines with WEBSERVER and SQL, you would think you could specify WEBSERVER, SQL as the tag filter in the phase tag filter. But since the tags are joined with an AND, you'll see the phase won't match any machines. So you'd have to add a NODE tag (or something similar) and apply it to both webservers and SQL machine and then target NODE for things you want to do on all the machines.

The above image shows the tag filtering on the Phase settings. Note too the parallelization settings.

Tip 3: Use Phases to Start New Sessions

At my customer we were using Windows 2012 R2 machines. However, we wanted to use PowerShell DSC for configuring the VMs and you need Windows Management Framework 5.0 to get DSC. So we executed a PowerShell task to upgrade the PowerShell to 5.x:

if ($PSVersionTable.PSVersion.Major -lt 5) {
    $powershell5Url = "https://go.microsoft.com/fwlink/?linkid=839516"
    wget -Uri $powershell5Url -OutFile "wmf51.msu"
    Start-Process .\wmf51.msu -ArgumentList '/quiet' -Wait
}

Notes:

• Line 1: This script checks the major version of the current PowerShell
• Lines 2,3: If it's less than 5, then the script downloads PowerShell 5.1 (the path to the installer can be update to whichever PowerShell version you need)
• Line 4: The installer is invoked with the quiet parameter

However, if we then called a task right after the update task, we'd still get the old PowerShell since all tasks within a phase are executed in the same session. We just added another phase with the same Deployment Group settings - the second phase started a new session and we got the upgraded PowerShell.

This doesn't work for environment variables though. When you set machine environment variables, you have to restart the agent. The VSTS team are working on providing a task to do this, but for now you have to reboot the machine. We'll cover how to do this in Tip 10.

Tip 4: Update Your PowerShell PackageProviders and Install DSC Modules

You really should be using PowerShell DSC to configure Windows. The notation is succinct and fairly easy to read and Windows plays nicely with DSC. However, if you're using custom modules (like xNetworking) you have to ensure that the modules are installed. You can pre-install all the modules so that your scripts can assume the modules are already installed. To install modules you'll need to update your Package Providers. Here's how to do it:

Import-Module PackageManagement
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force

You'll need to start a new Phase in order to pick up the new packages. Then you'll be able to install modules:

Install-Module -Name xActiveDirectory -Force
Install-Module -Name xNetworking -Force
Install-Module -Name xStorage -Force
Install-Module -Name xDSCDomainjoin -Force
Install-Module -Name xComputerManagement -Force

Not all machines need all the modules, but this step is so quick I found it easier to just enumerate and install all the modules anyway. That way I know that any machine could run any DSC script I throw at it.

Tip 5: Install Azure PowerShell

If you're going to do anything against Azure from the VMs (in our case we were downloading binaries from a blob store) then you'll want to use the Azure PowerShell task. This task provides an authenticated context (via a preconfigured endpoint) so you don't have to worry about adding passwords or anything to your script. However, for it to work, you'll need to install Azure PowerShell. Again this must be a separate phase so that subsequent phases can make use of the Azure cmdlets. To do this simply add a PowerShell task and run this line of script: Install-Module AzureRM -AllowClobber -Force

Tip 6: Use PowerShell DSC for OS Configuration

OS configuration can easily be specified by describing the state: is IIS installed or not? Which other OS roles are installed? So DSC is the perfect tool for this kind of work. You can use a single DSC script to configure a group of machines (or nodes, in DSC) but since we have the VSTS agent you can simply write your scripts for each machine using "node localhost". DSC script are also (usually) idempotent - so they work no matter what state the environment is in when the script executes. No messy if statements to check various conditions - DSC does it for you.

When you're doing DSC, you should first check if there is a "native" resource for your action - for example, configuring Windows Features uses the WindowsFeature resource. However, there are some custom actions you may want to perform. There are tons of extensions out there - we used xActiveDirectory to configure an Active Directory Domain Controller settings, for example.

There are times when you'll want to do some custom work that there simply is no custom module for. In that case, you'll need to use the Script resource. The script resource is composed of three parts: GetScript, TestScript and SetScript. GetScript is optional and should return the current state as an object if specified. TestScript should return a boolean - true for "the state is correct" or false for "the state is not correct". If TestScript returns a false, then the SetScript is invoked. Here's an example Script we wrote to configure SMB on a machine according to Security requirements:

Script SMBConfig
{
	GetScript = { @{ Result = Get-SmbServerConfiguration } }
	TestScript =
	{
		$config = Get-SmbServerConfiguration
		$needConfig = $config.EnableSMB2Protocol -and (-not ($config.EnableSMB1Protocol))
		if ($needConfig) {
				Write-Host "SMB settings are not correct." 
		}
		$needConfig
	}
	SetScript =
	{
		Write-Host "Configuring SMB settings" 
		Set-SmbServerConfiguration -EnableSMB1Protocol $false -Force
		Set-SmbServerConfiguration -EnableSMB2Protocol $true -Force
	}
}

Notes:

• Line 1: Specify the type of resource (Script) and a unique name
• Line 3: The GetScript returns an hash table with a Result property that describes the current state - in this case, the SMB settings on the machine
• Line 4: The start of the TestScript
• Line 6: Query the SMB settings
• Line 7: Determine if we need to configure anything or not - this is a check on the SMBProtocol states
• Lines 8-10: Write a message if we do need to set state
• Line 11: return the bool: true if the state is correct, false otherwise
• Lines 16-17: correct the state of the machine - in this case, set protocols accordingly

Tip 7: Use Plain PowerShell for Application Install and Config

Expressing application state and configuration as a DSC script can be challenging. I once wrote some DSC that could install SQL. However, I ended up using a Script resource - and the TestScript just checked to see if a SQL service was running. This check isn't enough to determine if SQL features are installed according to some config.

Instead of writing long Script resources, I just revert to "plain" PowerShell for app install and configuration. This is especially true for more complicated apps. Just make sure your script are idempotent - that is that they can run and succeed every time. For example, if you're installing a service, you may want to first check to see if the service exists before running the installer (otherwise the installer may fail since the service already exists). This allows you to re-run scripts again if other scripts fail.

Tip 8: Attaching Data Disks in Azure VMs

If you're creating data disks for your VMs, then you usually specify the size and type of disk in the ARM template. But even if you add a disk, you need to attach it in the OS. To do this, I used the xStorage DSC extension. This requires a disk number. When we started, the data disk was always disk 2. Later, we added Azure Disk Encryption - but this added another disk and so our disk numbers were off. We ended up needing to add some logic to determine the data disk number and pass that in as a parameter to the DSC configuration:

Configuration DiskConfig
{
	param
	(
		[Parameter(Mandatory)]
		[string]$dataDiskId
	)

	# import DSC Resources 
	Import-DscResource -ModuleName PSDscResources
	Import-DscResource -ModuleName xStorage

	Node localhost
	{
        LocalConfigurationManager
		{
			ActionAfterReboot = 'ContinueConfiguration'
			ConfigurationMode = 'ApplyOnly'
			RebootNodeIfNeeded = $true
		}

        xWaitforDisk DataDisk
        {
            DiskId = $dataDiskId
            RetryIntervalSec = 60
            RetryCount = 3
        }

        xDisk FVolume
        {
            DiskId = $dataDiskId
            DriveLetter = 'F'
            FSLabel = 'Data'
            DependsOn = "[xWaitforDisk]DataDisk"
        }
    }
}

# work out what disk number the data disk is on
$dataDisks = Get-Disk -FriendlyName "Microsoft Virtual Disk" -ErrorAction SilentlyContinue
if ($dataDisk -eq $null) {
	$dataDisk = Get-Disk -FriendlyName "Microsoft Storage Space Device" -ErrorAction SilentlyContinue
}
# filter to GPT partitions
$diskNumber = 2
Get-Disk | Out-Host -Verbose
$dataDisk = Get-Disk | ? { $_.PartitionStyle -eq "RAW" -or $_.PartitionStyle -eq "GPT" }
if ($dataDisk -eq $null) {
	Write-Host "Cannot find any data disks"
} else {
	if ($dataDisk.GetType().Name -eq "Object[]") {
		Write-Host "Multiple data disks"
		$diskNumber = $dataDisk[0].Number
	} else {
		Write-Host "Found single data disk"
		$diskNumber = $dataDisk.Number
	}
}
Write-Host "Using $diskNumber for data disk mounting"

DiskConfig -ConfigurationData .\ConfigurationData.psd1 -dataDiskId "$($diskNumber)"
Start-DscConfiguration -Wait -Force -Path .\DiskConfig -Verbose 

Notes:

• Lines 3-7: We specify that the script requires a dataDiskId parameter
• Lines 11,12: Import the modules we need
• Lines 23-28: Wait for disk with number $dataDiskId to be available (usually it was immediately anyway)
• Lines 30-36: Mount the disk and assign drive letter F with a label of Data
• Lines 41-43: Get potential data disks
• Lines 46-59: Calculate the data disk number, defaulting to 2
• Lines 62,63: Compile the DSC and the invoke the configuration manager to "make it so"

Tip 9: Configuring DNS on Azure VNets

In our example, we needed a Domain Controller to be on one of the machines. We were able to configure the domain controller using DSC. However, I couldn't get the other machines to join the domain since they could never find the controller. Eventually I realized the problem was a DNS problem. So we added the DNS role to the domain controller VM. We also added a private static IP address for the domain controller so that we could configure the VNet accordingly. Here's a snippet of the DSC script for this:

WindowsFeature DNS
{
        Ensure = "Present"
        Name = "DNS"
        DependsOn = "[xADDomain]ADDomain"
}

xDnsServerAddress DnsServerAddress
{
        Address        = '10.10.0.4', '127.0.0.1'
        InterfaceAlias = 'Ethernet 2'
        AddressFamily  = 'IPv4'
        DependsOn = "[WindowsFeature]DNS"
}

Notes:

• Lines 1-6: Configure the DNS feature
• Lines 8-14: Configure the network DNS NIC using the static private IP 10.10.0.4

Now we needed to configure the DNS on the Azure VNet to use the domain controller IP address. We used this script:

param($rgName, $vnetName, $dnsAddress)
$vnet = Get-AzureRmVirtualNetwork -ResourceGroupName $rgName -Name $vnetName
if ($vnet.DhcpOptions.DnsServers[0] -ne $dnsAddress) {
    $vnet.DhcpOptions.DnsServers = @($dnsAddress)
    Set-AzureRmVirtualNetwork -VirtualNetwork $vnet
}

Notes:

• Line 2: Get the VNet using the resource group name and VNet name
• Line 3: Check if the DNS setting of the VNet is correct
• Lines 4,5: If it's not, then set it to the internal IP address of the DNS server

This script needs to run as an Azure PowerShell script task so that it's already logged in to an Azure context (the equivalent of running Login-AzureRMAccount -ServicePrincipal). It's sweet that you don't have to provide any credentials in the script!

Now that we've set the DNS on the VNet, we have to reboot every machine on the VNet (otherwise they won't pick up the change). That brings us to the final tip.

Tip 10: Wait on Machines When They Reboot

You can easily reboot a machine by running this (plain) PowerShell: Restart-Machine -ComputerName localhost -Force. This is so simple that you can do it as an inline PowerShell task:

Rebooting the machine is easy: it's waiting for it to start up again that's more challenging. If you have a task right after the reboot task, the deployment fails since the agent goes offline. So you have to build in a wait. The simplest method is to add an agentless phase and add a Delay task:

However, you can be slightly more intelligent if you poll the machine states using some Azure PowerShell:

param (
    [string]$ResourceGroupName,
    [string[]]$VMNames = @(),
    $TimeoutMinutes = 2,
    $DelaySeconds = 30
)

Write-Host "Delay for $DelaySeconds seconds."
Start-Sleep -Seconds $DelaySeconds

if($VMNames.Count -eq 0)
{
    $VMNames = (Get-AzureRmVm -ResourceGroupName $ResourceGroupName).Name
    Write-Host "Getting VM names."
}

$seconds = 10
$desiredStatus = "PowerState/running"

foreach($vmName in $VMNames)
{
    $timer = [Diagnostics.Stopwatch]::StartNew()
    Write-Host "Getting statuses of VMs."
    $statuses = (Get-AzureRmVm -ResourceGroupName $ResourceGroupName -VMName $vmName -Status).Statuses
    $status = $statuses | Where-Object { $_.Code -eq $desiredStatus }
    while($status -eq $null -and ($timer.Elapsed.TotalMinutes -lt $TimeoutMinutes))
    {
        Write-Verbose "Retrying in $($seconds) seconds."
        Start-Sleep -Seconds $seconds
        $statuses = (Get-AzureRmVm -ResourceGroupName $ResourceGroupName -VMName $vmName -Status).Statuses
        $status = $statuses | Where-Object { $_.Code -eq $desiredStatus }
    }

    if($timer.Elapsed.TotalMinutes -ge $TimeoutMinutes)
    {
        Write-Error "VM restart exceeded timeout."
    }
    else
    {
        Write-Host "VM name $($vmName) has current status of $($status.DisplayStatus)."
    }
}

Notes:

• The script requires a resource group name, an optional array of machine names (otherwise it will poll all the VMs in the resource group), a delay (defaulted to 30 seconds) and a timeout (defaulted to 2 minutes)
• The script will delay for a small period (to give the machines time to start rebooting) and then poll them until they're all running or the timeout is reached.

This script has to run in an Azure PowerShell task in an agent phase from either the Hosted agent or a private agent - you can't run it in a Deployment Group phase since those machines are rebooting!

Conclusion

Deployment Groups are very powerful - they allow you to dynamically target multiple machines and execute configuration in a local context. This makes complex environment provisioning and configuration much easier to manage. However, it's always good to know limitations, gotchas and practical tips when designing a complex deployment workflow. Hopefully these tips and tricks make your life a bit easier.

Happy deploying!

If you've ever had to create a complex ARM template, you'll know it can be a royal pain. You've probably been tempted to split out your giant template into smaller templates that you can link to, only to discover that you can only link to a sub-template if the sub-template is accessible via some public URI. Almost all of the examples in the Template Quickstart repo that have links simply refer to the public Github URI of the linked template. But what if you want to refer to a private repo of templates?

Using Blob Containers

The solution is to use blob containers. You upload the templates to a private container in an Azure Storage Account and then create a SAS token for the container. Then you create the full file URI using the container URI and the SAS token. Sounds simple, right? Fortunately with VSTS Release Management, it actually is easy.

As an example, let's look at this template that is used to create a VNet and some subnets. First we'll look at the VNet template (the linked template) and then how to refer to it from a parent template.

The Child Template

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "vnetName": {
      "type": "string"
    },
    "vnetPrefix": {
      "type": "string"
    },
    "subnets": {
      "type": "object"
    }
  },
  "variables": {
  },
  "resources": [
    {
      "name": "[parameters('vnetName')]",
      "type": "Microsoft.Network/virtualNetworks",
      "location": "[resourceGroup().location]",
      "apiVersion": "2016-03-30",
      "dependsOn": [],
      "tags": {
        "displayName": "vnet"
      },
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[parameters('vnetPrefix')]"
          ]
        }
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/virtualNetworks/subnets",
      "tags": {
        "displayName": "Subnets"
      },
      "copy": {
        "name": "iterator",
        "count": "[length(parameters('subnets').settings)]"
      },
      "name": "[concat(parameters('vnetName'), '/', parameters('subnets').settings[copyIndex()].name)]",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[parameters('vnetName')]"
      ],
      "properties": {
        "addressPrefix": "[parameters('subnets').settings[copyIndex()].prefix]"
      }
    }
  ],
  "outputs": {
  }
}

Notes:

• There are 3 parameters: the VNet name and prefix (strings) and then an object that contains the subnet settings
• The first resource is the VNet itself - nothing complicated there
• The second resource uses copy to create 0 or more instances. In this case, we're looping over the subnets.settings array and creating a subnet for each element in that array, using copyIndex() as the index as we loop

There's really nothing special here - using a copy is slightly more advanced, and the subnets parameter is a complex object. Otherwise, this is plain ol' ARM json.

The Parent Template

The parent template has two things that are different from "normal" templates: it needs two parameters (containerUri and containerSasToken) that let it refer to the linked (child) template and it invokes the template by specifying a "Microsoft.Resources/deployments" resource type. Let's look at an example:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "containerUri": {
      "type": "string"
    },
    "containerSasToken": {
      "type": "string"
    }
  },
  "variables": {},
  "resources": [
    {
      "apiVersion": "2017-05-10",
      "name": "linkedTemplate",
      "type": "Microsoft.Resources/deployments",
      "properties": {
        "mode": "incremental",
        "templateLink": {
          "uri": "[concat(parameters('containerUri'), '/Resources/vNet.json', parameters('containerSasToken'))]",
          "contentVersion": "1.0.0.0"
        },
        "parameters": {
          "vnetName": { "value": "testVNet" },
          "vnetPrefix": { "value": "10.0.0.0/16" },
          "subnets": {
            "value": {
              "settings": [
                {
                  "name": "subnet1",
                  "prefix": "10.0.0.0/24"
                },
                {
                  "name": "subnet2",
                  "prefix": "10.0.1.0/24"
                }
              ]
            }
          }
        }
      }
    }
  ],
  "outputs": {}
}

Notes:

• There are two parameters that pertain to the linked template: the containerUri and the SAS token
• In the resources, there is a "Microsoft.Resources/deployment" resource - this is how we invoke the child template
• In the templateLink, the URI is constructed by concatenating the containerUri, the path to the child template within the container, and the SAS token
• Parameters are passed inline - note that even simple parameters look like JSON objects (see vNetName and vnetPrefix)

Initially I tried to make the subnets object an array: but this blew up on the serialization. So I made an object called "settings" that is an array. So the subnets value property is an object called "settings" that is an array. You can look back at the child template to see how I dereference the object to get the values: to get the name of a subnet, I use "parameters('subnet').settings[index].name" (where index is 0 or 1 or whatever). The copy uses the length() method to get the number of elements in the array and then I can use copyIndex() to get the current index within the copy.

Of course the parent template can contain other resources - I just kept this example really simple to allow us to zoom in on the linking bits.

Source Structure

Here's a look at how I laid out the files in the Azure Resource Group project:

You can see how the vNet.json (the child template) is inside a folder called "Resources". I use that as the relative path when constructing the URI to the child template.

The Release Definition

Now all the hard work is done! To get this into a release, we just create a storage account in Azure (that we can copy the templates to) and we're good to go.

Now create a new release definition. Add the repo containing the templates as a release artifact. Then in your environment, drop two tasks: Azure File Copy and Azure Resource Group Deployment. We configure the Azure File Copy task to copy all our files to the storage account into a container called templates. We also need to give the task two variable names: one for the containerUri and one for the SAS token:

Once this task has executed, the templates will be available in the (private) container with the same folder structure as we have in Visual Studio.

On the next task, we can select the parent template as the template to invoke. We can pass in any parameters that are needed - at the very least, we need the containerUri and SAS token, so we pass in the variables from the previous task using $() notation:

Now we can run the release and voila - we'll have a vNet with two subnets.

Conclusion

Refactoring templates into linked templates is good practice - it's DRY (don't repeat yourself) and can make maintenance of complicated templates a lot easier. Using VSTS Release Management and a storage container, we can quickly, easily and securely make linked templates available and it all just works ™.

Happy deploying!

I recently worked with a customer that had some containers running Python code. The code was written by data scientists and recently a dev had been hired to help the team get some real process in place. As I was helping them with their CI/CD pipeline (which used a Dockerfile to build their image, publish to Azure Container Registry and then spun up the containers in Azure Container Instances), I noted that there were no unit tests. Fortunately the team was receptive to adding tests and I didn't have to have a long discussion proving that they absolutely need to be unit testing.

Python Testing with PyTest

I've done a bit of Python programming, but am by no means an expert. However, after a few minutes of searching I realized there are a ton of good Python frameworks out there. One I came across is PyTest (which is cool since it's very pythonic). This great post (Python Testing 101: PyTest) from Andy Knight even had a Github repo with some code and tests. So when I got back to the hotel, I forked the repo and was able to quickly spin up a build definition in VSTS that runs the tests - with code coverage!

Continuous Testing with VSTS

At a high level, here are the steps that your build has to perform:

• Clone the repo
• Install PyTest and other packages required
• Run PyTest, instructing it to output the results to (JUnit) XML and produce (Cobertura) coverage reports
• Publish the test results
• (Optional) Fix the styling of the HTML coverage reports
• Publish the coverage reports

Why the "fix styles" step? When we create the HTML coverage reports (so that you can see which lines of code are covered and which are not) we publish them to VSTS. However, for security reasons VSTS blocks the css styling when viewing these reports in the Build Summary page. Fortunately if you inline the styles, you get much prettier reports - so we use a node.js package to do this for us.

Fortunately I've already done this and published the VSTS build definition JSON file in my forked repo. Here's how you can import the code, import the CI definition and run it so you can see this in action yourself.

Import the Repo from Github

The source code is in a Github repo - no problem! We'll import into VSTS and then we can mess with it. We can even fork the Github repo, then import it - that way we can sumbit pull requests on Github for changes we make. In this case, I'll just import the repo directly without forking.

Log in to VSTS and navigate to the Code hub. Then click on the repo button in the toolbar and click Import repository.

Enter the URL and click Import.

Now we have the code in VSTS! Remember, this is just Git, so this is just another remote (that happens to be in VSTS).

Import the Build Definition

Now we can import the build definition. First, navigate to the Github repo and clone it or download the PythonTesting-CI.json file. Then open VSTS and navigate to a team project and click on Build and Release (in the Blue toolbar at the top) to navigate to the build and release hub. Click on Builds (in the grey secondary toolbar) and click the "+Import" button.

In the import dialog, browse to the json file you downloaded previously and click Import.

You'll then see the build definition - there are a couple things that need to be fixed, but the steps are all there.

Note how the Agent queue is specifying "Hosted Linux Preview" - yep, this is running on a Linux VM. Now you don't have to do this, since Python will run on Windows, but I like the Linux agent - and it's fast too! Rename the definition if you want to.

Now we'll fix the "Get sources" section. Click on "Get sources" to tell the build where to get the sources from. Make sure the "This account" tile is selected and then set the repository to "python-testing-101" or whatever you named your repo when you imported. You can optionally set Tag sources and other settings.

One more addition: click on the Triggers tab and enable the CI trigger:

Now you can click Save and queue to queue a new build! While it's running, let's look at the tasks.

1. Install Packages: this Bash task uses curl to get the pip install script, then install pip and finally install pytest and pytest-cov packages. If you're using a private agent you may not have to install pip, but pip doesn't come out of the box on the Hosted Linux agent so that's why I install it.
2. Run Tests: invoke python -m pytest (which will run any tests in the current folder or subfolders), passing --junitxml=testresults.xml and the pycov args to create both an XML and HTML report
3. Install fix-styles package: this just runs "npm install" in the root directory to install this node.js package (you can see this if you look at the package.json file)
4. Fix styles: we run the "fix" script from the package.json file, which just invokes the fix-styles.js file to inline the styling into the HTML coverage reports
5. Publish Test Results: we publish the XML file, which is just a JUnit test result XML file. Note how under Control Options, this task is set to run "Even if a previous task has failed, unless the build was cancelled". This ensures that the publish step works even when tests fail (otherwise we won't get test results when tests fail).
6. Publish Code Coverage: This task published the XML (Cobertura) coverage report as well as the (now style-inlined) HTML reports

Really simple! Let's navigate to the build run (click on the build number that you just queued) and - oh dear, the tests failed!

Seems there is a bug in the code. Take a moment to see how great the test result section is - even though there are failing tests. Then click on Tests to see the failing tests:

All 4 failing tests have "subtract" in them - easy to guess that we have a problem in the subtract method! If we click on a test we can also see the stack trace and the failed assertions from the test failure. Click on the "Bug" button above the test to log a bug with tons of detail!

Just look at that bug: with a single button click we have exception details, stack traces and links to the failing build. Sweet!

Now let's fix the bug: click on the Code hub and navigate to example-py-pytest/com/automationpanda/example and click on the calc_func.py file. Yep, there's a problem in the subtract method:

Click on the Edit button and change that pesky + to a much better -. Note, this isn't what you'd usually do - you'd normally create a branch from the Bug, pull the repo, fix the bug and push. Then you'd submit a PR. For the sake of this blog, I'm just fixing the code in the code editor.

Click the Commit button to save the change. In "Work items to link" find the Bug that we created earlier and select it. Then click Commit.

The commit will trigger a new build! Click on Build and you'll see a build is already running. Click on the build number to open the build.

This time it's a success! Click on the build number to see the report - this time, we see all the tests are passing and we have 100% coverage - nice!

If you click on "Code Coverage*" just below the header, you'll see the (prettified) HTML reports. Normally you won't have 100% coverage and you'll want to see which methods have coverage and which don't - you would do so by browsing your files here and noting which lines are covered or not by the color highlighting:

Also note that we can see that this build is related to the Bug (under Associated work items). It's almost like we're professional developers…

Conclusion

Just because there is "Visual Studio" in the name, it doesn't mean that VSTS can't do Python - and do it really, really well! You get detailed test logging, continuous integration, code coverage reports and details - and for very little effort. If you're not testing your Python code - just do it ™ with VSTS!

This week at MVP summit, I showed some of my colleagues a trick that I use to manage identity hell. I have several accounts that I use to access VSTS and the Azure Portal: my own Microsoft Account (MSA), several org accounts and customer org accounts. Sometimes I want to open a release from my 10th Magnitude VSTS account so that I can grab some tasks to put into CustomerX VSTS release. The problem is that if I open the 10M account in a browser, and then open a new browser, I have to sign out of the 10M account and sign in with the CustomerX account and then the windows break… identity hell.

At first I used to open InPrivate or Incognito windows. That gave me the ability to get to 4 different profiles: IE and IE InPrivate, Chrome and Chrome Incognito. But then my incognito windows don't have cached identities or history or anything that I like to have in my browser. Hacky - very hacky.

Solution: Chrome People

About 2 years ago I stumbled onto Chrome People (or Profiles). This really simple "trick" has been fantastic and I almost never open Incognito anymore. In the upper right of the Chrome chrome (ahem) there is a little text that tells you what your current "person" is:

Click that text to open the People hub:

Here you can see that I have 5 People: ColinMSA, 10M, AdminNWC and NWC and another customer profile. To switch profiles, I just click on the name. To add a person, just click "Manage people".

I can easily add a new person from this view - and I can assign an icon to the person.

When you create a new person, Chrome creates a shortcut to that person's browser on the desktop. I end up clicking on that and adding it to my taskbar:

If I want to open up the Azure Portal or VSTS using my MSA, I click the ColinMSA icon and I'm there. If I need to open my customer VSTS or Portal, I just click that icon. Each window is isolated and my identities don't leak. Very neat, very clean. Under the hood, the shortcuts just add a small arg to the Chrome.exe launcher: --profile-directory="Profile 1". The first profile is Default, the second is Profile 1, the third Profile 2 and so on.

Final Thoughts

You can also do something similar in FireFox, but I like Chrome. This simple trick helps me sort out my identity hell and I can quickly switch to different identity contexts without having to sign in and out all the time. For my MSA I sign into my Google account, but I don't do that for the other browsers. All in all it's a great way to manage multiple identities.

Happy browsing!

I've been working on some pretty complicated infrastructure deployment pipelines using my release management tool of choice (of course): VSTS Release Management. In this particular scenario, we're deploying a set of VMs to a region. We then want to deploy exactly the same setup but in a different region. Conceptually, this is like duplicating infrastructure between different datacenters.

Here's what the DEV environment in a release could look like:

If we're duplicating this to 5 regions, we'd need to clone the environment another 4 times. However, that would mean that any updates to any tasks would need to be duplicated over all 5 regions. It's easy to forget to update or to fat-finger a copy - isn't there a better way to maintain sets of tasks? I'm glad you asked…

DRY - Don't Repeat Yourself

DRY (Don't Repeat Yourself) is a common coding practice - any time you find yourself copying code, you should extract it into a function so that you only have to maintain that logic in a single place. We can do the same thing in a release (or build) using Task Groups. Task Groups are like functions that you can call from releases (or builds) from many places - but maintain in a single place. Just like functions, they have parameters that you can set when you "call" them. Click the selector (checkmark icon to the right of each task) to select all the tasks you want to group, right-click and select "Create task group":

A popup asks for the name of the Task Group and bubbles up all the parameters that are used in the tasks within the group. You can update the defaults and descriptions and click Create (helpful hint: make variables for all the values so that the variable becomes the default rather than a hard-coded value - this will make it easier to re-use the Task Group when you clone environments later):

So far, so good:

However, there's a snag: looking at the parameters section, you'll notice that we don't have any parameter for the Azure Service Endpoint. Let's open the tasks and update the value in the dropdown to $(AzureSubscription):

Now you can see that the parameter is bubble up and surfaced as a parameter on the Task Group - it even has the dropdown with the Service Endpoints. Nice!

Consuming the Task Group

Open up the release again. You'll see that you now have a new parameter on the Task Group: the AzureSubscription. We'll select the DEV sub from the dropdown.

Also note how the phase is now a single "task" (which is just a call to the Task Group). Under the hood, when the release is created, Release Management deletes the task group and replaces it with the tasks from the Task Group - so any values that are likely to change or be calculated on the fly should be variables.

Let's now clone the DEV environment to UAT-WESTUS and to UAT-EASTUS.

If we edit the UAT-WESTUS, we can edit the service endpoint (and any other parameters) that we need to for this environment:

Excellent! Now we can update the Task Group in a single place even if we're using it in dozens of environments. Of course you'd need to update the other parameter values to have environment-specific values (Scopes) in the Variables section.

Conclusion

Task Groups are a great way to keep your releases (or builds) DRY - even allowing you to parameterize the Azure Service Endpoint so that you can duplicate infrastructure across different subscriptions or regions in Azure.

Happy deploying!

There are a lot of ALM MVPs that advocate the "One Team Project to Rule Them All" when it comes to Visual Studio Team Services (VSTS) and Team Foundation Server (TFS). I've been recommending it for a long time to any customer I work with. My recommendation was based mostly on experience - I've experienced far too much pain when organizations have multiple Team Projects, or even worse, multiple Team Project Collections.

While on a flight to New Jersey I watched a fantastic talk by Allan Kelley titled Continuous Delivery and Conway's Law. I've heard about Conway's Law before and know that it is applied to systems design. A corollary to Conway's Law, referred to as Inverse Conway Maneuver, is to structure your organization intentionally to promote a desired system architecture. This has a lot of appeal to me with regards to DevOps - since DevOps is not a tool or a product, but a culture: a way of thinking.

With these thoughts in mind, as I was watching Kelley's talk I had an epiphany: you can perform an Inverse Conway Maneuver by the way you structure your VSTS account or TFS install!

What is Conway's Law?

In April 1968, Mel Conway published a paper called "How Do Committees Invent?" The central thesis of this paper is this: "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." In other words, the design of your  organizational and team structures will impose itself on your system designs. At least, if they are out of sync, you will experience friction. The Inverse Conway Maneuver recognizes Conway's Law and makes it intentional: use organizational and team structure to promote desired systems design. For example, distributed teams tend to develop more modular products, while centralized teams tend to develop monoliths.

Historical side-note: Conway's paper was rejected by Harvard Business Review in 1967 since Conway had "failed to prove his thesis". Ironically, a team of MIT and Harvard Business School researchers published a paper in 2015 which found "strong evidence" to support the hypothesis.

How does this apply to VSTS and TFS? I'll explain, but it's awkward having to type "VSTS and TFS". For the remainder of this blog I'll just write VSTS - but the same principles apply to TFS: the VSTS account is like a TFS Team Project Collection. If we equate VSTS account to Team Project Collection, then the rest of the hierarchy (Team Project, Team etc.) is exactly equivalent. In short, when I say VSTS account I also mean TFS Team Project Collection.

One Objective: Deliver Value to End Users

In days gone by, IT was a service center to Business. Today, most organizations are IT companies - irrespective of the industry they operate in. Successful businesses are those that embrace the idea that IT is a business enabler and differentiator, not just a cost center. There should be very little (if any) division between "business" and "IT" - there is one team with one goal: deliver value to customers. Interestingly the definition of DevOps, according to Donovan Brown (Principal DevOps Manager at Microsoft), is "the union of people, process and products to enable continuous delivery of value to our endusers" (emphases mine).

One Objective means everyone is aligned to the overall goal of the business. If you look at two of the Principles behind the Agile Manifesto:

• Business people and developers must work together daily throughout the project.
• The best architectures, requirements, and designs emerge from self-organizing teams.

you'll see a common theme: aligning everyone to the One Objective. The point I'm making is that there needs to be a "one team" culture that permeates the organization. DevOps is cultural before it's about tools and products. But putting the thinking into practice is no easy task. Fortunately, having the correct VSTS structures supports an Inverse Conway Maneuver.

One Team Project

So how do you use VSTS for an Inverse Conway Maneuver? You have a single VSTS account with a single Team Project.

Having all the work in a single Team Project allows you to view work at a "portfolio" (or organizational) level - that is, across the entire organization. This is (currently) impossible to do with multiple VSTS accounts and very difficult with multiple Team Project Collections. Even viewing portfolio level information with  multiple Team Projects can be difficult. Work item queries are scoped to Team Projects by default; widgets, dashboards, builds, releases, package feeds, test plans - these all live at Team Project level. If you have multiple Team Projects you've probably experienced a fragmented view of work across the organization. Interestingly, that's probably not only from a VSTS point of view, but this structure (by Conway's Law) is probably responsible for silos within the organization.

Hence the recommendation for a single "Team Project to Rule Them All." Not only will this allow anyone to see work at a portfolio level, but this allows teams to share source repositories, build definitions, release definitions, reports and package feeds. It's a technical structure that encourages the One Objective.

Teams

I can hear you already: "But I have 500 developers/analysts/testers/DBAs/Ops managers (let's say engineers, shall we?) - how can I possibly organize them under a single team project?" That's where Teams come in. Teams allow organizations to organize work into manageable sets. When you're an engineer and you want to deliver value, you probably only need a general idea of the One Objective, rather than having to know the minutia of every bit of work across the entire organization. Having your team's work in a separate ring-fenced area allows you to focus on what you need day-to-day. You can go up to the portfolio level when you need a wider context - but you probably don't need that every day. Leadership will more likely spend most of their time looking at work at the portfolio level rather than all the way down to the minutia of the team-level work.

So how should you organize your teams? Again, Conway's Law is going to have enormous impact here. Do you have a 3-tier application? Then you might be tempted to create a DBA Team, a Service Team and a UI Team. Perhaps create a Mobile team and a Data Analytics Team too. Surely that's reasonable, right?

The answer, to quote Consultese (the dialect of the consultant) is: It Depends. Perhaps that is the way to go since that is how your application is architected. But that could be boxing you in: horizontally composed teams violate the Agile principle of cross-functional teams. A better approach is to have your teams composed around functional area or module. Where possible, they should be loosely coupled. Again by Conway's Law this will start reflecting in your app architecture - and you'll start seeing your applications become loosely coupled services. Have you ever wondered why micro-services are so popular today? Could it be the Agile movement started to break huge monolithic organizations into small, loosely-coupled, cross-functional and self-organizing teams, and now we're starting to see that reflected in our architecture? Inverse Conway Maneuvers at work.

In short, create Teams around functional areas and use Area Paths to denote ownership of that work. If an Epic/Feature/Story belongs to a team, put it in the Area Path for that team and it appears on their backlogs. Another tip is that your Area Paths should be durable (long-lived) while your work items should not: work items should have a definite start and end date. Don't make an Epic for "Security" since that's not likely to end at a specific date. Rather, have an Area Path for Security and place work items in that area path.

Organizational Dimensions in VSTS

There are four axes that most organizations use to organize work: functional area, iteration, release and team. Unfortunately, VSTS only really gives us two: Area and Iteration. While Release Management in VSTS is brilliant, there isn't yet a first-class citizen for the concept of a Release. And while you can create a custom Team Field in TFS and slice teams on that field, you can't do so in VSTS, so you have to munge Team and Area Path together. In my experience it's best not to fight these limits: use Area Path to denote Team, use iterations to time-box, and if you really need a Release concept, add a custom field.

Areas and Work Item States

Organizations will still need inter-team communication, but this should be happening far less frequently that intra-team communication. That's why we optimize for intra-team communication. It's also why co-locating a team wherever possible is so important. If you do this, then by Conway's Law you are more likely to end up with modules that are stable, resilient, independent and optimized.

We've already established that vertical Teams are tied to Area Paths. Each Team "owns" a root area path, typically with the same name as the Team. This is the area path for the team's backlog. The team can then create sub-areas if they need do (leave this up to the team - they're self-organizing after all). Kanban boards can be customized at the team level, so each team can decide on whatever columns and swim-lanes they want in order to optimize their day-to-day work. Again, leave this up to the team rather than dictating from the organizational level.

Work Item states can't be customized at Team level - only at the Team Project level. If you only have a single Team Project, that means every team inherits the same work item states. This is actually a good thing: a good design paradigm is to have standard communication protocols, and to have services have good contracts or interfaces, without dictating what the internals of the service should look like. This is reflected by the common "language" of work item state, but let's teams decide how to manage work internally via customized Kanban boards. Let Conway's Law work for you!

Iterations

While teams should have independent backlogs and areas, they should synchronize on cadence. That is, it's best to share iterations. This means that teams are independent during a sprint, but co-ordinate at the end of the sprint. This enforces the loose coupling: teams are going to have dependencies and you still want teams to communicate - you just want to streamline that communication. Sharing iterations and synchronizing on that heartbeat is good for the Teams as well as the software they're delivering.

Enterprise Alignment vs Team Autonomy

The VSTS team have a single Team Project Collection for their work. They speak about Enterprise Alignment vs Team Autonomy. I heard a great illustration the other day: the Enterprise is like a tanker - it takes a while to turn. Agile Teams are like canoes - they can turn easily. However, try to get 400 canoes pointed in the same direction! As you work to self-organizing teams, keep them on the One Objective so that they're pointed in the same direction. Again, that's why I like the One Team Project concept: the Team Project is the One Direction, while Teams still get autonomy in their Team Areas for daily work.

Organizing Source Code, Builds, Releases, Test Plans and Feeds

If you have a single Team Project, then you'll have a challenge: all repositories, builds, releases, test plans and package feeds are in a single place. Builds have the concept of Build Folders, so you can organize builds by folders if you need to. However, repos, releases, test plans and feeds don't have folders. That means you'll need a good naming strategy and make use of Favorites to manage the noise. In my opinion this is a small price to pay for the benefits of One Team Project.

Security

Often I come across organizations that want to set up restrictions on who can see what. In general: don't do this! Why do you care if Team A can see Team B's backlog? In fact it should be encouraged! Find out what other teams are working on so that you can better manage dependencies and eliminate double work. Same principle with Source Code: why do you care if Team C and see Team D's repos?

There are of course exceptions: if you have external contractors, you may want to restrict visibility for them. In VSTS, deny overrides allow, so in general, leave permissions as "Not Set" and then explicitly Deny groups when you need to. The Deny should be the exception rather than the rule - if not, you're probably doing something wrong.

Of course you want to make sure you Branch Policies (with Pull Requests) in your source code and approval gates in your Releases. This ensures that the teams are aware of code changes and code deployments. Don't source control secrets - store them in Release Management or Azure Key Vault. And manage by exception: every action in VSTS is logged in the Activity Log, so you can always work out who did what after the fact. Trust your teams!

Conclusion

Don't fight Conway's Law: make it work for you! Slim down to a single VSTS Account with a single Team Project and move all your Teams into that single Team Project. Give Teams the ability to customize their sub-areas, backlogs, boards and so on: this gives a good balance of Enterprise Alignment and Team Autonomy.

Here is a brief summary of how you should structure your VSTS account:

• A single VSTS Account (or TFS Team Project Collection)
• A single Team Project
• Multiple Teams, all owning their root Area Path
• Shared Iteration Paths
• Use naming conventions/favorites for Repos, Releases, Test Plans and Feeds
• Use folders for organizing Build Definitions
• Enforce Branch Policies in your Repos and use Approval Gates in Release Management
• Have simple permissions with minimal DENY (prefer NOT SET and ALLOW)

Happy delivering!

Visual Studio Team Services (VSTS) is a cloud platform. That means it's publicly accessible from anywhere - at least, by default. However, Enterprises that are moving from TFS to VSTS may want to ensure that VSTS is only accessed from a corporate network or some white-list of IPs.

To enable conditional access to VSTS, you'll have to have an Azure Active Directory (AAD) backed VSTS account. The conditional access is configured on AAD, not on VSTS itself, since that's where the authentication is performed. For instructions on how to do this, read this MSDN article.

Audit Access

However, this may be a bit heavy-handed. Perhaps you just want to audit access that isn't from a white-list of IPs, instead of blocking access totally. If you're an administrator, you may have come across the Usage page in VSTS. To get there, navigate to the landing page for your VSTS account, click the gear icon and select Usage:

This page will show you all your access to VSTS. To see the IPs, you have to add the "IP Address" column in the column options:

Nice, but what about other users? To get that you have to use the VSTS REST API.

Dumping an Exception Report with PowerShell

There is an (undocumented) endpoint for accessing user access. It's https://<your account>.visualstudio.com/_apis/Utilization/UsageSummary with a whole string of query parameters. And since it's a REST call, you'll need to be authenticated so you'll have to supply a header with a base-64 encoded Personal Access Token (PAT).

Using PowerShell, you can make a call to the endpoint and then filter results where the IP is not in the white-list of IPs. Fortunately for you, I've made a gist for you, which you can access here. When you call the script, just pass in your account name, your PAT, the start and end date and the white-list of IPs. Any access from an IP not in this list is dumped to a CSV.

Limitations

There are some limitations to the API:

1. You'll need to be a Project Collection or Account admin to make this call (since there's no documentation, I'm guessing here).
2. You can only go back 28 days, so if you need this as an official exception report you'll have to schedule the run.

Conclusion

VSTS knows the client IP for any access. Using the API, you can dump a list of access events that are not from a white-list of IPs.

Happy auditing!

Releases almost always require some kind of credentials - from service credentials to database usernames and passwords. There are a number of ways to manage credentials in VSTS release management. In this post I'll look at a couple of common techniques. For brevity, I'm going to refer to secrets as a proxy for secrets and credentials.

Don't Store Secrets in Source Control

One bad practice you want to steer away from is storing secrets in source control. A lot of teams I work with have their build process create multiple environment-specific packages, using tools like config transforms. I like to get teams to think of build and release as two separate (but linked) processes:

The point is that the build should be totally environment agnostic. Good unit tests use mocking or fakes, so they shouldn't need environment-specific information. That means that they need to create packages that are, as I like to call them, swiss cheese - they need to have holes or tokens that can have environment-specific values injected at deployment time. You don't need tokens if your deployment process is capable of doing variable substitution - like the IIS Deployment on Machine Group task or Azure App Service Deployment task that can both do inline variable replacement (see my earlier post on how to do this - and this also now applies to the IIS Deployment on Machine Group task).

Centralized vs Decentralized Secret Management

I see two broad categories of secret management: centralized and decentralized. Centralized secret management has the advantage of specifying/updating the secret once, even if it's used in many places - but has the disadvantage of being managed by a small subset of users (admins typically). This can also be an advantage, but can be a bottleneck. Decentralized secret management usually ends up in duplicated secrets (so updating a password leaves you hunting for every occurrence of that password) but removes the bottleneck of centralized management. Choosing a method will depend on your culture, auditing requirements and management overhead.

Decentralized Secret Management

Decentralized secret management is the easiest to consider, and there's really only one way to do it: in your release definition, define your secrets as variables that are locked and you're done. If you need to use the same secret in multiple definitions, you just create the same variable. Of course if you change the value, you have to change it in each release that uses it. But you don't have to log a ticket or wait for anyone to change the value for you - if it changes, you update it in place for each release and you're done.

Centralized Secret Management

There are three types of centralized secret management: Azure KeyVault, Variable Groups and Custom Key Vault. Let's consider each method.

The KeyVault and Variable Group methods both define a Variable Group - but if you use KeyVault, you manage the values in KeyVault rather than in the Variable Group itself. Otherwise they are exactly the same.

Go to the VSTS release hub and click on Library to see variable groups. Create a new Variable Group and give it a name. If this is a "plain" Variable Group, define all your secrets and their values - don't forget to padlock the values that you want to hide. If you're using KeyVault, first define a Service Endpoint in the Services hub for authenticating to the KeyVault. Then come back and link the Variable Group to the KeyVault and specify which Secrets are synchronized.

Now when you run define a release, you link the Variable Group (optionally scoping it) and voila - you have a centralized place to manage secrets, either directly in the Variable Group or via KeyVault.

The variable group can be linked to many releases, so you only ever have to manage the values in one place, irrespective of how many releases reference them. To use the values, just use $(SecretName) in your tasks.

The last method is Custom Key Vault. I worked with a customer a few months back that used some sort of third-party on-premises key vault. Fortunately this vault had a REST API and we were able to create a custom task that fetched secrets from this third-party key vault. If you do this, you need to remember to add in a custom task to get the values, but this was an elegant solution for my customer since they already had an internal key vault.

Conclusion

There are a number of ways to manage secrets and credentials in VSTS/TFS. The most robust is to use Azure KeyVault, but if you don't have or don't want one you can use Variable Groups in-line. Whatever method you choose, just make sure you don't store any secrets in source control!

Happy releasing!

I've done a fair amount of ARM template authoring. It's not as bad as XML, but the JSON can get laborious. A number of my colleagues use Terraform templates and I was recently on a project that was using these templates. I quickly did a couple PluralSight Terraform classes to get up to speed and then started hacking away. In this post I'll jot down a couple thoughts about how we structure Terraform projects and how to deploy them using VSTS. The source code for this post is on Github.

Stacks

When we create Terraform projects, we divide them into "stacks". These are somewhat independent, loosely-coupled components of the full infrastructure we're deploying. Let's take the example of an Azure App Service with deployment slots that connects to an Azure SQL database and has Application Insights configured. In this scenario, we have three "stacks": SQL, WebApp and AppInsights. We then have an additional "stack" for the Terraform remote state (an Azure blob) and finally a folder for scripts. Here's what our final folder structure looks like:

Follow the instructions in the README.md file for initializing the backend using the state folder. Note that backend.tfvars and secrets.tfvars are ignored by the .gitignore file so should not be committed to the repo.

Workspace = Environment

Thinking ahead, we may want to create different environments. This is where Terraform workspaces come in handy - we use them to represent different environments. That way we can have a single template and can re-use it in multiple environments. So if you look at webapp/variables.tf, you'll see this snippet:

variable "stack_config" {
  type = "map"

  default = {
    dev = {
      name             = "webapp"
      rg_name_prefix   = "cd-terra"
      plan_name_prefix = "cdterra"
      app_name_prefix  = "cdterraweb"
    }

    uat = {
      name             = "webapp"
      rg_name_prefix   = "cd-terra"
      plan_name_prefix = "cdterra"
      app_name_prefix  = "cdterraweb"
    }
  }
}

You can see how we have different maps for different workspaces. To consume the environment (or workspace) specific variables, we use the locals resource in our main.tf scripts. The common format is something like this:

locals {
  env        = "${var.environment[terraform.workspace]}"
  secrets    = "${var.secrets[terraform.workspace]}"
  stack      = "${var.stack_config[terraform.workspace]}"
  created_by = "${var.created_by}"
  stack_name = "${local.stack["name"]}"

  env_name   = "${terraform.workspace}"
  release    = "${var.release}"
  ...
  app_name   = "${local.stack["app_name_prefix"]}-${local.env_name}"
}

We create local variables for env, secrets and stack by dereferencing the appropriate workspace's map values. Now we can use "${local.app_name}" and the value will be an environment-specific value. Also note how we have a variable called "release" - we add this as a tag to all the Azure resources being created so that we can tie the resource to the release that created/updated it.

Releases in VSTS

Now we get to the really interesting bit: how we run the templates in a release pipeline in VSTS. I tried a couple of marketplace Terraform extensions, but wasn't happy with the results. The most promising one was Peter Groenewegen's extension, but it was not workspace aware and while it did download Terraform so that I could run Terraform using the hosted agents, it didn't preserve Terraform on the path. I eventually ditched it for a plain ol' bash script. To perform Terraform operations, we have to:

1. Replace tokens in the release.tfvars file
2. Download the Terraform executable
3. Run the Terraform apply bash script for each stack in order

I ended up using the Hosted Ubuntu 1604 hosted agent for the agent phase - I don't know for sure, but I suspect this is running in a container - in any case, it's super fast to start up and execute. Because I'm running the build on Linux, I wrote the scripts I used in bash - but you can easily create equivalent PowerShell scripts if you really want to - though VSTS will run bash scripts on Windows agents just fine - although the paths are different.

Artifact

For the release to work, it needs access to the terraform templates. I create a new release and use the source repo as the incoming artifact with an alias "infra". You can add multiple artifacts, so if you're going to deploy code after deploying infrastructure, then you can add in the build as another artifact.

Variables

In the release, I define a number of variables:

I tried to use the environment variable format for ARM_ACCESS_KEY, ARM_CLIENT_ID etc. but found I had to supply these explicitly - which I do via the release.tfvars file. The release.tfvars file has tokens that are replaces with the environment values at deploy time. If you add more environment-specific variables, then you need to add their tokens in the tfvars file and add the variable into the variables section of the release. One last note: I use $(Release.EnvironmentName) as the value for the Environment variable - but this needs a different value for the "destroy" environment (each environment I have in the pipeline has a corresponding "destroy" environment for destroying the resources). You can see how I specify "dev" as the Environment name for the "destroy dev" environment.

Download Terraform

This is only required if you're using the hosted agents - if you're using a private agent, then you're better off downloading terraform and adding it to the PATH. However, in the scripts folder I have a bash script (download-terraform.sh) that downloads Terraform using curl (from a URL specified in a variable) and untars it to the path specified in the TerraformPath variable. From that point on, you can use $(TerraformPath)\terraform for any Terraform operations.

Applying a Stack

The scripts folder contains the script for applying a stack (run-terraform.sh). Let's dig into the script:

#!/bin/bash -e

echo "*********** Initialize backend"
echo "access_key = \"${1}\"" > ../backend.tfvars
$2/terraform init -backend-config=../backend.tfvars -no-color

echo ""
echo "*********** Create or select workspace"
if [ $($2/terraform workspace list | grep $3 | wc -l) -eq 0 ]; then
  echo "Create new workspace $3"
  $2/terraform workspace new $3 -no-color
else
  echo "Switch to workspace $3"
  $2/terraform workspace select $3 -no-color
fi

echo ""
echo "*********** Run 'plan'"
$2/terraform plan --var-file=../global.tfvars --var-file=../release.tfvars -var="release=$4" --out=./tf.plan -no-color -input=false

echo ""
echo "*********** Run 'apply'"
$2/terraform apply -no-color -input=false -auto-approve ./tf.plan

Notes:

• Lines 3 - 5: Initialize the backend. Output the access_key to the root backend.tfvars file (remember this won't exist in the repo since this file is ignored in .gitignore)
• Lines 7-15: Create or select the workspace (environment)
• Lines 17-19: Run terraform plan passing in global variables from global.tfvars, environment-specific variables now encapsulated in release.tfvars and pass in the release number (for tagging)
• Lines 21-23: Run terraform apply using the plan generated in the previous command

Destroying a Stack

Destroying a stack is almost the same as applying one - the script (run-terraform-destroy.sh) just does a "plan -destroy" to preview the operations before calling terraform destroy.

The Release Steps

Now we can see what these blocks look like in the pipeline. Here's a pipeline with a dev and a "destroy dev" environment:

The dev environment triggers immediately after the release is created, while the "destroy dev" environment is a manual-only trigger.

Let's see what's in the dev environment:

There you can see 5 tasks: replace variables, download Terraform and then an apply for each stack (in this case we have 3 stacks). The order here is important only because the WebApp stack reads output variables from the state data of the SQL and AppInsights deployments (to get the AppInsights key and SQL connection strings). Let's take a closer look at each task:

Replace Variables

For this I use my trusty ReplaceTokens task from my build and release extension pack. Specify the folder (the root) that contains the release.tfvars file and the file search format, which is just release.tfvars:

Next we use a Shell task to run the download Terraform script, which expects the path to install to as well as the URL for the Terraform binary to download:

Finally we use a Shell task for each stack - the only change is the working folder (under Advanced) needs to be the stack folder - otherwise everything else stays the same:

Success! Here you can see a run where nothing was changed except the release tag - the templates are idempotent, so we let Terraform figure out what changes (if any) are necessary.

Conclusion

Terraform feels to me to be a more "enterprise" method of creating infrastructure as code than using pure ARM templates. It's almost like what TypeScript is to JavaScript - Terraform has better sharing and state awareness and allows for more maintainable and better structured code. Once I had iterated a bit on how to execute the Terraform templates in a release, I got it down to a couple really simple scripts. These could even be wrapped into custom tasks.

Happy deploying!

I've written before about Selenium testing (Parallel Testing in a Selenium Grid with VSTS and Running Selenium Tests in Docker using VSTS and Release Management). The problem with these solutions, however, is that you need a VM! However, I was setting up a demo last week and decided to try to solve this challenge using Azure Container Instances (ACI), and I have a neat solution.

So why ACI? Why not a Kubernetes cluster? This solution would definitely work in a k8s cluster, but I wanted something more light-weight. If you don't have a k8s cluster, spinning one up just for Selenium testing seemed a bit heavy handed. Also, I discovered that ACI now lets you spin up multiple containers in the same ACI group via a yaml file.

I've created a GitHub repo with all the source code for this post so you can follow along - it includes the following:

• a sample app (boilerplate File->New Project->MVC) with unit tests
• .NET Core Selenium test project and tokenized runsettings files
• ARM template for Azure Web App
• Tokenized yaml file for the ACI infrastructure
• CI yaml build definition (yay for build-as-code!)
• json release definition (release-as-code isn't yet available, so we'll have to do the release via the designer)

Architecture

Once we've built and package the application code, the release process (which we model in the Release Definition) is as follows:

1. Provision infrastructure - both for the app as well as the ACI group with the Selenium hub, worker nodes and VSTS agents
2. Deploy the app
3. Install .NET Core and execute "dotnet test", publishing test results
4. Tear down the ACI

The component architecture for the ACI is pretty straight-forward: we run containers for the following:

1. A Selenium Hub - this will listen for test requests and match the requested capabilities with a worker node
2. Selenium nodes - Chrome and Firefox worker nodes
3. VSTS Agent(s) - these connects to VSTS and execute tests as part of a release

To run in parallel I'm going to use a multi-configuration phase - so if you want to run Firefox and Chrome tests simultaneously you'll need at least 2 VSTS agents (and 2 pipelines!). ACI quotas will let you spin up groups with up to 4 processors and 14GB of memory, so for this example we'll use a Selenium hub, 2 workers and 2 VSTS agents each using .5 processor and .5GB memory.

One concern is security - how do you secure the test rig since it's running in the public cloud? Fortunately, this solution doesn't require any external ports - the networking is all internal (the worker nodes and VSTS agent connect to the hub using "internal" ports) and the VSTS agent itself connects out to VSTS and does not require incoming connections. So we don't have to worry about securing endpoints - we don't even expose any!

Let's take a look at the architecture of this containers in ACI:

Notes:

• We are running 5 containers: selenium-hub, selenium-chrome and selenium-firefox and 2 vsts-agents
• The only port we have to think about is the hub port, 4444, which is available internally - we don't have any external ports
• The vsts-agent is connected to our VSTS account, and again no external port is required

The trick to getting multiple Selenium nodes within the group to connect to the hub was to change the ports that they register themselves on - all taken care of in the yaml definition file.

As for the tests themselves, since I want to run the tests inside a vsts-agent container, they have to be written in .NET Core. Fortunately that's not really a big deal - except that some methods (like the screenshot method) are not yet implemented in the .NET Core Selenium libraries.

We'll get into the nitty-gritty of how to use this ACI in a release pipeline for testing your app, but before we do that let's quickly consider how to deploy this setup in the first place.

Permanent or Transient?

There are two ways you can run the ACI - permanently or transiently. The permanent method spins up the ACI and leaves it running full-time, while the transient method spins the ACI up as part of an application pipeline (and then tears it down again after testing). If you are cost-sensitive, you probably want to opt for the transient method, though this will add a few minutes to your releases. That shouldn't be too much of a problem since this phase of your pipeline is running integration tests which you probably expect to take a bit longer. If you are optimizing for speed, then you probably just want to spin the ACI up in a separate pipeline and just assume it's up when your application pipeline runs. Fortunately the scripts/templates are exactly the same for both methods! In this post I'll show you the transient method - just move the tasks for creating/deleting the ACI to a separate pipeline if you prefer the ACI to be up permanently.

Putting it all Together

To put this all together, you need the following:

• A VSTS account
• An Azure subscription
• A Service Endpoint to the Azure sub from VSTS
• Colin's ALM Corner Build Extension pack installed onto your VSTS account
• An agent queue in VSTS (the ACI VSTS agents will register onto this queue)
• A personal access token (PAT) to your VSTS account

For the source code you can link directly to my GitHub repo. However, if you want to change the code, then you'll either need to fork it to your own GitHub account or import the repo into your VSTS account.

The Build

The build is really simple - and since it's yaml-based, the code is already there. To create a build definition, enable the YAML build preview on your VSTS account, then browse to the Build page. Create a new YAML build and point to the .vsts-ci.yml file. The build compiles code, versions the assemblies, runs unit tests with code coverage and finally publishes the web app as a webdeploy package:

The Release

You'll need to import the release definition to create it. First download the WebApp.ReleaseDefinition.json file (or clone the repo) so that you have the file on disk. Now, if you're using the "old" release view, just navigate to the Releases page and click the + button at the top of the left menu, then select "Import release pipeline". If you're using the new preview release view, you'll need to create a dummy release (since the landing page doesn't show the toolbar). Once you've created a dummy release, click the "+ New" button in the toolbar and click "Import a pipeline". Then browse to the WebApp.ReleaseDefinition.json file and click import. Once it's imported, you'll need to fix up a few settings:

Variables

Click on the Variables tab and update the names for:

• RGName - the name of the resource group for all the infrastructure
• WebAppName - the name of the web app (must be globally unique in Azure)
• ACIName - the name for the Azure Container Instance
• Location - I'd suggest you leave this on WestUS for the ACI quotas
• VSTSAccount - the name of your VSTS account (i.e. the bit before .visualstudio.com)
• VSTSPool - the name of the agent pool that you created earlier
• VSTSToken - your PAT. Make sure to padlock this value (to make it secret)

Artifacts

Click on the Pipeline tab top open the designer. You're going to have to delete and recreate the artifacts, since the id's are specific to my VSTS, so I cleared them in the definition json file. The primary artifact (so add this first) is your web app build - so add a new artifact of type "Build" and point to the WebApp build. Make sure the "Source alias" of this artifact is set to "WebApp" to preserve the paths in the tasks. You can also enable the CD trigger (to queue a release when a new build is available) if you want to. Now add another artifact - this time point to the source code repo (either on GitHub or your VSTS account) and alias this artifact as "infra" to preserve paths.

Job Queues

Now click on the Tasks tab. Click on the "Provision Infrastructure" job header and then select "Hosted Linux Preview" for the agent pool (we're running some Azure CLI commands via bash, so we need an Ubuntu agent). You can repeat this for the last job "Tear down ACI".

The "Deploy using WebDeploy" task requires a Windows agent, so change the queue on this job to "Hosted VS2017". Finally, change the "Run Tests" queue to the agent queue you created earlier (with the same name as the VSTSPool variable).

Azure Endpoints

You'll need to click on each "Azure" task (the Azure CLI tasks, the Azure Resource Group Deployment task and the Azure App Service Deploy task and configure the correct Azure endpoint:

You should now be able to save the definition - remove "Copy" from the name before you do!

The Jobs

Let's have a quick look at the tasks and how they are configured:

Provision Infrastructure

This job provisions infrastructure for the web app (via ARM template) as well as the ACI (via Azure CLI). The first task executes the ARM template, passing in the WebAppName. The template creates a Free tier App Service Plan and the App service itself. Next we replace some tokens in the ACI yaml definition file - this is somewhat akin to a Kubernetes pod file. In the file we specify that we require 5 containers: the Selenium hub (which opens port 4444), the worker nodes (one firefox and one chrome, running on different ports and connecting to the hub via localhost:4444) and 2 VSTS agents (with environment variables for the VSTS account, pool and PAT) so that the agent can connect to VSTS. I had to specify agent names since both containers get the same hostname, so if you don't and the agents have the same name, the 2nd agent would override the 1st agent registration.

Finally we invoke the Azure CLI task using an inline script to create the ACI using the yaml file:

The script itself is really a one-liner to "az container create" and we pass in the resource group name, the ACI name and the path to the yaml file.

Deploy using WebDeploy

The deploy job is a single task: Deploy Azure App Service. This has to run on a windows agent because it's invoking webdeploy. We specify the App name from the variable and point to the webdeploy zip file (the artifact from the build).

Of course a real application may require more deployment steps - but this single step is enough for this demo.

Run Tests

This job should be executing on the agent queue that you've configured in variables - this is the queue that the ACI agents are going to join in the first job. The first task installs the correct .NET core framework. We then replace the tokens in the runsettings file (to set the browser and the BaseURL). Then we execute "dotnet test" and finally publish the test results.

You'll notice that I have unset "Publish test results" in the dotnet test task. This is because the run is always published as "VSTest Test Run" - there's no way to distinguish which browser the test run is for. We tweak the test run title in the Publish Test Results step:

You'll also notice that we only have a single job - so how does the parallelization work? If you click on the Job name, you'll see that we've configured the parallelization settings for the job:

We're "splitting" the values for the variable "Browser" - in this case it's set to "chrome,firefox". In other words, this job will spawn twice - once for Browser=chrome and once for Browser=firefox. I've set the maximum number of agents to 2 since we only have 2 anyway.

Teardown ACI

Finally we tear down the ACI in a single Azure CLI inline script where we call "az container delete" (passing in the resource group and ACI names):

To ensure that this job always runs (even if the tests fail) we configure the Advanced options for the job itself, specifying that it should always run:

Run It!

Now that we have all the pieces in place, we can run it! Once it has completed, we can see 2 "Run Test" jobs were spawned:

If we navigate to the Test tab, we can see both runs (with the browser name):

Conclusion

Using ACI we can get essentially serverless parallel Selenium tests running in a release pipeline. We're only charged for the compute that we actually used in Azure, so this is a great cost optimization. We also gain parallelization or just better test coverage (we are running the same tests in 2 browsers). All in all this proved to be a useful experiment!

Happy testing!

I spend a fair amount of time in consoles - specifically PowerShell and Bash (Windows Subsystem for Linux) on my Windows 10 machine. I also work with Git - a lot. So having a cool console that is Git aware is a must. I always recommend Posh-Git (a PowerShell prompt that shows you which Git branch you're on as well as the branch status). At Ignite this I saw some zsh consoles in VS Code on Mac machines. So I wondered if I could get my consoles to look as cool. And it's not just about the looks - seeing your context in the console is a productivity booster!

It turns out that other than installing some updated fonts for both PowerShell and Bash, you can get pretty sweet consoles fairly easily.

Updating Fonts

The fonts that the custom shells use are UTF-8, so you'll need UTF-8 fonts installed. You'll also need so-called "powerline" fonts. Fortunately, there's a simple script you can run to install a whole bunch of cool fonts that will work nicely on your consoles.

Here are the steps for installing the fonts. Open a PowerShell and enter the following commands:

git clone https://github.com/powerline/fonts.git
.\install.ps1

This took about 5 minutes on my machine.

PowerShell

So to pimp out your PowerShell console, you'll need to install a couple modules: Posh-Git and Oh-My-Posh. Run Install-Module Posh-Git and Install-Module Oh-My-Posh. Once both modules are installed, you need to edit your $PROFILE (you can run code $PROFILE to quickly open your profile file in VSCode). Add the following lines:

Install-Module posh-git
Install-Module oh-my-posh
Set-Theme Paradox

You can of course choose different themes - run Get-Theme to get a list of themes. One last thing to do - set the background color of your PowerShell console to black (I like to make the opacity 90% too).

Now if you cd to a git repo, you'll get a Powerline status. Sweet!

Bash

You can do the same thing for your Bash console. I like to use fish shell so you'll have to install that first. Once you have fish installed, you can install oh-my-fish - a visual package manager for fish (and yes, oh-my-posh is a PowerShell version of oh-my-fish). Once oh-my-fish is installed, use it to install themes. You can install agnoster by running omf install agnoster - I like bobthefish, so I just run omf install bobthefish. Now my bash console is pimped too!

Solarized Theme

One more change you may want to make: update your console colors to the Solarized theme. To do that, follow the instructions from this repo.

Conclusion

If you're going to work in a console frequently, you may as well work in a pretty one! Oh-My-Fish and Oh-My-Posh let you quickly and easily get great-looking consoles, and Posh-Git adds in Git context awareness. What's not to love?

Happy console-ing!

I'm currently doing some work with a customer that is integrating between ServiceNow and Azure DevOps (the artist formerly known as VSTS). I quickly spun up a development ServiceNow instance to play around a bit. One of the use-cases I could foresee was a release gate that only allows a release to continue if a Change Request (CR) is in the Implement state. So I had to do some investigation: I know there are a few out-of-the-box Azure DevOps release gates, including a REST API call - but I knew that you could also create a custom gate. I decided to see if I could create the gate without expecting the release author having to know the REST API call to ServiceNow or how to parse the JSON response!

Follow along to see the whole process - or just grab the code in the Github repo.

Finding the ServiceNow REST API

Part One of my quest was to figure out the REST API call to make to ServiceNow. The ServiceNow documentation is ok - perhaps if you understand ServiceNow concepts (and I don't have deep experience with them) then they're fine. But I quickly felt like I was getting lost in the weeds. Add to that many, many versions of the product - which all seem to have different APIs. After a couple hours I did discover that the ServiceNow instance has a REST API explorer - but I'm almost glad I didn't start there as you do need some knowledge of the product in order to really use the explorer effectively. For example, I was able to query the state of the CR if I had its internal sys_id, but I didn't expect the user to have that. I wanted to get the state of the CR by its number - and how to do that wasn't obvious from the REST API explorer.

Anyway, I was able to find the REST API to query the state of a Change Request:

https://<instance>.servicenow.com/api/now/table/change_request?sysparm_query=number=<number>&sysparm_fields=state&sysparm_display_value=true

A couple notes on the query strings:

• sysparm_query lets me specify that I want to query the change_request table for the expression "number=<number>", which lets me get the CR via its number instead of its sys_id
• sysparm_fields lets me specify which fields I want returned - in this case, just the state field
• sysparm_value=true expands the enums from ints to strings, so I get the "display value" of the state instead of the state ID

The next problem is authentication - turns out if you have a username and password for your ServiceNow instance, you can include a standard auth header using BasicAuth (this is over HTTPS, so that's ok). I tested this with curl and was able to get a response that looks something like this:

{"result":[{"state":"Implement"}]}

Creating a Custom Release Gate Extension

Now that I know the REST API call to ServiceNow, I turned to how to Part Two of my quest: create a custom Release Gate extension. Fortunately, I had Microsoft DevLabs' great Azure DevOps Extension extension as a reference (this was originally from Jesse Houwing) - and I use this all the time to package and publish my own Azure DevOps Build and Release extension pack.

It turns out that the release gate "task" itself is pretty simple, since the entire task is just a JSON file which specifies its UI and the expression to evaluate on the response packet. The full file is here but let's examine the two most important parts of this task: the "inputs" element and the "execution" element. First the inputs:

"inputs": [
  {
    "name": "connectedServiceName",
    "type": "connectedService:ServiceNow",
    "label": "Service Now endpoint",
    "required": true,
    "helpMarkDown": "Service Now endpoint connection."
  },
  {
    "name": "crNumber",
    "type": "string",
    "label": "Change Request number",
    "defaultValue": "",
    "required": true,
    "helpMarkDown": "Change Request number to check."
  },
  {
    "name": "validState",
    "type": "string",
    "label": "State",
    "defaultValue": "Implement",
    "helpMarkDown": "State that the CR should be in to pass the gate.",
    "required": true
  }
]

Notes:

• connectedServiceName is of type "connectedService:ServiceNow". This is the endpoint used to call the REST API and should handle authentication.
• crNumber is a string and is the CR number we're going to search on
• validState is a string and is the state the CR should be in to pass the gate

Given those inputs, we can look at the execute element:

"execution": {
  "HttpRequest": {
    "Execute": {
      "EndpointId": "$(connectedServiceName)",
      "EndpointUrl": "$(endpoint.url)/api/now/table/change_request?sysparm_query=number=$(crNumber)&sysparm_fields=state&sysparm_display_value=true",
      "Method": "GET",
      "Body": "",
      "Headers": "{\"Content-Type\":\"application/json\"}",
      "WaitForCompletion": "false",
      "Expression": "eq(jsonpath('$.result[0].state')[0], '$(validState)')"
    }
  }
}

Notes:

• The execution is an HttpRequest
• Endpoint is set to the connectedService input
• EndpointUrl is the full URL to use to hit the REST API
• The REST method is a GET
• The body is empty
• We're adding a Content-Type header of "application/json" - notice that we don't need to specify auth headers since the Endpoint will take care of that for us
• The expression to evaluate is checking that the state field of the first result is set to the value of the validState variable

And that's it! Let's take a look at the connected service endpoint, which is defined in the extension manifest (not in the task definition):

{
  "id": "colinsalmcorner-snow-endpoint-type",
  "type": "ms.vss-endpoint.service-endpoint-type",
  "targets": [
    "ms.vss-endpoint.endpoint-types"
  ],
  "properties": {
    "name": "ServiceNow",
    "displayName": "Service Now",
    "helpMarkDown": "Create an authenticated endpoint to a Service Now instance.",
    "url": {
      "displayName": "Service Now URL",
         "description": "The Service Now instance Url, e.g. `https://instance.service-now.com`."
    },
    "authenticationSchemes": [
    {
      "type": "ms.vss-endpoint.endpoint-auth-scheme-basic",
      "inputDescriptors": [
        {
          "id": "username",
          "name": "Username",
          "description": "Username",
          "inputMode": "textbox",
          "isConfidential": false,
          "validation": {
            "isRequired": true,
            "dataType": "string",
            "maxLength": 300
          }
        },
        {
          "id": "password",
          "name": "Password",
          "description": "Password for the user account.",
          "inputMode": "passwordbox",
          "isConfidential": true,
          "validation": {
            "isRequired": true,
            "dataType": "string",
            "maxLength": 300
          }
        }
      ]
    }
  ]
}

Notes:

• Lines 2-6: specify that this contribution is of type Service Endpoint
• Line 8: name of the endpoint type - this is referenced by the gate in the endpoint input
• Lines 9-10: description and help text
• Line 11-14: specify a URL input for this endpoint
• The rest: specify the authentication scheme for the endpoint

By default the ms.vss-endpoint.endpoint-auth-scheme-basic authentication scheme adds an Authorization header to any request made to the URL of the service endpoint. The value of the header is a base64 encoded munge of user:password. It's great that you don't have to mess with this yourself!

Putting It All Together

Now we have the service endpoint and the gate, we're ready to publish and install the extension! The readme.md in the repo has some detail on this if you want to try your own (or make changes to the code from mine), or you can just install the extension that I've published if you want to use the gate as-is. If you do publish it yourself, you'll need to change the publisher and the GUIDs before you publish.

For the release to work, you'll need to make the CR a variable. I did this by adding the variable and making it settable at queue time:

Now when I queue the release, I have to add the CR. Of course you could imagine a release being queued off from an automated process, and that can pass the CR as part of the body of the REST API call to queue the release. For now, I'm entering it manually:

So how do we specify the gate? Edit the release and click on the pre- or post-approval icon for the environment and open the Gates section. Click the + to add a new gate and select the "Change Request Status" gate. We can then configure the endpoint, the CR number and the State we want to pass on:

To create an endpoint, just click on "+ New" next to the Service Now endpoint drop-down - this will open a new tab to the Service Endpoints page where you can add a new ServiceNow endpoint.

Note how we set the Change Request number to the variable $(ChangeRequestNumber). That way this field is dynamic.

Finally, set the "Evaluation options" to configure the frequency, timeout and other gate settings:

Once the release runs, we can see the Gate invocations and results:

Note that the Gate has to pass twice in a row before it's successful and moves the pipeline on.

Conclusion

Creating release gates as extensions is not too hard once you have some of the bits in place. And it's a far better authoring experience than the out of the box REST API call - which leaves you trying to mess with auth headers and parsing JSON responses. If you want to get release authors to really fully utilize the power of gates, do them a solid and wrap the gate in an extension!

Happy gating!

I remember when I first learned about Git circa 2012. I was skeptical - you can change history? What kind of source control system let you change history? However, it seemed to have huge momentum and so I started learning how to use it. Once you get over the initial learning curve - and there is one when you switch from centralized version control systems like Team Foundation Version Control (TFVC) or Subversion - I started to see the beauty of Git. And now I believe that teams can benefit enormously if they migrate to Git. I believe that so strongly that I spoke about this very topic at VSLive! in Orlando earlier this month.

In this post I want to detail why I think migrating to Git makes sense, common objections I hear, and some common ways you can migrate to Git. Migrating to Git make business sense as well as technical sense - so I'll call out business value-adds along the way. I'll primarily be talking about migrating from TFVC, but similar principles apply if you're migrating from other centralized source control systems.

Why Git?

There are several reasons why I think Git is essential for modern teams:

1. Branches are cheap
2. Merging is better
3. Code review is baked in via Pull Request
4. Better offline workflow

Cheap Branches

The primary reason I love Git is that branches are cheap. We'll get to the technical reasons why this important next - but the main business benefit of cheap branches lies in the ability to easily isolate (and later merge) development streams. That should be exciting since it means that small changes can be completed, merged and deployed without having to be held hostage by larger, longer-running changes. Delays cost, so anything that eliminates delays is good!

In centralized version control, a branch is a complete copy of the source - so typically teams keep the number of branches small. With Git, branches are essentially pointers, so creating branches is cheap. This means teams can create a lot of branches. Why does this make a difference anyway? The idea of a branch is to isolate code changes. Typical TFVC branching strategy is "DEV-MAIN-PROD". This is an attempt to isolate code in development (DEV) from code that's being tested (MAIN) and code that's running in production (PROD). That seems at first glance to be exactly what we want branches for - however, there's a catch: what if we have two or ten or twenty features in development? I coach teams to check in early, check in often - but that means that at times the code that's checked in will be unstable. Teams expect this at in the DEV branch. In fact, there's a term for how stable a branch is: hardness. The DEV branch is considered "soft" since it's not always stable - while PROD is supposed to be "hard" - that is, stable. But this branching strategy is flawed in that it isolates code at too coarse a level. What we really want is to isolate more granularly - especially if we want to deploy smaller features when they're complete without having to wait for larger features to be ready for deployment.

Git allows teams to create a branch per feature - also commonly referred to as topic branching. You don't want to do this when each branch is an entire copy of the code-base - but since Git branches are pointers, we can create branches liberally. By using a good naming convention and periodically cleaning branches that are stale (that haven't been updated for long periods) teams can get very good at isolating changes, and that makes their entire application lifecycle more resilient and more agile and minimize costly delays.

Better Merging

Merge debt can also be costly - the further away two branches diverge, the more costly and risky merging them becomes. Again, thinking in "business terms", this means you can move faster, with better quality - and what business doesn't want that?

Let's imaging you have 20 features in flight on a single DEV branch, and you somehow manage to coordinate a merge when all the features are ready to go, you'll probably spend a lot of time working through the merge since there are so many changes. Also, features that are completed quickly are forced to wait until the slowest feature is complete - which is a lot of waste. Or teams decide to merge anyway, knowing that they're merging incomplete code.

Also, when a file changes in a Git repo, Git records the entire file, not just the diffs (like TFVC). This means that merging between arbitrary branches works. With TFVC, branches have to be related to merge - or you could try a dreaded "baseless merge", which is very error-prone. Even though Git stores the entire file for a change, it does so very efficiently, but because of this merging is far easier for the Git.

Empirically I find that Git teams have fewer merge conflicts and merge issues than TFVC teams. Let's now imaging that we are using Git and have 20 features in flight - and 3 are ready to be deployed, but we want to test them. If we want to test them individually, no problem - we do a build off the branch which is master (the stable code) plus the branch changes. We can queue 3 builds and test each feature in isolation. We can also merge any branch into any of the others (something you can't easily do in unrelated branches in TFVC), so we can also test them together and make sure that there are no breaking changes in the merge - even before we merge each branch to master! This let's teams deploy features much more rapidly and frequently, eliminating waste along the way.

Code Reviews

One of GitHub's engineers introduced the concept of Pull Requests (PRs) and it's since become ubiquitous in the Git world - even though you don't typically do PRs in your local Git repo. The PR lets developers advertise that their code is ready to be merged. Policies (and reviews) can be built around the PR so that only quality code is merged into master. PRs are dynamic - so if I am the reviewer and comment on some code that a developer has submitted in a PR, the developer can fix the code and I can see the changes "live" in the PR. In contrast, TFVC lets you submit a Code Review work item (only through the Visual Studio IDE) and if the code needs to be changed, a new Code Review needs to be created. The whole code review process is clunky and laborious. However, I find PRs to be unobtrusive - they let us check code quickly, respond and adapt, and finally merge in a really natural manner. The Azure DevOps PR interface is fantastic - and if you add branch policies (available in Azure DevOps) you can enforce links to work items, comment resolution, build verification and even external system checks before a PR is merged. This lets teams "shift left" and build quality into their process in a natural, powerful and unobtrusive manner.

Better Offline

Git is a distributed version control system - it's designed to be used locally and synchronized to a central repo for sharing changes. As such, disconnected workflows are natural and powerful - and since cloning a repository gets the entire history of the repo from day 0, and I can branch and merge locally, the disconnected experience is excellent. TFVC used to require connection to the server to do most source control operations - with local workspaces (circa 2013) some source control operations can be performed offline, but you still need to be connected to the server to branch and merge.

Common Objections

There are four common objections I often hear to migrating to Git:

1. I can overwrite history
2. I have large files
3. I have a very large repo
4. I don't want to use GitHub
5. There's a steep learning curve

Overwriting History

Git technically does allow you to overwrite history - but (as we know from Spiderman) with great power comes great responsibility! If your teams are careful, they should never have to overwrite history. And if you're synchronizing to Azure DevOps you can also add a security rule that prevents developers from overwriting history (you need the "Force Push" permission enabled to actually sync a repo that's had rewritten history). The point is that every source control system works best when the developers using it understand how it works and which conventions work. While you can't overwrite history with TFVC, you can still overwrite code and do other painful things. In my experience, very few teams have managed to actually overwrite history.

Large Files

Git works best with repos that are small and that do not contain large files (or binaries). Every time you (or your build machines) clone the repo, they get the entire repo with all its history from Day 0. This is great for most situations, but can be frustrating if you have large files. Binary files are even worse since Git just can't optimize how they are stored. That's why Git LFS was created - this lets you separate large files out of your repos and still have all the benefits of versioning and comparing. Also, if you're used to storing compiled binaries in your source repos - stop! Use Azure Artifacts or some other package management tool to store binaries you have source code for. However, teams that have large files (like 3D models or other assets) you can use Git LFS to keep your code repo slim and trim.

Large Repos

This used to be a blocker - but fortunately the engineers at Microsoft have been on a multi-year journey to convert all of Microsoft's source code to Git. The Windows team has a repo that's over 300GB in size, and they use Git for source control! How? They invented Virtual File System (VFS) for Git. VFS for Git is a client plugin that lets Git think it has the entire repo - but only fetches files from the upstream repo when a file is touched. This means you can clone your giant repo in a few seconds, and only when you touch files does Git fetch them down locally. In this way, the Windows team is able to use Git even for their giant repo.

Git? GitHub?

There is a lot of confusion about Git vs GitHub. Git is the distributed source control system created by Linus Torvalds in 2005 for the Linux kernel. If you create a repo, you have a fully functioning Git repo on your local machine. However, to share that code, you need to pick a central place that developers can use to synchronize their repos - so if I want your changes, you'd push your changes to the central repo, and I'd pull them from there. We're still both working totally disconnected - but we're able to share our code via this push/pull model. GitHub is a cloud service for hosting these sorts of centralized repos - made famous mostly because it's free for open source projects (so you can host unlimited public repos). You don't have to use GitHub to use Git - though it's pretty much the de-facto platform for open source code. They do offer private repos too - but if you're an enterprise, you may want to consider Azure Repos since you get unlimited private repos on Azure Repos. You can also create Git repos in Team Foundation Server (TFS) from TFS 2015 to TFS 2019 (now renamed to Azure DevOps Server).

Learning Curve

There is a learning curve - if you've never used source control before you're probably better off when learning Git. I've found that users of centralized source control (TFVC or SubVersion) battle initially to make the mental shift especially around branches and synchronizing. Once developers grok how Git branches work and get over the fact that they have to commit and then push, they have all the basics they need to be successful in Git. I've never once had a team convert to Git and then decide they want to switch back to centralized source control!

Git and Microservices

Microservices are all the rage today - I won't go into details in this post about why - there's plenty of material available explaining why the industry is trending towards microservices. Conway's Law tells us that the structure of our architecture is strongly influenced by the structure of our organization. The inverse, Conway's Inverse Maneuver, postulates that you can influence the structure of an organization by the way you architect your systems! If you've been battling to get to microservices within your organization, consider migrating to Git and decomposing your giant central repo into smaller Git repos as a method of influencing your architecture. Perhaps someone has already come up with a "law" for this - if not, I'll coin "Colin's Repo Law" which states that the way that you structure your source code will influence everything else in the DevOps lifecycle - builds, releases, testing and so on. So be sure to structure your source code and repos with the end goal in mind!

Migrating to Git

Before we get to how to migrate, we have to address the issue of history. When teams migrate source control systems, they always ask about history. I push back a bit and inform teams that their old source control system isn't going away, so you don't lose history. For a small period of time, you may have two places to check for history - but most teams don't check history further out than the last month regularly. Some teams may have compliance or regulatory burdens, but these are generally the exception. Don't let the fear of "losing history" prevent you from modernizing your source control!

Monorepo or Multirepo?

The other consideration we have to make is monorepo or multirepo? A monorepo is a Git repo that contains all the code for a system (or even organization). Generally, Git repos should be small - my rule of thumb is the repo boundary should be the deployment boundary. If you always deploy three services at the same time (because they're tightly coupled) you may want to put the code for all three services into a single repo. Then again, you may want to split them and start moving to decouple them - only you can decide what's going to be correct.

If you decide to split your repo into multiple Git repos, you're going to have to consider what to do with shared code. In TFVC, you have shared code in the same repo as the applications, so you generally just have project references. However, if you split out the app code and the common code, you are going to have to have a way to consume the compiled shared code in the application code - that's a good use case for package management. Depending on your source control structure, the complexity of your system and your team culture, this may not be easy to do - in that case you may decide to just convert to a monorepo instead of a set of smaller repos.

The Azure DevOps team decided to use a monorepo even though their system is composed of around 40 microservices. They did this because the source code for Azure DevOps (which Microsoft hosts themselves as a SaaS offering) is the same source code that is used for the on-premises out-of-the-box Azure DevOps Server (previously TFS). Their CI builds are triggered off paths in the repo instead of triggering a build for every component every time the repo is changed. If you decide to use a monorepo, make sure your CI system is capable of doing this - and make sure you organize your source code into appropriate folders for managing your builds!

Migrating

So how can you migrate to Git? There are at least three ways:

1. Tip migration
2. Azure DevOps single branch import
3. Git-tfs import

Tip Migration

Most teams I work with wish they could reorganize their source control structure - typically the structure the team is using today was set up by a well-meaning developer a decade ago but it's not really optimal. Migrating to Git could be a good opportunity to restructure your repo. In this case, it probably doesn't make sense to migrate history anyway, since you're going to restructure the code (or break the code into multiple repos). The process is simple: create an empty Git repo (or multiple empty repos), then get-latest from TFS and copy/reorganize the code into the empty Git repos. Then just commit and push and you're there! Of course if you have shared code you need to create builds of the shared code to publish to a package feed and then consume those packages in downstream applications, but the Git part is really simple.

Single Branch Import

If you're on TFVC and you're in Azure DevOps (aka VSTS) then you have the option of a simple single-branch import. Just click on "Import repository" from the Azure Repos top level drop-down menu to pop open the dialog. Then enter the path to the branch you're migrating (yes, you can only choose one branch) and if you want history or not (up to 180 days). Then add in a name for the repo and let 'er rip!

There are some limitation here: a single branch and only 180 days of history. However, if you only care about one branch and you're already in Azure DevOps, then this is a no-brainer migration method.

Git-tfs

What if you need to migrate more than a single branch and retain branch relationships? Or you're going to ignore my advice and insist on dragging all your history with you? In that case, you're going to have to use Git-tfs. This is an open-source project that is build to synchronize Git and TFVC repos. But you can use it to do a once-off migration using git tfs clone. Git-tfs has the advantage that it can migrate multiple branches and will preserve the relationships so that you can merge branches in Git after you migrate. Be warned that it can take a while to do this conversion - especially for large repos or repos with long history. You can easily dry-run the migration locally, iron out any issues and then do it for real. There's lots of flexibility with this tool, so I highly recommend it.

If you're on Subversion, then you can use Git svn to import your Subversion repo in a similar manner to using Git-tfs.

Conclusion

Modernizing source control to Git has high business value - most notably the ability to effectively isolate code changes, minimize merge debt and integrate unobtrusive code reviews which can improve quality. Add to this the broad user-base for Git and you have a tool that is both powerful and pervasive. With Azure DevOps, you can also add "enterprise" features like branch policies, easily manage large binaries and even large repos - so there's really no reason not to migrate. Migrating to Git will cause some short-term pain in terms of learning curve, but the long term benefits are well worth it.

Happy source controlling!

I read a great blogpost recently by my friend and fellow MVP Jakob Ehn. In this post he outlines how he created a multi-stage Dockerfile to run .NET Core tests. I've always been on the fence about running tests during a container build - I usually run the tests outside and then build/publish the container proper only if the tests pass. However, this means I have to have the test frameworks on the build agent - and that's where doing it inside a container is great, since the container can have all the test dependencies without affecting the host machine. However, if you do this then you'll have test assets in your final container image, which isn't ideal. Fortunately, with multi-stage Dockerfiles you can compile (and/or test) and then create a final image that just has the app binaries!

I was impressed by Jakob's solution, but I wanted to add a couple enhancements:

1. Jakob builds the container twice and runs the tests twice: one build for the test runs (in a shell task using the --target arg) and one to build the container proper - which would end up execute the tests again. I wanted to improve this if I could.
2. Add code coverage. I think that it's almost silly to not do code coverage if you have tests, so I wanted to see how easy it was to add coverage to the test runs too!

tl;dr

If you want the final process, have a look at my fork of the PartsUnlimited repo on Github (on the k8sdevops branch). You'll see the final Dockerfile and the azure-pipelines.yml build definition file there.

Adding Code Coverage

I wanted to take things one step further and add code coverage into the mix. Except that doing code coverage in .NET Core is non-trivial. For that it seems you have to use Coverlet. I ended up adding a coverlet.msbuild package reference to my test project and then I just configured the test args for "dotnet test" to specify coverage options in the "dotnet test" command - we'll see that in the Dockerfile next.

Removing the Redundancy

Jakob runs a shell script which builds the container only to the point of running the tests - he doesn't want to build the rest of the container if the tests fail. However, when I was playing with this I realized that if tests fail, then the docker build process fails too - so I didn't worry about the test and final image being in the same process. If the process completes, I know the tests have passed - if not, then I might have to diagnose to figure out if there is a build issue or a test issue, but logging in Azure pipelines is fantastic so that's not too much of a concern.

The next issue was getting the test and coverage files out of the interim image and have a clean final image without test artifacts. That's where labels come in. Let's look at the final Dockerfile:

FROM microsoft/dotnet:2.2-sdk AS build-env
WORKDIR /app
ARG version=1.0.0

# install npm for building
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - && apt-get update && apt-get install -yq nodejs build-essential make

# Copy csproj and restore as distinct layers
COPY PartsUnlimited.sln ./
COPY ./src/ ./src
COPY ./test/ ./test
COPY ./env/ ./env

# restore for all projects
RUN dotnet restore PartsUnlimited.sln

# test
# use the label to identity this layer later
LABEL test=true
# install the report generator tool
RUN dotnet tool install dotnet-reportgenerator-globaltool --version 4.0.6 --tool-path /tools
# run the test and collect code coverage (requires coverlet.msbuild to be added to test project)
# for exclude, use %2c for ,
RUN dotnet test --results-directory /testresults --logger "trx;LogFileName=test_results.xml" /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=/testresults/coverage/ /p:Exclude="[xunit.*]*%2c[StackExchange.*]*" ./test/PartsUnlimited.UnitTests/PartsUnlimited.UnitTests.csproj
# generate html reports using report generator tool
RUN /tools/reportgenerator "-reports:/testresults/coverage/coverage.cobertura.xml" "-targetdir:/testresults/coverage/reports" "-reporttypes:HTMLInline;HTMLChart"
RUN ls -la /testresults/coverage/reports

# build and publish
RUN dotnet publish src/PartsUnlimitedWebsite/PartsUnlimitedWebsite.csproj --framework netcoreapp2.0 -c Release -o out /p:Version=${version}

# Build runtime image
FROM microsoft/dotnet:2.2-aspnetcore-runtime
WORKDIR /app
EXPOSE 80
COPY --from=build-env /app/src/PartsUnlimitedWebsite/out .
ENTRYPOINT ["dotnet", "PartsUnlimitedWebsite.dll"]

Notes:

• Line 1: I'm getting the big bloated .NET Core SDK image which is required to compile, test and publish the app
• Line 6: install npm prerequisites. I could create a custom build image with this on it, but it's really quick if these dependencies don't exist. If you're running on a private agent, this layer is cached so you don't do it on every run anyway.
• Lines 9-12: copy app and test files into the container
• Line 15: restore packages for all the projects
• Line 19: add a label which we can use later to identify this layer
• Line 21: install the report generator tool for coverage reports
• Line 24: run "dotnet test" to invoke the test. I specify the results directory which I'll copy out later and specify a trx logger to get a VSTest results file. The remainder of the args are for coverage: the format is cobertura, I specify a folder and specify some namespaces to exclude (note how I had to use %2c for commas to get this to work correctly)
• Line 26: run the report generator tool to produce html coverage reports
• Line 30: publish the app - this is the only bit I really want in the final image
• Lines 33-37: copy the final binaries into an image based on the .NET Core runtime - which is far lighter than the SDK image the previous steps started on (about 10% of the size)
• Line 36: this is where we do the actual copy of any artifacts we want in the final image

When the build completes, we'll end up with a number of interim images as well as a final deployable image with just the app - this is the image we're going to push to container registries and so on. Doing some docker images queries shows how important slimming down the final image is:

$> docker images --filter "label=test=true" | head -2
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
<none>              <none>              13151da78ddb        2 hours ago         2.53 GB
$> docker images | grep partsunlimited
partsunlimitedwebsite                      1.0.1                    957346c64b03        2 hours ago         308 MB

You can see that we get a 2.53GB image for the SDK build process (it's repo and tag are both <none> since this is an intermediary layer). The final image is only 308MB!

You'll also note how we used the label in the filter expression to get only the layers that have a label "test=true". If we add the "-q" parameter, we'll get just the id of that layer, which is what we'll need to get the test and coverage files out to publish in the CI build.

The Azure Pipelines YML File

The CI definition turns out to be quite simple:

name: 1.0$(Rev:.r)

trigger:
- k8sdevops

pool:
  vmImage: 'Ubuntu-16.04'

variables:
  imageName: 'partsunlimitedwebsite:$(build.buildNumber)'

steps:
- script: docker build -f Dockerfile -t $(imageName) .
  displayName: 'docker build'
  continueOnError: true

- script: |
    export id=$(docker images --filter "label=test=true" -q | head -1)
    docker create --name testcontainer $id
    docker cp testcontainer:/testresults ./testresults
    docker rm testcontainer
  displayName: 'get test results'

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'VSTest'
    testResultsFiles: '**/test*.xml' 
    searchFolder: '$(System.DefaultWorkingDirectory)/testresults'
    publishRunAttachments: true
  displayName: 'Publish test results'

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: 'cobertura'
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/testresults/coverage/coverage.cobertura.xml'
    reportDirectory: '$(System.DefaultWorkingDirectory)/testresults/coverage/reports'
  displayName: 'Publish coverage reports'

Notes:

• Lines 13-15: build and tag the docker image using the Dockerfile
• Lines 17-22: get the id of the interim image and create a container. Then copy out the test results files and then delete the container.
• Lines 24-30: publish the test file
• Lines 32-37: publish the coverage results and reports

Final Results

The final results are fantastic. Below are screenshots of the summary page, the test results page, the coverage report and a drill-down to see coverage for a specific class:

Conclusion

Running tests (with code coverage) inside a container is actually not that bad - you need to do some fancy footwork after the build to get the test/coverage results, but all in all the process is pleasant. We're able to run tests inside a container (not that this mandates real unit tests - tests that have no external dependencies!), get the results out and publish a super-slim final image.

Happy testing!