background

Documentation

Technical overview

Architecture

The project consists of the following components:

  • A Front End Website written using ASP.NET MVC. It currently uses .NET Framework 4.6.1. Where interactive elements are required as part of the front end, Angular 1.6 is used to provide these.
  • A CMS written using ASP.NET MVC, also currently using .NET Framework 4.6.1. Where interactive elements are required as part of the front end, Angular 1.6 is used to provide these.
  • A Public API written using ASP.NET Web Api, also currently using .NET Framework 4.6.1.
  • A number of Azure WebJobs written using the Azure WebJobs C# SDK

Server Side

The Front End Website, Public API and CMS Back End share a common approach to development. All are based on the Storm SDK - an in-house, custom-built set of libraries on top of ASP.NET MVC. The SDK has been used across 100s of projects since it was first conceived in 2011 and has undergone continuous improvement since then. The SDK provides a solid base for applications as well as "battle-tested" patterns for writing scalable APIs in .NET.

All of the server side components communicate with a SQL Server database using NHibernate. The front end website uses a Redis Cache as a read store to reduce direct access to the database.

Key technologies

  • .NET Framework 4.6.1
  • NHibernate
  • Castle Windsor

Project Structure

  • ShowMS.Migrations.Runner - background WebJob for running Database Migrations
  • ShowMS.Migrations.Runner.Holder - containing website for the Migrations WebJob for deployment
  • ShowMS.WebJob - main WebJob for running background jobs
  • ShowMS.WebJob.Core - base code for building Azure WebJobs on top of the Storm SDK
  • ShowMS.WebJob.Holder - containing website for the main WebJob for deployment
  • ShowMS - front end website
  • ShowMS.Admin - CMS
  • ShowMS.API - Public API
  • ShowMS.Core - Application layer code
  • ShowMS.IntegrationTests - collection of 640 Integration Tests of the Front End Website and Public API
  • ShowMS.Migrations - code-based migrations for the database
  • ShowMS.UnitTests - collection of 900 unit tests for various aspects of the code

Front End

The front end of the website is built as a server-side rendered ASP.NET MVC application.

Where a more interaction is required, small pockets of AngularJS code are deployed. This can take the form of an interactive widget or an area of the site which functions as an entire single page application.

The AngularJS code is assembled using GulpJS.

Key technologies

  • AngularJS 1.6
  • GulpJS
  • NPM

Architecture Overview

image

Azure Architecture

ShowMS has been cloud native from day one, and is entirely cloud based. Wherever possible we have used cloud services/PaaS - indeed there isn't a single Virtual Machine (VM) that needs maintained across the entire ShowMS platform.

Pre-Prod (Dev, Staging, QA)

Each environment (for more details see Environments is deployed as a single Resource Group within Azure containing the following resources:

Production

For production, we operate 2 geo-redundant copies of the above infrastructure contained within a single resource group. The makeup of this resource group is as follows:

Web sites

At present, these services are run in a Primary/Secondary configuration using Azure Traffic Manager which manages request routing at the DNS level.

Data Storage

Geo resiliency for the SQL Server is managed using SQL Azure's Geo Replication feature. This means that the East Coast database is a readable secondary of the main, writeable primary database located on the West Coast. Failover to the secondary is achieved manually and is only intended to be used in the event of an extended outage.

Geo resiliency for the Redis Cache is managed using Redis Azure's Geo Replication feature. This means that the East Coast cache is a readable secondary of the main, writeable primary database located on the West Coast. Failover to the secondary is achieved manually and is only intended to be used in the event of an extended outage.

Blob Storage

Geo resiliency for Blob Storage is provided by using Azure Blob Storage's Geo redundant-storage feature. This means that in the event of an outage Microsoft is able to enable reads from the secondary region (in our case, East US) in the event of an outage in primary region (in our case, West US).

Others

Geo resiliency for Azure Key Vault is included with the service - please see here for details.

At present, there is no geo redundancy enabled for this service. Investigating steps to provide this is on our roadmap.

At present, there is no geo redundancy enabled for this service. Investigating steps to provide this is on our roadmap. This service is only used by background tasks in the CMS.

Geo redundancy for Cosmos DB is provided by using the built in capabilities of the service. Currently, there is a read only replica of the main, writeable primary database located on the West Coast. Failover to the secondary is manually and is only intended to be used in the event of an extended outage.

Overview

image

Deployment

Cake

The project uses the open source CakeBuild project to author their build script. CakeBuild allows developers to write their build script in C# which is compiled to either Powershell or Bash using the cross-platform .NET compiler.

build.cake contains the definitions of the tasks which are executed as part of the build. build.ps1 is the script which is executed to perform the build.

e.g.

./build.ps1 -Target CI -Verbosity Diagnostic

Tasks

  • Clean - clears the output artifacts directory
  • Restore-NuGet-Packages - Fetches the NuGet packages for the solution
  • Create-Version-Info - Creates an empty AssemblyInfo.cs file to later receive the version information
  • Update-Version-Info - Runs GitVersion the determine the current version of the entire project based on it's git history and git tags. Then writes this information into the AssemblyInfo.cs file created during Create-Version-Info so that when the project is built later in the process it's able to encode this information into the binary.
  • Update-AppVeyor-Build-Number - see AppVeyor. Sets the AppVeyor build number in line with the previously determined version.
  • Run-Unit-Tests - runs the unit tests defined for the project and reports any results as an AppVeyor artifact
  • Run-Integration-Tests - runs the integration tests defined for the project and reports any results as an AppVeyor artifact
  • Npm - installs the necessary Npm packages to build the front end website
  • Gulp - performs the build steps necessary to assemble the static assets needed for the front end website using the GulpJS build system
  • NpmAdmin - installs the necessary Npm packages to build the CMS front end
  • GulpAdmin - performs the necessary build steps to assemble the assets needed for the CMS front end
  • Build - builds every project in the solution using MSBuild
  • Create-Website-Deploy-Package - Uses MSBuild to generate a zipped WebDeploy package of the front end website ready for deployment to Azure App Service
  • Create-Admin-Deploy-Package - As above for the CMS website
  • Create-WebJob-Deploy-Package - As above for the WebJob used for background processing
  • Create-Migrations-Deploy-Package - As above for the WebJob used for running database migrations
  • Create-Api-Deploy-Package - As above for the website hosting the Public API
  • Upload-AppVeyor-Artifacts - aggregate task which uploads all of the outputs generated above as artifacts for deployment later
  • Default - aggregate task which builds the project and runs the tests.
  • CI - aggregate task which is executed in CI to build the project and generated the required artifacts

AppVeyor

The CI system AppVeyor is used to build the project and store any artifacts. The CI script simply executes:

./build.ps1 -Target CI -Verbosity Diagnostic

and stores the test results and the resultant zip files.

These artifacts can then be deployed using AppVeyor's concept of environments.

There is an AppVeyor Environment for each component of the system i.e. one for the front end website, one for the CMS, one for the Public API and one for each WebJob. Additionally, due to the architecture of the system having two globally redundant regions when deployed there is an additional AppVeyor environment for each region which supports this (currently East US and West US).

Additionally, all of the static assets required for each site have a corresponding environment as these are deployed to Azure Blob Storage and then accessed via Azure CDN.

Each environment allows any valid artifact produced during the build phase to be deployed to it. Currently, artifacts are not auto-deployed to any environment, however, as the suite of automated tests for the application improves we will be looking to introduce more Continuous Delivery process to this.

Manual deployment steps currently consist of:

  • selecting the correct environment which should be deployed to
  • selecting the correct artifact which should be deployed
  • waiting for the system to confirm the correct deployment

Automatic deployment recording is via Slack notifications and into NewRelic.

Azure

Configuration

Configuration is managed via the use of Azure App Service App Settings. These allow configuration to be set dynamically in the Azure portal and applied to the deployed website without the need to update a config file locally and deploy.

Hot-swapping

In order to minimize downtime during deployments to Production we make use of Azure App Service Deployment Slots. When deploying to production, therefore, the process is:

  1. Update Failover/East-US website
  2. Smoke test failover site
  3. Update Release Candidate Deployment Swap and initialize a "Swap with preview". This makes the new code available on a separate URL with all of the production settings applied to it. This website is then available to be tested and "warmed up" so that when it starts receiving the production traffic, it doesn't need to go through a "cold start" process.
  4. Smoke test release candidate slot, ensuring that site responds quickly by the end of the smoke test.
  5. Complete the swap to production
  6. Test production
  7. Post launch hypercare for any issues

If any issues are found during step 4, then it is possible to abort the deployment and not proceed with the swap.

This process allows us the ability to both test the website in its final form with its production settings without affecting the production traffic. Additionally the "warming" of the site minimizes any slowness as the deployment rolls out.

Database Migrations

The project contains an Azure WebJob which performs database migrations. Where these are required as part of a deployment, this is deployed to the relevant website and the migrations are automatically executed on completion of the deployment.

Environments

The application has 4 environments:

Dev

Staging

QA

Production

Monitoring

Monitoring is currently via NewRelic and Microsoft Azure

NewRelic

NewRelic integration is via the .NET Agent (currently version 8.1.709). NewRelic alerts are configured centrally by the DTCI Technology Operations Team.

Microsoft Azure

Microsoft Azure offers built in alerting via its Monitor service. It is currently set to alert on the following metrics:

  • Greater than 100 HTTP Server Errors in 5 minutes on the Public API
  • Greater than 100 HTTP Server Errors in 5 minutes on the Front End Website
  • Average response time is greater than 2s on the Front End Website

These alerts trigger emails and text messages to the ShowMS team who are then able to respond appropriately.

Others

The Front End website and API are also monitored by Catchpoint, configured by the DTCI Tech Ops team.

Pager Duty is also used for alerting members of the ShowMS team to any incidents. Again this is managed by the DTCI Tech Ops team.