mesosphere-icons-connected-carmesosphere-icons-connected-homemesosphere-icons-credit-cardmesosphere-icons-cursormesosphere-icons-shopping-bag-gearsmesosphere-icons-toolsArtboardmesosphere-icons-medium-cloud-serversmesosphere-icons-medium-database-gearmesosphere-icons-medium-graph-moneymesosphere-icons-medium-shield }}

Continuous deployment with Mesos, Marathon and Docker

Datacenter automation is one of the strongest features offered by the Mesosphere technology. This post is a follow on from our recent presentation at the Bay Area Infracoders meetup where we demonstrated how an organization can use Mesosphere to easily construct a simple continuous deployment pipeline from source code repository to your datacenter.

Goal

Continuous deployment is the holy grail of datacenter automation. In this post, we demonstrate a simplified example of how we use Mesos, Marathon, TeamCity, and Docker internally to deploy applications automatically to our internal staging environment. The team is automatically notified of new deployments by using Slack.

The following flow chart shows the high level sequence of actions. A check-in to GitHub triggers a build of a Docker image in TeamCity. On a successful build, a special TeamCity Deploy build is triggered, which uses Marathon’s REST API to trigger a new deployment.

Continuous Deployment with Marathon

Implementation

Our example project is a reveal.js presentation, a developer friendly HTML + JavaScript presentation format that Mesosphere engineers use for talks. We check these into a GitHub repository at github.com/mesosphere/presentations.

Project Setup

For this example, we are using the presentation located at ba-infracoders-2015:

Continuous Deployment with Marathon

The app folder contains our presentation along with a Dockerfile and marathon.json which describe how to build and deploy our presentation.

The simple Dockerfile reuses the nginx base container which provides the nginx webserver. Finally, we ADD the presentation directory to the container to be served by nginx.

FROM nginx
MAINTAINER Mesosphere support@mesosphere.com

EXPOSE 80

ADD app/ /usr/share/nginx/html

The simple Marathon app definition in marathon.json specifies our mesosphere/cd-demo-app DockerHub repository as the source of our Docker image. Note the $tag placeholder, which is required by TeamCity to deploy the latest built tag of our presentation.

We also specify health checks in this definition, which allows Marathon to gauge when the newly deployed container is up and running correctly. The health checks use HTTP requests to a defined endpoint. You can read more about how Marathon health checks work here.

{
  "id": "/mesosphere/cd-demo-app",
  "instances": 1,
  "cpus": 1,
  "mem": 512,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "mesosphere/cd-demo-app:$tag",
      "network": "BRIDGE",
      "portMappings": [
        {
          "servicePort": 28080,
          "containerPort": 80,
          "hostPort": 0,
          "protocol": "tcp"
        }
      ]
    }
  },
  "healthChecks": [
    {
      "gracePeriodSeconds": 120,
      "intervalSeconds": 30,
      "maxConsecutiveFailures": 3,
      "path": "/",
      "portIndex": 0,
      "protocol": "HTTP",
      "timeoutSeconds": 5
    }
  ],
  "constraints": [
    [ "hostname", "GROUP_BY" ]
  ]
}

TeamCity Setup

In TeamCity, we have a build that builds and pushes the Docker image for our presentation and another build that deploys the most recently built image to Marathon.

Tag Scheme

Before setting up the build you should develop a scheme to use for generating the Docker image tags.

Here’s the scheme we’re going to use for our demo: %env.BUILD_NUMBER%-%teamcity.build.branch%.%env.BUILD_VCS_NUMBER%.

The components of the tag are as follows:

  • %env.BUILD_NUMBER%: The build number so that we can always track down the TeamCity build that created the Docker Image
  • %teamcity.build.branch%: The branch of the Git repo that the Docker Image was built from
  • %env.BUILD_VCS_NUMBER%: The Git SHA of the code that was added to the Docker Image

This scheme ensures that we have a new tag for every build, meaning that we are creating immutable containers. Also, the scheme ensures that we have visibility into the branch that the code came from and the Git SHA of the commit.

Docker Image Build

The first build in TeamCity has these steps:

  1. docker login: log in to our DockerHub account
  2. docker build: build the specified Dockerfile with a new tag
  3. docker push: push the built image to DockerHub
  4. Report tag into new marathon.json: use jq to write the tag into the Marathon app definition

The first three steps are fairly standard, but the fourth step is the magic that makes this work. Our script looks like this:

mkdir -p target

echo %TAG% > target/docker-tag

cat marathon.json | \
  jq '.container.docker.image |= "%DOCKER_IMAGE%:%TAG%"' > target/marathon.json

After the build completes you can view the generated artifacts on the Artifacts tab for the specific build:

TeamCity Build Artifacts

Deploy

After the first build completes successfully and there are new artifacts, TeamCity automatically triggers a second build where we do two things:

  1. PUT to Marathon’s REST API
  2. Send a message to the team on Slack

Marathon’s REST API makes it easy to kick off new deployments of an application. Using HTTP PUT with the newly generated marathon.json artifact creates a new deployment. You can read more about Marathon deployments here. The deployment starts a new instance of the application and stops the old instance only when the new application is deemed “healthy”.

APP_ID=$(cat %MARATHON_JSON_FILE% | jq -r '.id')
http --check-status PUT \
  %SCHEME%://%MARATHON_HOST%:%MARATHON_PORT%/v2/%MARATHON_JSON_TYPE%/$APP_ID < %MARATHON_JSON_FILE%

Sending a message on Slack is easy because of their accessible API. The process involves pushing a JSON encoded message to a pre-configured Slack webhook URL (see Slack’s documentation for more details).

echo '{
  "username"   :"%SLACK_USERNAME%",
  "channel"    :"%SLACK_CHANNEL%",
  "text"       :"%SLACK_MESSAGE%",
  "icon_emoji" :"%SLACK_EMOJI%",
  "mrkdwn"     :%SLACK_MARKDOWN%
}' | http --print=HhBb --json POST %SLACK_WEBHOOK_URL%

Of course, by parameterizing all of these values, it is easy to create a TeamCity template and extend this to other services.

Name value
MARATHON_HOST
MARATHON_JSON_FILE marathon.json
MARATHON_JSON_TYPE apps
MARATHON_PORT 8080
SCHEME http
SLACK_CHANNEL #demo
SLACK_EMOJI :teamcity:
SLACK_MARKDOWN true
SLACK_MESSAGE Heads Up! %teamcity.build.triggeredBy% just deployed
SLACK_USERNAME TeamCity
SLACK_WEBHOOK_URL https://hooks.slack.com/services/

Results

Each time a successful Docker image build completes, a deploy build is triggered, the marathon.json is PUT to Marathon, and the app begins rolling out across your cluster. A notification that an application rollout is happening is sent to the team.

Artifact marathon.json

Here you can see the generated marathon.json that is sent to Marathon.

{
  "id": "/mesosphere/cd-demo-app",
  "instances": 1,
  "cpus": 1,
  "mem": 512,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "mesosphere/cd-demo-app:34-master.3fcfaa664108e75120c65c26e69fb57b8fefce3d",
      "network": "BRIDGE",
      "portMappings": [
        {
          "servicePort": 28080,
          "containerPort": 80,
          "hostPort": 0,
          "protocol": "tcp"
        }
      ]
    }
  },
  "healthChecks": [
    {
      "gracePeriodSeconds": 120,
      "intervalSeconds": 30,
      "maxConsecutiveFailures": 3,
      "path": "/",
      "portIndex": 0,
      "protocol": "HTTP",
      "timeoutSeconds": 5
    }
  ],
  "constraints": [
    [ "hostname", "GROUP_BY" ]
  ]
}

Marathon Rollout

The new version of the application is scaled up, the new task is shown in the Staged state here: Marathon Scale Up

After Marathon considers the new instance healthy, it cleans up old instances of the application: Marathon Cleanup Old Instances

Slack Message

A notification is then sent to the team indicating that a deploy has been triggered: Slack Notifications

Key Takeaways

  1. Set up an automatic build for your project and generate a new Docker tag for each build.
  2. Generate marathon.json as a build artifact.
  3. Configure TeamCity to collect the generated marathon.json.
  4. Configure firewalls to allow TeamCity build agents to communicate with Marathon.
  5. Create a TeamCity build to PUT marathon.json to Marathon.
  6. Notify your team of the deployment.

Summary

This post shows how to easily set up automated builds that deploy Dockerized applications automatically to a Mesosphere cluster. We use this same procedure in many of our projects to make life easier for our engineers. Give continuous deployment on Mesosphere a try today!