Continuously Deploying Containers with Jenkins Pipeline To a Docker Swarm Cluster


Viktor Farcic


@vfarcic

CloudBees.com

TechnologyConversations.com

Viktor Farcic

Continuous Deployment


Every commit to production


Nirvana

Docker Containers


Jenkins Pipeline


Docker Swarm

Docker Containers


Why?


Virtual machines are obese

Jenkins Pipeline


Why?


CD expressed as code

Docker Swarm


Why?


Cluster cannot be managed manually

Setup

Requirements: Vagrant
git clone https://github.com/vfarcic/ms-lifecycle.git

cd ms-lifecycle

vagrant plugin install vagrant-cachier

vagrant up cd swarm-master swarm-node-1 swarm-node-2

vagrant ssh cd

ansible-playbook /vagrant/ansible/swarm.yml -i /vagrant/ansible/hosts/prod

ansible-playbook /vagrant/ansible/jenkins-node-swarm.yml -i /vagrant/ansible/hosts/prod

ansible-playbook /vagrant/ansible/jenkins.yml -c local

git clone https://github.com/vfarcic/go-demo.git

cd go-demo

CD Steps

  • Test
  • Build
  • Deploy

Testing

Testing

Unit tests

  • Unreliable
  • Do not prove much
  • Without TDD == too late
  • TDD != testing technique
  • TDD == development process

Testing

Functional tests

  • Infrastructure problems
  • Unknown whether it works inside the system

Testing

docker-compose -f docker-compose-test.yml run --rm unit

ll

Testing

Building

  • We know how to do it
  • Self-sufficient?
  • Immutable?

Building

docker build -t vfarcic/go-demo .

# docker push vfarcic/go-demo

Building

Deployment

  • It's a cluster, not a server
  • Infrastructure?

Deployment

export DOCKER_HOST=tcp://swarm-master:2375

docker info

docker-compose up -d

docker-compose ps

docker-compose down

Deployment

Jenkins Pipeline

CD defined as code

node("cd") {
  git "https://github.com/vfarcic/go-demo.git"

  stage "pre-dep-tests"
  sh "docker-compose -f docker-compose-test.yml run --rm unit"

  stage "build"
  sh "docker build -t vfarcic/go-demo ."
  // sh "docker push vfarcic/go-demo"

  stage "deploy"
  withEnv(['DOCKER_HOST=tcp://swarm-master:2375']) {
    sh "docker-compose up -d"
    sh "docker-compose ps"
  }
}

Jenkins Pipeline

CD defined as code

Press the Build button
docker-compose -p go-demo-1 down

The End

Deployment

==

Downtime

Deployment

Blue-green deployment

requires

proxy

Deployment

Proxy

needs to be

dynamic

Deployment

Dynamic

==

0 configuration (files)

Deployment

0 configuration (files)

==

Service discovery

Deployment

Service discovery

==

Registry + registration + discovery

Deployment (revisited)

Deployment

Problems

  • Batteries included but removable
  • Swarm does not support/include...
    • Blue-green deployment
    • Service discovery
    • Dynamic proxy

Deployment

What needs to be done

  • Different targets for blue and green releases
  • Figure out which color to deploy
  • Service registry
  • Register/deregister services
  • Use service registry to reconfigure proxy

Deployment

How can it be done

Jenkins Pipeline

CD defined as code

...
  stage "deploy"
  withEnv(envs) {
    sh "docker-flow -p go-demo --flow=deploy --flow=proxy --flow=stop-old"
  }
...

Deployment

CD defined as code

http://10.100.198.200:8080/job/go-demo-2/configure
http://10.100.198.200:8080/job/go-demo-2
Press the Build button
curl -i cd/demo/hello

Testing

(before production)

==

unwarranted optimism

Testing

Blue-green deployment

enables

production testing

before and after proxy reconfiguration

Testing (revisited)

Testing (revisited)

Problems

  • Where are my services?
  • Parallel execution?

Testing (revisited)

What needs to be done

  • Discover the addresses
  • Run tests for each instance

Testing (revisited)

How can it be done

  • Request info from Consul
  • Leverage Pipeline's step parallel

Jenkins Pipeline

CD defined as code

...
  stage "deploy"
  withEnv(envs) {
    sh "docker-flow -p go-demo --flow=deploy --scale=\"${SCALE}\""
  }

  stage "pre-integ-tests"
  def addresses = getAddresses(consul)
  def tests = [:]
  for (address in addresses) {
    def host = "${address.ServiceAddress}:${address.ServicePort}"
    def index = "${address.CreateIndex}"
    tests[index] = { sh "HOST_IP=${host} docker-compose -p go-demo-tests-${index} -f docker-compose-test.yml run --rm production" }
  }
  parallel tests

  stage "integrate"
  withEnv(envs) {
    sh "docker-flow -p go-demo --flow=proxy --flow=stop-old"
  }

  stage "post-integ-tests"
  sh "HOST_IP=${proxy} docker-compose -f docker-compose-test.yml run --rm production"
                            ...

Testing (revisited)

CD defined as code

http://10.100.198.200:8080/job/go-demo-3/configure
http://10.100.198.200:8080/job/go-demo-3
Press the Build button
docker ps -a

curl -i cd/demo/hello

Rollback

Only if the flow fails

Afterwards, only roll forward

Rollback

Failed flow

==

Unchanged production

Rollback

Pre-deployment tests

==

No action

Rollback

Pre-integration tests

Proxy still configured with the old release

Stop the new release

Remove service data

Rollback

Post-integration tests

Proxy configured with the new release

Reconfigure the proxy

Stop the new release

Update service data

Rollback

Rollback

Databases?

Backward compatibility

Short iterations

Where to run?

In production, if necessary

No SSH

Where to run?

Parting words

Docker == immutability

Pipeline == CD as code

Swarm == cluster

Viktor Farcic


Viktor Farcic


@vfarcic


TechnologyConversations.com

Viktor Farcic


Amazon
LeanPub

Cleanup

exit

vagrant destroy -f