Microservices Lifecycle Workshop


Viktor Farcic


@vfarcic

TechnologyConversations.com

CloudBees.com

Viktor Farcic

Continuous deployment

  • Continuous integration
  • Continuous delivery
  • Continuous deployment

Continuous deployment

CI process

Continuous deployment

CI pipeline

Continuous deployment

Continuous delivery process

Continuous deployment

Continuous deployment pipeline

Containers (Docker)

Shipping container is an object for holding or transporting something
  • Isolated
  • Immutable
  • Reliable
  • Self-sufficient
  • Scalable

Containers (Docker)

VMs vs Containers

Containers (Docker)

Self-sufficient container

Containers (Docker)

Container with the separate DB

Containers (Docker)

Containers with the shared DB

Monolithic applications

  • Single unit
  • Time increases complexity and size
  • Time decreases development, testing and deployment speed
  • Layers
  • Change is hard and with risks
  • Scaling = multiplication of the entire application

Monolithic applications

Early

Monolithic applications

Later

Monolithic applications

Scaling

Microservices

Applications that fit into a screen

  • System composed of small services
  • Autonomy/independence
  • Data exchange through APIs
  • Bounded context

Microservices

Self-Sufficiency

Microservices

Gartner

Microservices are simpler, developers get more productive and systems can be scaled quickly and precisely, rather than in large monolithic globs. And I haven’t even mentioned the potential for polyglot coding and data persistence.
- Gary Olliffe

Microservices

Object-Oriented Design

The big idea is 'messaging'. The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.
- Alan Kay

Microservices

Single Responsibility Principle

Gather together those things that change for the same reason, and separate those things that change for different reasons
- Robert C. Martin

Microservices

Linux = Microservices


ps aux | grep jav[a] | awk '{print $2}' | xargs kill
						

Microservices

  • One thing or one functionality
  • Any tools or languages
  • Truly loosely coupled
  • Teams independence
  • Easier testing and CD
  • Decentralization

Microservices

Disadvantages

  • Increased operational and deployment complexity
    • Configuration Management
    • Containers (Docker)
    • Work shifted from development to DevOps
  • Remote process calls

Microservices

Advantages

  • Scaling
  • Resilience / fault isolation
  • Innovation
  • Size
  • Decoupling
  • Deployment
  • No need for long term commitment

Microservices

Best Practices

  • Containers (Docker)
  • Reverse proxy
  • Minimalist approach
  • CM is a must
  • Cross functional teams
  • API versioning

Deployment

  • Big vs small
  • Mutable vs immutable

Deployment

Mutable Monster Server

Deployment

Immutable Server

Deployment

Immutable Server

Deployment

Immutable Server

Deployment

Immutable Server

Deployment

Immutable Micro Services

Deployment

Immutable Micro Services

Deployment

Immutable Micro Services

Deployment

Immutable Micro Services

Deployment Pipeline

  1. Checkout the code
  2. Run pre-deployment tests
  3. Compile and/or package the code
  4. Build the container
  5. Push the container to the registry
  6. Deploy the container to the production server
  7. Integrate the container
  8. Run post-deployment tests

Deployment Pipeline: Initial Stages

Creating the CD VM

git clone https://github.com/vfarcic/ms-lifecycle.git

cd ms-lifecycle

vagrant plugin install vagrant-cachier

vagrant up cd --provision

vagrant ssh cd

Deployment Pipeline: Initial Stages

Checking out the code

git clone https://github.com/vfarcic/go-demo.git

cd go-demo

Deployment Pipeline: Initial Stages

Running pre-deployment tests and compiling and/or packaging the code

cat docker-compose-test.yml

docker-compose -f docker-compose-test.yml run --rm unit

docker ps -a

ll -t

Deployment Pipeline: Initial Stages

Building Docker containers

cat Dockerfile

docker build -t vfarcic/go-demo .

docker images

Deployment Pipeline: Initial Stages

Running containers

docker-compose up -d db app

docker-compose ps

PORT=<PORT> # Put the port from the ps command

docker-compose logs

docker-compose exec app ping -c 1 db

docker-compose exec db ping -c 1 app

Deployment Pipeline: Initial Stages

Running containers

curl -i localhost:$PORT/demo/hello

curl -XPUT localhost:$PORT/demo/person?name=Viktor

curl -XPUT localhost:$PORT/demo/person?name=Sara

curl localhost:$PORT/demo/person

docker-compose down

docker ps -a

Deployment Pipeline: Initial Stages

Pushing containers to the registry

# docker push vfarcic/go-demo

docker tag vfarcic/go-demo 10.100.198.200:5000/go-demo

docker push 10.100.198.200:5000/go-demo

exit

Deployment Pipeline: Initial Stages

Checklist

  1. Checkout the code
  2. Run pre-deployment tests
  3. Compile and/or package the code
  4. Build the container
  5. Push the container to the registry
  6. Deploy the container to the production server
  7. Integrate the container
  8. Run post-deployment tests

Configuration Management

Configuration Management

Configuring the Production Environment

vagrant up prod

vagrant ssh cd

cd /vagrant/ansible

ansible-playbook prod.yml -i hosts/prod

Configuration Management

Production environment Ansible playbook

cat prod.yml

cat roles/docker/tasks/main.yml

cat roles/docker/tasks/debian.yml

cat hosts/prod

exit

Deployment Pipeline: Intermediate Stages

  1. Checkout the code
  2. Run pre-deployment tests
  3. Compile and/or package the code
  4. Build the container
  5. Push the container to the registry
  6. Deploy the container to the production server
  7. Integrate the container
  8. Run post-deployment tests

Deployment Pipeline: Intermediate Stages

Deploying containers to the production server

vagrant ssh cd

export DOCKER_HOST=tcp://prod:2375

docker ps -a

cd go-demo

docker-compose up -d db app

docker-compose ps

Deployment Pipeline: Intermediate Stages

Post-deployment verification

docker inspect godemo_app_1

PORT=$(docker inspect \
	--format='{{(index (index .NetworkSettings.Ports "8080/tcp") 0).HostPort}}' \
	godemo_app_1)

echo $PORT

curl -XPUT prod:$PORT/demo/person?name=Viktor

curl -XPUT prod:$PORT/demo/person?name=Sara

curl prod:$PORT/demo/person

Deployment Pipeline: Intermediate Stages

Stopping Production Node

exit

vagrant halt prod

Deployment Pipeline: Intermediate Stages

Checklist

  1. Checkout the code
  2. Run pre-deployment tests
  3. Compile and/or package the code
  4. Build the container
  5. Push the container to the registry
  6. Deploy the container to the production server
  7. Integrate the container
  8. Run post-deployment tests

Service Discovery

Service Discovery

Single node

Service Discovery

Multiple nodes

Service Discovery

Service Discovery Elements

  • Service registry
  • Service registration
  • Service discovery

Service Discovery

Tools

Service Discovery

Consul Ansible playbook

vagrant up serv-disc-01 serv-disc-02 serv-disc-03

vagrant ssh cd

cat /vagrant/ansible/hosts/serv-disc

cat /vagrant/ansible/consul.yml

cat /vagrant/ansible/roles/consul/tasks/main.yml

cat /vagrant/ansible/roles/consul/defaults/main.yml

cat /vagrant/ansible/host_vars/10.100.194.201

Service Discovery

Setting up Consul

ansible-playbook \
    /vagrant/ansible/consul.yml \
    -i /vagrant/ansible/hosts/serv-disc

curl serv-disc-01:8500/v1/catalog/nodes | jq '.'

Service Discovery

Consul

Service Discovery

Key/value store: PUT

curl -X PUT -d 'this is a test' \
    serv-disc-01:8500/v1/kv/msg1

curl -X PUT -d 'this is another test' \
    serv-disc-02:8500/v1/kv/messages/msg2

curl -X PUT -d 'this is a test with flags' \
    serv-disc-03:8500/v1/kv/messages/msg3?flags=1234

Service Discovery

Key/value store: GET/DELETE

curl serv-disc-03:8500/v1/kv/?recurse | jq '.'

curl serv-disc-02:8500/v1/kv/msg1 | jq '.'

curl serv-disc-01:8500/v1/kv/msg1?raw

curl -X DELETE serv-disc-01:8500/v1/kv/messages/msg2

curl serv-disc-03:8500/v1/kv/?recurse | jq '.'

curl -X DELETE serv-disc-02:8500/v1/kv/?recurse

curl serv-disc-03:8500/v1/kv/?recurse | jq '.'

Service Discovery

Registrator

cat /vagrant/ansible/roles/registrator/tasks/main.yml

cat /vagrant/ansible/roles/registrator/defaults/main.yml

ansible-playbook \
    /vagrant/ansible/registrator.yml \
    -i /vagrant/ansible/hosts/serv-disc

Service Discovery

Consul / Registrator

Service Discovery

Registering service

docker -H tcp://serv-disc-01:2375 run -d \
    --name nginx \
    --env SERVICE_NAME=nginx \
    --env SERVICE_ID=nginx \
    -p 1234:80 \
    nginx:alpine

curl serv-disc-01:8500/v1/catalog/service/nginx-80 | jq '.'

Service Discovery

Registering service again

docker -H tcp://serv-disc-02:2375 run -d \
    --name nginx2 \
    --env "SERVICE_ID=nginx2" \
    --env "SERVICE_NAME=nginx" \
    --env "SERVICE_TAGS=balancer,proxy,www" \
    -p 1111:80 \
    nginx:alpine

curl serv-disc-02:8500/v1/catalog/service/nginx-80 | jq '.'

Service Discovery

Consul / Registrator / Consul Template

Service Discovery

Running Consul Template

curl serv-disc-01:8500/v1/catalog/service/nginx-80 | jq '.'

cat /data/consul-template/example2.ctmpl

consul-template \
    -consul serv-disc-01:8500 \
    -template "/data/consul-template/example2.ctmpl:/tmp/example.conf" \
    -once

cat /tmp/example.conf

Service Discovery

Cleanup

exit # Exit cd node

vagrant destroy -f serv-disc-01 serv-disc-02 serv-disc-03

Proxy Service

Proxy Service

nginx Ansible Playbook

vagrant up proxy

vagrant ssh cd

cat /vagrant/ansible/nginx.yml

cat /vagrant/ansible/roles/nginx/files/services.conf

ansible-playbook /vagrant/ansible/nginx.yml \
    -i /vagrant/ansible/hosts/proxy

export DOCKER_HOST=tcp://proxy:2375

docker ps

curl proxy:8500/v1/catalog/services | jq '.'

Proxy Service

Services Without Proxy

cd go-demo

docker-compose up -d db app

curl proxy/api/v1/books

docker-compose ps

PORT=$(docker inspect \
    --format='{{(index (index .NetworkSettings.Ports "8080/tcp") 0).HostPort}}' \
    godemo_app_1)

curl proxy:$PORT/demo/hello

exit

Proxy Service

Services Without Proxy

Proxy Service

Manually Setting Proxy

vagrant ssh proxy

PORT=$(docker inspect \
    --format='{{(index (index .NetworkSettings.Ports "8080/tcp") 0).HostPort}}' \
    godemo_app_1)

echo "location /demo {
    proxy_pass http://10.100.193.200:$PORT/demo;
}" | sudo tee /data/nginx/includes/go-demo.conf

docker kill -s HUP nginx

curl http://localhost/demo/hello

docker rm -f nginx

exit

Proxy Service

Manually Setting Proxy

Proxy Service

Docker Flow: Proxy

vagrant ssh cd

export DOCKER_HOST=tcp://proxy:2375

docker run -d \
    --name docker-flow-proxy \
    -e CONSUL_ADDRESS=10.100.193.200:8500 \
    -p 80:80 -p 8081:8080 \
    vfarcic/docker-flow-proxy

curl "proxy:8081/v1/docker-flow-proxy/reconfigure?serviceName=go-demo&servicePath=/demo" \
    | jq '.'

curl -i proxy/demo/hello

Proxy Service

Docker Flow: Proxy

cd go-demo

docker-compose ps

docker-compose scale app=3

docker-compose ps

curl "proxy:8081/v1/docker-flow-proxy/reconfigure?serviceName=go-demo&servicePath=/demo" \
    | jq '.'

curl -i proxy/demo/hello # Repeat a few times

docker-compose logs app

Deployment

Docker Flow: Proxy

Proxy Service

Cleanup

exit

vagrant halt proxy

Deployment Pipeline: The Late Stages

  1. Checkout the code
  2. Run pre-deployment tests
  3. Compile and/or package the code
  4. Build the container
  5. Push the container to the registry
  6. Deploy the container to the production server
  7. Integrate the container
  8. Run post-deployment tests

Deployment Pipeline: The Late Stages

Provision Production

vagrant up prod

vagrant ssh cd

cd /vagrant/ansible

ansible-playbook prod2.yml -i hosts/prod --skip-tags "nginx"

Deployment Pipeline: The Late Stages

Run Containers

export DOCKER_HOST=tcp://prod:2375

cd ~/go-demo

docker-compose up -d db app

curl prod:8500/v1/catalog/service/go-demo \
    | jq '.'
Open http://10.100.198.201:8500/ui.

Deployment Pipeline: The Late Stages

Reconfigure the Proxy

curl -i prod/demo/hello

docker run -d \
    --name docker-flow-proxy \
    -e CONSUL_ADDRESS=10.100.198.201:8500 \
    -p 80:80 -p 8081:8080 \
    vfarcic/docker-flow-proxy

curl "prod:8081/v1/docker-flow-proxy/reconfigure?serviceName=go-demo&servicePath=/demo" \
    | jq '.'

curl -i prod/demo/hello

Deployment Pipeline: The Late Stages

Run Integration Tests

cat docker-compose-test.yml

unset DOCKER_HOST

export HOST_IP=10.100.198.201

docker-compose \
    -f docker-compose-test.yml \
    run --rm production

# docker push vfarcic/go-demo

Deployment Pipeline: The Late Stages

Clean Up

export DOCKER_HOST=tcp://prod:2375

docker-compose down

curl "prod:8081/v1/docker-flow-proxy/remove?serviceName=go-demo" \
    | jq '.'

Deployment Pipeline: The Late Stages

Checklist

  1. Checkout the code
  2. Run pre-deployment tests
  3. Compile and/or package the code
  4. Build the container
  5. Push the container to the registry
  6. Deploy the container to the production server
  7. Integrate the container
  8. Run post-deployment tests

Deployment Pipeline: The Late Stages

Problems

  1. Downtime
  2. Not scalable
  3. Not failure tolerant
  4. Difficult to monitor
  5. Not automated

Standard Deployment

Standard Deployment

Standard Deployment

Blue-Green Deployment

Blue-Green Deployment

Blue-Green Deployment

Blue-Green Deployment

Blue-Green Deployment

Blue-Green Deployment

cat docker-compose-bg.yml

export NEXT_COLOR=blue

VERSION=:1.0 docker-compose -f docker-compose-bg.yml \
    up -d db app-${NEXT_COLOR}

curl "prod:8081/v1/docker-flow-proxy/reconfigure?serviceName=go-demo-${NEXT_COLOR}&servicePath=/demo" \
    | jq '.'

curl -i prod/demo/hello

docker-compose -f docker-compose-bg.yml logs

curl -XPUT -d $NEXT_COLOR prod:8500/v1/kv/go-demo/color

Blue-Green Deployment

Blue-Green Deployment

COLOR=$(curl prod:8500/v1/kv/go-demo/color?raw)

NEXT_COLOR=$(if [[ "$COLOR" == "blue" ]]; then echo "green"
else echo "blue"; fi)

VERSION=:1.1 docker-compose -f docker-compose-bg.yml \
    up -d db app-${NEXT_COLOR}

docker-compose -f docker-compose-bg.yml ps

curl -i prod/demo/hello

docker-compose -f docker-compose-bg.yml logs

Blue-Green Deployment

Blue-Green Deployment

curl "prod:8081/v1/docker-flow-proxy/reconfigure?serviceName=go-demo-${NEXT_COLOR}&servicePath=/demo" \
    | jq '.'

curl "prod:8081/v1/docker-flow-proxy/remove?serviceName=go-demo-${COLOR}" \
    | jq '.'

curl -XPUT -d $NEXT_COLOR prod:8500/v1/kv/go-demo/color

curl -i prod/demo/hello # Repeat

docker-compose -f docker-compose-bg.yml logs

docker-compose -f docker-compose-bg.yml stop app-${COLOR}

docker-compose -f docker-compose-bg.yml ps

Blue-Green Deployment

Blue-Green Deployment

docker-compose -f docker-compose-bg.yml down

curl "prod:8081/v1/docker-flow-proxy/remove?serviceName=go-demo-${NEXT_COLOR}" \
    | jq '.'

Docker Flow

export FLOW_PROXY_HOST=10.100.198.201

export FLOW_PROXY_RECONF_PORT=8081

export FLOW_CONSUL_ADDRESS=http://10.100.198.201:8500

export FLOW_PROXY_DOCKER_HOST=tcp://10.100.198.201:2375

docker-flow --flow=deploy --flow=proxy --flow=stop-old

docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

curl -i prod/demo/hello

Docker Flow

docker-flow --flow=deploy --flow=proxy --flow=stop-old

docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

curl -i prod/demo/hello

Clean Up

exit

vagrant destroy -f prod

Clustering And Scaling Services

Clustering And Scaling Services

Tools

  • Kubernetes
  • Mesos DCOS

Clustering And Scaling Services

Service Discovery

Clustering And Scaling Services

Master and Nodes

Clustering And Scaling Services

Deploying the First Container

Clustering And Scaling Services

The Second Container

Clustering And Scaling Services

All Nodes Full

Clustering And Scaling Services

Adding a New Node

Clustering And Scaling Services

Failed Node

Clustering And Scaling Services

Setting Up Docker Swarm

vagrant up swarm-master swarm-node-1 swarm-node-2

vagrant ssh cd

cat /vagrant/ansible/swarm.yml

cat /vagrant/ansible/roles/swarm/tasks/main.yml

ansible-playbook /vagrant/ansible/swarm.yml \
    -i /vagrant/ansible/hosts/prod

export DOCKER_HOST=tcp://10.100.192.200:2375

docker info

docker ps -a

Clustering And Scaling Services

Deploying with Docker Swarm

cd ~/go-demo

docker-compose up -d db app

docker-compose ps

curl swarm-master:8500/v1/catalog/services \
    | jq '.'

curl swarm-master:8500/v1/catalog/service/go-demo \
    | jq '.'

docker-compose down

Clustering and Scaling Services

Swarm with Docker Flow

export FLOW_PROXY_HOST=10.100.198.200

export FLOW_PROXY_RECONF_PORT=8081

export FLOW_CONSUL_ADDRESS=http://10.100.192.200:8500

export FLOW_PROXY_DOCKER_HOST=tcp://10.100.198.200:2375

docker-flow --flow=deploy --flow=proxy --flow=stop-old

curl -i localhost/demo/hello

Clustering and Scaling Services

Swarm With Docker Flow

Clustering and Scaling Services

Swarm With Docker Flow

docker-flow --flow=deploy --flow=proxy --flow=stop-old

docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

curl -i localhost/demo/hello

Clustering and Scaling Services

Swarm With Docker Flow

Clustering and Scaling Services

Swarm With Docker Flow

docker-flow --scale="+2" --flow=scale --flow=proxy

docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

curl -i localhost/demo/hello

docker-flow --scale="-2" --flow=scale --flow=proxy

docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Clustering and Scaling Services

Swarm With Docker Flow

Clustering And Scaling Services

Cleaning Up

docker rm -f $(docker ps -a | grep godemo | awk '{ print $1 }')

sudo docker rm -f docker-flow-proxy

CI Tools

  • GoCD
  • Bamboo
  • Travis
  • CircleCI
  • Shippable

CI Tools

Jenkins Setup

ansible-playbook /vagrant/ansible/jenkins-node-swarm.yml \
    -i /vagrant/ansible/hosts/prod

ansible-playbook /vagrant/ansible/jenkins.yml \
    -c local

cat /vagrant/ansible/jenkins.yml

cat /vagrant/ansible/roles/java/tasks/main.yml

cat /vagrant/ansible/roles/jenkins/tasks/main.yml

cat /vagrant/ansible/roles/jenkins/defaults/main.yml
Open http://10.100.198.200:8080

CI Tools

Jenkins Global Library

git clone http://10.100.198.200:8080/workflowLibs.git /tmp/workflowLibs

cd /tmp/workflowLibs

git checkout -b master

mkdir vars

cp ~/go-demo/jenkins/vars/dockerFlowWorkshop.groovy \
    /tmp/workflowLibs/vars/dockerFlow.groovy

git config --global user.name "vfarcic"

git add --all && git commit -a -m "Docker Flow"

git push --set-upstream origin master

CI Tools

Deployment Flow With Jenkins

cd ~/go-demo

cat jenkins/vars/dockerFlowWorkshop.groovy

cat Jenkinsfile

curl -XPUT -d "10.100.198.200" \
    http://swarm-master:8500/v1/kv/proxy/ip

CI Tools

Deployment With Jenkins

http://10.100.198.200:8080 > New Item
Item Name: go-demo; Type: Multibranch Pipeline > OK
Add Source > Git
Project Repository: https://github.com/vfarcic/go-demo.git > Save
http://10.100.198.200:8080/job/go-demo/branch/master/
http://10.100.198.200:8080/job/go-demo/branch/master/lastBuild/console

Say Again

  • over mutable deployments
  • over big
  • over manual procedures
  • over predefined configuration
  • over static
  • over do-it-all tools

That is, while there is value in the items on the , we value the items on the more.

Viktor Farcic


@vfarcic


TechnologyConversations.com

Viktor Farcic


Amazon
LeanPub

Cleanup

exit

vagrant destroy -f