The DevOps 2.1 Toolkit

Monitoring

Swarm Services

Viktor Farcic

@vfarcic

TechnologyConversations.com

CloudBees.com

Viktor Farcic

Changed everything

Monoliths

become microservices

Why now?

Changed everything

OS infrastructure

moves to containers

and becomes immutable

Consequences?

Changed everything

Servers

become cluster

The result?

Changed everything

Cluster

changes software architecture

into immutable, stateless services

that communicate through service discovery

The need for new processes!

Changed everything

Agile

changed how we develop software

but it failed to deliver it

It excluded operations!

Changed everything

DevOps

is new agile

It enables teams to be in full control

The result?

Changed everything

DevOps

enables continuous deployment

Changed everything

You are NOT a coder,

a tester, or an operator

You need to know everything

and then specialize

Hands-On Time

Setting Up

a Swarm Cluster

and

Running Services

Docker Swarm

Nodes Setup

for i in 1 2 3 4 5; do
  docker-machine create -d virtualbox swarm-$i
done

docker-machine ls

Provisioning

eval $(docker-machine env swarm-1)

docker swarm init --advertise-addr $(docker-machine ip swarm-1)

TOKEN=$(docker swarm join-token -q manager)

for i in 2 3; do
  eval $(docker-machine env swarm-$i)

  docker swarm join --token $TOKEN \
    --advertise-addr $(docker-machine ip swarm-$i) \
    $(docker-machine ip swarm-1):2377
done

Provisioning

TOKEN=$(docker swarm join-token -q worker)

for i in 4 5; do
  eval $(docker-machine env swarm-$i)

  docker swarm join --token $TOKEN \
    --advertise-addr $(docker-machine ip swarm-$i) \
    $(docker-machine ip swarm-1):2377
done

Networking

docker network create --driver overlay proxy

docker network create --driver overlay go-demo

docker network ls

Service

docker service create --name go-demo-db \
  --network go-demo --reserve-memory 100m mongo

docker service create --name go-demo -e DB=go-demo-db \
  --network go-demo --network proxy --reserve-memory 50m \
  --label com.df.notify=true --label com.df.distribute=true \
  --label com.df.servicePath=/demo --label com.df.port=8080 \
  vfarcic/go-demo

Reverse Proxy

docker service create --name swarm-listener \
  --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
  -e DF_NOTIF_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure \
  -e DF_NOTIF_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove \
  --constraint 'node.role==manager' --network proxy \
  vfarcic/docker-flow-swarm-listener

docker service create -e MODE=swarm -e LISTENER_ADDRESS=swarm-listener \
  -p 80:80 -p 443:443 --network proxy --replicas 3 \
  --name proxy vfarcic/docker-flow-proxy

docker service ls

curl -i $(docker-machine ip swarm-1)/demo/hello

Monitoring

  • Monitoring a single application
  • Monitoring services in single server
  • Monitoring many services in a cluster

Monitoring

What Do We Need?

Historical and (near) realtime information about our system

  • logs
  • hardware utilization
  • health checking
  • network traffic
  • ...

Monitoring

What Do We Need?

Decentralized data collectors

and centralized data storage

Logging

  • A way to parse data and send them to a central database in near real-time.
  • The capacity of the database to handle near real-time data querying and analytics.
  • A visual representation of the data through filtered tables, dashboards, and so on.

Logging

Tools

... or one of many other options

Hands-On Time

Ship logs

from any container

running anywhere inside the cluster

Elastic Search Service

eval $(docker-machine env swarm-1)

docker network create --driver overlay elk

docker service create --name elasticsearch --network elk \
  -p 9200:9200 --reserve-memory 500m elasticsearch:2.4

docker service ps elasticsearch

Log Stash Service

mkdir -p docker/logstash

cp conf/logstash.conf docker/logstash/logstash.conf

cat docker/logstash/logstash.conf

docker service create --name logstash \
  --mount "type=bind,source=$PWD/docker/logstash,target=/conf" \
  --network elk -e LOGSPOUT=ignore --reserve-memory 100m \
  logstash:2.4 logstash -f /conf/logstash.conf

docker service ps logstash

Testing Log Stash

LOGSTASH_NODE=$(docker service ps logstash | tail +2 | awk '{print $4}')

eval $(docker-machine env $LOGSTASH_NODE)

LOGSTASH_ID=$(docker ps -q --filter "ancestor=logstash:2.4")

docker logs $LOGSTASH_ID

Testing Log Stash

eval $(docker-machine env swarm-1)

docker service create --name logger-test --network elk \
  --restart-condition none \
  debian logger -n logstash -P 51415 hello world

eval $(docker-machine env $LOGSTASH_NODE)

docker logs $LOGSTASH_ID

eval $(docker-machine env swarm-1)

docker service rm logger-test

Log Spout Service

docker service create --name logspout \
  --network elk --mode global -e SYSLOG_FORMAT=rfc3164 \
  --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
  gliderlabs/logspout syslog://logstash:51415

docker service ps logspout

Testing Log Spout

docker service create --name registry \
  -p 5000:5000 --reserve-memory 100m \
  --mount "type=bind,source=$PWD,target=/var/lib/registry" \
  registry

docker service ps registry

eval $(docker-machine env $LOGSTASH_NODE)

docker logs $LOGSTASH_ID

Kibana Service

docker service create --name kibana \
  --network elk --network proxy --reserve-memory 50m \
  -e ELASTICSEARCH_URL=http://elasticsearch:9200 \
  --label com.df.notify=true --label com.df.distribute=true \
  --label com.df.servicePath=/app/kibana,/bundles,/elasticsearch \
  --label com.df.port=5601 \
  kibana:4.6

docker service ps kibana

open http://$(docker-machine ip swarm-1)/app/kibana

Metrics

  • A way to get hardware and software metrics
  • Without coupling with services
  • A way to query metrics
  • A way to visualize metrics
  • A way to alert based on metrics

Tools



Hands-On Time

Expose metrics from

hardware,

containers,

and the proxy

Node Exporter

eval $(docker-machine env swarm-1)

docker service create --name node-exporter \
  --mode global --network proxy \
  --mount "type=bind,source=/proc,target=/host/proc" \
  --mount "type=bind,source=/sys,target=/host/sys" \
  --mount "type=bind,source=/,target=/rootfs" \
  prom/node-exporter:0.12.0 -collector.procfs /host/proc -collector.sysfs /host/proc -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

docker service ps node-exporter

UTIL_ID=$(docker ps -q --filter label=com.docker.swarm.service.name=util)

docker exec -it $UTIL_ID apk add --update curl

docker exec -it $UTIL_ID curl http://node-exporter:9100/metrics

Container Exporter

docker service create --name cadvisor \
  --network proxy --mode global \
  --mount "type=bind,source=/,target=/rootfs" \
  --mount "type=bind,source=/var/run,target=/var/run" \
  --mount "type=bind,source=/sys,target=/sys" \
  --mount "type=bind,source=/var/lib/docker,target=/var/lib/docker" \
  google/cadvisor:v0.24.1

docker service ps cadvisor

docker exec -it $UTIL_ID curl http://cadvisor:8080/metrics

Proxy Exporter

docker service create --name proxy-exporter \
  --network proxy \
  prom/haproxy-exporter:v0.7.1 \
  -haproxy.scrape-uri="http://admin:admin@proxy/admin?stats;csv"

docker service ps proxy-exporter

docker exec -it $UTIL_ID curl http://proxy-exporter:9101/metrics

Hands-On Time

Execute ad-hoc queries

and create dashboards

Prometheus

cat conf/prometheus.yml

docker service create --name prometheus \
  --network proxy -p 9090:9090 \
  --mount "type=bind,source=$PWD/conf/prometheus.yml,target=/etc/prometheus/prometheus.yml" \
  prom/prometheus:v1.2.1

docker service ps prometheus

open http://$(docker-machine ip swarm-1):9090

# rate(haproxy_frontend_http_requests_total{job="proxy"}[1m])

for i in {1..20}; do
  sleep 0.$[ (((($RANDOM % 10) + 1) * 100)) ]s && curl $(docker-machine ip swarm-1)/demo/hello
done

Grafana

docker service create --name grafana \
  --network proxy -p 3000:3000 grafana/grafana:3.1.1

docker service ps grafana

open http://$(docker-machine ip swarm-1):3000

# DS: http://prometheus:9090
# DS: http://elasticsearch:9200

open https://grafana.net/dashboards/704

open https://grafana.net/dashboards/405

open https://grafana.net/dashboards/609
"Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves" -– Alan Kay

Viktor Farcic

@vfarcic

TechnologyConversations.com

Amazon
LeanPub
 
LeanPub

Cleanup

docker-machine rm -f \
    swarm-1 swarm-2 swarm-3 swarm-4 swarm-5