CloudBees Jenkins


Viktor Farcic


@vfarcic

TechnologyConversations.com

CloudBees.com

Viktor Farcic

Warning


  • I speak fast
  • I do things even faster
  • Stop me if you get lost
  • It's OK to ask questions at any time
  • You will not learn how to do something
  • ... but you will learn what can be done
  • Everything is online
  • I will make a mistake. Don't be judgemental!

Jenkins


  • Continuous Integration
  • Continuous Delivery
  • Continuous Deployment
  • Orchestrator
  • Awesome

Question


Can I set up a fully functioning production cluster with two Jenkins services registered in CJOC?

... and create a continuous deployment flow executing on each commit?

... and explain the process?

... and be over with it in 45 minutes?

What?


  • Create a Swarm cluster
  • Create production services
  • Create CJOC service
  • Create two Jenkins master services
  • Register Jenkins masters in CJOC
  • Install plugins across all masters
  • Create Jenkins agents

45 minutes?

CJOC


A central place to manage all masters

... because there might be too many

Hands-On Time


Setting Up CJOC and masters

Creating a Cluster


git clone https://github.com/vfarcic/vfarcic.github.io.git

cd vfarcic.github.io/jenkins

./bootstrap-cb.sh

eval $(docker-machine env swarm-1)

docker node ls

docker service ls

curl -i $(docker-machine ip swarm-1)/demo/hello

Creating CJOC Service


docker network create --driver overlay jenkins

mkdir -p docker/jenkins-oc

docker service create --name jenkins-oc \
  --network jenkins -p 8090:8080 -p 50000:50000 \
  --constraint 'node.labels.env == prod' --reserve-memory 300m \
  --mount "type=bind,source=$PWD/docker/jenkins-oc,target=/var/jenkins_home" \
  cloudbees/jenkins-operations-center:2.7.21.1

docker service ps jenkins-oc

Creating Jenkins Services


for i in 1 2; do
  mkdir -p docker/jenkins-$i

  docker service create --name jenkins-$i \
    --network jenkins -p 808$i:8080 -p 5000$i:50000 \
    --constraint 'node.labels.env == prod' --reserve-memory 300m \
    --mount "type=bind,source=$PWD/docker/jenkins-$i,target=/var/jenkins_home" \
    cloudbees/jenkins-enterprise:2.7.21.1
done

docker service ps jenkins-1

docker service ps jenkins-2

Setting Up CJOC


open http://$(docker-machine ip swarm-1):8090

cat docker/jenkins-oc/secrets/initialAdminPassword # CJOC password

cat license-key.secret

cat license-cert.secret

# Use admin/admin as user/pass

# Choose "Install Suggested Plugins"

Setting Up CJOC


# Create a new "Client Master" job "jenkins-1"

echo http://$(docker-machine ip swarm-1):8081 # Master address

cat docker/jenkins-1/secrets/initialAdminPassword # Master password

# Install suggested plugins

# Create a new "Client Master" job "jenkins-2"

echo http://$(docker-machine ip swarm-1):8082

cat docker/jenkins-2/secrets/initialAdminPassword # Master password

# Install suggested plugins

installing Plugins


  • Create new Cluster Operations job
  • Add Operation > Masters
  • Check From Operations Center Root
  • Add Step > Install Plugin
  • Plugin ID = swarm
  • Version = 2.2
  • Save > Run

Creating Agents


docker-machine ssh swarm-4

sudo mkdir -p /workspace && sudo chmod 777 /workspace && exit

docker-machine ssh swarm-5

sudo mkdir -p /workspace && sudo chmod 777 /workspace && exit

export USER=admin

export PASSWORD=admin

Creating Agents


docker service create --name jenkins-agent \
  --mode global --constraint 'node.labels.env == test' \
  --network jenkins \
  -e COMMAND_OPTIONS="-master http://jenkins-1:8080 -username $USER -password $PASSWORD -labels 'docker' -executors 5" \
  --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
  --mount "type=bind,source=$HOME/.docker/machine/machines,target=/machines" \
  --mount "type=bind,source=/workspace,target=/workspace" \
  vfarcic/jenkins-swarm-agent:latest

open http://$(docker-machine ip swarm-1):8081/computer/

docker service ps jenkins-agent

DRY


open http://$(docker-machine ip swarm-1):8081/configure

docker-machine ip swarm-1 # Env. var PROD_ID

# PROD_NAME = swarm-1

Production Services


docker service ps go-demo

Jenkins Pipeline


open http://$(docker-machine ip swarm-1):8081/newJob

# Name it "go-demo"

# Select "Pipeline" as job type

Jenkins Pipeline


node("docker") {

  git "https://github.com/vfarcic/go-demo.git"

  withEnv([
    "COMPOSE_FILE=docker-compose-test-local.yml",
    "COMPOSE_PROJECT_NAME=go-demo-master"
  ]) {

    stage "Unit"
    sh "docker-compose run --rm unit"
    sh "docker build -t go-demo ."

    stage "Staging"
    try {
      sh "docker-compose up -d staging-dep"
      sh 'HOST_IP=localhost docker-compose run --rm staging'
    } catch(e) {
      error "Staging failed"
    } finally {
      sh "docker-compose down"
    }

    stage "Publish"
    sh "docker tag go-demo localhost:5000/go-demo:2.${env.BUILD_NUMBER}"
    sh "docker push localhost:5000/go-demo:2.${env.BUILD_NUMBER}"

    stage "Production"
    withEnv([
      "DOCKER_TLS_VERIFY=1",
      "DOCKER_HOST=tcp://${env.PROD_IP}:2376",
      "DOCKER_CERT_PATH=/machines/${env.PROD_NAME}"
    ]) {
      sh "docker service update --image localhost:5000/go-demo:2.${env.BUILD_NUMBER} go-demo"
    }
	for (i = 0; i <5; i++) {
      sh "HOST_IP=${env.PROD_IP} docker-compose run --rm production"
    }
  }

}

Production Updated


docker service ps go-demo

Jenkinsfile


Decentralized job definitions

... in code repository

... where they belong

Hands-On Time


Defining jobs outside Jenkins

Creating a Jenkinsfile job


open https://github.com/vfarcic/go-demo/blob/devops21-dm/Jenkinsfile

open https://github.com/vfarcic/go-demo/blob/master/Jenkinsfile

open http://$(docker-machine ip swarm-1):8081/newJob

# Name it "go-demo-mb"

# Select "Multibranch Pipeline" as job type

# Add Source > GitHub

# Owner = vfarcic

# Repository = go-demo

Global Library


To not repeat yourself...

Hands-On Time


Defining global functions

Global Functions


open http://$(docker-machine ip swarm-1):8081/configure

# Global Pipeline Libraries > Add
# Name = my-lib
# Default Version = master
# Legacy SCM > Git
# Repository URL = https://github.com/vfarcic/go-demo.git

open http://$(docker-machine ip swarm-1):8081/job/go-demo-mb/job/global-lib/

Artifactory


In case you need to store stuff...

Hands-On Time


Storing artifacts in Artifactory

Artifactory Service


docker service create --name artifactory \
  --constraint 'node.labels.env == prod' -p 8091:8081 \
  docker.bintray.io/jfrog/artifactory-oss:latest

docker service ps artifactory

open http://$(docker-machine ip swarm-1):8091/artifactory/webapp/#/admin/repository/local/new

# Use admin/password as user/pass

# Add "Generic" repository called "go-demo"

Artifactory in Jenkins


open http://$(docker-machine ip swarm-1):8081/pluginManager/available

# Install "Artifactory Plugin"

open http://$(docker-machine ip swarm-1):8081/configure

# Artifactory servers > Add

# Server ID = artifactory-1

# Username/Password = admin/password

echo http://$(docker-machine ip swarm-1):8091/artifactory # URL

Running


open https://github.com/vfarcic/go-demo/blob/artifactory/Jenkinsfile

open http://$(docker-machine ip swarm-1):8081/job/go-demo-mb/job/artifactory/

open http://$(docker-machine ip swarm-1):8091/artifactory/webapp/#/artifacts/browse/tree/General/go-demo

RBAC


When you have trust issues...

Hands-On Time


Securing Jenkins with RBAC

RBAC Setup


open http://$(docker-machine ip swarm-1):8081/pluginManager/available

# Install the "Mock Security Realm" plugin

open http://$(docker-machine ip swarm-1):8081/configureSecurity

# Select "Enable Security"

# Select "Mock Security Realm"

# Specify "god admin", "admin admin" and "developer devs" as users and groups

# Select "Role-based matrix authorization strategy"

RBAC Setup


open http://$(docker-machine ip swarm-1):8081/plugin/nectar-rbac/manage/
# Add "administrator" role and select all permissions
# Add "developer" role and select "Overall Read", "Job Build & Read" permissions
open http://$(docker-machine ip swarm-1):8081/groups/newGroup
# Type "admins" as "Group name"
# Select "Granted" to "administrator"
# Click "Add user/group" and type "admin"
# Click "Back to Groups" > "New Group"
# Type "developers" as "Group name"
# Select "Granted" to "developer"
# Click "Add user/group" and type "dev"
open http://$(docker-machine ip swarm-1):8081/plugin/nectar-rbac/manage/
# Deselect all permissions to "authenticated"
# Explore logging in as "dev" or "admin"

Folders


Things should be organized...

Hands-On Time


Organizing Jenkins jobs through folders

Folders Setup


open http://$(docker-machine ip swarm-1):8081

# Click "New Item"

# Name if "my-folder" and select "Folder" as type

open http://$(docker-machine ip swarm-1):8081/job/go-demo/

# Click "Move/Copy/Promote", write "/my-folder" as "Destination", click "Move"

Input


Because you have trust issues...

Hands-On Time


Waiting for a confirmation

Input step in a pipeline


open https://github.com/vfarcic/go-demo/blob/trust-issues/Jenkinsfile

Notifications


To stop you from doing something else...

Hands-On Time


Sending email on buil failure

SMTP Service


docker service create --name postfix --network jenkins -p 25:25 \
    -e maildomain=mail.acme.com -e smtp_user=admin:admin \
    catatnight/postfix:latest

open http://$(docker-machine ip swarm-1):8081/configure

# SMTP server = postfix
# User Name / Password = admin/admin
# SMTP Port = 25

open https://github.com/vfarcic/go-demo/blob/notif/Jenkinsfile

open http://$(docker-machine ip swarm-1):8081/job/go-demo-mb/job/notif/

Jenkins Best Practices

  • Always secure Jenkins
  • Don't build on the master
  • Backup Jenkins Home regularly
  • Setup Jenkins on a fast hard disk
  • Cleanup (remove) old builds
  • Create a single Pipeline job for a project
  • Fail fast
  • Fixing a broken build has the highest priority
  • DRY, define pipeline shared libraries
  • Do not pull SCM, use web hooks
  • Use Jenkinsfile

Pipeline Best Practices

  • Treat pipelines as code
  • Always work inside stages
  • Do all material work inside nodes
  • Run steps in parallel
  • Define nodes inside parallel steps
  • Don't use input inside a node
  • Wrap inputs with timeouts
  • Use withEnv instead global env. vars
  • Use stash/unstash to share files between nodes

Viktor Farcic


@vfarcic


TechnologyConversations.com

Amazon
LeanPub

 
LeanPub

Cleanup

docker-machine rm -f swarm-1 swarm-2 swarm-3 swarm-4 swarm-5

rm -rf docker go-demo