Skip to content

Setup Jenkins Pipeline and Blue Ocean in Kubernetes

Jenkins Pipeline (or simply "Pipeline" with a capital "P") is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Blue Ocean rethinks the user experience of Jenkins. Designed from the ground up for Jenkins Pipeline, but still compatible with freestyle jobs, Blue Ocean reduces clutter and increases clarity for every member of the team.

Jenkins pipeline

Install Jenkins in K8s

Create namespace

kubectl create namespace devops

Create Jenkins containers

Create pv/pvc/deployment/service/ingress by yaml files 

kubectl create -f ops-storage.yml
kubectl create -f rbac.yml
kubectl create -f jenkins.yml

Storage yaml

The content of ops-storage.yml is as bellow. 

apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node
              operator: In
              values:
                - dev
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: devops
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 20Gi

RBAC yaml

The content of rbac.yml is as bellow. 

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: devops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: devops

Jenkins yaml

The content of jenkins.yml is as bellow. Remember to replace the Traefik ingress configuration [domain] to your domain. 

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: devops
spec:
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: jenkinshome
              subPath: jenkins
              mountPath: /var/jenkins_home
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      securityContext:
        fsGroup: 1000
      volumes:
        - name: jenkinshome
          persistentVolumeClaim:
            claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: devops
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
    - name: web
      port: 8080
      targetPort: web
      nodePort: 30002
    - name: agent
      port: 50000
      targetPort: agent
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: devops
spec:
  rules:
    - host: [domain]
      http:
        paths:
          - path: /
            backend:
              serviceName: jenkins
              servicePort: web

Wait until running

Check if pods runs 

kubectl get pods -n devops

Login Jenkins

Check the Jenkins password and login, then install plugins. 

#Please check the jenkins startup logs or check the content in this file
cat /mnt/data/jenkins/secrets/initialAdminPassword

Install plugins

Installed plugins include: * Blue Ocean * Kubernetes plugin (Job and Stage Monitoring Plugin) * NodeJS Plugin * Embeddable Build Status Plugin

Jenkins Slave Image

Jenkins Slave jobs will run in your Kubernetes cluster, so it is important that the kubectl version for you Kubernetes cluster is the same with the Jenkins Slave docker image, otherwise you may meet some version conflict issues for Jenkins Pipelines jobs.

I build my own Jenkins Slave docker image hustakin/jenkins-slave:latest with the kubectl version v1.14.2. For different version, you should build your own jenkins slave docker image with another kubectl version based on image cnych/jenkins:jnlp.

Kuberctl replace

The easiest way to customize different version kubectl is to replace the executable file "kubectl" from image cnych/jenkins:jnlp. You can find the file in your kubernetes cluster or download it. 

#Find current kubectl
which kubectl
#/usr/bin/kubectl

#Download v1.14.2 kubectl and modify it to executable
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.14.2/bin/darwin/amd64/kubectl
chmod +x ./kubectl

Dockerfile

Copy the kubectl file to current folder and build the Dockerfile. 

FROM cnych/jenkins:jnlp
MAINTAINER Frankie Fan "hustakin@gmail.com"
ENV REFRESHED_AT 2019-6-26
COPY kubectl /usr/local/bin/kubectl
ENTRYPOINT ["jenkins-slave"]

Jenkins Configurations

Configure Kubernetes

Click Manage Jenkins->Configure System->Add a new cloud->Kubernetes. Test the connection and input namespace and Jenkins inner url by format (..svc.cluster.local:8080) Jenkins k8s

Configure Jenkins Slave k8s template

Add pod template, and remember to clean the "Command to run" and "Arguments to pass to the command" (This's really important!).

The image hustakin/jenkins-slave:latest has the kubectl version of v1.14.2 Jenkins k8s pod

Configure Jenkins Slave k8s volumes

Add volumes in the pod template including docker, .kube config files as well as the maven cache. (Each node of the cluster will cache the maven into /home/ec2-user/.m2) Jenkins k8s volume

Configure service account

The service account is created in previous RBAC yaml step. Jenkins service account

Configure global tool

Click Manage Jenkins->Global Tool Configuration. Configure maven installations. Jenkins maven

Configure NodeJS installations

Click Manage Jenkins->Global Tool Configuration. Configure NodeJS installations. Jenkins nodejs

Configure DockerHub credentials

Click Jenkins->Credentials->(global)->Add credentials. Specify your DockerHub username and password. Remember the ID here will be used in your Jenkinsfile to specify "credentialsId". Jenkins dockerhub

Configure Pipelines

Write Dockerfile and Jenkinsfile and k8s.yaml in your project root folder and push it to GitHub. Create the repository in Dockerhub for your project in order to save docker built images.

Jenkinsfile content for SpringBoot

The Jenkinsfile content is as bellow 

node('jenkins-jnlp') {
    env.MVN_HOME = "${tool 'Maven'}"
    env.PATH="${env.MVN_HOME}/bin:${env.PATH}"

    stage('Prepare') {
        echo "1.Prepare Stage"
        checkout scm
        script {
            build_tag = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()
            if (env.BRANCH_NAME != 'master') {
                build_tag = "${env.BRANCH_NAME}-${build_tag}"
            }
        }
    }
    stage('Compile') {
        echo "2.Compile SpringBoot App Stage"
        sh "mvn clean package -Dmaven.test.skip=true"
    }
    stage('Build') {
        echo "3.Build Docker Image Stage"
        sh "docker build -t hustakin/test:${build_tag} ."
    }
    stage('Push') {
        echo "4.Push Docker Image Stage"
        withCredentials([usernamePassword(credentialsId: 'dockerHub', passwordVariable: 'dockerHubPassword', usernameVariable: 'dockerHubUser')]) {
            sh "docker login -u ${dockerHubUser} -p ${dockerHubPassword}"
            sh "docker push hustakin/test:${build_tag}"
        }
    }
    stage('Deploy') {
        echo "5. Deploy To K8S Cluster Stage"
        sh "sed -i 's/<BUILD_TAG>/${build_tag}/' k8s.yaml"
        sh "sed -i 's/<BRANCH_NAME>/${env.BRANCH_NAME}/' k8s.yaml"
        sh "kubectl apply -f k8s.yaml --record"
    }
}

Dockerfile content for SpringBoot

The Dockerfile content is as bellow 

FROM openjdk:8-jre
MAINTAINER Frankie Fan "hustakin@gmail.com"
ENV REFRESHED_AT 2019-7-3
VOLUME /tmp
ADD web/target/test.jar test.jar
ENTRYPOINT ["java","-XX:-BytecodeVerificationLocal","-XX:-BytecodeVerificationRemote","-XX:CICompilerCount=3","-XX:InitialHeapSize=268435456","-XX:+ManagementServer","-XX:MaxHeapSize=4263510016","-XX:MaxNewSize=2147483648","-XX:MinHeapDeltaBytes=524288","-XX:NewSize=89128960","-XX:OldSize=179306496","-XX:TieredStopAtLevel=1","-XX:+UseCompressedClassPointers","-XX:+UseCompressedOops","-XX:-UseLargePagesIndividualAllocation","-XX:+UseParallelGC","-Djava.security.egd=file:/dev/./urandom","-Dspring.profiles.active=dev","-jar","/test.jar"]
EXPOSE 80 443

K8s content for SpringBoot

The k8s.yaml content is as bellow 

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: test-server
  namespace: agric
spec:
  replicas: 2
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: test-server
    spec:
      containers:
        - name: test-server
          image: hustakin/test:<BUILD_TAG>
          env:
            - name: branch
              value: <BRANCH_NAME>
          ports:
            - containerPort: 80
              protocol: TCP
              name: web
            - containerPort: 443
              protocol: TCP
              name: https
          # inventory probe
          readinessProbe:
            httpGet:
              path: /ready
              port: 80
            initialDelaySeconds: 15
            periodSeconds: 5
            failureThreshold: 5
      imagePullSecrets:
        - name: docker-hub

Jenkinsfile content for Angular

The Jenkinsfile content is as bellow 

node('jenkins-jnlp') {
    env.NODEJS_HOME = "${tool 'Node'}"
    env.PATH="${env.NODEJS_HOME}/bin:${env.PATH}"

    stage('Prepare') {
        echo "1.Prepare Stage"
        checkout scm
        script {
            build_tag = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()
            if (env.BRANCH_NAME != 'master') {
                build_tag = "${env.BRANCH_NAME}-${build_tag}"
            }
        }
    }
    stage('Compile') {
        echo "2.Compile Angular App Stage"
        sh "npm install"
        sh "npm install node-sass"
        sh "npm run prod"
    }
    stage('Build') {
        echo "3.Build Docker Image Stage"
        sh "docker build -t hustakin/test-front:${build_tag} ."
    }
    stage('Push') {
        echo "4.Push Docker Image Stage"
        withCredentials([usernamePassword(credentialsId: 'dockerHub', passwordVariable: 'dockerHubPassword', usernameVariable: 'dockerHubUser')]) {
            sh "docker login -u ${dockerHubUser} -p ${dockerHubPassword}"
            sh "docker push hustakin/test-front:${build_tag}"
        }
    }
    stage('Deploy') {
        echo "5. Deploy To K8S Cluster Stage"
        sh "sed -i 's/<BUILD_TAG>/${build_tag}/' k8s.yaml"
        sh "sed -i 's/<BRANCH_NAME>/${env.BRANCH_NAME}/' k8s.yaml"
        sh "kubectl apply -f k8s.yaml --record"
    }
}

Dockerfile content for Angular

The Dockerfile content is as bellow 

FROM nginx:1.16.0
MAINTAINER Frankie Fan "hustakin@gmail.com"
ENV REFRESHED_AT 2019-6-5
COPY nginx/default.conf /etc/nginx/conf.d/
RUN rm -rf /usr/share/nginx/html/*
COPY /dist /usr/share/nginx/html
CMD ["nginx", "-g", "daemon off;"]
EXPOSE 80

Nginx conf content for Angular

The nginx/default.conf content is as bellow. 

server {
  listen 80;
  sendfile on;
  default_type application/octet-stream;
  gzip on;
  gzip_http_version 1.1;
  gzip_disable      "MSIE [1-6]\.";
  gzip_min_length   256;
  gzip_vary         on;
  gzip_proxied      expired no-cache no-store private auth;
  gzip_types        text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
  gzip_comp_level   9;
  root /usr/share/nginx/html;
  location / {
    try_files $uri $uri/ /index.html =404;
  }
}

K8s content for Angular

The k8s.yaml content is as bellow 

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: test-front-dev
  namespace: agric
spec:
  replicas: 2
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: test-front-dev
    spec:
      containers:
        - name: test-front-dev
          image: hustakin/test-front:<BUILD_TAG>
          env:
            - name: branch
              value: <BRANCH_NAME>
          ports:
            - containerPort: 80
              protocol: TCP
              name: web
            - containerPort: 443
              protocol: TCP
              name: https
      imagePullSecrets:
        - name: docker-hub

Create Github Pipeline

Click Jenkins->Open Blue Ocean->New Pipeline, and create pipeline for your project. Github pipeline

Create Github Pipeline

The jenkins pipeline should work now. Enter your jenkins job and master branch, you could see the stage view here. Github pipeline branch

Trigger pipeline

Push your code changes to Github. Click Jenkins->Open Blue Ocean. You can trigger the build for all pipelines you created by clicking the Run button. Blue Ocean pipeline

Blue Ocean build

Click Jenkins->Open Blue Ocean, and click the pipeline you just created, you will see the Jenkinsfile stages and related logs here. Blue Ocean build