28 Apr 2019

Running PostgreSQL in a Cloud on Oracle Containers Engine for Kubernetes

In this post I am going to show a few steps to deploy and run PostgreSQL database in a K8S cluster on OKE.

The deployment is going to be based on postgres:11.1 Docker image which requires a few environment variables to be configured: POSTGRES_DB (database name), POSTGRES_USER and POSTGRES_PASSWORD. I am going to store values of these variables in K8S ConfigMap and Secret:
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgre-db-config
data:
  db-name: flexdeploy

apiVersion: v1
kind: Secret
metadata:
  name: postgre-db-secret
stringData:
  username: creator
  password: c67

These configuration K8S resources are referenced by the StatefulSet which actually takes care of the lifespan of a pod with postgres-db container:
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: postgre-db
  labels:
    run: postgre-db
spec:
  selector:
      matchLabels:
        run: postgre-db
  serviceName: "postgre-db-svc"
  replicas: 1
  template:
    metadata:
      labels:
        run: postgre-db
    spec:
      containers:
      - image: postgres:11.1
        volumeMounts:
           - mountPath: /var/lib/postgresql/data
             name: db     
        env:
          - name: POSTGRES_DB
            valueFrom:
              configMapKeyRef:
                   name: postgre-db-config
                   key: db-name                  
          - name: POSTGRES_USER
            valueFrom:
              secretKeyRef:
                   name: postgre-db-secret
                   key: username                  
          - name: POSTGRES_PASSWORD
            valueFrom:
              secretKeyRef:
                   name: postgre-db-secret
                   key: password                  
          - name: PGDATA
            value: /var/lib/postgresql/data/pgdata           
        name: postgre-db
        ports:
        - containerPort: 5432
  volumeClaimTemplates:
   - metadata:
       name: db
     spec:
       accessModes: [ "ReadWriteOnce" ]
       resources:
         requests:
           storage: 3Gi  


StatefulSet K8S resource has been specially designed for stateful applications like database that save their data to a persistent storage. In order to define a persistent storage for our database we use another K8s resource Persistent Volume and here in the manifest file we are defining a claim to create a 3Gi Persistent Volume with name db. The volume is called persistent because its lifespan is not maintained by a container and not even by a pod, it’s maintained by a K8s cluster. So it can outlive any containers and pods and save the data. Meaning that if we kill or recreate a container or a pod or even the entire StatefulSet, the data will be still there. We are referring to this persistence volume in the container definition mounting a volume on path /var/lib/postgresql/data. This is where PostgreSQL container stores its data.

In order to access the database we are going to create a service:
apiVersion: v1
kind: Service
metadata:
  name: postgre-db-svc 
spec:
  selector:
    run: postgre-db
  ports:
    - port: 5432
      targetPort: 5432 
  type: LoadBalancer   

This is a LoadBalancer service which is accessible from outside of the K8S cluster:
$ kubectl get svc postgre-db-svc
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
postgre-db-svc   LoadBalancer   10.96.177.34   129.146.211.77   5432:32498/TCP   39m

Assuming that we have created a K8s cluster on OKE and our kubectl is configured to communicate with it we can apply the manifest files to the cluster:
kubectl apply -f postgre-db-config.yaml
kubectl apply -f postgre-db-secret.yaml
kubectl apply -f postgre-db-pv.yaml
kubectl apply -f postgre-db-service.yaml

Having done that, we can connect to the database from our favorite IDE on our laptop

The manifest files for this post are available on GitHub.

That's it!

20 Apr 2019

Deployment Strategies with Kubernetes and Istio

In this post I am going to discuss various deployment strategies and how they can be implemented with K8s and Istio. Basically the implementation of all strategies is based on the ability of K8s to run multiple versions of a microservice simultaneously and on the concept that consumers can access the microservice only through some entry point. At that entry point we can control what version of a microservice the consumer should be routed to.

The sample application for this post is going to be a simple Spring Boot application wrapped into a Docker image. So there are two images superapp:old and superapp:new representing an old and a new versions of the application respectively:

docker run -d --name old -p 9001:8080 eugeneflexagon/superapp:old
docker run -d --name new -p 9002:8080 eugeneflexagon/superapp:new


curl http://localhost:9001/version
{"id":1,"content":"old"}

curl http://localhost:9002/version
{"id":1,"content":"new"}


Let's assume the old version of the application is deployed to a K8s cluster running on Oracle Kubernetes Engine with the following manifest:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: superapp
spec:
  replicas: 3
  template:
    metadata:
       labels:
         app: superapp
    spec:
      containers:
        - name: superapp
          image: eugeneflexagon/superapp:old
          ports:
            - containerPort: 8080
So there are three replicas of a pod running the old version of the application. There is also a service routing the traffic to these pods:
apiVersion: v1
kind: Service
metadata:
  name: superapp
spec:
  selector:
    app: superapp   
  ports:
    - port: 8080
      targetPort: 8080      

Rolling Update
This deployment strategy updates pods in a rolling update way, changing them one by one.


This is a default strategy handled by a K8s cluster itself, so we just need to update the superapp deployment with a reference to the new image:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: superapp
spec:
  replicas: 3
  template:
    metadata:
       labels:
         app: superapp
    spec:
      containers:
        - name: superapp
          image: eugeneflexagon/superapp:new
          ports:
            - containerPort: 8080
However, we can fine-tune the rolling update algorithm by providing parameters for this deployment strategy in the manifest file:
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
       maxSurge: 30%
       maxUnavailable: 30%   
  template:
  ...

The maxSurge parameter defines the maximum number of pods that can be created over the desired number of pods. It can be either a percentage or an absolute number. Default value is 25%.
The maxUnavailable parameter defines the maximum number of pods that can be unavailable during the update process. It can be either a percentage or an absolute number. Default value is 25%.


Recreate
This deployment strategy kills all old pods and then creates the new ones.

spec:
  replicas: 3
  strategy:
    type: Recreate
  template:
  ...

Very simple.

Blue/Green
This strategy defines an old version of the application as a green one and a new version as a blue one. Users always have access only to the green version.


apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: superapp-01
spec:
  template:
    metadata:
       labels: 
         app: superapp
         version: "01"
...



apiVersion: v1
kind: Service
metadata:
  name: superapp
spec:
  selector:
    app: superapp 
    version: "01"
...

The service routes the traffic only to pods with label version: "01".

We deploy a blue version to a K8s cluster and make it available only for QAs or for a test automation tool (via a separate service or direct port-forwarding).



apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: superapp-02
spec:
  template:
    metadata:
       labels:
         app: superapp
         version: "02"
...


Once the new version is tested we switch the service to it and scale down the old version:



apiVersion: v1
kind: Service
metadata:
  name: superapp
spec:
  selector:
    app: superapp 
    version: "02"
...


kubectl scale deployment superapp-01 --replicas=0

Having done that,  all users work with the new version.

So there is no Istio so far. Everything is handled by a K8s cluster out-of-the box. Let's move on to the next strategy.


Canary
I love this deployment strategy as it lets the users test the new version of the application and they don't even know about that. The idea is that we deploy a new version of the application and route 10% of the traffic to it. The users have no idea about that.


If it works for a while, we can balance the traffic 70/30, then 50/50 and eventually 0/100.
Even though this strategy can be implemented with K8s resources only by playing with the number of old and new pods, it is way more convenient to implement it with Istio.
So the old and the new applications are defined as the following deployments:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: superapp-01
spec:
  template:
    metadata:
       labels: 
         app: superapp
         version: "01"
...

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: superapp-02
spec:
  template:
    metadata:
       labels:
         app: superapp
         version: "02"
...

The service routes the traffic to both of them:
apiVersion: v1
kind: Service
metadata:
  name: superapp
spec:
  selector:
    app: superapp 
...

On top of that we are going to use the following Istio resources: VirtualService and  DestinationRule.
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: superapp
spec:
  host: superapp
  subsets:
  - name: green
    labels:
      version: "01"
  - name: blue
    labels:
      version: "02"
---     
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: superapp
spec:
  hosts:
    - superapp   
  http:
  - match:
    - uri:
        prefix: /version
    route:
    - destination:
        port:
          number: 8080
        host: superapp
        subset: green
      weight: 90
    - destination:
        port:
          number: 8080
        host: superapp    
        subset: blue  
      weight: 10
The VirtualService will route all the traffic coming to the superapp service (hosts) to the green and blue pods according to the provided weights (90/10).

A/B Testing
With this strategy we can precisely control what users, from what devices, departments, etc. are routed to the new version of the application.

For example here we are going to analyze the request header and if its custom tag "end-user" equals "xammer" it will be routed to the new version of the application. The rest of the requests will be routed to the old one:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: superapp
spec:
  gateways:
    - superapp
  hosts:
    - superapp   
  http:
  - match:
    - headers:
        end-user:
          exact: xammer                 
    route:
    - destination:
        port:
          number: 8080
        host: superapp
        subset: blue
  - route:
    - destination:
        port:
          number: 8080
        host: superapp
        subset: green

All examples and manifest files for this post are available on GitHub so you can play with various strategies and sophisticated routing rules on your own. You just need a K8s cluster (e.g. Minikube on your laptop) with preinstalled Istio. Happy deploying!

That's it!

13 Apr 2019

Load Testing of a Microservice. Kubernetes way.

Let's assume there is a microservice represented by a composition of containers running on a K8s cluster somewhere in a cloud, e.g. Oracle Kubernetes Engine (OKE). At some point we want to quickly stress test a specific microservice component or the entire microservice. So we want to know how it works under the load, how it handles many subsequent requests coming from many parallel clients. The good news is that we have already a tool for that. Up and running. This is the Kubernetes cluster itself.

We're going to use Kubernetes Job for this testing described in the following manifest file:
apiVersion: batch/v1
kind: Job
metadata:
   name: job-load
spec:
   parallelism: 50   
   template:
     spec:
       containers:
         - name: loader
           image: eugeneflexagon/aplpine-with-curl:1.0.0
           command: ["time", "curl", "http://my_service:8080/my_path?[1-100]"]     
       restartPolicy: OnFailure   
This job is going to spin up 50 pods running in parallel and sending 100 requests each to my_service on port 8080 and with path my_path. Having the job created and started by invoking
kubectl  apply -f loadjob.yaml
We can observe all 50 pods created by the job using

kubectl get pods -l job-name=job-load
NAME             READY     STATUS      RESTARTS   AGE
job-load-4n262   1/2       Completed   1          12m
job-load-dsqtc   1/2       Completed   1          12m
job-load-khdn4   1/2       Completed   1          12m
job-load-kptww   1/2       Completed   1          12m
job-load-wf9pd   1/2       Completed   1          12m
...

If we look at the logs of any of these pods

kubectl logs job-load-4n262

We'll see something like the following:
[1/100]: http://my_service.my_namespace:8080/my_path?1 --> <stdout>
{"id":456,"content":"Hello world!"}

[2/100]: http://my_service.my_namespace:8080/my_path?2 --> <stdout>
{"id":457,"content":"Hello world!"}

[3/100]: http://my_service.my_namespace:8080/my_path?3 --> <stdout>
{"id":458,"content":"Hello world!"}

....

real    0m 10.04s
user    0m 0.00s
sys     0m 0.04s

That's it!