In this episode Vipul, Rahul and Vishal discuss deployment lifecycle of Rails on Kubernetes includingRolling restart with zero downtime.Key Points From This Episode Image building and Image caching Moving from docker hub to Jenkins Reduction in build time after moving from docker hub to Jenkins Tools involved in managing cluster Kubernetes namespacing and labellingLinks Mentioned in This Episode BigBinary Website BigBinary on Twitter Vipul on Twitter Vishal on Twitter Rahul on Twitter Kubernetes Rancher Docker Cloud OpenShiftTranscript[Hi Vishal, hi Rahul.[[[How exactly – what exactly is this app and how it get – what exactly is this real app and how it gets built or deployed? What other things are involved in this?[Obviously, we have our code base on Github and whenever we commit, we have to deploy that change in any of the deployment app, maybe it is a EC2 or Heroku with containers, we have to build an image and we have to deploy it on Kubernetes.Yeah, first thing is like, build an image then push it to some Docker registry, maybe Docker hub, quay.io or self hosted Docker registry. Then trigger your Kubernetes deployment so that your Docker image is pulled and deploy it in rolling restart. Meaning that without zero downtime.This is all the basics steps to what we have in our deployment lifecycle and important part where we had to work on was about image building, automation of image building and caching and also deploying all the apps and zero downtime. Taking care of somethings like configuration changes, how to do that in run time and how we can take care of all the interlinked services like databases and other things.Yeah, I think we can talk more about how we have achieved image building and caching. Vishal, can you brief about like how we started on image building?[The standard where is to update command as well as update install command on the same length using – you can mention it as the update and person on person update install and the list of packages of the databases.Because of this, the daily instruction so this line will be colorized instruction in Docker file and this instruction will be cached when you try to build another image from the same Docker file and the cached instruction will be used and start of, you know, doing the same thing over again when you build another image. After that…[[We build different – we build the same image but instead, we are passing as an argument to container while running those images. Let’s say if we want to run only web image, we’ll just pass an argument, “Hey, this container needs to be started with a pod type or container web and it will start only unicorn process.” Same applies for sidekiq and other processes as well. We are using the same image for running all the four services and that is helping out studio the build time and deploy time. This is where we are achieving caching and image building.There is good support from Docker Hub for image caching and whenever we are pushing a commit to build an image, it automatically uses the previous build and it took – uses it from cache and our image is built. While we were building an image, I think when Vishal used all the build and stuff, he used to stuck on like on image building while using Docker Hub. Because Docker Hub used to take around 10, 15 minutes to build our image and it did not support a parallel rails in our initial release plan.We searched for some tools which could speed up our deployment process and I think this is where we led to use something like Jenkins. With Jenkins, really moved our image building process faster and that we started to run out of build in parallel and reduced the build time as well. Yeah, we Vishal, how easy was it from moving Docker building thing from Docker Hub to Jenkins?[The same image, I was trying to build on my local macro machine, it was taking just around 5 to 10 minutes but on Docker Hub, it was taking around 20 to 30 minutes and that too was happening sequentially, which means if I submit more than two builds, there were actually – and in Docker Hub was picking up one by one.Yeah, actually, we were discussing about like how caching was done. Docker has so many ways to reduce the build time and optimize the build size as well. There are some standard ways. In most of the cases what happens — so Docker, what the lines which we write in Docker file, those are called as instructions and in most cases, the instructions are, you know — I mean, the caching is performed by comparing just the instructions and that could help Docker to decide whether to use previously cached instruction to invalidate that cache and rebuild the same. For each instruction, cache actually – the child image will be produced and that will be considered as a leap of cache leap or that particular instruction.That will be later on can be used by Docker as the cache product particular instruction. This happens but most of this happens but most of this is like there are instructions. One is to – one instruction can be from them we can specify the base image tag to promise one of the instruction and that is – let’s say expose them command.These are kind of instructions which are, you know, simply compared or directly by Docker to check whether they can be – there is a cache layer available whether we use it or not. There are some minor instructions like [inaudible], which I mean, that there is some different approach that’s taken by Docker to check whether to use the cache layer generator to those instructions.What happens, all the copy instructions are used to copy or add the tiles form the build context to the build that is being generated. So what happens is that Docker looks at the last modified and the last access, these kind of some meta data which Docker looks at it and something is changed and it actually generates check sum best kind of meta data and this meta data is modified somehow, which is build context for building the image, it will be used to compare and desire for their two use, you know, previously cache layer for that particular ad or instruction.[[Both of these services, we were not able to see the improvements and we also looked at some other options like grey and what not but talked about that, let’s try to build it on our own, or Jenkins server. I mean, we setup Jenkins and installed a few packages like Git to pull the source code from git hub.Another package was cloud based Docker build and published plugin, we’ll mention the link to this again in the notes of this podcast. This cloud-base Docker build and publish plugin on Jenkins it is used to – it provides a UI to configure like from which Docker file. So actually we can specify the path to the Docker file and the directory that we have cloned using git plugin. We can specify the part and in the options provided by this cloud based Docker build and publish plugin, we can specify that.Docker build command should be run and that Docker file along with some options like what should be built on the names and what not. Also, we can use this plugin to push the build image to any registry. We still use Docker platform, I mean, Docker Hub to push our build images and actually it happens in just a few seconds before Jenkins, which needs on AWS where there in no network problem actually.The initial improvement we saw is that actually it was our own machine so we allotted high, I mean, we used machine which had high configurations or high resource, or I mean more resources actually. Because of that, we were able to run more builds in parallel actually. The best implement we saw is that we ca manually trigger the builds. As compared to Docker where it wasn’t allowed to modify us the Docker build command, which is actually run on their service. Here on Jenkins, we were able to modify the Docker build command as per our need. We can specify the environment where we say build arguments on Jenkins.Another thing was, we were able to see like what is being cached and I mean, it was easy to debug like what is really happening and we were, you know, it was easy to fix it. There are any issues while building the image and another improvement was now, it was our own server and we had more resources as per our need, we can scale it up or down as per our need based on the usage of resources and based on the usage that we were able to see on that machine. Now, the build was taking just around four to five minutes and that was a big improvement for us.[[[On Jenkins, we used, in startup, we’re using simple build. We use parameterized build on Jenkins. I mean, the parameterized build is basically build triggered using an URL, which we can specify developers which are already known to Jenkins server and those parameters, I mean, we use the Jenkins parameterized to URL from outside. Actually, our use cases to trigger Jerkins build from our own internal tool from there we trigger the build.We ask user, like prompt user, “From which branch do we want to build the image?” And usually this is or this app from this particular branch and for this particular environment like production and changing or something like that. I want to build the image and perform and deploy.So all of these import is submitted to Jenkins as part of the parameterized build and that parameterized build then are Docker build command using our cloud based Docker build and publish plug in. That information is then used in our instructions to actually run some commands while building the image. Environment is one thing that we –[[User should be able to build based on, I mean, the ticket transitions between you know, some three different steps then we would be able to trigger the Jenkins spheres and perform and deploy.[[We obviously checked in Kubernetes and community and it has a really good tool, called “helm” and it manages all the deployment releases and that what’s really take. We are on public cloud and we were deploying all our application on to our Kubernetes cluster.As per our need, we wanted something in the early assemble phase. In this particular project case, the climate was to have a single click deployment or single command deployment. Though, helm was good, even but we wanted to make our Kubernetes cluster most simpler even for non technical people.Even the helm was good, we thought of using it as like a deployment tool so we came up with our own small tool, we — it works just like helm, it updates all the manifests and using that UI which is like coupled to our cloud provider, it also modifies the database instances and other things.That is just a tool, but just talking about helm, there is also Red Hat’s Openshift, which manages the deployment things as well and that is also one of the good tools. If you are not – I will again say that this is really opinionated on each user like wich tool to use, there are community tools from Netflix, from ZenDesk and all. People are coming up with their own tools and utilities as per the application need.[RAHUL: Yup. This is all about deployment tools if we want to talk about like cluster provisioning, maybe there are various ways to deploy a cluster as well. When we talk about deploying apps on public cloud, I don’t think there are – well known plugin providers provide Kubernetes.At least AWS doesn’t, there are some tool out or there are too some community tools to build up your own cluster and maintain it high availability. We explored kubespray, we explored kubeadm, we explored kops and decided to go with kops. kops is a Kubernetes cluster provisioning tool, we are using which we can implement HA and disaster recovery on Kubernetes cluster. We started off with it and it seemed really good.More towards upgrading that you want to upgrade form one Kubernetes version to higher Kubernetes version, we can do it without down time. Also, it’s a supports private networking using calico like a flannel, weave and other tools but we don’t use as kubenet. The important part we were impressed using kops is like it’s supported private sub netting out of the box. In our traditional architecture, we had all the productions machines that were behind the VPN and we had our natting instance using which are traffic used to go.Using private sub netting using kops, we deployed all our nodes in private subnet even after some private subnet and we had only one jump box or bastion which has traffic flow.Using that tool down the line – behind the scene it uses, private sub nets of AWS as we won’t be going that much into details and we used calico as our networking. Instead of kube net, kubnet is the default networking of kubernetes.That reallly made provisioning cluster easier and we were managing our Kubernetes clusters using kops, that is one of the important tool. I would mention helm, kops and some of the tools like open shift who were like –[[[[Let’s say we have three pod but there’s no point running db:migrate on all of the three pods even though they’re identical. We came across something called Kubernetes jobs which helps us running some custom commands and while starting deployment, let’s say like, our deployment way is push the commit on manual trigger, we have build in Jenkins and pushed our Docker image to docker registry.We want to deploy, we have pulled our latest image in our deployment manifest and we’re deploying. While deploying, we just – when we pull an image but before applying a deployment for other service, we just executed, our DB migrate job which just runs only ones before deployment.That helps us to keep the DB migrate task out of web pods or web containers and executed only once. This is how we achieve custom tasks.[[[[And even though the services were really managed beautifully in Kubernetes using service name spaces and service labels, that really helped us. But for cloud providers, Kubernetes gives us a service type of load balancer especially for our web service as a load balancer, it automatically clears an ELB or AWS managers security group and our application is up and running on ELB URL. But this is where we obviously, we don’t want to give a client…[[So we have to use some DNS, meaningful DNS and we have to keep an update of the DNS either way even the services recreated on Kubernetes or ABC.com should serve the same application you wanted. So there are some add-ons like route 53 add on for AWS which keeps our configures that outfit with route 53 domains with Kubernetes ELB as well. So this is how like when our app is deployed and our services up and running, our ELB end point is mapped with a route 53 domain and client accesses it using route 53 domain.[[So in Kubernetes there are name spaces which helps you in isolating the naming convention and there is something called labeling. So we can label our nodes meaning that in our cluster there are 10 nodes. We will label them, two of the nodes will be labled like client is equal to ABC and when we are deploying or deploying manifest, we will assign that label to our pod specification. Meaning that when I am mentioning client is equal to ABC, all of my ABC client pods that are deployed only on node which has label ABC. So that helped us really like segregating the environments for different clients and labeling is one of the things. Also with name spaces they had –[[And all the related secrets and all are in name space or specific client and the labeling feature is also a native feature from Kubernetes. We can label our nodes and we can use that node selector using node selector parameter between our deployment manifest. We can specify to our pods say go and run on this pod so that Kubernetes schedules that pod which matches the label.[[So as we are on AWS, we choose to use RDS and for our background jobs, we use Redis so elastic cache also. The elastic cache is the one which is available from AWS. So interesting part also is like when we were talking about single click deployment or single command deployment, if we are provisioning resources for Kubernetes, we can just create it from existing deployment.So here it comes, we started off with some declarative tools like Ansible and Chef. So initially we started off if we are creating one new application we will create a database RDS for him. We’ll retrieve the end point of that RDS and that end point will be created as in secret on Kubernetes cluster and the name of that secret will be embedded in our deploying manifest. The same goes for elastic cache as well. On first provisioning, we have to create all other related resources if you are using some object storage like S3 create to S3 bucket for it and give the appropriate permissions and also restore the credentials.So there are services like where we need to use S3 credentials and this is where something like kube2iam, which allows us not to feed our AWS keys or any application since we are using it into portal deployment specs. Instead, you create an AWS role and just to give some role specific permission to the kube2iam. You can just Google it out and that is a good add on. So this all to get started but initially if you are provisioning, you have to come up with your own scripts.So then further when we move to our internal tool, we choose the cloud provider API like AWS, SDK to manage all of these database stuff. But I would say that is only a one-time thing when we are creating the app and further it is our Kubernetes deployment templates itself.[[The first thing is when we are using persistent volume claims, we can’t scale up and down because volume is attached to single node of single pod and when we are scaling up, AWS doesn’t allow us to scale a single volume with multiple instances. So that is first and prime limitation by AWS and up on when we recreate a pod and we reapply the plugin there are chances that AWS stucks your volume meaning that how fastly the Kubernetes applies its deployment templates, AWS sometimes does not able to detach the volume from the running node and attach it back to the next running node. Where Kubernetes should be able to run the next pod.So these are some of the runtime issues and we will first take some time and this is where we don’t use it for critical applications but yeah, we are effectively using it for some of the applications where really we have to store some GB’s of data like 40, 50 GB’s of data onto this and that is working good as far as we don’t really deploy frequently on those app. So persistent volume claim is one of the good choice.But I would say if you are really interested in using stateful apps I would rather say try, you can try stateful apps. Now also there are something like open EBS coming up and Rook as well. So I would say what is happening all over the company to get it up and running even Kubernetes 1.7 and 1.8 says that they should officially people should move to use Kubernetes for stateful apps as well.[[The other day when traffic was in peak, we have to spin up something numbers of web pods. If we would have been using persistent volume over there either we would have to on runtime we would have to spin up 10 new persistent volumes. Our AWS would have limited over there so that you can share a single wall under 10 new ports.[[So YAMAL and just on manifest are in control of our Ruby app because those are the really important thing when we are trying to deploy. So the Ruby apps manages it really well but people like Openshift and Helm, if you are just starting off you can either choose Helm or Openshift, which is just the management of manifests.[[[[END]