September 25, 2020
Gitlab offers a Kubernetes runner integration that you can use to create and monitor a kubernetes cluster on gitlab following a few instructions on their site. We wanted to use spot instances for our Kubernetes cluster to help keep costs down, so we did things a bit differently.
Essentially we set up an EKS cluster, added an auto-scaling node group that uses spot instances and then connected it to Gitlab and installed their integration.
AWS has a pretty thorough walk through of how to create an EKS cluster with spot instances.
If you follow their guide, you will have an EKS cluster with spot instances, here are somethings you should consider that we discovered while building our cluster:
(An example eksctl create cluster yaml file)
(example of spot instance nodegroup config file)
If you followed the guide from AWS you now have a EKS cluster with spot instances that has the node termination handler, and autoscaler installed.
You can delete the 'normal' nodegroup with: eksctl delete nodegroup --cluster=<clusterName> --name=<nodegroupName>.
Next, we followed Gitlabs instructions for adding an ‘Existing Kubernetes cluster’ to our organisation. After it has joined, which can take some time, we installed the ‘Gitlab runner’ and ‘Prometheus’, for monitoring, from the ‘Applications’ tab, and there you have it!
From here all that's needed is to tell your builds to use the new runner.
We found the process pretty easy to setup, and using config files means we can keep a record of our setup and build it again if we need to. We also connected our cluster to Rancher which is a great tool for monitoring your Kubernetes cluster.
If you are building docker images I suggest having a look at using Kaniko, a tool from Google to build images from dockerfiles inside a kubernetes cluster. Gitlab has some good documentation on how to implement it.
AWS spot instance EKS:
Gitlab kuberenetes instructions
Gitlab and Kaniko
DevOps Engineer for nerd.vision and other teams. When I am not doing that I like traveling and playing drums.