Lesson learnt from launching a Kubernetes cluster from scratch

There are many tools for launching a kubernetes cluster, after trying many tools my current favorite one is Kargo. My main reason is Kargo gives you both the control on provisioning the instances (using kargo-cli) and installing kubernetes (using ansible directly). However, to understand more about how kubernetes is setup, i challenge myself to do a setup from scratch manually. The motivation is that I can have a better understand on kubernetes components and dependencies.

There are already many posts on the internet about setting up a kubernetes cluster from scratch (like this good one), so I will mainly summarize my lessons learnt during the process, not diving on the detail of the how.

The installation process is quite simple

I did a some preparation before starting the installation, reading the document from kubernetes, also read part of kargo source and kube-deploy/docker-multinode source. But it is simpler than my expectation.

In one sentence, you will install docker, etcd, flannel, then finally use hyperkube for launching kubelet. etcd is the database of kubernetes, flannel is network plugin to provide overlay network for docker. The steps are repeatable for a multi node cluster. Just for information, I use Ubuntu 16.04 for host OS, and use flannel because i’m familiar most with these tools.

The nice thing is I learn out of the installation process is Kubelet is amazingly kool. Kubelet is an agent that runs on all of kubernetes cluster nodes, and its job is quite simple: it consumes the PodSpec provides by various sources, through a folder or API server, then manage the described containers. In someway, it is like a supervisor for containers. There are some very good articles about kubelet here and here. So when you deploy a kubernetes master, you just need to deploy kubelet, then point to a manifest folder and use PodSpec to describe other kubernetes services like apiserver, controller and scheduler. Kubelet will setup these up for you. For a kubernetes slave, it will pull the PodSpec from apiserver.

Another interesting thing is that you can also make kubelet to manage an addon-manager, which is similar to kubelet but it monitors addon folder and creates kube-proxy and kube-dns addon on top of kubernetes cluster. The whole setup can be visualized as the below picture, the services inside dashed box are managed by kubelet.

Beware of the default value in Hyperkube

Hyperkube is quite helpful, since it is an all-in-one binary docker has everything: kubelet, apiserver, controller manager, scheduler, proxy …It is easy to get started with hyperkube but once I want to make some modification such as changing the cluster CIDR I found myself a bit struggle in making the change. To change cluster CIDR, I need to mount both the manifest and addon into hyperkube, since the default CIDR is and the default DNS is also hardcoded to Also another thing is the default manifest makes apiserver opened to the world. And If you want to use your own certs, you will need to rewrite all the manifests and addons configuration to use your certs. By the way, Makecerts is quite useful to generate certs for kubernetes. The setup-files.sh also does not make sense to me. If I do the setup again, next time i will just use the binary files directly.

Make sure you enable ip forwarding for both ipv4 and ipv6

Another lesson i learnt is you need to enable ip forwarding for both ipv4 and ipv6 packet forwarding for all kubernetes nodes! The surface problem is that I can not access services exposed via NodePort, but they are still accessible from inside the cluster! I thought it was my setup, something wrong with kube-proxy. I spent many hours before finding out the actual problem. Ubuntu 16.04 has ipv4 port forwarding enabled by default, but not ipv6 by default. Kubernetes listens on ipv6, the socket will work for both ipv4 and ipv6 but since the forwarding is not enabled so the packet will be dropped externally. But thanks to that I learn about iptables debugging using TRACE, which is very useful for debugging iptables, and this picture will help you to understand all the chains and steps inside iptables.


The whole installation took me 2 days to set up a proper cluster. (half of that is finding out the problem with ip-forwarding …). For production environment, i will surely just use Kargo since it is a lot easier and less error prone. However, going through the process helps me understand more about kubernetes components and dependencies. I would recommend anyone to try to install kubernetes manually, i’m sure that you will learn something out of this!

Originally published at www.nqbao.com on March 19, 2017.

I write, so I learn.