Lesson learnt from launching a Kubernetes cluster from scratch

The installation process is quite simple

I did a some preparation before starting the installation, reading the document from kubernetes, also read part of kargo source and kube-deploy/docker-multinode source. But it is simpler than my expectation.

Beware of the default value in Hyperkube

Hyperkube is quite helpful, since it is an all-in-one binary docker has everything: kubelet, apiserver, controller manager, scheduler, proxy …It is easy to get started with hyperkube but once I want to make some modification such as changing the cluster CIDR I found myself a bit struggle in making the change. To change cluster CIDR, I need to mount both the manifest and addon into hyperkube, since the default CIDR is 10.0.0.0/16 and the default DNS is also hardcoded to 10.0.0.10. Also another thing is the default manifest makes apiserver opened to the world. And If you want to use your own certs, you will need to rewrite all the manifests and addons configuration to use your certs. By the way, Makecerts is quite useful to generate certs for kubernetes. The setup-files.sh also does not make sense to me. If I do the setup again, next time i will just use the binary files directly.

Make sure you enable ip forwarding for both ipv4 and ipv6

Another lesson i learnt is you need to enable ip forwarding for both ipv4 and ipv6 packet forwarding for all kubernetes nodes! The surface problem is that I can not access services exposed via NodePort, but they are still accessible from inside the cluster! I thought it was my setup, something wrong with kube-proxy. I spent many hours before finding out the actual problem. Ubuntu 16.04 has ipv4 port forwarding enabled by default, but not ipv6 by default. Kubernetes listens on ipv6, the socket will work for both ipv4 and ipv6 but since the forwarding is not enabled so the packet will be dropped externally. But thanks to that I learn about iptables debugging using TRACE, which is very useful for debugging iptables, and this picture will help you to understand all the chains and steps inside iptables.

Conclusion

The whole installation took me 2 days to set up a proper cluster. (half of that is finding out the problem with ip-forwarding …). For production environment, i will surely just use Kargo since it is a lot easier and less error prone. However, going through the process helps me understand more about kubernetes components and dependencies. I would recommend anyone to try to install kubernetes manually, i’m sure that you will learn something out of this!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store