As most tech people, I like playing around with technologies. I am well familiar with docker, and it was time to play around with some container orchestration. There are a lot of options out there Mesosphere Marathondocker swarm, cattlekubernetes, etc. I started playing with kubernetes as it is backed by google and is growing in popularity. I wanted to see what the fuzz is about.

Setting up

If you want to play around with kubernetes, they have a product called minikube. It makes it easy to play around with kubernetes. However it is a single node. The tech person that I am, i require a cluster, for reasons! Not wanting to spend any money, I will be using my home KVM server to setup kubernetes. I want to have 1 master and 2 nodes, which are using LAN IP addresses, 10.0.0.100,10.0.0.101,10.0.0.102 . Because I am most familiar with ubuntu, i first started using ubuntu 16.04 to run my cluster on. However I ran into a lot of issues doing this. So I changed my setup to ferdora 25,  it is better supported by kubernetes. Logical, since red hat invests in kubernetes. On your local machine, checkout the kubernetes contrib repository.  It contains ansible scripts to provision your master and nodes with. Ansible is a simple way to provision and deploy software onto servers.

Create inventory

Once you have checked out the contrib repository, create a file within ansible/inventory called inventory and add the following content:

[masters]
10.0.0.100

[etcd]
10.0.0.100

[nodes]
10.0.0.101
10.0.0.102
10.0.0.103
10.0.0.104

Change your ip addresses to your ferdora instances. However all my VM’s are running from within my own home, i want them to only handle the private network. By using internal private network IP’s, your kubernetes nodes become special snowflakes. It makes more sense to use hostnames for provisioning, however in my case IP’s suffice.

Change some settings

Change the file ansible/inventory/group_vars/all.yml. This file contains all specifics for your cluster. For example how ansible should connect via ssh and if you wish to install addons. My advise would be to keep the kube_ui and kube_dash disabled, they are outdated and do not work properly out of the box. And i will later in this post, install them.

Provisioning

Once you are setup run the ansible script “ansible/scripts/deploy-cluster.sh” Ansible will do his magic and provision your machines. However i ran in to several issues The first being a missing python2-dnf library, however this is only applicant when installing the nodes separately

fatal: [10.0.0.101]: FAILED! => {"changed": false, "failed": true, "msg": "`python2-dnf` is not installed, but it is required for the Ansible dnf module."}

The fix is easy, login to your server and

sudo yum install python2-dnf

Done, run ansible again. The following error was interesting, it could not start flannel daemon due to timeouts. The problem sat in the configuration to start flannel with For some reason in the file /etc/sysconfig/flanneld, the FLANNEL_ETC environment variable was empty. It could not start etcd, and therefor flanneld could not start When running ansible again, your cluster should be configured, yey!

Enable addons

Ssh into your kubernetes master. Dashboard Before installing the kubernetes dashboard remove cockpit-ws from each fedora VM. The port mapping interferes with the dashboard, besides who needs server management in a gui.

sudo yum remove cockpit-ws

Enable the dashboard, so we could see if it works by clicking! Server management in a gui, pffff, but container management in a gui, awesome!

kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

Expose the dashboard to the outside world.

Because i do not have any loadbalancer in front of my kubernetes cluster I will expose the dashboard based on node IP addresses. This will make your node a special snowflake, which it already was. And you might want to do some security checking if your kubernetes cluster is exposed to the internet. Check the IP address of the pod:

kubectl get pods --namespace=kube-system
kubectl describe pod kubernetes-dashboard-3095304083-e8di9 --namespace=kube-system

change kubernetes-dashboard-3095304083-e8di9 to your pod identifier whats visible in the “get pods” command. My output:

Name:		kubernetes-dashboard-3095304083-e8di9
Namespace:	kube-system
Node:		10.0.0.102/10.0.0.102
Start Time:	Tue, 13 Dec 2016 21:07:07 +0100
Labels:		app=kubernetes-dashboard
    pod-template-hash=3095304083
Status:		Running
IP:		172.16.84.10
Controllers:	ReplicaSet/kubernetes-dashboard-3095304083
Containers:
  kubernetes-dashboard:
    Container ID:	docker://3e8ca0727c9947246e5c8d674d703e1a0f11bd4a6b39a56616b3d3fb9cb3111f
    Image:		gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0
    Image ID:		docker://sha256:e5133bac8024ac6c916f16df8790259b5504a800766bee87dcf90ec7d634a418
    Port:		9090/TCP
    State:		Running
      Started:		Wed, 14 Dec 2016 15:48:23 +0100
    Last State:		Terminated
      Reason:		Error
      Exit Code:	1
      Started:		Tue, 13 Dec 2016 21:07:40 +0100
      Finished:		Wed, 14 Dec 2016 15:46:03 +0100
    Ready:		True
    Restart Count:	1
    Liveness:		http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vycbj (ro)
    Environment Variables:	<none>
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	True 
  PodScheduled 	True 
Volumes:
  default-token-vycbj:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-vycbj
QoS Class:	BestEffort
Tolerations:	dedicated=master:Equal:NoSchedule

As you can see my dashboard is assigned to node 10.0.0.102. I will expose the dashboard to that external ip. Exposing suggests that your pod is reachable through that ip addressing

kubectl expose deployment kubernetes-dashboard --namespace=kube-system --type=LoadBalancer --name=kubernetes-dashboard --external-ip=10.0.0.102

Your dashboard should be visible at http://10.0.0.102:9090. Once you logged in you should see a few pods crashing due to “no credentials” Fix your pods For example sky-dns

Failed to pull image "gcr.io/google_containers/skydns:2015-10-13-8c72f8c": image pull failed for gcr.io/google_containers/skydns:2015-10-13-8c72f8c, this may be because there are no credentials on this request. details: (manifest unknown: Failed to fetch "2015-10-13-8c72f8c" from request "/v2/google_containers/skydns/manifests/2015-10-13-8c72f8c".)
Error syncing pod, skipping: failed to "StartContainer" for "skydns" with ErrImagePull: "image pull failed for gcr.io/google_containers/skydns:2015-10-13-8c72f8c, this may be because there are no credentials on this request. details: (manifest unknown: Failed to fetch \"2015-10-13-8c72f8c\" from request \"/v2/google_containers/skydns/manifests/2015-10-13-8c72f8c\".)"
Liveness probe failed: HTTP probe failed with statuscode: 503

My first thought it was actually a credentials issue, however the true issue is that skydns from the contrib repository is very outdated. You want to update this. On your kubernetes master change the file /etc/kubernetes/addons/dns/skydns-rc.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
spec:
  # replicas: not specified here:
  # 1\. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2\. Default is 1.
  # 3\. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
        scheduler.alpha.kubernetes.io/tolerations: '[{"key":"CriticalAddonsOnly", "operator":"Exists"}]'
    spec:
      containers:
      - name: kubedns
        image: gcr.io/google_containers/kubedns-amd64:1.9
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthz-kubedns
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - --domain=cluster.local.
        - --dns-port=10053
        - --config-map=kube-dns
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
      - name: dnsmasq
        image: gcr.io/google_containers/kube-dnsmasq-amd64:1.4
        livenessProbe:
          httpGet:
            path: /healthz-dnsmasq
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --cache-size=1000
        - --no-resolv
        - --server=127.0.0.1#10053
        - --log-facility=-
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details
        resources:
          requests:
            cpu: 150m
            memory: 10Mi
      - name: dnsmasq-metrics
        image: gcr.io/google_containers/dnsmasq-metrics-amd64:1.0
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 10Mi
      - name: healthz
        image: gcr.io/google_containers/exechealthz-amd64:1.2
        resources:
          limits:
            memory: 50Mi
          requests:
            cpu: 10m
            # Note that this container shouldn't really need 50Mi of memory. The
            # limits are set higher than expected pending investigation on #29688.
            # The extra memory was stolen from the kubedns container to keep the
            # net memory requested by the pod constant.
            memory: 50Mi
        args:
        - --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
        - --url=/healthz-dnsmasq
        - --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null
        - --url=/healthz-kubedns
        - --port=8080
        - --quiet
        ports:
        - containerPort: 8080
          protocol: TCP
      dnsPolicy: Default  # Don't use cluster DNS.

Delete all the existing pods for dns, and it should automatically reload the dns addon. The same goes for kibana logging and heapster In /etc/kubernetes/addons/cluster-logging/kibana-controller.yaml change the container image version from:

gcr.io/google_containers/kibana:1.3

to

gcr.io/google_containers/kibana:v4.6.1

  In /etc/kubernetes/addons/cluster-monitoring/heapster-controller.yaml change the container image version from:

gcr.io/google_containers/heapster:v1.0.2

to

gcr.io/google_containers/heapster:v1.2.0

and

gcr.io/google_containers/addon-resizer:1.0

to

gcr.io/google_containers/addon-resizer:1.6

Delete the heapster pods and your kubernetes cluster should be up and running in no time! Interesting side note, if you have issues that node A could not reach node B through flannel cluster IP range. The solution could be restarting docker services. ( or reboot ).

Conclusion

If you just want to run some containers at your home cluster, dont use kubernetes :). To be honest it is quite tedious to setup, and is prone to errors. This is probably due to setting this up on your local KVM, with a local IP range. For simple container running, just use docker-compose or hashicorp nomad. Setting up multi-node hashicorp nomad takes you about an hour. If you still want to run containers in kubernetes and a single node is sufficient use minikube!