Kubernetes

We currently use Mesos/Marathon for deploying containers on our infrastructure, see the services repo for a list of services we deploy with Marathon.

Unfortunately, Marathon has proved less than ideal for us. It has mysterious downtime events, and too much control rests in the web interface, where we’d rather have as much configuration as possible in Git.

Kubernetes is an industry standard and well established alternative to Marathon that we should use. While there’s plenty of documentation for using Kubernetes, it’s rare for users to set up their own clusters-- they usually use a cloud offering (like Amazon’s EKS or Google’s GKE). We’ll have to figure out how to use puppetlabs-kubernetes to set up our cluster, and then migrate services over.

Kubernetes can also be used for more advanced workflows we’d like, such as cron-like scheduling. We could also use it to host containers for OCF users, possibly replacing our apphosting offering!

Services on Kubernetes

  • templates
  • pma
  • grafana
  • mastodon

Services Still on Marathon

  • create
  • ircbot
  • metabase
  • ocfweb
  • puppetboard
  • rt
  • slackbridge
  • snmp-exporter
  • sourcegraph
  • thelounge
1 Like

This is a Hard project. If you’re interested in this, here’s some reading material:

The more that people try this, the easier it gets.

Daniel has laid out the OCF’s dream goals and uses of Kubernetes, and I’m commandeering this thread for staff tutorials, question answering, and good-practices accumulation.

People of interest, who [helped set up the cluster, are working on OCF kube projects, use kubernetes at work]:

@fydai (mastodon) @baisang (work / Kafka) @cooperc (work) @mcint (work) @wporr (create, Ceph) @adityasri @jameszhu. Or see dev- namespace in kubernetes.

Getting a dev- namespace

A root staffer can create a namespace for users who want to test development, as follows (thanks to @dkessler):
https://gist.github.com/dkess/8015ea87c53887acf78527f8b3c3cfcd

DEVUSER=oskitorvalds
kubectl create namespace dev-$DEVUSER
kubectl -n dev-$DEVUSER create rolebinding dev-$DEVUSER --clusterrole=edit --user=$DEVUSER

Why kubernetes?

Declarative infrastructure, in 1 place. That yaml file automates everything from scheduling web server work and maintaining a set of ready instances to auto-scaling on cpu load/memory/*-metric and staged (test-and-check) deployments. Declaring the desired state of the entire system (within the format provided) allows the management software to put the system into the desired state or explain why it can’t. This is the dream anyway, reality is close behind.

First explorations

See kubectl help, kubectl help [cmd], or kubectl [cmd] [-h|--help] early, often, and on a whim. Take notes, and share them.
kubectl get all

Basic commands

Play with & learn

I’m currently working (at long last) on getting a Kubernetes config for JupyterHub, which will allow users to spin up multiple Jupyter Notebook servers for large groups of students. My use-case would be running HKN’s EECS decal, where our current strategy of getting everyone to download a repo from Github / install a Python virtualenv could be replaced with accessing a Jupyter notebook.

The JupyterHub developers recommend Zero to Kubernetes for installing JupyterHub on Kubernetes (Github), which uses Helm to install a Kubernetes config from a package host.

However, as I currently understand, using Helm to download (somewhat) untrusted code, to install an arbitrary code executor (JupyterHub), is not quite desirable. Helm has an entire page devoted to securing a helm installation, which is not done by default.

My current plan, instead, is to use Helm to do a dry-run install of JupyterHub, which will allow it to instantiate all of the yaml template files, which we can then inspect / commit to Github / install.

Feel free to look into Helm, but I suspect that the overhead involved in installing Helm and integrating it into our infrastructure will end up being much larger the effort of installing JupyterHub “manually” (without Helm).