Daniele Polencic
Daniele Polencic

Coding a real-time dashboard for Kubernetes

April 2020

Coding a real-time dashboard for Kubernetes

TL;DR: In Kubernetes you can use the Shared Informer — an efficient code pattern to watch for changes in Kubernetes resources. In this article you will learn how it works and how you can build a real-time dashboard for Kubernetes with it.

In Kubernetes, you can monitor changes to Pods in real-time with the --watch flag:

bash

kubectl get pods --watch

The --watch flag is part of the Kubernetes API, and it is designed to dispatch update events incrementally.

If you tried the command in the past, you might have noticed how the output is often confusing:

`kubectl get pods --watch`

How many more Pods were created?

Two, but you had to parse the output a couple of times to be sure.

Why is the command not updating the output in place?

Let's dive into what happens when you execute that command.

kubectl watch

When you type kubectl get pods --watch, a request is issued to:

GET https://api-server:8443/api/v1/namespaces/my-namespace/pods?watch=1

The response is temporarily empty and hangs.

The reason is straightforward: this is a long-lived request, and the API is ready to respond with events as soon as there's one.

Since nothing happened, the connection stays open.

Let's test this with a real cluster.

You can start a proxy to the Kubernetes API server on your local machine with:

bash

kubectl proxy
Starting to serve on 127.0.0.1:8001

Kubectl proxy creates a tunnel from your local machine to the remote API server.

It also uses your credentials stored in KUBECONFIG to authenticate.

From now on, when you send requests to 127.0.0.1:8001 kubectl forwards them to the API server in your cluster.

You can verify it by issuing a request in another terminal:

bash

curl localhost:8001
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1",
    // more APIs ...
  ]
}

It's time to subscribe for updates with:

bash

curl localhost:8001/api/v1/pods?watch=1

Notice how the request does not complete and stays open.

In another terminal, create a Pod in the default namespace with:

bash

kubectl run my-pod --image=nginx --restart=Never

Observe the previous command.

There's output this time! — and a lot of it.

output.json

{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}

What happens when you change the image for that Pod?

Let's try:

bash

kubectl set image pod/my-pod my-pod=busybox

There's another entry in the watch output:

output.json

{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}

You can already guess what happens when you delete the Pod with:

bash

kubectl delete pod my-pod

The output from the watch command has another entry:

output.json

{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}
{"type":"DELETED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}

In other words, every time you use the watch=1 query string, you can expect:

  1. The request stays open.
  2. There is an update every time a Pod is added, deleted or modified.

If you recall, that's precisely the output from kubectl get pods --watch.

There are three events created:

  1. The ADDED event is fired when a new resource is created.
  2. The MODIFIED event is fired when an existing resource is changed.
  3. The DELETED event is fire when the resource is removed from etcd.

And every update is a JSON response delimited by a new line — nothing complicated.

Can you use those events above to track changes to your Pods in real-time?

Building a real-time dashboard for Pods

Imagine you want to build a real-time dashboard that tracks the location of your Pods in your Nodes.

Something like this:

K8bit — the tiny Kubernetes dashboard

When a new Pod is added, a green block is created in a Node.

When an existing Pod is deleted, a green block is removed from a Node.

Where do you start?

Since the dashboard is web-based, in this article, you will focus on working with the Kubernetes API with Javascript.

But the same API calls and code patterns can be applied to any other language.

Let's start.

If you want to jump to the code, you can check out this repository.

Before you can use the API, you need to:

  1. Host a static web page where you can serve the HTML, CSS and Javascript.
  2. Access the Kubernetes API

Thankfully, kubectl has a command that combines both.

Create a local directory with an index.html file:

bash

mkdir k8bit
cd k8bit
echo "<!DOCTYPE html><title>⎈</title><h1>Hello world!" > index.html

In the same directory, start a kubectl proxy that also serves static content with:

bash

kubectl proxy --www=.
Starting to serve on 127.0.0.1:8001

You learned already that kubectl proxy creates a tunnel from your local machine to the API server using your credentials.

If you use the flag --www=<folder> you can also serve static content from a specific directory.

The static content is served at /static by default, but you can customise that too with the flag --www-prefix='/<my-url>/'.

Please note that serving the API and the static assets under the same domain saves you from dealing with CORS permissions.

You can open your browser at http://localhost:8001/static to see the Hello World! page.

Let's see if you can connect to the Kubernetes API too.

Create a Javascript file named app.js with the following content:

app.js

fetch(`/api/v1/pods`)
  .then((response) => response.json())
  .then((podList) => {
    const pods = podList.items
    const podNames = pods.map(it => it.metadata.name)
    console.log('PODS:', podNames)
  })

You can include the script in the HTML with:

bash

echo '<script src="app.js"></script>' >> index.html

If you reload the page in your browser and inspect Chrome Dev Tools, Firefox Web Console or Safari Developer Tools, you should see a list of Pods from your cluster.

Next step, real-time updates!

As you probably guessed, you could use the watch query string and receive timely updates about Pods added or deleted.

The code in Javascript could look like this:

dashboard.js

fetch(`/api/v1/pods?watch=1`).then((response) => {
  /* read line and parse it to json */
})

While the initial call to the API is similar, handling the response is more complicated.

Since the response never ends and stays open, you have to parse the incoming payloads as they come.

You also have to remember to parse the JSON responses every time there's a new line.

Here's an example of a stream of bytes:

stream

{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}\n
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1",/* more json */}}\n

Please notice that you're not guaranteed to receive one line at the time.

You could have a stream that is interrupted in between JSON responses like this:

stream

{"type":"ADDED","object":{"kind":"Pod","apiVer
                              ---------------^
                              interrupted here

sion":"v1",/* more json */}}\n
^-----------
resumed here

That means that:

  1. You should buffer all incoming payloads.
  2. As the buffer grows, check if there are new lines.
  3. Every time there's a new line, parse it as a JSON blob.
  4. Call a function that prints the event in the console.

The following code handles the reading, buffering and splitting lines:

app.js

fetch(`/api/v1/pods?watch=1`)
  .then((response) => {
    const stream = response.body.getReader()
    const utf8Decoder = new TextDecoder('utf-8')
    let buffer = ''

    // wait for an update and prepare to read it
    return stream.read().then(function onIncomingStream({ done, value }) {
      if (done) {
        console.log('Watch request terminated')
        return
      }
      buffer += utf8Decoder.decode(value)
      const remainingBuffer = findLine(buffer, (line) => {
        try {
          const event = JSON.parse(line)
          const pod = event.object
          console.log('PROCESSING EVENT: ', event.type, pod.metadata.name)
        } catch (error) {
          console.log('Error while parsing', chunk, '\n', error)
        }
      })

      buffer = remainingBuffer

      // continue waiting & reading the stream of updates from the server
      return stream.read().then(onIncomingStream)
    })
  })

function findLine(buffer, fn) {
  const newLineIndex = buffer.indexOf('\n')
  // if the buffer doesn't contain a new line, do nothing
  if (newLineIndex === -1) {
    return buffer
  }
  const chunk = buffer.slice(0, buffer.indexOf('\n'))
  const newBuffer = buffer.slice(buffer.indexOf('\n') + 1)

  // found a new line! execute the callback
  fn(chunk)

  // there could be more lines, checking again
  return findLine(newBuffer, fn)
}

If you wish to dive more into the details of the above code, you should check out the browser Streaming API.

If you include the above snippet in your app.js, you can see real-time updates from your cluster!

There's something odd, though.

The API call includes a few of the Pods that were already listed by the first call.

If you inspect the console, you should find:

output

PODS: ['nginx-deployment-66df5b97b8-fxl7t', 'nginx-deployment-66df5b97b8-fxxqd']
^--------------------
First call to the API

PROCESSING EVENT: ADDED nginx-deployment-66df5b97b8-fxl7t
PROCESSING EVENT: ADDED nginx-deployment-66df5b97b8-fxxqd
^----------------------------
Those two pods are duplicates
as you've already seen them

There's a Pod is listed twice:

  1. In the "list all the Pods" API request and
  2. In the "stream the updates for all Pods" request.

Isn't the watch API supposed to stream only updates?

Why is it streaming events that happened in the past?

Tracking changes reliably

The watch API tracks only updates and it has a memory of 5 minutes.

So you could receive updates for Pods that were created or deleted up to 5 minutes ago.

How do you track only new changes reliably?

Ideally, you want to track all changes that happen after the first call to the API.

Fortunately, every Kubernetes object has a resourceVersion field that represents the version of the resource in the cluster.

You can inspect the field in your existing cluster with:

bash

kubectl get pod <my-pod> -o=jsonpath='{.metadata.resourceVersion}'
464927

The resource version is incremental, and it is included in the events from the watch API.

When you list all your Pods, the same resourceVersion is included in the response too:

bash

curl localhost:8001/api/v1/pods | jq ".metadata.resourceVersion"
12031

You can think about the resourceVersion number as a number that increments every time you type a command or a resource is created.

The same number can be used to retrieve the state of the cluster in a given point in time.

You could list all the Pods from the resourceVersion number 12031 with:

bash

curl localhost:8001/api/v1/pods?resourceVersion=12031
# ... PodList response

The resourceVersion could help you make your code more robust.

Here's what you could do:

  1. The first request retrieves all the Pods. The response is a list of Pods with a resourceVersion. You should save that number.
  2. You start the Watch API from that specific resourceVersion.

The code should change to:

dashboard.js

fetch('/api/v1/pods')
  .then((response) => response.json())
  .then((response) => {
    const pods = podList.items
    const podNames = pods.map(it => it.metadata.name)
    console.log('PODS:', podNames)
    return response.metadata.resourceVersion
  })
  .then((resourceVersion) => {
    fetch(`/api/v1/pods?watch=1&resourceVersion=${resourceVersion}`).then((response) => {
      /* read line and parse it to json */
      const event = JSON.parse(line)
      const pod = event.object
      console.log('PROCESSING EVENT: ', event.type, pod.metadata.name)
    })
  })

The code now works as expected and there are no duplicate Pods.

Congrats!

If you add or delete a Pod in the cluster, you should be able to see an update in your web console.

The code is reliable, and you only receive updates for new events!

Can you track the Node where each Pod is deployed?

Keeping a local cache

Since every Pod exposes a .spec.nodeName field with the name of the Pod, you could use that to construct a pair pod - node.

Well almost every Pod exposes .spec.nodeName.

When a Pod is created:

  1. It is stored in the database.
  2. An "ADDED" event is dispatched.
  3. The Pod is added to the scheduler queue.
  4. The scheduler binds the Pod to a Node.
  5. The Pod is updated in the database.
  6. The "MODIFIED" event is dispatched.

So you can keep a list of all Pods, but filter the list only for Pods that a .spec.nodeName.

You can keep track of all Pods in your cluster with a Map.

app.js

const pods = new Map()

fetch('/api/v1/pods')
  .then((response) => response.json())
  .then((response) => {
    const pods = podList.items
    const podNames = pods.map(it => it.metadata.name)
    console.log('PODS:', podNames)
    return response.metadata.resourceVersion
  })
  .then((resourceVersion) => {
    fetch(`/api/v1/pods?watch=1&resourceVersion=${resourceVersion}`).then((response) => {
      /* read line and parse it to json */
      const event = JSON.parse(line)
      const pod = event.object
      console.log('PROCESSING EVENT: ', event.type, pod.metadata.name)
      const podId = `${pod.metadata.namespace}-${pod.metadata.name}`
      pods.set(podId, pod)
    })
  })

You can display all Pods assigned to a Node with:

app.js

const pods = new Map()

// ...

function display() {
  Array.from(pods)
  .filter(pod => pod.spec.nodeName)
  .forEach(pod => {
    console.log('POD name: ', pod.metadata.name, ' NODE: ', pod.spec.nodeName)
  })
}

At this point, you should have a solid foundation to build the rest of the dashboard.

Please note that the current code is missing:

  1. A friendly user interface.
  2. Retries when a request is terminated prematurely.

Rendering the HTML and writing the CSS are omitted in this tutorial.

You can find the full project (including a friendly user interface) in this repository, though.

However, the retry mechanism is worth discussing.

Handling exceptions

When you make a request using the watch flag, you keep the request open.

But does it always stay connected?

Nothing in life lasts forever.

The request could be terminated for a variety of reasons.

Perhaps the API was restarted, or the load balancer between you and the API decided to terminate the connection.

You should handle this edge case — when it happens.

And when you decide to reconnect, you should only receive updates after the last one.

But how do you know what was the last update?

Again, the resourceVersion field is here to the rescue.

Since each update has a resourceVersion field, you should always save the last one you saw.

If the request is interrupted, you can initiate a new request to the API starting from the last resourceVersion.

You can change the code to keep track of the last resourceVersion with:

dashboard.js

let lastResourceVersion

fetch('/api/v1/pods')
  .then((response) => response.json())
  .then((response) => {
    const pods = podList.items
    const podNames = pods.map(it => it.metadata.name)
    console.log('PODS:', podNames)
    lastResourceVersion = response.metadata.resourceVersion
  })
  .then((resourceVersion) => {
    fetch(`/api/v1/pods?watch=1&resourceVersion=${lastResourceVersion}`).then((response) => {
      /* read line and parse it to json */
      const event = JSON.parse(line)
      const pod = event.object
      lastResourceVersion = pod.metadata.resourceVersion
      console.log('PROCESSING EVENT: ', event.type, pod.metadata.name)
    })
  })

The last change is including a fallback mechanism to restart the connection.

In this part, you should refactor the code like this:

dashboard.js

function initialList() {
  return fetch('/api/v1/pods')
    .then((response) => response.json())
    .then((response) => {
      /* store resource version and list of pods */
      return streamUpdates()
    })
}

function streamUpdates(){
  fetch(`/api/v1/pods?watch=1&resourceVersion=${lastResourceVersion}`).then((response) => {
    /* read line and parse it to json */
  })
  .then(() => {
    // request gracefully terminated
    return streamUpdates()
  })
  .catch((error) => {
    // error, reconnect
    return stremUpdates()
  })
}

Now you can be sure that the dashboard will keep streaming updates even after the connection with the API was lost.

Kubernetes Shared Informer

A quick recap of the code changes that you did:

  1. You listed all Pods and stored the resourceVersion.
  2. You started a long-lived connection with the API and asked for updates. Only the updates after the last resourceVersion are streamed.
  3. You keep a local dictionary with all the Pods that you've seen so far.
  4. You handled reconnections when the connection is (abruptly) terminated.

If you were to extend the same logic to Service and Deployments or any other Kubernetes resource, you probably want to have a very similar code.

It's a good idea to encapsulate the above logic in a library, so you don't have to keep reinventing the wheel every time you wish to track objects.

That's what the Kubernetes community thought too.

In Kubernetes, there's a code pattern called the Shared Informer.

A shared informer encapsulates:

  1. The initial request to retrieve a list of resources.
  2. A Watch API request that starts from the previous resourceVersion.
  3. An efficient cache mechanism to store the resources locally in a dictionary.
  4. Reconnections when the connection is lost

You can find an example of the shared informer in several programming languages:

Using the official Javascript client library for Kubernetes you can refactor your code in less than 20 lines:

informer.js

const listFn = () => listPodForAllNamespaces();
const informer = makeInformer(kc, '/api/v1/pods', listFn);

informer.on('add', (pod) => { console.log(`Added: ${pod.metadata!.name}`); });
informer.on('update', (pod) => { console.log(`Updated: ${pod.metadata!.name}`); });
informer.on('delete', (pod) => { console.log(`Deleted: ${pod.metadata!.name}`); });
informer.on('error', (err) => {
  console.error(err)
  // Restart informer after 5sec
  setTimeout(() => informer.start(), 5000)
})

informer.start()

Please note that the above code works only on Node.js and can't be run in the browser at the moment. This is a limitation of the client library.

Summary

All code written so far runs against kubectl proxy.

However, the same code could be repackaged and deployed inside your cluster.

Imagine being able to track Pods, Deployments, Services, DaemonSets, etc. from within a cluster.

That's precisely what happens when you deploy an operator or a controller in Kubernetes.

What else can you build?

I connected a Google Spreadsheet to Kubernetes, and I was able to change replicas for my Deployments with formulas.

Odd, right?

That's to show how powerful the Kubernetes API is.

That's all folks

What can you connect the Kubernetes API?

Do you have a brilliant idea on how to leverage the real-time updates in Kubernetes?

Let us know!

A special thank you goes to Daniel Weibel and Chris Nesbitt-Smith that reviewed the content of this article.

Your source for Kubernetes news

You are in!

More K8s news, events and jobs.