Dynamic Node Labeling via Node Feature Discovery on Kubernetes

6 min readApr 8, 2023

Introduction

At its core, Node Feature Discovery (NFD) is an application that automatically detects hardware features and system configuration of a node in a Kubernetes cluster. Think of it as a treasure hunter, scouring the nooks and crannies of your cluster to find the hidden gems of hardware and configuration features that can make a big difference in how your workloads are scheduled and run.

Node Feature Discovery can help the Kubernetes scheduler identify the nodes in your cluster that have specific features, such as GPUs or SSDs, and then schedule workloads to run on those nodes. This can help you make the most of your resources and ensure that your workloads are running as efficiently as possible.

My Use Case

I have a self-hosted K3s Kubernetes cluster. One application I host is Frigate NVR which consumes live video feeds from all my security cameras. When Frigate detects motion within a video stream, it isolates that motion into a small clip. The object creating the motion can be identified by off-loading the image to a Google Coral TPU USB accelerator such as “Person”, “Dog”, Cat” and this information can be used to make decisions.

I only want Frigate NVR to be scheduled on nodes that have a Coral TPU USB plugged in and I want to be able to move this USB device around as needed without having to update YAML files and making configuration changes to the cluster. NFD easily solves this for me.

About Node Feature Discovery

NFD detects hardware features available on each node in a Kubernetes cluster, and advertises those features using node labels. Of course, you could manually create node labels and schedule pods using them, but why do that if you don’t have to?

NFD consists of three software components:

NFD-Master is a daemon responsible for communication with the Kubernetes API (receives labeling requests from the worker and modifies node objects accordingly).
NFD-Worker is a daemon responsible for feature detection. One instance per node within the cluster. It will forward its detections to the NFD-Master to perform the actual node labeling.
NFD-Topology-Updater is a daemon responsible for examining allocated resources on a worker node to account for resources available to be allocated to new pod.

Feature Discovery is based on a wide range of domain-specific sources: CPU, Kernel, Memory, Network, PCI, Storage, System, USB, etc. Each feature source is responsible for detecting its respective features which are turned into node feature labels.

Installation

The installation can be straight forward:

$ kubectl apply -k https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref=v0.12.2

  namespace/node-feature-discovery created
  serviceaccount/nfd-master created
  clusterrole.rbac.authorization.k8s.io/nfd-master created
  clusterrolebinding.rbac.authorization.k8s.io/nfd-master created
  configmap/nfd-worker-conf created
  service/nfd-master created
  deployment.apps/nfd-master created
  daemonset.apps/nfd-worker created

Or installation via Helm:

$ export NFD_NS=node-feature-discovery

$ helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts
$ helm repo update
$ helm install nfd/node-feature-discovery -f <path/to/custom/values.yaml> \
  --namespace $NFD_NS --create-namespace

Where -f <path/to/custom/values.yaml> specifies a custom configuration to overwrite default values.

Check Pods are Running

Quick check to see pods are running:

$ kubectl -n node-feature-discovery get all  

  pod/node-feature-discovery-master-6b8f9f4d78-ch9h2          1/1     Running   0              29h
  pod/node-feature-discovery-worker-48z7z                     1/1     Running   48 (29h ago)   39d
  pod/node-feature-discovery-worker-cx9qd                     1/1     Running   35 (29h ago)   39d

Check Node Labels Created

By default the node feature labels are a bit ugly, they will look something like:

$ kubectl get nodes -o json | jq .items[].metadata.labels

  {
    "feature.node.kubernetes.io/pci-0106_1022.present": "true",
    "feature.node.kubernetes.io/pci-0108_144d.present": "true",
    "feature.node.kubernetes.io/pci-0300_1002.present": "true",
    "feature.node.kubernetes.io/usb-ef_0e8d_0608.present": "true",
    "feature.node.kubernetes.io/usb-ff_18d1_9302.present": "true",
    ...
  }

Customized Node Labels

One of the best things about NFD is that it’s highly customizable. You can configure it to detect custom features, allowing you to extend its functionality to support your own hardware or software features. This is particularly useful for organizations with unique hardware or software requirements.

We can limit which nodes devices that NFD should detect by specifying sources with the configuration parameter deviceClassWhitelist. This is used to specify a list of device classes that NFD should scan for on a node.

We can also create our own custom source, which can be used to map discovered Class, Vendor and Device codes to simpler human readable labels. All of this can be stored neatly within our values.yaml specified during installation.

As an example, below is the values.yaml file I used to limit NFD to just PCI and USB devices. For each PCI and USB source, a defined deviceClassWhitelist is specified to filter on. I also included comments on websites I used that help me determine what the codes meant. Then based on the class, vendor and device codes are define “custom” human readable label names:

worker:
  config:
    core:
      sources: ["custom", "pci", "usb"]
    sources:
      pci:
        # https://pci-ids.ucw.cz/read/PD
        deviceClassWhitelist: ["01", "03", "0b40", "12"]

      usb:
        # https://www.usb.org/defined-class-codes
        deviceClassWhitelist: ["02", "03", "0e", "ef", "fe", "ff"]

      custom:
        - name: "intel-gpu" # Intel integrated GPU
          matchOn:
            - pciId:
                class: ["0300"]
                vendor: ["8086"]
        - name: "amd-gpu-barcelo" # AMD Barcelo Integrated GPU
          matchOn:
            - pciId:
                class: ["0300"]
                vendor: ["1002"]
                device: ["15e7"]
        - name: "amd-gpu-raven-ridge" # AMD Raven Ridge Integrated GPU
          matchOn:
            - pciId:
                class: ["0300"]
                vendor: ["1002"]
                device: ["15dd"]

        - name: "google-coral-device" # USB Coral TPU
          matchOn:
            - usbId:
                vendor: ["18d1", "1a6e"]
        - name: "bluetooth-radio" # MediaTek Bluetooth Radio
          matchOn:
            - usbId:
                vendor: ["0e8d"]
                device: ["0608"]
        - name: "hp-usb-keyboard" # HP Slim Keyboard
          matchOn:
            - usbId:
                vendor: ["03f0"]
                device: ["2f4a"]

Once this is applied, the node labels start to make more sense:

$ kubectl get nodes -o json | jq .items[].metadata.labels

  {
    "feature.node.kubernetes.io/custom-amd-gpu-barcelo": "true",
    "feature.node.kubernetes.io/custom-bluetooth-radio": "true",
    "feature.node.kubernetes.io/custom-hp-usb-keyboard": "true",
    ...
    
    "feature.node.kubernetes.io/custom-google-coral-device": "true",
    ...
    
    "feature.node.kubernetes.io/custom-intel-gpu": "true",
    "feature.node.kubernetes.io/custom-rdma.capable": "true",
    ...
  }

Now the Kubernetes Scheduler can easily determined which nodes have Intel vs AMD GPUs, which node has a Coral TPU USB device, Bluetooth radio, etc. We can even determine which node has the current focus of the KVM Switch by looking at which node has the attached keyboard.

How to determine class, vendor and device codes

For USB Devices we can start with Linux utility lsusb:

$ lsusb

Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 0e8d:0608 MediaTek Inc. Wireless_Device
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 18d1:9302 Google Inc.

Looking at the line with “Google Inc.” the column 18d1:9302 contains the vendor and device code. The 18d1 is a vendor code for Google and 9302 is a device code. Being that Google does not make a lot of USB equipment our filter does not need to be that precise. Just filtering on the vendor code will be fine for our detection of the Google Coral TPU USB device:

- name: "google-coral-device" # USB Coral TPU
  matchOn:
    - usbId:
        vendor: ["18d1", "1a6e"]

NOTE: I was aware of an alternate vendor code for this device, I included it within the vendor match just in case I happen to get more of these modules.

For PCI Devices we can use lspci Linux utility, and limit our search to VGA devices using grep:

$ lspci -nn | grep VGA

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo [1002:15e7] (rev c2)

Within the [0300], the 0300 is the class for VGA compatible controller, and within [1002:15e7] the 1002 is vendor code for AMD and 15e7 is the device code. Being that AMD makes a lot of devices, we can be very specific with the device definition:

- name: "amd-gpu-barcelo" # AMD Barcelo Integrated GPU
  matchOn:
    - pciId:
        class: ["0300"]
        vendor: ["1002"]
        device: ["15e7"]

Scheduling Pods Based on Dynamic Labels

This is just scratching the surface of what Node Feature Discovery can do. This was sufficient for my needs to determine where Frigate NVR could be scheduled. Within Frigate NVR’s Helm values.yaml file, a nodeSelector can be defined using the dynamic labels created by NFD:

nodeSelector:
  kubernetes.io/role: "worker"
  feature.node.kubernetes.io/custom-amd-gpu-barcelo: "true"
  feature.node.kubernetes.io/custom-google-coral-device: "true"

Conclusion

In conclusion, Node Feature Discovery is a valuable tool for any Kubernetes user looking to optimize their workloads. With its ability to automatically detect hardware and software features, NFD can help you make the most of your resources and ensure that your workloads are running as efficiently as possible. So if you haven’t already, give NFD a try and see how it can help you navigate the waters of Kubernetes.