Ordering Containers within Pod

Why do you want to order containers inside the pod?

Ordering of pod can be a use case where we need certain containers to be up and running before we start our application code. Let’s say we have a Java application that needs a database(Mysql), cache(Aerospike/Redis) and Kafka to serve traffic. At the same time, we also need these dependencies to be instance-specific or local to the application stack. In such cases, Kubernetes did not provide an out-of-the-box solution before v1.28. For clusters with version < 1.28, there is no formal way of doing it. In order to mitigate this issue we have another lesser known open source workaround called Kubexit.

Introducing Kubexit: A Coordinated Container Startup Solution

Kubexit is an open-source project developed to provide a coordinated way to start and terminate containers inside a pod.

InitContainer cannot be used here as the containers declared in initContainers need to complete (container status should be complete) before the usual containers (declared in the Container section) start. For example, if you declare a MySQL container in the initContainer section then the pod will be stuck in PodInitialisation status, as the other containers declared in the Container section are waiting forever for the initContainers to complete.

Kubexit is a binary that we need to use for internal container ordering by declaring it in the initContainer section of the deployment.yaml. For Kubexit to work as expected we need to understand how it does what it does.

Kubexit allows you to declare 2 types of dependencies:
1. Birth Dependency: This dependency allows you to declare the birth order of the containers.

2. Death Dependency: This dependency allows you to declare the death ordering of the containers.

Integrating Kubexit with Deployments:

To use Kubexit within a pod we need to configure certain things.

• Declare the kubexit in initContainer so it downloads the binary into the pod. The /kubexit directory is where we download and store the binary inside the pod.

• We also need to override the entrypoint and/or args of the images pod for all containers among which we need to order. Append the keyword kubexit before the entrypoint or args as shown below.

• We need to create and mount a shared volume on all those containers which need to have ordering. The /graveyard is the directory which needs to be shared among all containers participating in ordering.

• Also define the other config such as :
  KUBEXIT_NAME: Container name from the Kubexit perspective.
  KUBEXIT_BIRTH_DEPS: Name of the container which needs to be up and running before the current container starts(this can be a comma separate list). The name mentioned in this is the name that was declared in KUBEXIT_NAME of the container.

Working of Kubexit:

• Kubexit requires a ServiceAccount and Role to monitor pod containers and shared volumes. It watches over the shared volume within a pod, enabling it to determine the status of containers and notify other containers if dependencies exist.
• To achieve this, the shared volume must be mounted in all containers requiring coordination among themselves. This configuration allows Kubexit to monitor container status using the readiness probe. It accomplishes this by appending /kubexit/kubexit (path to the binary) to the entrypoint/args of the container.
• Once the readiness probe confirms that the container has started, Kubexit marks the birth of the relevant container by placing a tombstone in the shared volume (e.g., /graveyard in the given example). Likewise, when a container ceases to exist, Kubexit adds a tombstone to indicate the container’s demise.
• Other containers can then monitor the shared volume to check if their dependencies are up, enabling them to initiate their own startup processes.

Note: Kubernetes however has provided support for such a use-case where we can keep the initContainer running as SideCarContainers in v1.28(link).

Reference:

https://github.com/karlkfi/kubexit
https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter
• Visit our other platforms: Stackademic | CoFeed | Venture | Cubed
• More content at PlainEnglish.io