Customize the kubeadm image repository
A deep dive into the Kubernetes source code
As of version 1.26.0, by default, kubeadm pulls images from registry.k8s.io for the cluster components.
root@host:~# kubeadm config images list
registry.k8s.io/kube-apiserver:v1.26.0
registry.k8s.io/kube-controller-manager:v1.26.0
registry.k8s.io/kube-scheduler:v1.26.0
registry.k8s.io/kube-proxy:v1.26.0
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.6-0
registry.k8s.io/coredns/coredns:v1.9.3I needed to set up a proxy-cache repository for these images using Harbor to proxy and cache images from registry.k8s.io and to provide the alternative imageRepository to be used instead of registry.k8s.io.
I did so by using kubeadm with a configuration file.
# kubeadm-config.yml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
imageRepository: my.customrepository.io/subpathAfter this change, I expected to see this list as a result:
root@host:~# kubeadm config images list --config=./kubeadm-config.yml
my.customrepository.io/subpath/kube-apiserver:v1.26.0
my.customrepository.io/subpath/kube-controller-manager:v1.26.0
my.customrepository.io/subpath/kube-scheduler:v1.26.0
my.customrepository.io/subpath/kube-proxy:v1.26.0
my.customrepository.io/subpath/pause:3.9
my.customrepository.io/subpath/etcd:3.5.6-0
my.customrepository.io/subpath/coredns/coredns:v1.9.3Instead, I got this list as a result:
root@host:~# kubeadm config images list --config=./kubeadm-config.yml
my.customrepository.io/subpath/kube-apiserver:v1.26.0
my.customrepository.io/subpath/kube-controller-manager:v1.26.0
my.customrepository.io/subpath/kube-scheduler:v1.26.0
my.customrepository.io/subpath/kube-proxy:v1.26.0
my.customrepository.io/subpath/pause:3.9
my.customrepository.io/subpath/etcd:3.5.6-0
my.customrepository.io/subpath/coredns:v1.9.3Everything looks good except the CoreDNS image; try pulling my.customrepository.io/subpath/coredns:v1.9.3 , and you will encounter the not found error!
Harbor converts my.customrepository.io/subpath/coredns:v1.9.3 to registry.k8s.io/coredns:v1.9.3 and tries to pull it, but there is no image available with this reference as it is moved from registry.k8s.io/coredns to registry.k8s.io/coredns/coredns .
Wait, what just happened?!
I will dive deep into the kubeadm source code to answer the question above.
TL;DR
If you need the solution, use this ClusterConfiguration instead:
# kubeadm-config.yml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
dns:
imageRepository: my.customrepository.io/subpath/coredns
imageRepository: my.customrepository.io/subpathDeep Dive
How is the images list generated?
Let’s dive in!
Running the command kubeadm config images list calls the following method defined for ImagesList struct.
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/cmd/config.go#L368
return imagesList.Run(out, printer)The ImagesList struct holds InitConfiguration .
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/cmd/config.go#L389
// ImagesList defines the struct used for "kubeadm config images list"
type ImagesList struct {
cfg *kubeadmapi.InitConfiguration
}And the InitConfiguration struct holds ClusterConfiguration .
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/apis/kubeadm/types.go#L32
// InitConfiguration contains a list of fields that are specifically "kubeadm init"-only runtime
// information. The cluster-wide config is stored in ClusterConfiguration. The InitConfiguration
// object IS NOT uploaded to the kubeadm-config ConfigMap in the cluster, only the
// ClusterConfiguration is.
type InitConfiguration struct {
metav1.TypeMeta
// ClusterConfiguration holds the cluster-wide information, and embeds that struct (which can be (un)marshalled separately as well)
// When InitConfiguration is marshalled to bytes in the external version, this information IS NOT preserved (which can be seen from
// the `json:"-"` tag in the external variant of these API types.
ClusterConfiguration `json:"-"`
// BootstrapTokens is respected at `kubeadm init` time and describes a set of Bootstrap Tokens to create.
BootstrapTokens []bootstraptokenv1.BootstrapToken
// NodeRegistration holds fields that relate to registering the new control-plane node to the cluster
NodeRegistration NodeRegistrationOptions
// LocalAPIEndpoint represents the endpoint of the API server instance that's deployed on this control plane node
// In HA setups, this differs from ClusterConfiguration.ControlPlaneEndpoint in the sense that ControlPlaneEndpoint
// is the global endpoint for the cluster, which then loadbalances the requests to each individual API server. This
// configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible
// on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process
// fails you may set the desired value here.
LocalAPIEndpoint APIEndpoint
// CertificateKey sets the key with which certificates and keys are encrypted prior to being uploaded in
// a secret in the cluster during the uploadcerts init phase.
CertificateKey string
// SkipPhases is a list of phases to skip during command execution.
// The list of phases can be obtained with the "kubeadm init --help" command.
// The flag "--skip-phases" takes precedence over this field.
SkipPhases []string
// Patches contains options related to applying patches to components deployed by kubeadm during
// "kubeadm init".
Patches *Patches
}The underlying ImagesList struct for the Run function is returned by the following function, which uses configutil.LoadOrDefaultInitConfiguration() to either get the default InitConfiguration or load it from a file (the file passed via --config flag in the kubeadm config images list command).
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/cmd/config.go#L377
// NewImagesList returns the underlying struct for the "kubeadm config images list" command
func NewImagesList(cfgPath string, cfg *kubeadmapiv1.ClusterConfiguration) (*ImagesList, error) {
initcfg, err := configutil.LoadOrDefaultInitConfiguration(cfgPath, cmdutil.DefaultInitConfiguration(), cfg)
if err != nil {
return nil, errors.Wrap(err, "could not convert cfg to an internal cfg")
}
return &ImagesList{
cfg: initcfg,
}, nil
}Now let’s take a look at the Run function.
It calls images.GetControlPlaneImages() with ClusterConfiguration from the ImagesList struct, which returns a list of container images kubeadm expects to use on a control plane node.
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/cmd/config.go#L422
// Run runs the images command and writes the result to the io.Writer passed in
func (i *ImagesList) Run(out io.Writer, printer output.Printer) error {
imgs := images.GetControlPlaneImages(&i.cfg.ClusterConfiguration)
if err := printer.PrintObj(&outputapiv1alpha2.Images{Images: imgs}, out); err != nil {
return errors.Wrap(err, "unable to print images")
}
return nil
}Let’s take a look at GetControlPlaneImages() function.
For each core Kubernetes image, GetKubernetesImage() function is called with the component name and the ClusterConfiguration , which generates and returns the image for the components managed in the Kubernetes main repository, including the control-plane components and kube-proxy.
// https://github.com/kubernetes/kubernetes/blob/1edbb8cf1a864c2dd71a1e5b7282de4bc3d8f5fa/cmd/kubeadm/app/images/images.go#L90
// GetControlPlaneImages returns a list of container images kubeadm expects to use on a control plane node
func GetControlPlaneImages(cfg *kubeadmapi.ClusterConfiguration) []string {
images := make([]string, 0)
// start with core kubernetes images
images = append(images, GetKubernetesImage(constants.KubeAPIServer, cfg))
images = append(images, GetKubernetesImage(constants.KubeControllerManager, cfg))
images = append(images, GetKubernetesImage(constants.KubeScheduler, cfg))
images = append(images, GetKubernetesImage(constants.KubeProxy, cfg))
// pause is not available on the ci image repository so use the default image repository.
images = append(images, GetPauseImage(cfg))
// if etcd is not external then add the image as it will be required
if cfg.Etcd.Local != nil {
images = append(images, GetEtcdImage(cfg))
}
// Append the appropriate DNS images
images = append(images, GetDNSImage(cfg))
return images
}Let’s take a look at GetKubernetesImage() function:
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/images/images.go#L35
// GetKubernetesImage generates and returns the image for the components managed in the Kubernetes main repository,
// including the control-plane components and kube-proxy.
func GetKubernetesImage(image string, cfg *kubeadmapi.ClusterConfiguration) string {
repoPrefix := cfg.GetControlPlaneImageRepository()
kubernetesImageTag := kubeadmutil.KubernetesVersionToImageTag(cfg.KubernetesVersion)
return GetGenericImage(repoPrefix, image, kubernetesImageTag)
}This function calls the GetGenericImage() function with the image repository, name, and tag to generate and return a platform-agnostic image.
The image repository is obtained using the cfg.GetControlPlaneImageRepository() function.
Let’s take a look at it:
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/apis/kubeadm/types.go#L389
// GetControlPlaneImageRepository returns name of image repository
// for control plane images (API,Controller Manager,Scheduler and Proxy)
// It will override location with CI registry name in case user requests special
// Kubernetes version from CI build area.
// (See: kubeadmconstants.DefaultCIImageRepository)
func (cfg *ClusterConfiguration) GetControlPlaneImageRepository() string {
if cfg.CIImageRepository != "" {
return cfg.CIImageRepository
}
return cfg.ImageRepository
}This function will return the ImageRepository field of its received ClusterConfiguration struct, which is the imageRepository we defined in kubeadm-config.yml
So long, so good!
Now let’s get back to GetControlPlaneImages() function.
// https://github.com/kubernetes/kubernetes/blob/1edbb8cf1a864c2dd71a1e5b7282de4bc3d8f5fa/cmd/kubeadm/app/images/images.go#L90
// GetControlPlaneImages returns a list of container images kubeadm expects to use on a control plane node
func GetControlPlaneImages(cfg *kubeadmapi.ClusterConfiguration) []string {
images := make([]string, 0)
// start with core kubernetes images
images = append(images, GetKubernetesImage(constants.KubeAPIServer, cfg))
images = append(images, GetKubernetesImage(constants.KubeControllerManager, cfg))
images = append(images, GetKubernetesImage(constants.KubeScheduler, cfg))
images = append(images, GetKubernetesImage(constants.KubeProxy, cfg))
// pause is not available on the ci image repository so use the default image repository.
images = append(images, GetPauseImage(cfg))
// if etcd is not external then add the image as it will be required
if cfg.Etcd.Local != nil {
images = append(images, GetEtcdImage(cfg))
}
// Append the appropriate DNS images
images = append(images, GetDNSImage(cfg))
return images
}As you can see, the CoreDNS image reference is retrieved from the GetDNSImage() function.
Here’s the GetDNSImage() function:
// https://github.com/kubernetes/kubernetes/blob/b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d/cmd/kubeadm/app/images/images.go#L43
// GetDNSImage generates and returns the image for CoreDNS.
func GetDNSImage(cfg *kubeadmapi.ClusterConfiguration) string {
// DNS uses default image repository by default
dnsImageRepository := cfg.ImageRepository
// unless an override is specified
if cfg.DNS.ImageRepository != "" {
dnsImageRepository = cfg.DNS.ImageRepository
}
// Handle the renaming of the official image from "registry.k8s.io/coredns" to "registry.k8s.io/coredns/coredns
if dnsImageRepository == kubeadmapiv1beta2.DefaultImageRepository {
dnsImageRepository = fmt.Sprintf("%s/coredns", dnsImageRepository)
}
// DNS uses an imageTag that corresponds to the DNS version matching the Kubernetes version
dnsImageTag := constants.CoreDNSVersion
// unless an override is specified
if cfg.DNS.ImageTag != "" {
dnsImageTag = cfg.DNS.ImageTag
}
return GetGenericImage(dnsImageRepository, constants.CoreDNSImageName, dnsImageTag)
}This function uses the same imageRepository we defined in kubeadm-config.yml but with one difference!
Check this if clause:
// Handle the renaming of the official image from "registry.k8s.io/coredns" to "registry.k8s.io/coredns/coredns
if dnsImageRepository == kubeadmapiv1beta2.DefaultImageRepository {
dnsImageRepository = fmt.Sprintf("%s/coredns", dnsImageRepository)
}If the imageRepository is the same as DefaultImageRepository, /coredns is attached to it; Since my imageRepository is not the same as DefaultImageRepositoryI missed this little tricky change!
The fix is simple; set imageRepository under dns key in the kubeadm-config.yml file to match this if clause:
// unless an override is specified
if cfg.DNS.ImageRepository != "" {
dnsImageRepository = cfg.DNS.ImageRepository
}You may ask:
Why didn’t the Kubernetes contributors handle this?!
The answer is simple: Backward Compatibility.
See this PR for more info.
I hope it helps you.
Please let me know your thoughts in the comments section below.
Thank you for your time and attention.
