Kubernetes tips & tricks: usefull kubectl commands

5 min readNov 14, 2023

Some commands may turn out to be everyday routine for some readers, but if there are also those who discover new things for themselves, thereby improving their effectiveness, the purpose of the article will be achieved.

Getting lists of pods and nodes

Many may already be familiar with retrieving all Pods from every namespace using the — all-namespaces flag. However, some users might not have noticed the introduction of a more concise alternative: -A (since the release of Kubernetes 1.15), but its existence may have eluded those accustomed to the longer syntax.
Discovering all pods facing issues and not in a running state (i.e., not Running) can be achieved:

kubectl get pods -A --field-selector=status.phase!=Running | grep -v Complete

By the way, taking a closer look at — field-selector is generally very useful (see the docs).

3. Getting a list of nodes indicating the amount of their RAM:

kubectl get no -o json | \
  jq -r '.items | sort_by(.status.capacity.memory)[]|[.metadata.name,.status.capacity.memory]| @tsv'

4. Getting a list of nodes and the number of pods on them:

kubectl get po -o json --all-namespaces | \
  jq '.items | group_by(.spec.nodeName) | map({"nodeName": .[0].spec.nodeName, "count": length}) | sort_by(.count)'

5. At times, DaemonSet may fail to schedule a pod on a node for various reasons. Manually hunting down these nodes can be a cumbersome task. To streamline the process, consider using the following mini-script to generate a list of nodes facing this issue:

ns=my-namespace
pod_template=my-pod
kubectl get node | grep -v \"$(kubectl -n ${ns} get pod -A -o wide | fgrep ${pod_template} | awk '{print $8}' | xargs -n 1 echo -n "\|" | sed 's/[[:space:]]*//g')\"

6. Utilizing kubectl top, you can easily retrieve information about pods that are consuming the highest amounts of processor or memory. Here’s how:

# cpu
kubectl top pods -A | sort --reverse --key 3 --numeric
# memory
kubectl top pods -A | sort --reverse --key 4 --numeric

7. Sorting the pod list — specifically, by the number of restarts:

kubectl get pods --sort-by=.status.containerStatuses[0].restartCount

8. Sorting can be performed based on other fields as well:

kubectl get pod --sort-by=.status.phase
kubectl get pod --sort-by=.status.startTime
kubectl get pod --sort-by=.spec.nodeName

Getting other information

When troubleshooting Ingress, the inevitable step involves navigating to the service and then searching for pods through its selector. Initially, I would search for the selector in the service manifest, but over time, I found it more effective to use it later with -o wide

kubectl get svc -o wide

In this case, we promptly obtain a selector that the service uses to locate the required pods.

2. Retrieve the limits and requests for each container in every pod using the following command:

kubectl get pods -n namespace -o=custom-columns='NAME:spec.containers[*].name,MEMREQ:spec.containers[*].resources.requests.memory,MEMLIM:spec.containers[*].resources.limits.memory,CPUREQ:spec.containers[*].resources.requests.cpu,CPULIM:spec.containers[*].resources.limits.cpu'

3. The kubectl run command, along with create, apply, and patch, offers a valuable feature to preview changes before they take effect — achieved through the use of the — dry-run=client flag. Additionally, when combining with -o yaml, you can obtain a manifest for the desired entity. For example:

kubectl create deploy graph --image=grafana --dry-run=client -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: graph
  name: graph
spec:
  replicas: 1
  selector:
    matchLabels:
      app: graph
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: graph
    spec:
      containers:
      - image: grafana
        name: grafana

All that’s left is to save it to a file ( > deploy.yaml), remove a few system/unnecessary fields, and it’s ready for further use.

4. Retrieve a detailed explanation of a resource’s manifest:

kubectl explain hpa
GROUP:      autoscaling
KIND:       HorizontalPodAutoscaler
VERSION:    v2

DESCRIPTION:
    HorizontalPodAutoscaler is the configuration for a horizontal pod
    autoscaler, which automatically manages the replica count of any resource
    implementing the scale subresource based on the metrics specified.
    
FIELDS:
  apiVersion    <string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

  kind  <string>
    Kind is a string value representing the REST resource this object
    represents. Servers may infer this from the endpoint the client submits
    requests to. Cannot be updated. In CamelCase. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

  metadata      <ObjectMeta>
    metadata is the standard object metadata. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

  spec  <HorizontalPodAutoscalerSpec>
    spec is the specification for the behaviour of the autoscaler. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status.

  status        <HorizontalPodAutoscalerStatus>
    status is the current information about the autoscaler.

Networks

Obtain internal IP addresses of cluster nodes:

kubectl get nodes -o json | \
  jq -r '.items[].status.addresses[]? | select (.type == "InternalIP") | .address' | \
  paste -sd "\n" -

2. Display a comprehensive list of services along with the NodePort assignments they occupy:

kubectl get -A svc -o json | \
  jq -r '.items[] | [.metadata.name,([.spec.ports[].nodePort | tostring ] | join("|"))]| @tsv'

3. In cases where CNI issues arise, such as those with Flannel, pinpointing a problematic pod requires examining the routes. The subnets of pods utilized in the cluster can be particularly helpful in this context:

kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}' | tr " " "\n"

Logs

Obtaining pod logs with a human-readable timestamp if not already present:

kubectl -n my-namespace logs my-pod --timestamps

2. Avoid waiting for the complete output of the pod container log — utilize the —tail option:

kubectl -n my-namespace logs my-pod --tail=50

3. Retrieve logs from all containers within a pod:

kubectl -n my-namespace logs my-pod --all-containers

4. Fetch logs from all pods identified by a specific label:

kubectl -n my-namespace logs -l app=nginx

5. Retrieve logs from the preceding container, such as the one that may have crashed:

kubectl -n my-namespace logs my-pod --previous

Other quick actions

How to copy all secrets from one namespace to another:

kubectl get secrets -o json --namespace namespace-old | \
  jq '.items[].metadata.namespace = "namespace-new"' | \
  kubectl create -f  -

2. Quickly create a self-signed certificate for tests:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=grafana.mysite.ru/O=MyOrganization"
kubectl -n myapp create secret tls mysecret --key=tls.key --cert=tls.crt

Useful links on the topic

Instead of a conclusion, there is a small list of similar materials and collections that I used, among other things, to prepare for the exam

Official cheatsheet from Kubernetes documentation;
A very extensive list of teams from Blue Matador, divided into sections;
A selection of links in Gist to tables with commands, articles on the topic and some commands;
Another GitHub repository contains only commands broken down into categories;
GitHub repo kubectl-aliases;
My personal GitHub with usefull commands;