How to Debug Kubernetes App Errors Like a Pro 1/3

In today’s rapidly advancing technological landscape, migrating from a monolithic to microservices architecture is becoming increasingly common. However, for those who are less experienced in the field, adapting to these new resources can present a significant challenge.

Whether you’re part of a development team, DevOps, infrastructure, or any other technical team, this article will provide valuable insights on how to optimize your debugging process.

You’ll learn about the tools and strategies I personally use to save time and improve efficiency when debugging microservices.

Keep reading to discover expert tips and tricks on how to debug microservices like a pro!

Before diving into the world of debugging errors in Kubernetes, I highly recommend reading my previous article, “Kubernetes: Another Perspective.” Once you’ve done that, I want to emphasize that the key to debugging errors in Kubernetes is to have a perfect understanding of the application. It isn’t just a matter of knowing that dependencies exist between other applications, or that a certain port needs to be exposed, or that there’s a requirement for connection to another application within the same Kubernetes cluster.

To truly succeed in debugging Kubernetes errors, it’s crucial to fully immerse yourself in the world of developers, and gain a deep understanding of how these applications work and what they need in order to function properly. By doing so, you’ll be better equipped to tackle any errors that may arise, and ensure that your applications run smoothly and efficiently.

So take the time to fully understand your applications and their dependencies, and you’ll be well on your way to becoming a Kubernetes debugging expert!

This post is primarily focused on what to do when your application is already within Kubernetes and is experiencing issues, not when the application itself has errors or when an external factor is blocking it. We’ll also cover some common error scenarios.

When it comes to debugging, I have two approaches depending on the criticality and the level of familiarity with the application: top-down and bottom-up.

Top-down approach.

This method is used when we know the application well and are confident that the issue lies within the Kubernetes side of things and the emergency calls and solving the problem is a priority.

We’ll begin by reviewing the ingress.

user to ingress

As we can see in the upper image, a user is attempting to connect to the Kubernetes ingress via a domain called “testing.jjotah.com.” Typically, if there is an error, it’s because the user is unable to connect to the application, or they are encountering some application errors. As we can see in the image, when an error occurs, the user attempts to connect with the ingress, and that may or may not be where the problem lies.

I’ll share with you some commands that work really well for me.

• nc → a command-line utility for reading and writing data between two computer networks
• curl → a command line tool that enables data exchange between a device and a server through a terminal
• kubectl → a command line tool for communicating with a Kubernetes cluster’s control plane, using the Kubernetes API. (alias k)

In this case, we tried to check if the port is open with the nc command

nc -v test.jjotah.com 443

When debugging issues related to domain verification with ports, it’s important to consider all the factors involved. In this case, when the error “Name or service not known” appears, I recommend following the next steps:

1. Verify the DNS settings in your domain provider. For this scenario, we need to confirm that the domain “test.jjotah.com” is pointing to the load balancer created by the ingress controller. To check this, we can use the “dig” command and verify that it’s pointing to the IP of the load balancer created with MetalLB in our kind cluster, as shown in the image.

In my case was that I didn’t create the DNS, so after create the DNS I pointed the DNS name into my load balancer ip address.

dig test.jjotah.com | grep test.jjotah.com

dig command

2. Ensure that external network connections at the cluster level are properly configured. This means reviewing any security settings or other factors that could be impacting the client machine’s DNS, particularly if the issue is only affecting one individual. By carefully checking all of these factors, we can effectively debug the issue and get our application back up and running as quickly as possible.

INGRESS PART

When we have already created the DNS we can test again the connection to “443 port”

Now this problem is “No route to host”, so I recommend checking:

1. If you have something between user and ingress. Like firewalls, CDN, or something that is working outside the cluster
2. Ingress definition

kubectl get ing

kubectl get ing

Here we find the first problem: we are doing checks to see if “443 port” is up and we have access to “80 port”, so we must try “80 port”.

If checking the “80 port" you are facing the same issue, maybe you should start with the bottom-up approach.

Let’s start by checking the container. As we know, the minimum resources in Kubernetes are pods that contain containers. Hence, we need to check if the pod is running correctly. Assuming you are working in the namespace where your application is located, we can use the following command to check the pod’s status:

kubectl get pod

In this case, the application was restarted three times, and the last restart was six minutes ago. Therefore, we need to identify the root cause of the issue. To do so, execute the following commands, replacing POD-NAME with your application pod’s name.

kubectl logs POD-NAME

kubectl describe pod POD-NAME

kubectl get events

Check for any errors, and if there is nothing, you can switch to the kube-system namespace and check the main components, executing the same commands on the following pods.

• ETCD
• Apiserver
• Controller-Manager
• Scheduler

If the issue persists, you might have to connect to the container to check the application. Although the following command is deprecated, it can still be used to connect to the container:

I know that the following command is DEPRECATED, but I’m an old school boy hahaha

kubectl exec -it POD-NAME bash

Inside the container, we need to check if our application is running (in my case, I checked if “nginx” was running and exposing the correct port). We can use the curl command on localhost to verify if the application is working correctly at the container level.

curl localhost

However, some containers may not have the required permissions to install commands like curl/nc/telnet or execute them. These containers are referred to as distroless. In such cases, it is best to create a Sidecar inside our application definition to install everything required for debugging.

If we have just a pod with “nginx”, we should convert it into a deployment and add a sidecar to debug it. To do so, we can delete the “nginx” pod and create a deployment definition.

kubectl delete pod nginx

In the next part, we’ll continue learning how to debug like a pro. Stay tuned and don’t miss it! Follow the link to the next page, see you!