Kubernetes Observability Part 3: Events, Logs & integration with Slack, OpenAI and Grafana

Build an interactive Slack message and investigate the root cause of a Kubernetes Events with the help of OpenAI Chat API.

TL;DR: Make an interactive Slack message that communicates with the Kubernetes custom controller via WebSockets and prompt OpenAI API in order to provide solutions for Kubernetes Events.

Introduction

In this articles series we will explore an one-stop-shop Observability solution for Kubernetes, where a custom Controller collects all the Kubernetes Events occuring in a cluster, forwards them to Grafana Loki via Promtail, decorates the one that indicate a problem with the Pods’ Logs that were recorded in the timeframe that each Event took place and sends a an interactive message to a Slack channel/bot that on demand could ask OpenAI Chat API to help us solving the error, based on the original Event message.

The series will consist out of three parts:

1. Part 1: Create a custom Controller, forward Events to Grafana Loki, present Events & Logs on a Grafana dashboard
2. Part 2: Build the integration with Slack and forward alerts.
3. Part 3: Build the integration with OpenAI Chat API and get solution proposal.

This is the third part of the series.

What is OpenAI Chat API ?

OpenAI Chat API is a platform offered by OpenAI that enables developers to build conversational interfaces, such as chatbots, virtual assistants, and messaging apps. It is a powerful natural language processing tool that can understand and generate human-like language, allowing chatbots to carry on more intelligent and natural conversations with users.

With OpenAI Chat API, developers can integrate pre-trained language models or build their custom models, depending on their specific needs and use cases. The API can be used for a wide range of applications, including customer service, sales, education, healthcare, and more.

The API offers several features, including advanced natural language processing capabilities, contextual understanding, sentiment analysis, and the ability to integrate with different platforms and channels.

Overall, OpenAI Chat API can help businesses and developers create more engaging and personalized interactions with users while improving efficiency and reducing costs.

OpenAI

What is an interactive Slack message ?

An interactive Slack message is a message within the Slack platform that allows users to interact with it and perform actions without leaving the chat interface. It is a type of message that includes interactive elements such as buttons, menus, and forms that users can interact with to trigger actions or provide input.

For example, an interactive Slack message could include buttons that allow users to approve or reject a task, select options from a menu, or fill out a form to provide information.

Interactive Slack messages are created using Slack’s Block Kit framework, which provides a set of tools and components for building interactive messages. Developers can use Block Kit to create custom layouts and designs for messages, as well as add interactive elements to enhance the user experience.

Overall, interactive Slack messages can improve communication and collaboration within teams by providing a more interactive and engaging way to perform tasks and share information within the Slack platform.

Making messages interactive

In the lifecycle of a typical interactive message flow, at some point Slack will send a request to a designated Request URL for your app, including all the context needed to identify the originating message, the user executing the action, and the specific values you’ve associated with the action. This request also contains a response_url you can use to continue interacting with the user or channel. In our case, as long as we are running inside a Kubernetes Cluster (namely in the context of a controller) that would be extremely cumbersome (don’t translate cumbersome as impossible). For that matter we are going to take another road to establish a two way communication between the controller and the interactive Slack messages. And that would be Slack Socket Mode.

Slack Socket Mode is a feature provided by Slack that allows developers to build custom integrations and bots that can interact with the Slack platform using a WebSocket connection instead of a traditional HTTP request-response cycle.

With Socket Mode, developers can create applications that can listen to real-time events and send messages to Slack without requiring a publicly accessible web server. This can be particularly useful for integrating with systems or devices that are behind a firewall or cannot be easily exposed to the internet.

Socket Mode uses a secure, encrypted WebSocket connection between the developer’s application and Slack’s servers. It allows developers to receive real-time events such as message updates, reactions, and presence changes, and respond to them in real-time. Developers can also send messages to Slack channels, users, and bots using the same WebSocket connection.

Using Socket Mode, developers can build custom integrations and bots that can listen and respond to events in real-time, without requiring a public URL or webhook. It provides a more secure and efficient way to build real-time applications on top of the Slack platform.

Enable Socket Mode for your app

Open https://api.slack.com/apps and under Socket Mode, enable Socket mode:

Follow the wizard, create an app token and as long as is created save its value as we are going to need it.

For the time being let’s put a pin on that and return back to our controller. What we want to accomplish are the following:

1. Add a button to our Slack message payload.
2. Wire this button back to the controller and get a dummy response.
3. Make a request to OpenAI Chat API asking guidance on resolving the issue.

Add a button the the Slack message

Adding a simple button to the payload of a message is fairly simple process. All you need to do is create a Slack Attachment , add to it a button as an AttachmentAction and pass it asMsgOptionAttachments to the PostMessage method of the Slack client. Let’s implement this in the forwardEvent function of event_controller.go:

func forwardEvent(
 level string,
 note string,
 commonLabels map[string]string,
 extraLabels map[string]string,
 firstSeen time.Time,
 lastSeen time.Time,
 logs string,
) error {

{{...omitted for brevity, as it's the same as before...}}

chatGptAttachment := slack.Attachment{
  Pretext:    "🆘 *Use OpenAI Chat API  to analyse the Event and suggest you a course of action:*",
  Fallback:   "Your client is not supported",
  CallbackID: "askGPT",
  Color:      "#3AA3E3",
  Actions: []slack.AttachmentAction{
   slack.AttachmentAction{
    Name:  "askgpt_action",
    Text:  "💬 Ask ChatGPT for help",
    Type:  "button",
    Value: note,
    Style: "primary",
   },
  },
 }

 attachments := slack.MsgOptionAttachments(chatGptAttachment)

 _, _, err := slackClient.PostMessage(
  channelID,
  msgOptionBlocks,
  attachments,
  slack.MsgOptionAsUser(true))
 if err != nil {
  return err
 }

{{...omitted for brevity, as it's the same as before...}}

return nil
}

Run your controller and let’s see what are we receiving to our channel’s end:

We successfully got a button as part of our alert message. At the time being does nothing, as it is not wired to any functionality back in our controller. If you click it, you’ll get a Slackbot message let you know that something went wrong:

Wire the button back to the controller

Here starts getting slightly more complicated. If you remember we chose to go with Slack Socket Mode, for the reason we explained above. Slack Socket Mode requires another kind of client to establish the requires connections through WebSockets, a socketMode.Client. Add in the imports of your event_controller.go the following package:

 "github.com/slack-go/slack/socketmode"

then, add slackSocketClient and slackAppToken (the app token we acquired after we enabled Socket Mode) in your vars:

var (
 slackBotToken     = "xoxb-XXXXXX"
 slackAppToken     = "xapp-XXXXXX"
 channelID         = "C0XXXXXXXXX"
 slackClient       *slack.Client
 slackSocketClient *socketmode.Client
)

and add the following functions to event_controller.go:

func initSlackModeClient() {

 slackSocketClient = socketmode.New(
    slackClient, 
    socketmode.OptionDebug(false))

 ctx, cancel := context.WithCancel(context.Background())
 defer cancel()

 go func(ctx context.Context, 
         slackClient *slack.Client, 
         socketModeClient *socketmode.Client) {
  logger := log.FromContext(ctx).WithName("socketMode")
  logger.Info("starting slack socketmode listener")

  for {
   select {
   case <-ctx.Done():
    logger.Info("shutting down slack socketmode listener")
    return
   case event := <-socketModeClient.Events:
    switch event.Type {
    case socketmode.EventTypeInteractive:
     icb, ok := event.Data.(slack.InteractionCallback)
     if !ok {
      logger.Info("could not typecast: %t\n", icb)
      continue
     }

     socketModeClient.Ack(*event.Request)
     prompt := icb.ActionCallback.AttachmentActions[0].Value
     
     response, err := askChatGpt(prompt)
     if err != nil {
      logger.Error(err, "failed to complete with chatgpt")
     }

     if strings.TrimSpace(response) != "" {
      err := forwardChatGptResponse(prompt, response)
      if err != nil {
       logger.Error(err, "failed to write back to slack channel")
      }
     }
    }
   }
  }

 }(ctx, slackClient, slackSocketClient)

 slackSocketClient.Run()
}

The function initSlackModeClient will accomplish three things. It will create a socketMode.Client, it will start the socketMode.Client:

slackSocketClient.Run()

and it will spawn a goroutine that will observe the context for termination or cancellation and the socketModeClient.Events for any incoming interactive event.Type(EventTypeInteractive). The question we want to ask ChatGPT is held in variable prompt, which is in practice the Event.Note value of the correlated Kubernetes Event in the original Slack message payload.

Second function we want to add is forwardChatGptResponse, which does nothing more than preparing a new message with the ChatGPT response as payload and sending it to the designated Slack channel:

func forwardChatGptResponse(prompt string, response string) error {
 headerText := fmt.Sprintf("🚧 *ChatGPT response for the event:* %s ", prompt)
 headerTextBlock := slack.NewTextBlockObject("mrkdwn", headerText, false, false)
 headerSection := slack.NewSectionBlock(headerTextBlock, nil, nil)

 chatGptTextBlock := slack.NewTextBlockObject("mrkdwn", response, false, false)
 chatGptSection := slack.NewSectionBlock(chatGptTextBlock, nil, nil)

 msgOption := slack.MsgOptionBlocks(
  headerSection,
  chatGptSection,
 )

 _, _, err := slackClient.PostMessage(
  channelID,
  msgOption,
  slack.MsgOptionAsUser(true))
 if err != nil {
  return err
 }

 return nil
}

Third function we need to add is askChatGpt. This one will perform later the interaction with OpenAI Chat API, but for the time being we will just return a dummy message.

func askChatGpt(prompt string) (string, error) {
 return "dummy response", nil
}

Last piece of the puzzle, change the initSlackClient function so it look like this:

func initSlackClient() {
 //slackClient = slack.New(slackBotToken, slack.OptionDebug(false))
 slackClient = slack.New(
  slackBotToken,
  slack.OptionDebug(false),
  slack.OptionAppLevelToken(slackAppToken),
 )
}

and in your reconciliation loop method, Reconcile, let it initialise the slackSocketClient as well:

if slackClient == nil {
   initSlackClient()
  }

  if slackSocketClient == nil {
   initSlackModeClient()
   go initSlackModeClient()
  }

Let’s run now our controller and click the “Ask ChatGPT for help” button of one of our messages:

That’s pretty cool, we just made an interactive Slack message and wired its functionality inside our Kubernetes customer Controller.

Make a request to OpenAI Chat API

First thing we need to do is create an API Key in order to consume OpenAI Chat API from our controller.

Go to https://platform.openai.com/account/api-keys and create your key and add it to your controller vars:

var(
  {{...the rest vars are omitted for brevity...}}

  chatGptApiKey         = "sk-XXXXXX"
)

Now, replace the body of the askGptChat function with this:

func askChatGpt(prompt string) (string, error) {
 client := openai.NewClient(chatGptApiKey)
 resp, err := client.CreateChatCompletion(
  context.Background(),
  openai.ChatCompletionRequest{
   Model: openai.GPT3Dot5Turbo,
   Messages: []openai.ChatCompletionMessage{
    {
     Role:    openai.ChatMessageRoleUser,
     Content: prompt,
    },
   },
  },
 )
 
 if err != nil {
  return "", err
 }
 
 return resp.Choices[0].Message.Content, nil
}

and add in your import section the following package:

openai "github.com/sashabaranov/go-openai" 

Now let’s run our controller for the last time, and if everything went well we should have an actual response from OpenAI Chat API using the model ChatGPT3.5Turbo:

That’s pretty amazing, if you consider that we started literally from zero building this Kubernetes custom Controller. We can now observe in real-time Events that indicate a problem either in the cluster or in an application running on the cluster, bundle them with their corresponding logs, send them to a designated Slack channel and give the user the opportunity to ask ChatGPT for guidance or even a solution plan!

Follow up

I’ve mentioned this many times through this series, that this is a quick-n-dirty prototype and it doesn’t evangelise neither best practices in developing Kubernetes custom controllers nor best practices in Goland. Nevertheless, is a conceptually a pretty solid MVP and stepping on that lot of things done scrappy can be corrected e.g. eliminate magic-string, put configuration values in ConfigMaps, put sensitive configuration values in Secrets and many more. Next steps, besides fulfilling the ones above is mainly decoupling the Controller from Slack client and create an extensible versatile basis in order to integrate with more clients like Microsoft Teams, Mattermost and many more.

If you liked this series don’t forget to clap 👏 and follow. Stay tuned and drop a comment.

You can find the source code here:

GitHub — akyriako/kvnts: A Kubernetes custom Controller to watch & forward Kubernetes Events to…