Using Leader Election with Spring Cloud Kubernetes and Spring Scheduler

Introduction

In microservices environments, it’s often necessary to ensure that only one instance performs scheduled tasks, such as batch processing, cache updates, or other periodic activities. To address this requirement, we can combine the concept of “leader election” with Spring Cloud Kubernetes and Spring Scheduler. In this article, we will explore how to implement this pattern and schedule tasks only on the leader instance.

An application running on kubernetes with 3 replicas

Prerequisites

Before proceeding, it’s important to have a basic understanding of microservices, the Spring Framework (including Spring Boot, Spring Cloud Kubernetes, and Spring Scheduler), and Kubernetes. Make sure you have properly set up your development environment with the necessary dependencies.

How does it work

The Spring Cloud Kubernetes leader election mechanism implements the leader election API of Spring Integration using a Kubernetes ConfigMap.

Multiple application instances compete for leadership, but leadership will only be granted to one. When granted leadership, a leader application receives an OnGrantedEvent application event with leadership Context. Applications periodically attempt to gain leadership, with leadership granted to the first caller. A leader will remain a leader until either it is removed from the cluster, or it yields its leadership. When leadership removal occurs, the previous leader receives OnRevokedEvent application event. After removal, any instances in the cluster may become the new leader, including the old leader.

Implementation

To illustrate the use of leader election with Spring Cloud Kubernetes and Spring Scheduler, let’s consider a scheduler called “ScheduledTasks”. Our goal is to schedule a sample task to be executed only on the leader instance.

1. First, add the necessary dependencies to the pom.xml file of your Spring Boot project:

  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter</artifactId>
  </dependency>

  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
  <dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-kubernetes-fabric8-leader</artifactId>
   <version>3.0.2</version>
  </dependency>
  <dependency>
   <groupId>org.awaitility</groupId>
   <artifactId>awaitility</artifactId>
   <version>3.1.2</version>
   <scope>test</scope>
  </dependency>

2. Next, create a class named ScheduledTask that will be responsible for performing the scheduled task. Annotate this class with @Component from Spring and configure it to run periodically using the @Scheduled annotation.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import java.text.SimpleDateFormat;
import java.util.Date;

@Component
public class ScheduledTasks {
    private static final Logger log = LoggerFactory.getLogger(ScheduledTasks.class);

    private static final SimpleDateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");

    private ManagerContext managerContext;
    public ScheduledTasks(ManagerContext managerContext){
        this.managerContext = managerContext;
    }
    @Scheduled(fixedRate = 5000)
    public void reportCurrentTime() {

        if (isLeader()) {
            // Perform the task
            log.info("The time is now {}", dateFormat.format(new Date()));
        } else {
            // Not the leader instance, wait until becoming the leader
            log.info("I am not an leader");
        }

    }

    private boolean isLeader() {
        // Logic to check if this instance is the leader
        return managerContext.getContext() != null; // Modify the logic to check real leadership
    }
}

3. Now, configure Spring Cloud Kubernetes to manage leader election, . Edit the application.properties file and add the following configurations:

spring.cloud.kubernetes.leader.role=world
# Configmap to which leader election metadata will be saved
spring.cloud.kubernetes.leader.config-map-name=my-config-map

In this example, we set “SchedulerService” as the role for which we want to elect a leader and the name of the configmap used for leader election.

4. Finally, enable the leader election functionality in Spring Cloud Kubernetes by creating a configuration class named LeaderController:

import br.com.jpmm.sampleleaderelectionwithspringcloud.support.ManagerContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.event.EventListener;
import org.springframework.http.ResponseEntity;
import org.springframework.integration.leader.Context;
import org.springframework.integration.leader.event.OnGrantedEvent;
import org.springframework.integration.leader.event.OnRevokedEvent;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.net.InetAddress;
import java.net.UnknownHostException;

@RestController
@RequestMapping("/leader")
public class LeaderController {


 private static final Logger log = LoggerFactory.getLogger(LeaderController.class);
 private final String host;

 @Value("${spring.cloud.kubernetes.leader.role}")
 private String role;

 private Context context;

 private ManagerContext managerContext;

 public LeaderController(ManagerContext managerContext) throws UnknownHostException {
  this.host = InetAddress.getLocalHost().getHostName();
  this.managerContext =managerContext;
 }

 /**
  * Return a message whether this instance is a leader or not.
  * @return info
  */
 @GetMapping
 public String getInfo() {
  if (this.context == null) {
   return String.format("I am '%s' but I am not a leader of the '%s'", this.host, this.role);
  }
  return String.format("I am '%s' and I am the leader of the '%s'", this.host, this.role);
 }

 /**
  * PUT request to try and revoke a leadership of this instance. If the instance is not
  * a leader, leadership cannot be revoked. Thus "HTTP Bad Request" response. If the
  * instance is a leader, it must have a leadership context instance which can be used
  * to give up the leadership.
  * @return info about leadership
  */
 @PutMapping
 public ResponseEntity<String> revokeLeadership() {
  if (this.context == null) {
   String message = String.format("Cannot revoke leadership because '%s' is not a leader", this.host);
   return ResponseEntity.badRequest().body(message);
  }
  this.context.yield();
  String message = String.format("Leadership revoked for '%s'", this.host);
  return ResponseEntity.ok(message);
 }

 /**
  * Handle a notification that this instance has become a leader.
  * @param event on granted event
  */
 @EventListener
 public void handleEvent(OnGrantedEvent event) {
  System.out.println(String.format("'%s' leadership granted", event.getRole()));
  this.context = event.getContext();
  managerContext.setContext(this.context);
 }

 /**
  * Handle a notification that this instance's leadership has been revoked.
  * @param event on revoked event
  */
 @EventListener
 public void handleEvent(OnRevokedEvent event) {
  System.out.println(String.format("'%s' leadership revoked", event.getRole()));
  this.context = null;
  managerContext.setContext(null);
 }

}

Now, when starting the application, Spring Cloud Kubernetes will handle leader election among the instances of the SchedulerTasks service. Only the leader instance will execute the logic of the scheduled task, while the other instances will wait until they become the leader.

You can get if that microservice is the leader, in addition to being able to revoke the leadership through an endpoint

Deploying on Kubernetes

After uploading your image to some registry, in this case I already uploaded it to the docker hub, let’s deploy it to kubernetes

1. Creating the role that will allow the application to call the Kubernetes APIs

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: leader
  labels:
    app: kubernetes-leader-election-example
    group: org.springframework.cloud
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - watch
  - get
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - watch
  - get
  - update
  # resourceNames:
  #   - <config-map name>

2. Creating the rolebinding for the default service account

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: kubernetes-leader-election-example
    group: org.springframework.cloud
  name: leader
roleRef:
  apiGroup: ""
  kind: Role
  name: leader
subjects:
- kind: ServiceAccount
  name: default
  apiGroup: ""

3. Creating the configMap that will be used by the application

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config-map
data: {}

4. Creating the deployment for kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: kubernetes-leader-election-example
  name: kubernetes-leader-election-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-leader-election-example
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kubernetes-leader-election-example
    spec:
      containers:
      - image: joaomilhome/poc-leader-election
        name: poc-leader-election
        ports:
        - containerPort: 8080
        resources: {}
status: {}

4. After the deploy you can verify two situations the first one that the configMap was filled with the name of the pod, second that in the log of the container it shows that it assumed the leadership

kubectl logs kubernetes-leader-election-example-7bb45f4d8b-vkpgg

Configmap with value

After that scale the deployment to 3 replicas

kubectl scale deployment kubernetes-leader-election-example --replicas=3

You will see a message in the other pods

2023-05-22T19:19:37.524Z  INFO 1 --- [   scheduling-1] b.c.j.s.support.ScheduledTasks           : I'm not the leader

You can kill the leader to verify that leadership will be taken over by another pod.

GITHUB

Follow all the implementation code

https://github.com/Jpmmdf/leader-election-spring

Conclusion

In this article, we explored how to use leader election with Spring Cloud Kubernetes and Spring Scheduler to schedule tasks only on the leader instance of a service. This approach ensures that only one instance performs scheduled tasks in microservices environments. By combining Spring Cloud Kubernetes and the leader election pattern, we can achieve efficient task scheduling and coordination within a distributed system.

By leveraging the Spring Cloud Kubernetes framework, we can easily integrate leader election capabilities into our applications. This allows us to designate a single instance as the leader, responsible for executing critical or specific tasks, while the other instances wait for their turn. The leader election process is managed by Kubernetes, ensuring fault tolerance and high availability.

Additionally, by utilizing the Spring Scheduler, we can schedule recurring tasks or jobs within our application. By combining this with leader election, we can ensure that these scheduled tasks are only executed by the leader instance. This guarantees that the tasks are performed reliably and avoids duplicate or conflicting executions across the instances.

Overall, the combination of leader election, Spring Cloud Kubernetes, and Spring Scheduler empowers developers to build scalable and resilient applications. By leveraging the capabilities of Kubernetes and the Spring ecosystem, we can effectively manage task execution, improve efficiency, and enhance the overall reliability of our microservices architecture.