Understanding gRPC Load Balancing in Kubernetes with istio

gRPC (gRPC Remote Procedure Calls) is a cross-platform open source high-performance remote procedure call (RPC) framework, which uses HTTP/2 for transport.gRPC has many advantages over traditional HTTP mechanism such as Multiplexing many requests on one connection (HTTP/2). This article explains this behavior in the combination of Kubernetes(RedHat Openshift) and (Redhat Openshift Service Mesh).

Environment Preparation:

• Create two new namespace

oc new-project grpc
oc new-project test

• Deploy all the server-specific artifacts in grpc namespace.This example is based upon Quarkus quick start with gRPC.
• Create a configmap.

kind: ConfigMap
apiVersion: v1
metadata:
  name: grpcconfig
  namespace: grpc
data:
  application.properties: |
    numrequest: 25
    quarkus.grpc.clients.hello.host: grpc.grpc.svc.cluster.local
    teststring: prom

• Create a new deployment based on the following.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: grpc-v1
  namespace: grpc
  labels:
    app: grpc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grpc
  template:
    metadata:
      labels:
        app: grpc
        app.kubernetes.io/version: l1   
      annotations:
        sidecar.istio.io/inject: 'true'
        sidecar.istio.io/logLevel: debug           
    spec:
      volumes:
        - name: quarkus-config-volume
          configMap:
            name: grpcconfig
            defaultMode: 420    
      containers:
        - name: custom-grpc-hello
          image: 'quay.io/shrishs/grpc-plain-text-quickstart-jvm'
          ports:
            - name: grpc
              containerPort: 9000
              protocol: TCP
          volumeMounts:
            - name: quarkus-config-volume
              mountPath: /deployments/config                      
      restartPolicy: Always

• Create two more deployment grpc-v2 & grpc-v3 with the same config(as above) except providing a different version.

app.kubernetes.io/version: l2
&
app.kubernetes.io/version: l3

• Create a Kubernetes service based on the above deployment.

kind: Service
apiVersion: v1
metadata:
  name: grpc
  namespace: grpc
  labels:
    app: grpc
spec:
  ports:
    - name: grpc
      protocol: TCP
      port: 9000
      targetPort: 9000
  selector:
    app: grpc

• Create another configmap(as above) & deployment as grpc-client(using the same above definition) in test namepspace.It will be used for calling grpc service.testString, and numRequest are defined in configmap.

    @GET
    @Path("/grpc")
    public String testGRPC() throws InterruptedException
    {
            MyClient client = new MyClient(host, 9000);
            try {
                client.makeMultipleRequests(testString, numRequest);
            } finally {
                client.shutdown();
            }
            return "Hello";
        }

public class MyClient {
    private final ManagedChannel channel;
    private final GreeterGrpc.GreeterBlockingStub blockingStub;

    public MyClient(String host, int port) {
        this(ManagedChannelBuilder.forAddress(host, port).usePlaintext().build());
    }

    private MyClient(ManagedChannel channel) {
        this.channel = channel;
        this.blockingStub = GreeterGrpc.newBlockingStub(channel);
    }

    public void shutdown() throws InterruptedException {
        channel.shutdown().awaitTermination(5, TimeUnit.SECONDS);
    }

    public void makeMultipleRequests(String name, int numRequests) {
        for (int i = 0; i < numRequests; i++) {
            HelloRequest request = HelloRequest.newBuilder().setName(name + i).build();
            HelloReply response = blockingStub.sayHello(request);
            System.out.println(response.getMessage());
        }
    }

Kubernetes gRPC Service without Istio

• Verify if the application is running in grpc namespace.

bash-3.2$ oc -n grpc get pods -l app=grpc
NAME                       READY   STATUS    RESTARTS   AGE
grpc-v1-7cc77c5cfc-tfbhf   1/1     Running   0          9m15s
grpc-v2-977b698f6-rb9z6    1/1     Running   0          9m14s
grpc-v3-8679b7f6ff-hxx9c   1/1     Running   0          9m14s

• Test the Request from the test namespace.

oc -n test exec deploy/grpc-client -- curl http://localhost:8080/hello/grpc

• Check the logs of all the pods and it is observed that only one pod received all the requests.

The reason for the above behavior is that gRPC is built on HTTP/2, and HTTP/2 is designed to have a single long-lived TCP connection, across which all requests are multiplexed — meaning multiple requests can be active on the same connection at any point in time. Normally, this is great, as it reduces the overhead of connection management. However, it also means that (as you might imagine) connection-level balancing isn’t very useful. Once the connection is established, there’s no more balancing to be done. All requests will get pinned to a single destination pod.

Kubernetes gRPC Service with Istio

• Create virtual service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grpc-vs
  namespace: grpc
spec:
  hosts:
    - grpc.grpc.svc.cluster.local
  http:
    - route:
        - destination:
            host: grpc
            port:
              number: 9000

• Create a Destination rule.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: grpc-dr
  namespace: grpc
spec:
  host: grpc
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        http2MaxRequests: 1
        maxRequestsPerConnection: 1
      tcp:
        maxConnections: 1  
  subsets:
  - name: v1
    labels:
      app.kubernetes.io/version: l1
  - name: v2
    labels:
      app.kubernetes.io/version: l2
  - name: v3
    labels:
      app.kubernetes.io/version: l3

• Add grpc and test namespace in ServiceMeshMemberRoll.

apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
  name: default
  namespace: istio-system
spec:
  members:
    - grpc
    - test

• Restart the pods in grpc and test namespace. Now envoy proxy is attached as sidecar.

bash-3.2$ oc -n grpc get pods -l app=grpc
NAME                       READY   STATUS    RESTARTS   AGE
grpc-v1-7cc77c5cfc-fm9m4   2/2     Running   0          33s
grpc-v2-977b698f6-hrmhq    2/2     Running   0          33s
grpc-v3-8679b7f6ff-lw9fp   2/2     Running   0          32s

bash-3.2$ oc -n test get pods -l app=grpc
NAME                           READY   STATUS    RESTARTS   AGE
grpc-client-65bb4d7c64-4ftpz   2/2     Running   0          37s

• Test the service again.

oc -n test exec deploy/grpc-client -- curl http://localhost:8080/hello/grpc

• Verify the graph in Kiali. Now it is observed that load balancing is happening across all 3 pods.

The reason for the above behavior is that istio(envoy proxy) open an HTTP/2 connection to each destination(all 3 pods), and balance requests across these connections.

• Verify it in jaeger.

ConnectionPoolSetting

The behavior of load balancing can be changed using DestinationRule ConnectionPool settings.

• Pick up the request-id from the above trace and look up into istio proxy logs.

Observe the line above (Creating a new connection). This is because of as above in destination Rule connectionPool setting (http2MaxRequests & maxRequestPerConnection is defined as 1). On each request a new connection is created and the connection is destroyed after serving the request.

• Change the destinationRule ConnectionPool setting(http2MaxRequests: 50,maxRequestsPerConnection: 100)

spec:
  host: grpc
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        http2MaxRequests: 50
        maxRequestsPerConnection: 100
      tcp:
        maxConnections: 1  

• Test the service again

oc -n test exec deploy/grpc-client -- curl http://localhost:8080/hello/grpc

• Pick up the request id from jaeger and look into envoy proxy logs. As now we have requests per connection more than one, It is using an existing connection to serve the request.

Accessing gRPC service via ingress gateway

Openshift uses HAProxy as ingress controller.HTTP2 is not enabled by default. It can be enabled as follows.

oc annotate ingresses.config/cluster ingress.operator.openshift.io/default-enable-http2=true

• Once HAProxy is enabled for HTTP/2 , Create a gateway definition with mutual TLS.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grpc-gateway
  namespace: grpc
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - grpc-istio-system.apps.ocp-XXX.com
    port:
      number: 443
      protocol: https
      name: HTTPS
    tls:
      mode: MUTUAL
      credentialName: istio-ingressgateway-certs

• Openshift Service Mesh automatically creates a passthrough route definition.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: grpc-grpc-gateway
  namespace: istio-system
  labels:
    maistra.io/gateway-name: grpc-gateway
    maistra.io/gateway-namespace: grpc
    maistra.io/generated-by: ior
  annotations:
    maistra.io/original-host: grpc-istio-system.apps.ocp-XXX.com
spec:
  host: grpc-istio-system.apps.ocp-XXX.com
  to:
    kind: Service
    name: istio-ingressgateway
    weight: 100
  port:
    targetPort: https
  tls:
    termination: passthrough
  wildcardPolicy: None

• Verify if the gRPC Service is working.

$ grpcurl --proto helloworld.proto -v  --insecure --cert istio/files/cert/client.example.com.crt --key istio/files/cert/client.example.com.key -d '{"name": "test"}' grpc-istio-system.apps.ocp-XXX.com:443 helloworld.Greeter/SayHello

Resolved method descriptor:
// Sends a greeting
rpc SayHello ( .helloworld.HelloRequest ) returns ( .helloworld.HelloReply );

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
date: Fri, 19 May 2023 15:40:14 GMT
grpc-accept-encoding: gzip
server: istio-envoy
x-envoy-upstream-service-time: 12

Response contents:
{
  "message": "Hello test"
}

Response trailers received:
(empty)
Sent 1 request and received 1 response

NOTE: Sometimes it does not work as the definition is not synchronized in envoy proxy. In that case, restart the ingress gateway

oc -n istio-system delete pod -l app=istio-ingressgateway

• Verify the same in Kiali.