Istio Service Mesh with ALB in EKS
Istio is one of the most popular Service Mesh available for Load Balancing, Service Discovery, Rate Limiting etc between microservices. If your K8s cluster is in EKS and if you install Istio using the default installation steps it creates a Classic Load Balancer and not an Application Load Balancer(ALB), as DevOps engineer there are situations where you need to install ALB for exposing your services. I have written this article based on inspiration, after reading this AWS blog. I have made some major changes to make the Istio set up to be Production-ready, I will show you how to implement Istio with some additional features that are not present in the above reference blog and create a Production ready Istio Mesh that will:
- expose multiple services via Internet-facing Application Load Balancer.
- expose service via Internal Load Balancer for calls that stay within V.P.C and are not exposed over the internet.
- expose multiple services using a single ALB Ingress attached with wildcard SSL from AWS ACM.
- implement mTLS to encrypt traffic from ALB to Istio Gateway as well.
Prerequisites
- working EKS cluster with proper access created via eksctl, terraform or manual via the console, here in this example my cluster is created via eksctl
- ALB ingress controller installed, in this demo example, I install it using eksctl
- both private and public subnets associated with EKS are tagged accordingly, if the EKS cluster is created via terraform or other methods ensure to tag the subnets accordingly.
- access to create AWS resources like route 53, certificate manager etc, it is better to have admin access to the AWS console.
- access to register and create domains and update DNS for the domains associated with your ALB, here we create two domains, for example, “some-domain.cloud” to be associated with public traffic ALB and “some-internal-domain.cloud” which is a privately hosted domain attached to a VPC for internal traffic ALB ( you might be wondering why is the need to register a privately hosted domain, this is just for creating the AWS ACM SSL certificate, for the ACM SSL verification a domain needs to be public as we use the CNAME method to verify it, this ACM SSL arn can be associated with internal ALB for mTLS, but the DNS mapping will be added in internal hosted route53 domain only so the DNS resolution stay within VPC).
As you can see we create 3 domains, 1 “domain.cloud” which is public and twice the same domain name “internal-domain.cloud” one public(just for ACM SSL cert verification using cname) and one internal associated with EKS VPC for internal DNS resolutions within VPC.
Istio gateway with internet-facing ALB(Application Load Balancer)
Once each of the prerequisites above is completed let's proceed ahead with the implementation.
- First, you need to deploy the application, here we deploy two services, for example, a simple Java spring boot application and a nginx welcome home page just to show the deployment of multiple services, sharing the manifest files for both java and the nginx service.
Please note that: In this example, we deploy two sample applications just for reference, in real-world production environment scenarios it will be complex applications that will serve a REST API application or perform some CRUD operations, here the manifest file can be taken by the reader for just reference for deploying such complex applications.
2. Install Istio with default profile and type NodePort, verify Istio installation using kubectl get po -n istio-system, you should see pods running.
istioctl install \
--set profile=default \
--set values.gateways.istio-ingressgateway.type=NodePort \
--set values.global.proxy.includeIPRanges="10.33.0.0/16"IMPORTANT : #determine-the-internal-ip-ranges-for-your-platform and dont forget to change the above "10.33.0.0/16" with the cidr with your private vpc subnet
3. Next, attach a label to the application namespace, in this case, its called “learn”. This labelling will tell Istio to inject a proxy sidecar to pods running in the namespace. You will need to delete existing pods in the namespace, if you deployed the application before labelling the namespace.
# label namespace
kubectl label ns learn istio-injection=enabled --overwrite##Do the below steps if you deployed the application before labelling the namespace, otherwise dont proceed with below steps.# delete existing pods so that Istio can inject sidecar
kubectl delete po -n learn --all# get list of pods
kubectl get po -n learn
4. Generate self-signed certificates. We will use a key/pair to encrypt traffic from ALB to Istio Gateway.
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
-keyout key.pem -out cert.pem -subj "/CN=*.yourdomain.com"Example :- #replace "*.yourdomain.com" with the wilcard domain name for your application for example in my case it will be like below:openssl req -x509 -newkey rsa:4096 -sha256 -days 36500 -nodes \
-keyout key.pem -out cert.pem -subj "/CN=*.some-domain.cloud"
5. Create a k8s secret containing key.pem and cert.pem. We will use it with Istio Gateway to implement traffic encryption.
kubectl create -n istio-system secret generic tls-secret \
--from-file=key=key.pem \
--from-file=cert=cert.pem6. Create a wildcard SSL cert for your domain using AWS ACM and save the “arn” of that resource for later use.
7. Now we have to configure traffic routing for Istio using the gateway and virtual services, apply the manifest files mentioned below,
- Istio Gateway resource is only created once, refer to this manifest file.
Istio Virtual Service resource is created per application:
- For the Java Virtual service, external application refer.
- For the sample nginx external application refer.
8. Next step is to create an ingress resource. For creating an external ingress controller download this repo and install the helm chart inside “helm-external” which accepts ACM certificate arn and hostname as parameters, and generate and install ingress correctly, ensure to change the host with some “dummy.some-domain.com” and certificate arn with the correct arn of the AWS ACM that we created earlier.
helm install alb-istio-ingress ./helm-external/istio-ingress \
--set host=dummy.some-domain.com \
--set certificate_arn=arn:aws:acm:xxxxxx:999999999999:certificate/xxxxxxxxOnce ingress is installed, it will provision an internet-facing AWS Application Load Balancer, bind it with ACM certificate for HTTPS traffic and forward traffic to Istio resources inside EKS cluster. You can get generated manifest of Ingress resource using
kubectl get ingress gw-ingress -n istio-system -o yaml9. Now you might be wondering why we created a wildcard SSL ACM in previous step 6 and in above step 8 we only attach a subdomain, how we can use a single Ingress to expose multiple services?
So to answer that due to templating issues we cannot pass the wildcard domain in the helm chart above, so for fixing that what we do is once the ingress is created we edit the ingress resource to configure annotations for the wildcard domain:
- edit the ingress that we created in the previous step
kubectl edit ingress gw-ingress -n istio-systemgo to spec.rules.host and it will currently be like below:
change it to the wildcard domain:
that is replace dummy.some-domain.cloud to wildcard ‘*.some-domain.cloud’ (don't forget the single inverted comma)
10. Last step is to create a wild card route 53 entry for your internet-facing ALB, so whenever any virtual service with the subdomain DNS is created it will get by default get routed to ALB and from there the Istio Gateway to Virtual service and will serve the traffic.
echo $(kubectl get ingress gw-ingress -n istio-system \
-o jsonpath="{.status.loadBalancer.ingress[*].hostname}")#we should get output similar to this:k8s-istiosys-xxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com
11. The final step to verify that your services are running fine, here in the above example I created and exposed two services publicly one Java service via the domain “java.some-domain.cloud” and nginx home page using “nginx.some-domain.cloud”, lets test both applications if they are accessible.
Java application:
Nginx application:
As you can see both services are deployed and accessible over their respective endpoints.
Istio gateway with internal ALB
In the previous example, we created Istio with a public internet-facing ALB, in various production environments there is a requirement to create internal domains that are not exposed over the internet and accessible within only VPC and your VPN like internal UI dashboard etc for that you need to create an Internal ALB with Istio, so I will show how this can be achieved.
- Follow steps 1–6 in the previous steps where we created the internet-facing ALB.
- To ensure mTLS for internal domains we need to create an AWS SSL ACM for the private domain too, so you would need to create a public domain first with the same name as your private hosted zone domain name, this is clearly mentioned in the last step of the prerequisite at the beginning of this blog.
- Now to configure routes:
- If the Istio Gateway is not created, create it only once, refer the step 7 in the previous step, if it's created don't repeat it.
- The only thing you need to repeatedly create is the virtual service, for example for exposing the internal Java service, you need to create a new virtual service refer.
4. Next step is to install the Ingress, using helm, before running the helm command from this repo we have to make some changes, we have to change the ALB schema to internal and change the ingress resource name to something new so it doesn't contradict with the public ingress we created in the previous step(step 8).
Post these changes we can deploy the helm chart for the Ingress installation.
helm install alb-istio-ingress ./helm-internal/istio-ingress \
--set host=dummy.some-internal-domain.com \
--set certificate_arn=arn:aws:acm:xxxxxx:999999999999:certificate/xxxxxxxxHere as you can see I create a new ingress using the ‘helm-internal’ chart.
5. Like in step 9 in previous, we need to edit ingress annotations to change the host to a wildcard internal domain(*.some-internal-domain.cloud):
kubectl edit ingress gw-ingress-internal -n istio-system 
6. Once ingress is installed, it will provision AWS Application Load Balancer, bind it with ACM certificate for HTTPS traffic and forward traffic to Istio resources inside EKS cluster. You can get generated manifest of internal ALB Ingress resource using:
kubectl get ingress gw-ingress-internal -n istio-system -o yamlOnce the helm finishes it will create an internal application load balancer like below:
7. Follow step 10 in the previous example to create a wildcard route 53 domain entry towards your ALB.
As you can see in the red highlighted box, the DNS is added to the privately hosted zone domain.
8. Finally we need to test if its working, as DNS is added to the domain hosted in the private hosted zone it won't resolve and also as it's attached to a private ALB it won't connect, here I’m testing to access it from my local terminal over the internet and as you can see it can't connect.
So to test it , you need to access the domain from inside any of the EC2 created within the VPC where the EKS is hosted or if it's behind a VPN, you can test it out:
As you can see I call my internal hosted domain java application via a curl request within EC2 and I’m getting a response.
PS: if external service entry is needed then add the “ServiceEntry” as mentioned in this link.
Conclusion
If you check any of the default installations of Istio in EKS it will only let you create a Classic Load Balancer in AWS and also it's not production ready.
Here in this article, I showed how to create a Production ready External internet facing and Internal Application Load Balancer with Istio Gateway for routing the traffic also showed how to expose multiple services using a single ALB ingress attached with wildcard SSL from ACM.