Provisioning Kubernetes clusters on GCP with Terraform and GKE

Updated in January 2023

TL;DR: In this article you will learn how to create clusters on the GCP Google Kubernetes Engine (GKE) with the gcloud CLI and Terraform. By the end of the tutorial, you will automate creating three clusters (dev, staging, prod) complete with the GKE Ingress in a single click.

GKE is a managed Kubernetes service, which means that the Google Cloud Platform (GCP) is fully responsible for managing the cluster's control plane.

In particular, GCP:

• Manages Kubernetes API servers and the etcd database.
• Runs the Kubernetes control-plane single or multiple availability zones.
• Scales the control-plane as you add more nodes to your cluster.
• Provides a mechanism to upgrade your control plane and nodes to a newer version.
• Rotates certificates and keys.

If you're running your cluster, you should still build all of those features.

However, when you use GKE, you outsource the tasks for a price of USD0.10 per hour per cluster.

The Ingress controller is enabled by default, and it's included in the pricing with the cluster. However, deploying a Load Balancer to serve traffic with that Ingress incurs in extra load balancing charges.

Google Cloud Platform has a 3-month free tier promotion that includes USD300 to freely spend on any service when you sign up for a new account.

If you use the free tier, you will not incur in any additional charge.

The rest of the guide assumes that you have an account on the Google Cloud Platform.

If you don't, you can sign up here.

Lastly, this is a hands-on guide — if you prefer to look at the code, you can do so here.

Table of contents

• Three popular options to provision a GKE cluster
• But first, let's set up the GCP account
• gcloud: the quickest way to provision a GKE cluster
• You can provision a GKE cluster with Terraform too
• gcloud CLI vs Terraform — pros, and cons
• Testing the cluster by deploying a simple Hello World app
• Routing traffic into the cluster with an Ingress
• Fully automated Dev, Staging, and Production environments with Terraform modules
• Kubectl kubeconfig tricks
• Summary and next steps

Three popular options to provision a GKE cluster

There are three popular options to run and deploy a GKE cluster:

1. You can create a cluster from the GCP web interface.
2. You can use the gcloud command-line utility.
3. You can define the cluster using code with a tool such as Terraform.

Even if it is listed as the first option, creating a cluster using the GCP interface is discouraged.

There are plenty of configuration options and screens that you have to complete before using the cluster.

When you create the cluster manually, can you be sure that:

• You did not forget to change one of the parameters?
• You can repeat precisely the same steps while creating a cluster for other environments?
• When there is a change, you can apply the same modifications in sequence to all clusters without any mistake?

The process is error-prone and doesn't scale well if you have more than a single cluster.

A better option is defining a file containing all the configuration flags and using it as a blueprint to create the cluster.

And that's precisely what you can do with infrastructure as code tools such as Terraform.

But first, let's set up the GCP account

Before you start using the gcloud CLI and Terraform, you have to install the Google Cloud SDK bundle.

The bundle includes all that are necessary tools to authenticate your requests to your account on.

You can find the official documentation on installing Google Cloud SDK here.

After you install the gcloud CLI you should run:

gcloud --version
Google Cloud SDK 412.0.0
bq 2.0.83
bundled-python3-unix 3.9.12
core 2022.12.09
gsutil 5.17

If you can see the above component output, that means the installation is successful.

Next, you need to link your account to the gcloud CLI, and you can do this with:

gcloud init
Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to [default]

# truncated output
* Run `gcloud topic --help` to learn about advanced features…

This will open a login page where you can authenticate with your credentials.

Once completed, you should see the "You are now authenticated with the Google Cloud SDK!" message.

One more authentication step is necessary to complete the setup:

gcloud auth application-default login
# truncated output
Quota project "k8s-tutorial-185508" was added to ADC which can be used by Google…

The command will automatically open a browser, and you will be once again prompted to log in with your Google credentials.

Before continuing here, you can find all of the available regions that GCP supports here.

Next, you will be prompted to use the default project or create a new one (if you are unsure, create a new project).

Before continuing there are two more steps that need to be done.

Enabling the billing for the project and activating the required APIs.

1. For the billing part, navigate to the web console and follow the steps here.
2. For enabling the API's follow the steps from the official documentation here.

The required API's that need to be enabled are the compute and container ones.

You can enable them with:

gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com

You can now try listing all your GKE clusters with:

gcloud container clusters list

An empty list.

That makes sense since you haven't created any clusters yet.

Provisioning a cluster using the gcloud CLI

Unlike in the case of AWS and their managed Kubernetes offering EKS, in GCP, you don't need an additional tool to provision a cluster.

It is already integrated into the gcloud command utility.

Let's explore the tool with:

gcloud container clusters create --help
NAME
    gcloud container clusters create - create a cluster for running containers

SYNOPSIS
    gcloud container clusters create NAME
        [--accelerator=[type=TYPE,[count=COUNT],...]]
# truncated output

You will get a lot of info and options that you can use to create clusters.

Let's test it by creating a cluster:

gcloud container clusters create learnk8s-cluster --zone europe-west1-b
Creating cluster learnk8s-cluster in europe-west1-b...
Cluster is being deployed...
Cluster is being health-checked...

Be patient on this step since it may take some time for GKE to provision all the resources.

While you are waiting for the cluster to be provisioned, you should download kubectl — the command-line tool to connect and manage the Kubernetes cluster.

Kubectl can be downloaded from here.

You can check that the binary is installed successfully with:

kubectl version --client
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4",...}

Back to the cluster.

As soon as gcloud container clusters create returns, you will find a kubeconfig in the current directory.

You can use it to connect to the new cluster.

Note how the cluster was created with the following default values:

• Cluster type: zonal
• Region: europe-west-1
• Zone: B
• Node count: 3 nodes, single-zone
• Autoscaling: off
• Instance type: e2-medium(CPU: 2; RAM: 4GB)
• Disk per node: 100GB

You can always choose different settings if the above isn't what you had in mind.

You can inspect the cluster's nodes with:

kubectl get nodes
# output truncated
NAME                                              STATUS   ROLES    AGE   VERSION
gke-learnk8s-cluster-default-pool-c0655b99-54br   Ready    <none>   50m   v1.24.7-gke.900
gke-learnk8s-cluster-default-pool-c0655b99-78jv   Ready    <none>   50m   v1.24.7-gke.900
gke-learnk8s-cluster-default-pool-c0655b99-q9hh   Ready    <none>   50m   v1.24.7-gke.900

And you can change settings for your cluster with the gcloud container clusters update command.

For example, you can enable the auto-scaling from 3 to 5 nodes with:

gcloud container clusters update learnk8s-cluster \
  --enable-autoscaling \
  --node-pool default-pool \
  --zone europe-west1-b \
  --project learnk8s-tutorial \
  --min-nodes 3 \
  --max-nodes 5

Updating learnk8s-cluster...

Be patient and wait for the update to finish.

Please note that --min-nodes and --max-nodes refer to the minimum and maximum node count per zone. If your cluster resides in two zones, the total number of nodes will double and range from 6 to 10.

Now, if you are interested in the details, you can further inspect the cluster with:

gcloud container clusters describe learnk8s-cluster --zone europe-west1-b
addonsConfig:
  kubernetesDashboard:
    disabled: true
  networkPolicyConfig:
    disabled: true
clusterIpv4Cidr: 10.0.0.0/14
# truncated output

Excellent! You've successfully created a cluster and even modified it to enable autoscaling.

You can delete the cluster now, as you will learn another way to deploy and manage it.

gcloud container clusters delete learnk8s-cluster --zone europe-west1-b
Deleting cluster learnk8s-cluster...done.

Going further, provisioning a cluster using Terraform

Terraform is an open-source Infrastructure as a Code tool.

Instead of writing the code to create the infrastructure, you define a plan of what you want to be executed, and you let Terraform create the resources on your behalf.

The plan isn't written in YAML though.

Instead Terraform uses a language called HCL - HashiCorp Configuration Language.

In other words, you use HCL to declare the infrastructure you want to be deployed, and Terraform executes the instructions.

Terraform uses plugins called providers to interface with the resources in the cloud provider.

This further expands with modules as a group of resources and are the building blocks you will use to create a cluster.

But let's take a break from the theory and see those concepts in practice.

Before you can create a cluster with Terraform, you should install the binary.

You can find the instructions on how to install the Terraform CLI from the official documentation.

Verify that the Terraform tool has been installed correctly with:

terraform version
Terraform v1.3.6

Let's start writing some code.

Terraform step by step guide

Create a new folder with the following files:

• variables.tf - to define the parameters for the cluster.
• main.tf - to store the actual code for the cluster.
• outputs.tf - to define the outputs.

In the main.tf file, copy and paste the following code:

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "4.47.0"
    }
  }
}

module "gke_auth" {
  source = "terraform-google-modules/kubernetes-engine/google//modules/auth"
  version = "24.1.0"
  depends_on   = [module.gke]
  project_id   = var.project_id
  location     = module.gke.location
  cluster_name = module.gke.name
}

resource "local_file" "kubeconfig" {
  content  = module.gke_auth.kubeconfig_raw
  filename = "kubeconfig-${var.env_name}"
}

module "gcp-network" {
  source       = "terraform-google-modules/network/google"
  version      = "6.0.0"
  project_id   = var.project_id
  network_name = "${var.network}-${var.env_name}"

  subnets = [
    {
      subnet_name   = "${var.subnetwork}-${var.env_name}"
      subnet_ip     = "10.10.0.0/16"
      subnet_region = var.region
    },
  ]

  secondary_ranges = {
    "${var.subnetwork}-${var.env_name}" = [
      {
        range_name    = var.ip_range_pods_name
        ip_cidr_range = "10.20.0.0/16"
      },
      {
        range_name    = var.ip_range_services_name
        ip_cidr_range = "10.30.0.0/16"
      },
    ]
  }
}

data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = "https://${module.gke.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(module.gke.ca_certificate)
}

module "gke" {
  source                 = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
  version                = "24.1.0"
  project_id             = var.project_id
  name                   = "${var.cluster_name}-${var.env_name}"
  regional               = true
  region                 = var.region
  network                = module.gcp-network.network_name
  subnetwork             = module.gcp-network.subnets_names[0]
  ip_range_pods          = var.ip_range_pods_name
  ip_range_services      = var.ip_range_services_name
  
  node_pools = [
    {
      name                      = "node-pool"
      machine_type              = "e2-medium"
      node_locations            = "europe-west1-b,europe-west1-c,europe-west1-d"
      min_count                 = 1
      max_count                 = 2
      disk_size_gb              = 30
    },
  ]
}

Create the variables.tf with the following content:

variable "project_id" {
  description = "The project ID to host the cluster in"
}
variable "cluster_name" {
  description = "The name for the GKE cluster"
  default     = "learnk8s-cluster"
}
variable "env_name" {
  description = "The environment for the GKE cluster"
  default     = "prod"
}
variable "region" {
  description = "The region to host the cluster in"
  default     = "europe-west1"
}
variable "network" {
  description = "The VPC network created to host the cluster in"
  default     = "gke-network"
}
variable "subnetwork" {
  description = "The subnetwork created to host the cluster in"
  default     = "gke-subnet"
}
variable "ip_range_pods_name" {
  description = "The secondary ip range to use for pods"
  default     = "ip-range-pods"
}
variable "ip_range_services_name" {
  description = "The secondary ip range to use for services"
  default     = "ip-range-services"
}

As you can see, all the code blocks get repeated here.

You just define which variable to hold what value.

Finally, create the outputs.tf file to define which data from the cluster you want to inspect after the cluster is created.

output "cluster_name" {
  description = "Cluster name"
  value       = module.gke.name
}

That's a lot of code!

But don't worry, I will explain everything as soon as you create the cluster.

Continue and in the same folder run:

terraform init

The command will initialize Terraform and create two more folders as well as a state file.

The state file is used to keep track of the resources that have been created already.

Consider this as a checkpoint, without it Terraform won't know what has been already created or updated.

There is another command that you can utilise in your undertaking with Terraform.

To quickly check if the configuration doesn't have any configuration errors you can do so with:

terraform validate
Success! The configuration is valid.

Next, you should run:

terraform plan
# output truncated
Plan: 12 to add, 0 to change, 0 to destroy.

Terraform will perform a dry-run and will prompt you with a detailed summary of what resources are about to create.

If you feel confident that everything looks fine, you can create the cluster with:

terraform apply
# output truncated
Apply complete! Resources: 12 added, 0 changed, 0 destroyed.
Outputs:
cluster_name = "learnk8s-cluster-prod"

After issuing the apply command, you will be prompted to enter the project-id from the GCP project you created earlier — i.e. where you want the cluster to be created and located.

Next, you will be prompted one more time to confirm your actions so just type yes.

If you do not like to get prompted, you can add the project ID to the project_id variable:

variable "project_id" {
  default     = "YOUR-PROJECT-ID"
  description = "The project ID to host the cluster in"
}

As mentioned before, creating or modifying clusters takes time.

So be patient here as the process can take up to 20 minutes for everything to be up and running.

When it's complete, if you inspect the current folder, you should notice a few new files:

tree .
.
├── kubeconfig-prod
├── main.tf
├── output.tf
├── terraform.tfstate
└── variables.tf

Terraform uses the terraform.tfstate to keep track of what resources were created.

The kubeconfig-prod is the kubeconfig for the newly created cluster.

Inspect the cluster pods using the generated kubeconfig file:

kubectl get pods --all-namespaces --kubeconfig=kubeconfig-prod
NAMESPACE     NAME                       READY   STATUS    AGE
kube-system   coredns-56666f95ff-l5ppm   1/1     Running   23m
kube-system   coredns-56666f95ff-w9brq   1/1     Running   23m
kube-system   kube-proxy-98vcw           1/1     Running   10m
# truncated output

If you prefer to not prefix the KUBECONFIG environment variable to every command, you can export it with:

export KUBECONFIG="${PWD}/kubeconfig-prod"

The export is valid only for the current terminal session.

Now that you've created the cluster, it's time to go back and discuss the Terraform file.

The Terraform file that you just executed is divided into several blocks, so let's look at each one of them.

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "4.47.0"
    }
  }
}

# ...

The first one is the provider block, where you define that you will use the Google provider for GCP.

module "gke_auth" {
  source = "terraform-google-modules/kubernetes-engine/google//modules/auth"
  version = "24.1.0"
  depends_on   = [module.gke]
  project_id   = var.project_id
  location     = module.gke.location
  cluster_name = module.gke.name
}

# ...

The next one is a module.

A module is a collection of resources that you can use over and over.

As the module's name might give away, in this part you configure the authentication and authorisation of the cluster, as well as how to fetch the details for the kubeconfig file.

resource "local_file" "kubeconfig" {
  content  = module.gke_auth.kubeconfig_raw
  filename = "kubeconfig-${var.env_name}"
}

# ...

In the above resource block, you define a local file that will store the necessary info such as certificate, user and endpoint to access the cluster.

This is your kubeconfig file for the cluster.

module "gcp-network" {
  source       = "terraform-google-modules/network/google"
  version      = "6.0.0"
  project_id   = var.project_id
  network_name = "${var.network}-${var.env_name}"
  # ...
}

The gcp-network is another module.

The GCP networking module creates a separate VPC dedicated to the cluster.

It also sets up separate subnet ranges for the pods and services.

module "gke" {
  source                 = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
  version                = "24.1.0"
  project_id             = var.project_id
  name                   = "${var.cluster_name}-${var.env_name}"
  # ...
}

Finally comes the GKE module, where you define the cluster and node pool specifications such as the name, regions, storage, instance type, images, etc.

This is where all the magic happens!

Next up are the variables, where you define the values that you can customise.

In this example, the main.tf is pulling the values referenced in the variables.tf file.

The main.tf has the gke module with the following inputs:

module "gke" {
  source                 = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
  project_id             = var.project_id
  name                   = "${var.cluster_name}-${var.env_name}"
  regional               = true
  region                 = var.region
  network                = module.gcp-network.network_name
  subnetwork             = module.gcp-network.subnets_names[0]
  ip_range_pods          = var.ip_range_pods_name
  ip_range_services      = var.ip_range_services_name
  node_pools = #...
}

The variables are noted as var.<variable-name>.

In the above example, you want to pass the value from the variable named region (on the right) to the input called region (on the left).

Through the variable defined as:

variable "region" {
  description = "The region to host the cluster in"
  default     = "europe-west1"
}

The gcloud utility vs Terraform — pros and cons

You can already tell the main differences between the gcloud utility and Terraform:

• Both create a GKE cluster.
• Both export a valid kubeconfig file.
• The configuration with the gcloud CLI is more straightforward and more concise.
• Terraform goes to great detail and is more granular. You have to craft every single resource carefully.

So which one should you use?

For smaller experiments, when you need to spin a cluster quickly you should consider using the gcloud CLI.

With a short command, you can easily create it.

For production-grade infrastructure where you want to configure and tune every single detail of your cluster, you should consider using Terraform.

But there's another crucial reason why you should prefer Terraform: incremental updates.

Let's imagine that you want to add a second pool of nodes to your cluster.

Perhaps you want to add another - CPU optimized node pool to your cluster for your compute hungry applications.

You can edit the file, and the new node pool as follows:

module "gke" {
  # ...
  ip_range_services      = var.ip_range_services_name
  node_pools = [
    {
      name                      = "node-pool"
      machine_type              = "e2-medium"
      node_locations            = "europe-west1-b,europe-west1-c,europe-west1-d"
      min_count                 = 1
      max_count                 = 2
      disk_size_gb              = 30
    },
    {
      name                      = "high-cpu-pool"
      machine_type              = "n1-highcpu-4"
      node_locations            = "europe-west1-b"
      min_count                 = 1
      max_count                 = 1
      disk_size_gb              = 100
   }
  ]
}

You can dry run the changes with:

terraform plan
# output truncated
Plan: 2 to add, 0 to change, 2 to destroy.

Once you're ready, you can apply the changes with:

terraform apply
# output truncated
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

You can verify now that the cluster has two node pools with:

kubectl get nodes --kubeconfig=kubeconfig-prod

NAME                                               
gke-learnk8s-cluster-pr-high-cpu-pool-5e00f2be-89ds
gke-learnk8s-cluster-prod-node-pool-8ab7b428-8p4g  
gke-learnk8s-cluster-prod-node-pool-bba42cdd-brv0  
gke-learnk8s-cluster-prod-node-pool-e7bf9d7e-fzq9  

Excellent, it works!

You've gone to great lengths to configure a production-ready cluster, but you haven't deployed any application yet.

Let's do that next.

Testing the cluster by deploying a simple Hello World app

You can create a Deployment with the following YAML definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-kubernetes
spec:
  selector:
    matchLabels:
      name: hello-kubernetes
  template:
    metadata:
      labels:
        name: hello-kubernetes
    spec:
      containers:
      - name: app
        image: paulbouwer/hello-kubernetes:1.10.1
        ports:
          - containerPort: 8080
        env:
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

You can also find all the files for the demo app here.

You can submit the definition to the cluster with:

kubectl apply -f deployment.yaml

To see if your application runs correctly, you can connect to it with kubectl port-forward.

First, retrieve the name of the Pod:

kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
hello-kubernetes-79b94d97c7-rzkhm   1/1     Running   0          13s

You can connect to the Pod with:

kubectl port-forward hello-kubernetes-79b94d97c7-rzkhm 8080:8080

Or you can use the tricker option that will automatically get the pod's name:

kubectl port-forward $(kubectl get pod -l name=hello-kubernetes --no-headers | awk '{print $1}') 8080:8080

The kubectl port-forward command connects to the Pod with the name hello-kubernetes-rzkhm.

And forwards all the traffic from port 8080 on the Pod to port 8080 on your computer.

If you visit http://localhost:8080 on your computer, you should be greeted by the applications web page.

Great!

Exposing the application with kubectl port-forward is an excellent way to test the app quickly, but it isn't a long term solution.

If you wish to route live traffic to the Pod, you should have a more permanent solution.

In Kubernetes, you can use a Service of type: LoadBalancer to expose your Pods.

You can use the following code:

apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080
  selector:
    name: hello-kubernetes

And submit the YAML with:

kubectl apply -f service-loadbalancer.yaml

As soon as you submit the command, GKE provisions an L4 Load Balancer and connects it to your pod.

It might take a while for the load balancer to be provisioned.

Eventually, you should be able to describe the Service and retrieve the load balancer's IP address.

Running:

kubectl get services
NAME               TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)
hello-kubernetes   LoadBalancer   10.30.34.77   35.195.53.227   80:32151/TCP

If you visit the external IP address in your browser, you should see the application.

Excellent!

There's only an issue, though.

The load balancer that you created earlier serves one service at a time.

Also, it has no option to provide intelligent routing based on paths.

So if you have multiple services that need to be exposed, you will need to create the same load balancers.

Imagine having ten applications that have to be exposed.

If you use a Service of type: LoadBalancer for each of them, you might end up with ten different L4 Load Balancers.

That wouldn't be a problem if those load balancers weren't so expensive.

How can you get around this issue?

Routing traffic into the cluster with an Ingress

In Kubernetes, there's another resource that is designed to solve that problem: the Ingress.

The Ingress has two parts:

1. The first is the Ingress object which is the same as Deployment or Service in Kubernetes. This is defined by the kind part of the YAML manifest.
2. The second part is the Ingress controller. This is the actual part that controls the load balancers, so they know how to serve the requests and forward the data to the Pods.

In other words, the Ingress controller acts as a reverse proxy that routes the traffic to your Pods.

The Ingress routes the traffic based on paths, domains, headers, etc., which consolidates multiple endpoints in a single resource that runs inside Kubernetes.

With this, you can serve multiple services simultaneously from one exposed endpoint - the load balancer.

There're several Ingress controllers that you can use:

1. GKE Ingress
2. Nginx Ingress
3. Ambassador
4. Traefik
5. And more.

In this part, you will use the first one from the list, the GKE Ingress controller.

A convenient feature from the Google Cloud Platform is that when you deploy a GKE cluster, the Ingress controller is automatically deployed.

So you won't have to install it manually.

Also, when you create an Ingress object, the GKE Ingress controller creates a Google Cloud HTTP(S) Load Balancer and configures it according to the information in the Ingress and its associated Services.

Going from there, your only responsibility is to define the Ingress manifest and define the rules and paths on how to reach your applications.

In other words, when you create an Ingress manifest in Kubernetes, the GKE Ingress controller converts the request into something that the Google Cloud Load Balancer understands (Forwarding Rules, Target Pools, URL maps etc.)

Let's have a look at an example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-kubernetes
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-kubernetes
            port:
              number: 80

The following Kubernetes Ingress manifest routes all the traffic from path /* to the Pods targeted by the hello-kubernetes Service.

As soon as you submit the resource to the cluster, the GKE Ingress controller is notified of the new resource.

And it creates:

• A backend service for each Kubernetes instance.
• A URL map for the /* path in the ingress resource. This ensures traffic to a specific path is routed to the correct Kubernetes Service.
• A forwarding rule defining the IP address and ports on the Load Balancer.
• And a target proxy that routes the requests to the URL maps.

You can verify the creation of above resources through the UI or by checking the ingress resource's annotations.

As with every Ingress controller, the GKE Ingress controller provides convenience since you can control your infrastructure uniquely from Kubernetes — there's no need to fiddle with GCP anymore.

Now that you know about the Ingress, you can give it a try and deploy it.

Create the Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-kubernetes
  annotations:
    cloud.google.com/load-balancer-type: "External"
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-kubernetes
            port:
              number: 80

Pay attention to the following annotation fields:

• kubernetes.io/ingress.class: "gce" is used to select the right Ingress controller in the cluster.
• cloud.google.com/load-balancer-type: "External", specifies the public-facing load balancer (the other option is to use the "Internal" type.)

You can submit the Ingress manifest to your cluster with:

kubectl apply -f ingress.yaml

It may take a few minutes (the first time) for the Ingress to be created, but eventually, you will see the following output from kubectl describe ingress:

kubectl describe ingress hello-kubernetes
Name:             hello-kubernetes
Namespace:        default
Address:          35.241.59.69
Default backend:  default-http-backend:80 (10.20.2.4:8080)
Rules:
  Host     Path  Backends
  ----        ----  --------
  *               /*   hello-kubernetes:80 (10.20.0.5:8080)

And if you look at the annotations section, you'll see the aforementioned resources (backend service, url map, etc.):

Annotations:  ingress.kubernetes.io/backends: {"k8s-be-30982":"HEALTHY","k8s-be-31404":"HEALTHY"}
ingress.kubernetes.io/forwarding-rule: k8s2-fr-okejq4pe-default-hello-kubernetes-ibq0nteg
ingress.kubernetes.io/target-proxy: k8s2-tp-okejq4pe-default-hello-kubernetes-ibq0nteg
ingress.kubernetes.io/url-map: k8s2-um-okejq4pe-default-hello-kubernetes-ibq0nteg

Excellent, you can use the IP from the Address field to visit your application in the browser.

Congratulations!

You've managed to deploy a fully working cluster that can route live traffic!

Please note that the current configuration only works if your Services are of type NodePort or LoadBalancer.

We already discussed why type: LoadBalancer might not be a good idea, so you should use only NodePort services with the GKE Ingress.

But there's also another option: container-native load balancing.

Container-native load balancing

If you use the GKE controller without any modification, the traffic routed by the load balancer reaches one of the nodes on their NodePort ports.

At that point, kube-proxy routes the traffic to the correct Pod.

While this is convenient and straightforward to use, it has a few drawbacks:

1. Every Service has to be exported as NodePort. However, NodePort ports have limited port ranges from port 30000 to 32767.
2. There's an extra hop. All the traffic must be routed twice: once with the load balancer, the other with the kube-proxy.

To work around those limitations, the GKE cluster and the ingress controller can be configured to route traffic directly to the Pods using container-native load balancing.

When you enable container-native load balancing, each Service in the cluster can be decorated with an extra annotation:

apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    name: hello-kubernetes

GKE uses the annotation to create a Network Endpoint Group (NEG) — a collection of IP addresses that the load balancer can use as backend.

From this point onwards, all the traffic routed from the Ingress does not reach the Node, but the Pod directly.

Why?

The Ingress uses the Network Endpoint Groups to route the traffic directly to the Pods.

Container-native load balancing gives you better performance, but you need to make sure that:

1. You created the cluster in a VPC-native cluster.
2. The VPC is not shared.

You can find more about the limitations here.

Thankfully, the Terraform code that you used so far has all of the above already implemented!

So you can tag any Service with the right annotation and befit from lower latencies.

Fully automated Dev, Staging, Production environments with Terraform modules

One of the most common tasks when provisioning infrastructure is to create separate environments.

It's a popular practice to provision three environments:

1. A development environment where you can test your changes and integrate them with other colleagues.
2. A staging environment used to sign-off requirements.
3. A production environment.

Since you want your apps to progress through the environments, you might want to provision not one, but three clusters, once for each environment.

When you don't have infrastructure as code, you are forced to click on the user interface and repeat the same choice.

But since you've mastered Terraform, you can refactor your code and create three (or more) environments with a single command!

The beauty of Terraform is that you can use the same code to generate several clusters with different names.

You can parametrise the name of your resources and create clusters that are exact copies.

You can reuse the existing Terraform code and provision three clusters simultaneously using Terraform modules and expressions.

Before you execute the script, it's a good idea to destroy any cluster that you created previously with terraform destroy.

You explored Terraform variables earlier, but let's revisit them.

The expression syntax is straightforward.

You define a variables like this:

variable "cluster_name" {
  description = "The name for the GKE cluster"
  default     = "learnk8s-cluster"
}
variable "env_name" {
  description = "The environment for the GKE cluster"
  default     = "dev"
}

You can append the definition in the variables.tf file.

Later, you can reference and chain the variables in the GKE module like this:

module "gke" {
  # output truncated for clarity
  name                   = "${var.cluster_name}-${var.env_name}"
}

Terraform will interpolate the string to "learnk8s-cluster-dev".

When you execute the usual terraform apply command, you can override the variable with a different name.

For example:

terraform apply -var="cluster_name=my-cluster" -var="env_name=staging"

This will provision a cluster with the name of my-cluster-staging.

In isolation, expressions are not particularly useful.

Let's have a look at an example.

If you execute the following commands, what do you expect?

Is Terraform creating two clusters or updating the dev cluster to a staging cluster?

terraform apply -var="env_name=prod"
# and later
terraform apply -var="env_name=staging"

The code updates the dev cluster to a staging cluster.

It's overwriting the same cluster!

But what if you wish to create a copy?

You can use the Terraform modules to your advantage.

Terraform modules use variables and expressions to encapsulate resources.

Move your main.tf, variables.tf, and output.tf in a subfolder, such as another folder named main and create an empty main.tf.

mkdir -p main
mv main.tf variables.tf output.tf cluster
tree .
.
├── main
│   ├── variables.tf
│   └── main.tf
│   └── output.tf
└── main.tf

From now on, you can use the code that you've created as a reusable module.

In the root main.tf add the project_id variable:

variable "project_id" {
  description = "The project ID to host the cluster in"
}

Note: since it's only one variable you can add it in the main.tf, but for more clarity, it's a good practice to isolate them in a separate variables.tf file.

And you can reference to that module like this:

module "prod_cluster" {
    source     = "./main"
    env_name   = "prod"
    project_id = "${var.project_id}"
}

And since the module is reusable, you can create more than a single cluster:

module "dev_cluster" {
    source     = "./main"
    env_name   = "dev"
    project_id = "${var.project_id}"
}

module "staging_cluster" {
    source     = "./main"
    env_name   = "staging"
    project_id = "${var.project_id}"
}

module "prod_cluster" {
    source     = "./main"
    env_name   = "prod"
    project_id = "${var.project_id}"
}

You can preview the changes with:

terraform plan
# output truncated
Plan: 36 to add, 0 to change, 0 to destroy.

Warning! Before applying you should be aware that there are quota limits for the free tier account on GCP. The most critical is tied to IP address quota. To circumvent that for this tutorial purposes the Terraform code for running multiple clusters is changed to include only a single node zone instead of the default - three.

You can apply the changes and create three clusters that are exact copies with:

terraform apply
# output truncated
Applied: 36 to added, 0 to change, 0 to destroy.

All three clusters have the GKE Ingress Controller installed automatically, so they are ready to handle production traffic.

What happens when you update the cluster module?

When you modify a property, Terraform will update all clusters with the same property.

If you wish to customize the properties on a per environment basis, you should extract the parameters in variables and change them from the root main.tf.

Let's have a look at an example.

You might want to run the same instance type such as e2-medium in the dev and staging environments but change to n2-highmem-2 instance type for production.

As an example you an refactor the code and extract the instance type as a variable:

variable "instance_type" {
  default = "e2-medium"
}

And add the corresponding change in the GKE module like:

#...
  node_pools = [
    {
      name                      = "node-pool"
      machine_type              = var.instance_type
# ...

Later, you can modify the root main.tf file with the instance type:

module "dev_cluster" {
    source     = "./main"
    env_name   = "dev"
    project_id = "${var.project_id}"
    instance_type = "e2-medium"
}

module "staging_cluster" {
    source     = "./main"
    env_name   = "staging"
    project_id = "${var.project_id}"
    instance_type = "e2-medium"
}

module "prod_cluster" {
    source     = "./main"
    env_name   = "prod"
    project_id = "${var.project_id}"
    instance_type = "n2-highmem-2"
}

variable "project_id" {
  description = "The project ID to host the cluster in"
}

If you wish you can apply the changes, and verify each cluster with its corresponding kubeconfig file.

Excellent!

As you can imagine, you can add more variables in your module and create environments with different configurations and specifications.

This marks the end of your journey!

A recap on what you've built so far:

1. So you've created a GKE cluster with Terraform.
2. You used the GKE Ingress controller and resource to route live traffic.
3. You parameterized the cluster and created a reusable module.
4. You used the module as an extension to provision multiple copies of your cluster (one for each environment: dev, staging and production).
5. You made the module more flexible by allowing small customizations such as changing the instance type.

Well done!

Kubectl kubeconfig tips and tricks

When you have several clusters, you might need to switch the kubeconfig files to access the right cluster and issue commands.

Unless you write your kubeconfig file in ~/.kube/kubeconfig, you can export the location of your kubeconfig file as an environment or pass it on with every command.

So you can either do:

export KUBECONFIG="${PWD}/kubeconfig-prod"

And then issue commands for the duration of that session or do it the other way with:

kubectl get pods --kube-config=kubeconfig-prod

A better and permanent solution is to write and merge the generated kubeconfig file(s) to be permanently accessible.

First list all the contents where the kubeconfig files are located with the tree command:

$ tree .
.
├── kubeconfig-dev
├── kubeconfig-prod
├── kubeconfig-staging
├── main
# [other files truncated from output]

Note: If you have other clusters, it's a good idea to make a backup of your current kubeconfig file.

Now you can chain them to the KUBECONFIG environment and save them to the .kubeconfig location:

KUBECONFIG=kubeconfig-dev:kubeconfig-staging:kubeconfig-prod kubectl config view --merge --flatten > $HOME/.kube/config

This command will consolidate all kubeconfig files and merge them into one.

With this, you can now switch contexts easily and issue commands to the clusters.

kubectl get nodes --context learnk8s-cluster-dev
NAME                                               STATUS   ROLES    AGE   VERSION
gke-learnk8s-cluster-dev-node-pool-42794894-q4kc   Ready    <none>   15m   v1.24.7-gke.900

kubectl get nodes --context learnk8s-cluster-prod
NAME                                                STATUS   ROLES    AGE   VERSION
gke-learnk8s-cluster-prod-node-pool-810c3002-mkr7   Ready    <none>   17m   v1.24.7-gke.900

You can now use a tool like kubectx to make the context switching even more comfortable.

Summary and next steps

Having the infrastructure defined as code makes your job easier.

If you wish to change the cluster version, you can do it in a centralized manner and have it applied to all clusters.

The setup described above is only the beginning, if you're provisioning production-grade infrastructure you should look into:

1. How to structure your Terraform in global and environment-specific layers.
2. Managing Terraform state and how to work with the rest of your team.
3. How to use External DNS to link your Route53 to your ALB Ingress controller.
4. How to set up TLS with cert-manager.
5. How to set up a private Elastic Container Registry (ECR).

And the beauty is that External DNS and Cert Manager are available as charts, so you could integrate them with your Helm provider and have all the clusters updated at the same time.