Kristijan Mitevski
Kristijan Mitevski

Provisioning Kubernetes clusters on Linode with Terraform

Published in April 2021


Provisioning Kubernetes clusters on Linode with Terraform

TL;DR: In this article, you will learn how to create Kubernetes clusters on Linode Kubernetes Engine (LKE) with the Linode CLI and Terraform. By the end of the tutorial, you will automate creating three clusters (dev, staging, and prod) (complete with an Ingress controller ready to serve live traffic).

Linode offers a managed Kubernetes service where you can request a cluster, connect to it, and use it to deploy applications.

Linode Kubernetes Engine (LKE) is a managed Kubernetes service, which means that the Linode platform is fully responsible for managing the cluster control plane.

In particular, LKE will:

When you use LKE, you outsource the management of the control plane to Linode at no cost.

You read that right.

Your clusters are subject to no management fees.

You only pay for what you use by the worker nodes — Linodes.

Linode offers a signup promotion that includes USD100 credit to freely spend on any service in the next 60 days after your registration.

If you use the promotion, you will not incur any additional charges when following this tutorial.

The rest of the guide assumes that you have an account on Linode.

And if you prefer to look at the code, you can do so here.

Table of contents

Linode offers four options to run and deploy an LKE cluster:

  1. You can create a cluster via the web-based LKE cloud manager.
  2. You can use the Linode API to create a cluster programmatically.
  3. You can use the LKE command-line utility.
  4. And finally, you can define the cluster using code with a tool such as Terraform.

Even if it is on the list as the first option, creating a cluster through the Linode portal is discouraged.

There are plenty of configuration options and screens that you have to complete before using the cluster.

When you create the cluster manually, can you be sure that:

The process through the user interface is error-prone and doesn't scale well if you have more than a single cluster.

A better option is defining a file containing all the configuration flags and using it as a blueprint to create the cluster.

And that is what you can do with the Linode CLI and infrastructure as code tools such as Terraform.

Setting up the Linode CLI

Before you start creating clusters, it's a good idea to install the Linode CLI.

You can find the official documentation on installing the Linode CLI here.

After you complete the installation, typing any command will prompt you for an initial setup:

bash

linode-cli show-users
Welcome to the Linode CLI. This will walk you through some initial setup.

After pressing enter, the Linode webpage will prompt you to log in and authenticate.

Once authenticated, you can return to the terminal to finish the rest of the setup.

To verify the setup, you can list the available regions with:

bash

linode-cli regions list
┌──────────────┬─────────┬────────────────────────────────────────────────────────────────────────────────┬────────┐
│ id           │ country │ capabilities                                                                   │ status │
├──────────────┼─────────┼────────────────────────────────────────────────────────────────────────────────┼────────┤
│ ap-west      │ in      │ Linodes, NodeBalancers, Block Storage, GPU Linodes, Kubernetes, Cloud Firewall │ ok     │
│ ca-central   │ ca      │ Linodes, NodeBalancers, Block Storage, Kubernetes, Cloud Firewall              │ ok     │
│ ap-southeast │ au      │ Linodes, NodeBalancers, Block Storage, Kubernetes, Cloud Firewall              │ ok     │
# output truncated

Great work!

You've set up the Linode CLI and can now proceed to create an LKE cluster.

The quickest way to provision an LKE cluster

You can use the Linode CLI to create a Kubernetes cluster.

Let's explore the command:

bash

linode-cli lke cluster-create --help

linode-cli lke cluster-create
Kubernetes Cluster Create

Arguments:
  --label: (required) This Kubernetes cluster's unique label for display purposes only.
  --region: (required) This Kubernetes cluster's location.
  --k8s_version: (required) The desired Kubernetes version for this Kubernetes cluster in the format of <major>.
  --tags: An array of tags applied to the Kubernetes cluster.
  --node_pools.type: A Linode Type for all of the nodes in the Node Pool.
  --node_pools.count: The number of nodes in the Node Pool.
  --node_pools.disks: **Note**: This field should be omitted except for special use cases.

There are three required parameters:

The rest of the arguments are necessary to specify which type of nodes you wish to run.

If you want to check which Kubernetes versions are available on Linode, you can do so with:

bash

linode-cli lke versions-list
┌──────┐
│ id   │
├──────┤
│ 1.20 │
│ 1.19 │
│ 1.18 │
│ 1.17 │
└──────┘

And, to check which node types are available, the command is:

bash

linode-cli linodes types
┌──────────────────┬──────────────┬─────────┬────────┬───────┬────────┬─────────┐
│ id               │ label        │ disk    │ memory │ vcpus │ hourly │ monthly │
├──────────────────┼──────────────┼─────────┼────────┼───────┼────────┼─────────┤
│ g6-nanode-1      │ Nanode 1GB   │ 25600102410.00755.0     │
│ g6-standard-1    │ Linode 2GB   │ 51200204810.01510.0    │
│ g6-standard-2    │ Linode 4GB   │ 81920409620.0320.0    │
│ g6-standard-4    │ Linode 8GB   │ 163840819240.0640.0    │
│ g6-standard-6    │ Linode 16GB  │ 3276801638460.1280.0# output truncated

Excellent!

You now have all the information to create an LKE cluster.

If you are not sure what instance type you should use for the node pool, you should use three nodes of type g6-standard-2.

Finally, The command to create the cluster is:

bash

linode-cli lke cluster-create \
  --label learnk8s \
  --region eu-west \
  --k8s_version 1.20 \
  --node_pools.count 3 \
  --node_pools.type g6-standard-2

Be patient; the cluster could take a few minutes to be created.

While you are waiting for the cluster to be provisioned, you should go ahead and download kubectl — the command-line tool to connect and manage the Kubernetes cluster.

Kubectl can be downloaded from here.

You can check that the binary is installed successfully with:

bash

kubectl version --client
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5"}

Once the cluster is created, you will get a table output with its ID.

bash

┌───────┬──────────┬─────────┐
│ id    │ label    │ region  │
├───────┼──────────┼─────────┤
│ 22754 │ learnk8s │ eu-west │
└───────┴──────────┴─────────┘

It's a good idea to append the ID to a variable for easier use.

bash

export CLUSTER_ID=22754

To connect to the LKE cluster, you will first need to fetch your credentials.

The Linode CLI provides a way to retrieve the kubeconfig, but since the file is base64 encoded, you will have to decode it.

First, retrieve the kubeconfig with:

bash

linode-cli lke kubeconfig-view $CLUSTER_ID --text

Next, you will need to remove the first line and then decode it.

Piping the above output through tail and base64 -d will get you the result you need.

bash

linode-cli lke kubeconfig-view $CLUSTER_ID --text | tail +2 | base64 -d

apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
    certificate-authority-data: LS...
# truncated output

Save the output as a file name kubeconfig.

You can use that file with the --kubeconfig argument or export it as the KUBECONFIG environment variable.

For the time being, you can use for the current terminal session with:

bash

export KUBECONFIG=<path to your kubeconfig>

You can test the connection to the cluster with:

bash

kubectl get nodes
NAME                          STATUS   ROLES    AGE   VERSION
lke22754-28916-605f77024efc   Ready    <none>   20m   v1.20.5
lke22754-28916-605f7702a3be   Ready    <none>   20m   v1.20.5
lke22754-28916-605f7702f85c   Ready    <none>   20m   v1.20.5

You can also use the Linode CLI to change the cluster after it was created.

For example, if you wish to resize the node pool from three to six worker nodes, you can do so with:

bash

linode-cli lke pool-update --help

linode-cli lke pool-update [CLUSTERID] [POOLID]
Node Pool Update

Arguments:
  --count: The number of nodes in the Node Pool.

You can retrieve the current pool id with:

bash

linode-cli lke pools-list $CLUSTER_ID
┌───────┬───────────────┬────────────────────┬─────────────┬────────┐
│ idtypeid                 │ instance_id │ status │
├───────┼───────────────┼────────────────────┼─────────────┼────────┤
│ 28916 │ g6-standard-2 │ 28916-605f77024efc │ 25548071    │ ready  │
│ 28916 │ g6-standard-2 │ 28916-605f7702a3be │ 25548075    │ ready  │
│ 28916 │ g6-standard-2 │ 28916-605f7702f85c │ 25548076    │ ready  │
└───────┴───────────────┴────────────────────┴─────────────┴────────┘

The final command is:

bash

linode-cli lke pool-update $CLUSTER_ID 28916 --count 6

The output may be a bit gibberish; that's because the CLI will try and output it in table format.

Although Linode is super quick on node scheduling, be patient and wait for the additional nodes to be added.

Try and list the nodes again:

bash

kubectl get nodes
NAME                          STATUS   ROLES    AGE    VERSION
lke22754-28916-605f77024efc   Ready    <none>   71m    v1.20.5
lke22754-28916-605f7702a3be   Ready    <none>   70m    v1.20.5
lke22754-28916-605f7702f85c   Ready    <none>   71m    v1.20.5
lke22754-28916-605f8756c236   Ready    <none>   109s   v1.20.5
lke22754-28916-605f87571a94   Ready    <none>   109s   v1.20.5
lke22754-28916-605f8757707b   Ready    <none>   93s    v1.20.5

Nice!

You have successfully created and updated an LKE cluster through the Linode CLI!

You can now delete the cluster, as you will learn other ways to deploy and manage it.

bash

linode-cli lke cluster-delete $CLUSTER_ID

Note: Once you hit enter, the cluster will be immediately deleted!

Provisioning an LKE cluster programmatically

Linode also offers an additional way to provision resources on the cloud.

That option is programmatic, with standard HTTP requests sent to the Linode API.

The API endpoints provide great flexibility for managing your infrastructure and services.

There is extensive documentation on the API, along with a lot of examples to get you started.

But before you start, you'll need to create a Bearer token to authenticate to the API.

The Linode CLI offers a convenient command for that:

bash

linode-cli profile token-create
┌──────────┬────────┬─────────────────────┬───────────────┬─────────────────────┐
│ id       │ scopes │ created             │ token         │ expiry              │
├──────────┼────────┼─────────────────────┼───────────────┼─────────────────────┤
│ 25906259 │ *      │ 2021-03-28T18:02:55 │ 74d9f518afxx  │ 2999-12-12T05:00:00 │
└──────────┴────────┴─────────────────────┴───────────────┴─────────────────────┘

Keep in mind that the command will display the token only once, and you won't be able to retrieve it afterwards.

You should also treat it like a password and store it safely!

Let's export the token as a variable:

bash

export TOKEN=74d9f518af26ffb4...

Now you can try to retrieve the available LKE clusters with:

bash

curl -H "Authorization: Bearer $TOKEN" \
  https://api.linode.com/v4/lke/clusters

{"data": [], "page": 1, "pages": 1, "results": 0}

The data field is empty, meaning you don't have any clusters running yet.

Let's deploy a cluster that has the same spec as the previous one:

bash

curl -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -X POST -d '{
    "label": "learnk8s",
    "region": "eu-west",
    "k8s_version": "1.20",
    "node_pools": [
      {
        "type": "g6-standard-2",
        "count": 3
      }
    ]
  }' \
  https://api.linode.com/v4/lke/clusters

{
  "id": 22806,
  "status": "ready",
  "created": "2021-03-28T18:59:08",
  "updated": "2021-03-28T18:59:08",
  "label": "learnk8s",
  "region": "eu-west",
  "k8s_version": "1.20",
  "tags": []
}

If you want, you can connect to it using the previous command.

bash

linode-cli lke kubeconfig-view 22806 --text | tail +2 | base64 -d

apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
    certificate-authority-data: LS...
# truncated output

Save the output as kubeconfig and export it as the KUBECONFIG environment variable.

bash

export KUBECONFIG=<path to your kubeconfig>

You can delete the cluster now, as you will learn how to use Terraform to deploy it.

Delete it using the following HTTP request:

bash

curl -H "Authorization: Bearer $TOKEN" \
  -X DELETE \
  https://api.linode.com/v4/lke/clusters/22806

Provisioning an LKE cluster with Terraform

Terraform is an open-source Infrastructure as a Code tool.

Instead of writing the code to create the infrastructure, you define a plan of what you want to be executed, and you let Terraform create the resources on your behalf.

The plan isn't written in YAML, though.

Instead, Terraform uses a language called HCL - HashiCorp Configuration Language.

In other words, you use HCL to declare the infrastructure you want to be deployed, and Terraform executes the instructions.

Terraform uses plugins called providers to interface with the resources in the cloud provider.

This further expands with modules as a group of resources and are the building blocks you will use to create a cluster.

But let's take a break from the theory and see those concepts in practice.

Before you can create a cluster with Terraform, you should install the binary.

You can find the instructions on how to install the Terraform CLI from the official documentation.

Verify that the Terraform tool has been installed correctly with:

bash

terraform version
Terraform v0.14.9

Before moving forward with the Terraform code, you will need to generate an access token.

A token must be provided to Terraform to authenticate and execute instructions on your behalf.

If you keep the token from the previous section, you can reuse it.

Otherwise, create a new token using the following command:

bash

linode-cli profile token-create
┌──────────┬────────┬─────────────────────┬───────────────┬─────────────────────┐
│ id       │ scopes │ created             │ token         │ expiry              │
├──────────┼────────┼─────────────────────┼───────────────┼─────────────────────┤
│ 25906259 │ *      │ 2021-03-28T18:02:55 │ 74d9f518afxx  │ 2999-12-12T05:00:00 │
└──────────┴────────┴─────────────────────┴───────────────┴─────────────────────┘

You have two options for using the token with Terraform:

  1. You can insert it directly in the HCL configuration file.
  2. You can expose it through an environment variable.

It's safer to use it through an environment variable, that way there would be no consequences if you decide to push the code to a public repository.

Now assign the token to environment variable named $LINODE_TOKEN:

bash

export LINODE_TOKEN=<YOUR_TOKEN_HERE>

Keep in mind that exporting variables this way will be available only for the duration of the session.

And create a file named main.tf with the following content:

main.tf

terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "=1.16.0"
    }
  }
}

provider "linode" {
}

resource "linode_lke_cluster" "lke_cluster" {
    label       = "learnk8s"
    k8s_version = "1.20"
    region      = "eu-west"

    pool {
        type  = "g6-standard-2"
        count = 3
    }
}

Terraform commands

In the same directory run:

bash

terraform init

The command initializes Terraform and executes a few crucial tasks:

  1. It downloads the Linode provider that is necessary to translate the Terraform instructions into API calls.
  2. It will create two more folders as well as a state file. The state file is used to keep track of the resources that have been already created.

Consider the state files as a checkpoint; Terraform won't know what has been already created or updated without them.

If you further want to validate if the configuration is correct, you can do so with the terraform validate command. If the config is valid, you'll get Success! The configuration is valid. output.

You're now ready to create an LKE cluster using Terraform.

Two commands are frequently used in succession.

The first is:

bash

terraform plan
Plan: 1 to add, 0 to change, 0 to destroy.

Terraform will perform a dry-run and will prompt you with a detailed summary of what resources is about to create.

It's always a good idea to double-check and verify what will happen to your infrastructure before you commit the changes.

You don't want to accidentally destroy a database because you forgot to add or remove a resource.

Once you are happy with the changes, you can create the resources for real with:

bash

terraform apply
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

On the prompt, confirm with yes, and Terraform will create the cluster.

Congratulations, you just utilized infrastructure as code and Terraform to provision a Kubernetes cluster!

If you are interested in exploring other resources, you can do so here.

For now, delete the existing cluster, as you will repeat the same experiment but one step at the time.

You can delete the cluster with:

bash

terraform destroy
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Terraform will print a list of resources that are ready to be deleted.

As soon as you confirm, Terraform destroys all resources.

Terraform step by step

Create a new folder with the following files:

If you went through the previous step, just copy over the main.tf file in the new folder.

In the main.tf file, copy and paste the following code:

main.tf

terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "=1.16.0"
    }
  }
}

provider "linode" {
}

resource "linode_lke_cluster" "lke_cluster" {
    label       = "learnk8s"
    k8s_version = "1.20"
    region      = "eu-west"

    pool {
        type  = "g6-standard-2"
        count = 3
    }
}

And in the outputs.tf add the following:

outputs.tf

resource "local_file" "kubeconfig" {
  depends_on   = [linode_lke_cluster.lke_cluster]
  filename     = "kube-config"
  content      = base64decode(linode_lke_cluster.lke_cluster.kubeconfig)
}

The code may look intimidating, but you have nothing to worry about.

I will explain every section.

To initialize Terraform and download the necessary resources.

bash

terraform init

To perform a dry-run and verify what will be created by Terraform.

bash

terraform plan
Plan: 2 to add, 0 to change, 0 to destroy.

Finally, to apply and create the resources:

bash

terraform apply
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Usually, the cloud providers need 10 to 20 minutes to provision the resources, but that is not the case with Linode!

Linode is lightning-fast when it comes to provisioning a cluster!

By the time you've read that sentence, your LKE cluster is created.

Now, if you inspect the current folder, you will notice a few new files:

bash

tree .
.
├── kube-config
├── main.tf
├── outputs.tf
├── terraform.tfstate
└── terraform.tfstate.backup

Terraform uses the terraform.tfstate to keep track of what resources were created.

The kube-config is the kube configuration file allowing you to access the newly created cluster.

Inspect the cluster pods using the generated kube config file:

bash

kubectl get nodes --kubeconfig kube-config
NAME                          STATUS   ROLES    AGE     VERSION
lke23221-29518-606847e5cfec   Ready    <none>   3m50s   v1.20.5
lke23221-29518-606847e62866   Ready    <none>   3m53s   v1.20.5
lke23221-29518-606847e67d8a   Ready    <none>   3m38s   v1.20.5

If you prefer to not prefix the --kubeconfig environment variable to every command, you can export the KUBECONFIG variable as:

bash

export KUBECONFIG="${PWD}/kube-config"

The export is valid only for the current terminal session.

Since the cluster is up and running now, let's dive in and discuss the Terraform files.

The Terraform files that you just executed are divided into several blocks, so let's look at each one of them.

The first two blocks of code are the required providers (Terraform v0.13+) and provider.

main.tf

terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "=1.16.0"
    }
  }
}

provider "linode" {
}

resource "linode_lke_cluster" "lke_cluster" {
    label       = "learnk8s"
    k8s_version = "1.20"
    region      = "eu-west"

    pool {
        type  = "g6-standard-2"
        count = 3
    }
}

This block is where you define your Terraform configuration for the cloud provider.

The source and versions are self-explanatory: you define the URL where to download the provider and which version to use.

In this case, the provider is Linode, and the version is 1.16.0.

If you want to learn more about version constraints, you can take a look here.

Let's move on to the next definition.

main.tf

resource "linode_lke_cluster" "lke_cluster" {
    label       = "learnk8s"
    k8s_version = "1.20"
    region      = "eu-west"

    pool {
        type  = "g6-standard-2"
        count = 3
    }
}

The linode_lke_cluster is the actual resource that manages the Linode Kubernetes cluster.

The lke_cluster is the locally given name for that resource.

If you want to reference to the cluster in the rest of your code, you can do so using its name.

The next three are the required arguments that you must supply:

Next is the pool section, where you define the worker node pool details.

The pool has two required arguments, the type and node count.

In this case, the pool deployed will be with 3 nodes, each of type g6-standard-2.

Definitions in the outputs.tf, as its name suggests, will output some requested data.

You will utilize the output to generate a kube config file required for cluster access.

outputs.tf

resource "local_file" "kubeconfig" {
  depends_on   = [linode_lke_cluster.lke_cluster]
  filename     = "kube-config"
  content      = base64decode(linode_lke_cluster.lke_cluster.kubeconfig)
}

The resource here will create a local file populated with the kube configuration.

The depends_on is a meta-argument that sets a dependency on something either a resource or module before another code block gets executed.

In this instance, the depends_on waits for the clusters before it creates the kubeconfig.

The other required parameters are filename and content.

The content holds the credentials of the cluster.

Since you need to reference the cluster, you can use:

To read the value from the cluster.

And since the content is base64 encoded, you will have to use the base64decode helper before saving it as the kube-config file.

The Linode CLI vs Terraform — pros and cons

You can already tell the main differences between the Linode CLI and Terraform:

So which one should you use?

For smaller experiments, when you need to spin a cluster quickly, you should consider using the Linode CLI.

With a short command, you can easily create it.

For production-grade infrastructure where you want to configure and tune every single detail of your cluster, you should consider using Terraform.

But there is another crucial reason why you should prefer Terraform — incremental updates.

Let's imagine that you want to add a second pool to your cluster.

Perhaps you want to add another - more CPU-optimized node pool to your cluster for your compute-hungry applications.

It's simple as adding another pool block and defining the new worker nodes.

main.tf

# ...
    pool {
        type  = "g6-standard-2"
        count = 3
    }
    pool {
        type  = "g6-dedicated-4"
        count = 3
    }
}

Proceed with the previous commands to plan and apply the new changes:

bash

terraform plan
Plan: 0 to add, 1 to change, 0 to destroy.

And you can apply the changes with:

bash

terraform apply
linode_lke_cluster.lke_cluster: Modifying...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

After a couple of minutes, you can verify that the new node pool is added with:

bash

kubectl get nodes --kubeconfig kube-config
NAME                          STATUS   ROLES    AGE     VERSION
lke23221-29518-606847e5cfec   Ready    <none>   1h28m   v1.20.5
lke23221-29518-606847e62866   Ready    <none>   1h28m   v1.20.5
lke23221-29518-606847e67d8a   Ready    <none>   1h28m   v1.20.5
lke23221-29550-6068b05d846f   Ready    <none>   2m54s   v1.20.5
lke23221-29550-6068b05ddb9b   Ready    <none>   2m52s   v1.20.5
lke23221-29550-6068b05e30b8   Ready    <none>   2m59s   v1.20.5

Excellent!

You've managed not only to create a cluster but modify it and add node pool, all through Terraform!

Now you can take this a step further and deploy an actual application to the cluster.

Testing the cluster by deploying a simple Hello World app

You can create a Deployment with the following YAML definition:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-kubernetes
spec:
  selector:
    matchLabels:
      name: hello-kubernetes
  template:
    metadata:
      labels:
        name: hello-kubernetes
    spec:
      containers:
        - name: app
          image: paulbouwer/hello-kubernetes:1.8
          ports:
            - containerPort: 8080

You can find all the manifests for the demo app in this repository.

NOTE: To make it easier, issuing commands to the cluster without specifying the --kubeconfig parameter each time. You can either export or move the generated kubeconfig to ~/.kube/config.

You can deploy the manifest with:

bash

kubectl apply -f deployment.yaml
deployment.apps/hello-kubernetes created

A quick way to check that the application runs correctly is to connect to it using kubectl port-forward.

But first, you need to retrieve the name of the pod with:

bash

kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
hello-kubernetes-6db5bf56c6-m9w9f   1/1     Running   0          112s

You can connect to the Pod with:

bash

kubectl port-forward hello-kubernetes-6db5bf56c6-m9w9f 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080

Or, with a single command:

bash

kubectl port-forward $(kubectl get pod -l name=hello-kubernetes --no-headers | awk '{print $1}') 8080:8080

The kubectl port-forward command connects to the Pod with the name hello-kubernetes-6db5bf56c6-m9w9f and forwards all the traffic from port 8080 on the Pod to port 8080 on your computer.

Please notice that kubectl port-forward opens the first port on your computer, and the second is the target port on the container. In this example, both are 8080.

Now, if you visit http://localhost:8080 on your computer, you should be greeted by the application's web page.

Exposing the application with kubectl port-forward is an excellent way to test the app quickly, but it isn't a long-term solution.

If you want to serve live traffic to the Pod, you will need a more permanent solution.

In Kubernetes, you can use a Service of type: LoadBalancer to start up a load balancer to expose your Pods.

You can use the following code:

service-loadbalancer.yaml

apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080
  selector:
    name: hello-kubernetes

And submit the YAML with:

bash

kubectl apply -f service-loadbalancer.yaml
service/hello-kubernetes created

As soon as you submit the service manifest, Linode will provision a Load Balancer and connect it to your pod. In Linode terms, load balancers are named Node Balancers.

You can list the services and retrieve the Load Balancer's IP address.

bash

kubectl get services
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
hello-kubernetes   LoadBalancer   10.128.201.3   185.3.92.242   80:31036/TCP   43s
kubernetes         ClusterIP      10.128.0.1     <none>         443/TCP        20m

If you visit the external IP address in your browser, you should see the application.

Excellent!

There is only one issue, though.

The load balancer that you created earlier serves one service at a time.

Also, it has no option to provide intelligent routing based on paths.

So if you have multiple services that need to be exposed, you will need to create the same number of load balancers.

Imagine having ten different applications that need to be exposed.

If you use a Service of type: LoadBalancer for each of them, you might end up with ten different Load Balancers.

This wouldn't have been a problem if those load balancers weren't so expensive — especially when running a myriad of them.

Not to worry, though!

You will learn another way to solve this challenge.

For now, delete the load balancer with:

bash

kubectl delete svc hello-kubernetes

Routing traffic into the cluster with an Ingress

In Kubernetes, another resource is designed to solve routing traffic inside the cluster: the Ingress.

The Ingress has two parts:

  1. The first is the Ingress object which is the same as Deployment or Service in Kubernetes. This is defined by the kind part of the YAML manifest.
  2. The second part is the Ingress controller. This is the actual part that controls the load balancers, so they know how to serve the requests and forward the requests to the Pods.

In other words, the Ingress controller acts as a reverse proxy that routes the traffic to your Pods.

The Ingress in Kubernetes

The Ingress routes the traffic based on paths, domains, headers, etc., which consolidates multiple endpoints in a single resource that runs inside Kubernetes.

With this, you can serve multiple services simultaneously from one exposed endpoint - the load balancer.

There're lots of Ingress controllers that you can choose from:

  1. Nginx Ingress
  2. Ambassador
  3. Traefik
  4. And more.

However, in this guide, you will deploy the first one - the Nginx Ingress Controller and use it to route live traffic to your application.

Deploying an Ingress Controller

There are multiple ways to deploying the Ingress controller.

The most common way is to use Helm and deploy everything in one command.

Helm is a package manager for Kubernetes.

Helm provides you with an excellent way to bundle up multiple YAML files and install (or remove) them in one go.

You can install the Helm binary by following the official instructions.

Helm will automatically fetch your cluster credentials — there is no further authentication needed.

After which, you can add the ingress chart repository:

bash

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

The repository contains all the instructions to install the Nginx controller.

Next, you can install the ingress controller with:

bash

helm install ingress ingress-nginx/ingress-nginx
# truncated output
NOTES:
The ingress-nginx controller has been installed.

Great!

Now that you have an Ingress controller installed on the cluster, you can utilize it to serve requests more efficiently.

You can read more on the Nginx Ingress Controller here.

With this, the nginx Ingress controller automatically provisions its load balancer.

And it will use it as the main entry point for all traffic.

You can verify this by checking on the services:

bash

kubectl get services
ingress-ingress-nginx-controller    LoadBalancer   10.128.111.243   185.3.93.28   80:30523/TCP,443:30177/TCP   56s

And since you deleted the previous service for the hello-world app, you will need to create a new one of type: ClusterIP.

The ClusterIP service makes the application only available inside the cluster.

It doesn't expose it to the outside world like the Service of type NodePort or LoadBalancer.

You will use this service and connect it to the ingress.

The Ingress controller will take care of forwarding the traffic inside the cluster.

Apply the ClusterIP service with:

bash

kubectl apply -f service.yaml
service/hello-kubernetes created

Let's check on the ingress manifest now:

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-kubernetes
  annotations:
    kubernetes.io/ingress.class: 'nginx'
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: hello-kubernetes
                port:
                  number: 80

That's a lot of lines!

Let's break it down.

And inside the service property:

Note: The service port can be different from the target or container port. The ingress doesn't care what your container port is. Its interest is in the port of the service.

And if you want to learn about the different path types, take a look at the following link.

You can now apply the ingress object:

bash

kubectl apply -f ingress.yaml
ingress.networking.k8s.io/hello-kubernetes created

You can describe the ingress details with:

bash

kubectl describe ingress hello-kubernetes
Name:             hello-kubernetes
Namespace:        default
Address:          185.3.93.28
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /   hello-kubernetes:80 (10.2.0.4:8080)
Annotations:  kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  Sync    92s (x2 over 2m2s)  nginx-ingress-controller  Scheduled for sync

As soon as you submit the resource to the cluster, the Ingress controller is notified of the new resource.

  • Consider the following cluster with three Nodes and two pods with the web application.
    1/4

    Consider the following cluster with three Nodes and two pods with the web application.

  • When you install the Ingress, an Ingress Nginx pod is created in your cluster.
    2/4

    When you install the Ingress, an Ingress Nginx pod is created in your cluster.

  • Ingress Nginx is exposed to external traffic with a Service of type: LoadBalancer. Linode provisions a Node Balancer and routes traffic to the Nginx Pod.
    3/4

    Ingress Nginx is exposed to external traffic with a Service of type: LoadBalancer. Linode provisions a Node Balancer and routes traffic to the Nginx Pod.

  • When you create an Ingress manifest, the Ingress routes the incoming traffic to your apps.
    4/4

    When you create an Ingress manifest, the Ingress routes the incoming traffic to your apps.

Great job!

You provisioned a cluster and made it ready to serve traffic using an Ingress!

If you now follow the IP in the Address field, you will be able to visit the application's web page.

Fully automated Development, Staging, and Production environments with Terraform

The most common infrastructure setup for software projects is divided into three environments:

  1. Development — Where the code is initially deployed by developers and tested for common bugs.
  2. Staging — The next stage where the more polished code goes for tests by the QA team.
  3. Production — Where the code gets deployed to production after having a green light by QA and tested that it's stable.

Since you want your apps to progress through the environments, you might want to provision multiple clusters, one for each environment.

When you are not utilizing infrastructure as code, you will be forced to click repeatedly on the user interface to create the environments.

But when you use infrastructure is code, you can parametrize the name of your resources and create clusters that are exact copies.

You can reuse the existing Terraform code and provision all three clusters simultaneously using Terraform modules and expressions.

Before you execute the script, it's a good idea to destroy any cluster that you created previously with terraform destroy.

The expression syntax is straightforward.

First, you define variables like this:

variables.tf

variable "cluster_name" {
  description = "The name for the LKE cluster"
  default     = "learnk8s"
}

variable "env_name" {
  description = "The environment for the LKE cluster"
  default     = "dev"
}

Terraform variables are usually defined in a separate variables.tf file.

Later, you can reference and link the variables in the main.tf like this:

main.tf

#...
resource "linode_lke_cluster" "lke_cluster" {
    label       = "${var.cluster_name}-${var.env_name}"
#...

Terraform will interpolate the string to "learnk8s-dev".

When you execute the usual terraform apply command, you can pass arguments to override the variable with a different name.

For example:

bash

terraform apply -var="cluster_name=my-cluster" -var="env_name=staging"

Passing the vars as above will provision a cluster with the name of my-cluster-staging.

But variables might not always work the way you expect.

Look at this code snippet:

bash

terraform apply -var="env_name=dev"
# and later
terraform apply -var="env_name=staging"

If you execute the commands in quick succession, what happens?

Is Terraform creating two clusters or updating the dev cluster to be a staging cluster?

The answer is: it will overwrite the dev cluster and make it staging!

But if you don't want that?

Is there a way to create separate clusters?

This is where the Terraform modules come in.

Move your main.tf, variables.tf, and outputs.tf in a subfolder and create an empty main.tf.

bash

mkdir -p cluster_module
mv main.tf variables.tf outputs.tf cluster_module
tree .
.
├── main.tf
└── cluster_module
    ├── main.tf
    ├── outputs.tf
    └── variables.tf

From now on, you can use the code in the cluster_module folder as a reusable module.

Since you probably want clusters with different names, let's introduce a few parameters.

In the cluster_module folder where the main.tf file is located, replace the cluster name with a variable and also append the env_name like so:

main.tf

#other code truncated
resource "linode_lke_cluster" "lke_cluster" {
    label       = "${var.cluster_name}-${var.env_name}"
    k8s_version = "1.20"
    region      = "eu-west"
    tags        =  [ var.env_name ]

Notice the difference between chaining multiple variables and assigning a single one. Since you aren't chaining two or more variables, there is no need to declare it with ${}. Instead, you can append the tag value directly with var.variable_name.

You will also need to create a unique kube config filename to differentiate it between the clusters.

Append the env_name there as well:

outputs.tf

resource "local_file" "kubeconfig" {
  depends_on   = [linode_lke_cluster.lke_cluster]
  filename     = "kube-config-${var.env_name}"
  content      = base64decode(linode_lke_cluster.lke_cluster.kubeconfig)
}

Now you can reference all the code from the root main.tf like this:

main.tf

module "prod_cluster" {
    source       = "./cluster_module"
    env_name     = "prod"
    cluster_name = "learnk8s"
}

And since the module is reusable, you can create more than a single cluster:

main.tf

module "dev_cluster" {
    source       = "./cluster_module"
    env_name     = "dev"
    cluster_name = "learnk8s"
}
module "staging_cluster" {
    source       = "./cluster_module"
    env_name     = "staging"
    cluster_name = "learnk8s"
}
module "prod_cluster" {
    source       = "./cluster_module"
    env_name     = "prod"
    cluster_name = "learnk8s"
}

Preview the changes with:

terraform plan
Plan: 6 to add, 0 to change, 0 to destroy.

Apply the changes and create the three environments that are exact copies with:

terraform apply
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

This is good stuff, but now you have to go manually cluster by cluster and install the Ingress controller.

Can Terraform help you to automate that as well?

Yes, it can!

There is a Helm provider designed explicitly for this purpose.

Amend the main.tf in the cluster_module folder to include the Helm provider and the Nginx Ingress Controller:

main.tf

terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "=1.16.0"
    }
    helm = {
      source = "hashicorp/helm"
      version = "=2.1.0"
    }
  }
}

provider "linode" {
}

provider "helm" {
  kubernetes {
    config_path = "kube-config-${var.env_name}"
  }
}

resource "linode_lke_cluster" "lke_cluster" {
    label       = "${var.cluster_name}-${var.env_name}"
    k8s_version = "1.20"
    region      = "eu-west"
    tags        =  [ var.env_name ]

    pool {
        type  = "g6-standard-2"
        count = 3
    }
}

resource "helm_release" "ingress-nginx" {
  depends_on   = [local_file.kubeconfig]
  name       = "ingress"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
}

The code for this section can be found in the repository.

It's the same drill as before; you define the provider, a resource, and which chart should Helm install.

You can proceed to update the clusters.

But since there is a new provider defined, you must use terraform init to initialize Terraform once more.

Note: Adding the Helm provider this way may recycle the node pools.

Now execute a terraform plan and check what resources will change with:

terraform plan
Plan: 3 to add, 0 to change, 0 to destroy.

Finally, apply the changes to your clusters and include the Ingress controller with:

bash

terraform apply
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

It will take some time for all the controllers to be installed.

Using the kube configs, verify that the NGINX Ingress Controllers are deployed:

bash

kubectl get pod --kubeconfig=kube-config-dev
NAME                                                READY   STATUS    RESTARTS   AGE
ingress-ingress-nginx-controller-867f748bf7-n8f82   1/1     Running   0          68s

kubectl get pod --kubeconfig=kube-config-staging
NAME                                                READY   STATUS    RESTARTS   AGE
ingress-ingress-nginx-controller-867f748bf7-j72w4   1/1     Running   0          69s

kubectl get pod --kubeconfig=kube-config-prod
NAME                                                READY   STATUS    RESTARTS   AGE
ingress-ingress-nginx-controller-867f748bf7-6w7dw   1/1     Running   0          111s

Excellent!

Now all of your environments are ready to serve traffic in real-time using an Ingress controller.

If you want to include an Ingress controller like this with a single cluster, the files are available here.

One more thing to cover is the updates.

What happens when you update the cluster module?

When you modify a property or add a resource as you've done for the Nginx controller, Terraform will update all clusters with the same property.

If you wish to customize the properties on a per-environment basis, you should extract the parameters in variables and change them from the root main.tf.

Let's have a look at an example.

You might want to run the dev and staging environments with the current instance types but add a more powerful one for production.

As an example you can refactor the code and extract the instance type as a variable:

variables.tf

variable "instance_type" {
  description = "The node pool instance type"
  default = "g6-standard-2"
}

And amend the main.tf file to reflect that:

main.tf

#...
pool {
    type  = var.instance_type
    count = 3
}
#...

Later, you can modify the root main.tf file with the instance type:

main.tf

module "dev_cluster" {
    source          = "./cluster_module"
    env_name        = "dev"
    cluster_name    = "learnk8s"
    instance_type   = "g6-standard-2"
}
module "staging_cluster" {
    source          = "./cluster_module"
    env_name        = "staging"
    cluster_name    = "learnk8s"
    instance_type   = "g6-standard-2"
}
module "prod_cluster" {
    source          = "./cluster_module"
    env_name        = "prod"
    cluster_name    = "learnk8s"
    instance_type   = "g6-dedicated-4"
}

If you wish, you can proceed to apply the changes and verify that the node pool in the production cluster is changed:

bash

kubectl get nodes --kubeconfig=kube-config-prod
NAME                          STATUS   ROLES    AGE     VERSION
lke23924-30462-6072eacc6374   Ready    <none>   4m12s   v1.20.5
lke23924-30462-6072eaccbb8a   Ready    <none>   4m16s   v1.20.5
lke23924-30462-6072eacd1126   Ready    <none>   3m51s   v1.20.5

Be patient here as replacing the node pool may take a couple of minutes.

Excellent!

As you can imagine, you can add more variables to your module and create environments with different configurations and specifications.

This marks the end of your journey!

Summary

A recap on what you've built so far:

Well done!

Be the first to be notified when a new article or Kubernetes experiment is published.

*We'll never share your email address, and you can opt-out at any time.

There are more articles like this!

Be notified every time we publish articles, insights and new research on Kubernetes!

You are in!

We're also maintain an active Telegram, Slack & Twitter community!