Securing Kubernetes Applications with CrowdSec Intrusion Detection System
Detect and Defend with CrowdSec IDS
📌 introduction:
Kubernetes applications are increasingly being targeted by attackers. The microservice architecture of Kubernetes clusters creates a large attack surface, with every application opening a new potential entry point for attackers.
CrowdSec is an open-source intrusion detection system (IDS) that can be used to protect Kubernetes applications from attack. CrowdSec can be configured to monitor application logs and detect malicious activity. When an attack is detected, CrowdSec can block the attacker’s IP address or take other remediation actions.
This blog post will show you how to install and configure CrowdSec in a Kubernetes cluster. We will also discuss how to use CrowdSec to detect attacks on Kubernetes applications.
▶️ Architecture:
Here’s an architecture overview of CrowdSec inside a K8s cluster.
In this part, I will show you how you can install CrowdSec Intrusion Detection System (IDS) inside a Kubernetes cluster.
⛳️ Prerequisites:
- installed Kubernetes cluster.
- installed nginx ingress controller.
- kubectl.
- helm.
🌕 Installing the HelloWorld Application:
To demonstrate how to install an application using CrowdSec, we have released a HelloWorld application that can be deployed using the Nginx ingress controller.
This Helm chart is hosted in thse charts repository.
helm repo add crowdsec https://crowdsecurity.github.io/helm-charts
"crowdsec" has been added to your repositoriesThen update the repositories to get the new charts
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "crowdsec" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈Now, we can install the HelloWorld chart with default values in the default namespace.
$ helm install helloworld crowdsec/helloworld
NAME: helloworld
LAST DEPLOYED: Mon Sep 30 10:12:21 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
http://helloworld.local/To access the webUI, you need to retrieve the public URL:
$ kubectl get ingresses.networking.k8s.io
NAME CLASS HOSTS ADDRESS PORTS AGE
helloworld helloworld.local a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com 80 2m18shost a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com
a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com has address 52.31.225.95
a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com has address 54.73.240.30
a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com has address 176.34.92.134
This command might take some time to get a result when using the EKS AWS service.
We can modify the hosts’ file and add one of the public IP addresses:
echo "52.31.225.95 helloworld.local" | sudo tee -a /etc/hostsOur environment is ready. Let’s deep dive into the exciting part.
✳️ Install CrowdSec:
Crowdsec chart is also available in these charts repositories.
First, we need to create a new namespace so that CrowdSec will be isolated:
$ kubectl create ns crowdsec
namespace/crowdsec createdWe want to monitor Nginx ingress controller logs because our application is deployed behind the Nginx ingress controller.
We can create a new file crowdsec-values.yaml, containing the CrowdSec chart configuration.
agent:
# To specify each pod you want to process it logs (pods present in the node)
acquisition:
# The namespace where the pod is located
- namespace: ingress-nginx
# The pod name
podName: ingress-nginx-controller-*
# as in crowdsec configuration, we need to specify the program name so the parser will match and parse logs
program: nginx
# Those are ENV variables
env:
# As it's a test, we don't want to share signals with CrowdSec so disable the Online API.
- name: DISABLE_ONLINE_API
value: "true"
# As we are running Nginx, we want to install the Nginx collection
- name: COLLECTIONS
value: "crowdsecurity/nginx"
lapi:
env:
# As it's a test, we don't want to share signals with CrowdSec, so disable the Online API.
- name: DISABLE_ONLINE_API
value: "true"Now we can install CrowdSec using our config file in the CrowdSec namespace we created previously.
helm install crowdsec crowdsec/crowdsec -f crowdsec-values.yaml -n crowdsec
kubectl get pods -n crowdsecAnd we can see that our LAPI (CrowdSec’s local API) and agent are running:
$ kubectl get pods -n crowdsec
NAME READY STATUS RESTARTS AGE
crowdsec-agent-kf9fr 1/1 Running 0 34s
crowdsec-lapi-7455dw9947-dewr 1/1 Running 0 34s🚧 Demo:
To test whether CrowdSec detects attacks, we will simulate an attack on the HelloWorld application using Nikto and see CrowdSec metrics and alerts.
./nikto.pl -host http://helloworld.localNow we can get a shell into the CrowdSec agent pod and see metrics and alerts:
kubectl get pods -n crowdsec
kubectl -n crowdsec exec -it crowdsec-agent-kf9fr -- sh/ # cscli metrics
INFO[30-09-2023 13:39:50 PM] Buckets Metrics:
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-bad-user-agent | 3 | 183 | 186 | 369 | - |
| crowdsecurity/http-crawl-non_statics | - | 7 | 9 | 351 | 2 |
| crowdsecurity/http-path-traversal-probing | - | - | 1 | 2 | 1 |
| crowdsecurity/http-probing | 1 | - | 2 | 2 | 1 |
| crowdsecurity/http-sensitive-files | - | 3 | 4 | 17 | 1 |
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[30-09-2023 13:39:50 PM] Acquisition Metrics:
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/containers/ingress-nginx-controller-fd7bb8d66-llxc9_ingress-nginx_controller-c536915796f13bbf66d1a8ab7159dbd055773dbbf89ab4d9653043591dfaef1f.log | 371 | 371 | - | 741 |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[30-09-2023 13:39:50 PM] Parser Metrics:
+--------------------------------+------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs | 1113 | 738 | 375 |
| child-crowdsecurity/nginx-logs | 371 | 371 | - |
| crowdsecurity/dateparse-enrich | 371 | 371 | - |
| crowdsecurity/docker-logs | 371 | 371 | - |
| crowdsecurity/geoip-enrich | 371 | 371 | - |
| crowdsecurity/http-logs | 371 | 360 | 11 |
| crowdsecurity/nginx-logs | 371 | 371 | - |
| crowdsecurity/whitelists | 371 | 371 | - |
+--------------------------------+------+--------+----------+Metrics show files read by CrowdSec (in the acquisition table) and how many are parsed/unparsed, all scenarios triggered by the logs (in the bucket table). As we installed the collection crowdsecurity/Nginx, it comes with multiple scenarios that detect HTTP attacks.
Now let’s see if the CrowdSec agent detects something:
/ # cscli alerts list
+----+------------------+--------------------------------------+---------+---------+-----------+--------------------------------+
| ID | VALUE | REASON | COUNTRY | AS | DECISIONS | CREATED AT |
+----+------------------+--------------------------------------+---------+---------+-----------+--------------------------------+
| 3 | Ip:82.125.X.X | crowdsecurity/http-sensitive-files | FR | Orange | ban:1 | 30-09-2023 13:42:50.624461519 |
| | | | | | | +0000 UTC |
| 2 | Ip:82.125.X.X | crowdsecurity/http-crawl-non_statics | FR | Orange | ban:1 | 30-09-2023 13:42:50.104034837 |
| | | | | | | +0000 UTC |
| 1 | Ip:82.125.X.X | crowdsecurity/http-bad-user-agent | FR | Orange | ban:1 | 30-09-2023 13:42:50.104018425 |
| | | | | | | +0000 UTC |
+----+------------------+--------------------------------------+---------+---------+-----------+--------------------------------+Those are alerts raised by the CrowdSec agent following our Nikto scan. We can see that several scenarios were triggered and the agent sent ban decisions to the LAPI. It means that it will be stored and shared with the bouncers to block this IP.
🏗 Remediation:
So after detecting attacks from the previous article, we can now delete all the alerts to start from a clean CrowdSec database.
# Get a shell on the LAPI pod
kubectl -n crowdsec exec -it crowdsec-lapi-7444d9ff6-pqm4q -- sh
# Delete all decisions
cscli alerts delete -aInstall Crowdsec Lua bouncer plugin:
To install a bouncer, we need to generate a bouncer API key, so the bouncer can communicate with the CrowdSec API to know if it needs to block the IP or not. Still, in the same crowdsec-lapi container shell, generate the bouncer API key using this command:
$ cscli bouncers add ingress-nginx
Api key for 'ingress-nginx':
e00b2155a7e43dd8e8d9294305bd9741
Please keep this key since you will not be able to retrieve it!You will get an API key, you need to keep it and save it for the ingress-nginx bouncer.
Now we can patch our ingress-nginx helm chart to add and enable the crowdsec lua plugin using the following configuration (the API_KEY and API_URL for the bouncer to communicate with crowdsec LAPI). You can put this configuration in a file crowdsec-ingress-bouncer.yaml.
controller:
extraVolumes:
- name: crowdsec-bouncer-plugin
emptyDir: {}
extraInitContainers:
- name: init-clone-crowdsec-bouncer
image: crowdsecurity/lua-bouncer-plugin
imagePullPolicy: IfNotPresent
env:
- name: API_URL
value: "http://crowdsec-service.crowdsec.svc.cluster.local:8080"
- name: API_KEY
value: "e00b2155a7e43dd8e8d9294305bd9741"
- name: BOUNCER_CONFIG
value: "/crowdsec/crowdsec-bouncer.conf"
command: ['sh', '-c', "sh /docker_start.sh; mkdir -p /lua_plugins/crowdsec/; cp -R /crowdsec/* /lua_plugins/crowdsec/"]
volumeMounts:
- name: crowdsec-bouncer-plugin
mountPath: /lua_plugins
extraVolumeMounts:
- name: crowdsec-bouncer-plugin
mountPath: /etc/nginx/lua/plugins/crowdsec
subPath: crowdsec
config:
plugins: "crowdsec"
lua-shared-dicts: "crowdsec_cache: 50m"Once we have this patch we can upgrade the ingress-nginx chart
helm -n ingress-nginx upgrade -f ingress-nginx-values.yaml -f crowdsec-ingress-bouncer.yaml ingress-nginx ingress-nginx/ingress-nginxNow we have our ingress controller patched with CrowdSec Lua bouncer plugin. We’ll start an attack again using Nikto on http://helloworld.local.
./nikto.pl -host http://helloworld.local/By getting a shell in the CrowdSec agent pod and listing the alerts, you’ll see your IP is attacking the helloworld app.
$ kubectl -n crowdsec exec -it crowdsec-agent-zjlr8 -- sh
/ # cscli decisions list
+----+----------+-------------------+--------------------------------------+--------+---------+-------------+--------+--------------------+----------+
| ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID |
+----+----------+-------------------+--------------------------------------+--------+---------+-------------+--------+--------------------+----------+
| 3 | crowdsec | Ip:86.X.X.X | crowdsecurity/http-crawl-non_statics | ban | FR | 0123 Orange | 43 | 3h59m44.053908518s | 3 |
+----+----------+-------------------+--------------------------------------+--------+---------+-------------+--------+--------------------+----------+Now, if we try to access the helloworld app using CURL
$ curl -v http://helloworld.local
* Trying 3.248.157.157:80...
* TCP_NODELAY set
* Connected to helloworld.local (3.248.157.157) port 80 (#0)
> GET / HTTP/1.1
> Host: helloworld.local
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Date: Mon, 27 Dec 2021 16:14:26 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 403 Forbidden
403 Forbidden
nginx
* Connection #0 to host helloworld.local left intactAnd tehre you go! We can see that the Nginx ingress controller blocked our IP (by sending us a 403 HTTP code), and we cannot access the helloworld application.
To make the app accessible again, from the crowdsec-agent pod, we just need to delete the decision on our IP.
$ cscli decisions delete --ip 86.X.X.X
INFO[30-09-2023 14:10:50 PM] 4 decision(s) deletedAnd CURL the helloworld app again.
$ curl -v http://helloworld.local
* Trying 3.248.157.157:80...
* TCP_NODELAY set
* Connected to helloworld.local (3.248.157.157) port 80 (#0)
> GET / HTTP/1.1
> Host: helloworld.local
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sat, 30:09:2023 15:39:50 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 13
< Connection: keep-alive
< X-App-Name: http-echo
< X-App-Version: 0.2.3
<
helloworld !
* Connection #0 to host helloworld.local left intactAnd we can see that we have access again.
🗒 Closing Thoughts:
In this tutorial, we explored the fundamental steps of setting up CrowdSec in a Kubernetes environment, highlighting its proficiency in detecting attacks. However, it’s crucial to remember that detection alone is only a part of the equation.
To complete the security process, we showcased a guide on how to configure the essential remediation component.
Thank you for Reading, see you in the next post. 🤟
🚀 Feel free to connect with me :
♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/
♻️ Twitter : https://twitter.com/rajhisaifeddine
The end ✌🏻
