At Last week’s Kubecon EU I was fortunate enough to be on stage with Ian Coldwater, Brad Geesaman, and Duffie Cooley presenting a talk called “Malicious Compliance: Reflections on Trusting Container Scanners”.
In this talk, one of the things we looked at was how it would be possible for a malicious container image to bypass container vulnerability scanners. We built up a set of techniques to demonstrate how this worked, with the details hosted on GitHub.
As with any talk we came up with quite a few ideas that we couldn’t fit in, for time, so I wanted to quickly talk about one I came up with as it’s a quick (and somewhat silly) way to bypass container vulnerability scanners.
What do container vulnerability scanners look for?
As we talked about during the presentation, there are a number of files and directories that contain information used by vulnerability scanners and SBOM tools to identify what software is installed in a container image, and what vulnerabilities are present in that software.
So obviously anything which stops the scanner look at those locations could interfere with their operation. With that idea in mind, I was thinking of some ways to do this, and I came up with a simple one.
Runtime image extraction
If we have a container image which only creates the container filesystem at runtime then scanners (which operate statically on images, rather than executing them) won’t be able to see that filesystem, and will likely just return an empty set of results.
The way I came up with to do this was a simple golang program which takes a tarball and a command as arguments, then extracts the tarball and executes the command.
Before I talk about the code, I’ll make it clear this is purely a PoC it’s not something anyone should use in production!
The basic code looks like this :-
package main
import (
"fmt"
"os"
"os/exec"
archiver "github.com/mholt/archiver/v3"
)
func main() {
// Check if the correct number of arguments are provided
if len(os.Args) < 3 {
fmt.Println("Usage: go run main.go <tar_file> <file to execute>")
os.Exit(1)
}
tarFile := os.Args[1]
//password := os.Args[2]
destination := "."
z := archiver.Tar{
MkdirAll: true,
ContinueOnError: true,
OverwriteExisting: true,
ImplicitTopLevelFolder: false,
}
// Use archiver library to open and extract the ZIP file
err := z.Unarchive(tarFile, destination)
if err != nil {
fmt.Println("Failed to unzip:", err)
os.Exit(1)
}
fmt.Println("Unzip completed successfully!")
out, err := exec.Command(os.Args[2]).Output()
//err = cmd.Run()
if err != nil {
fmt.Println("Failed to execute:", err)
os.Exit(1)
}
fmt.Printf("output %s", out)
}
The code is pretty simple essentially we just take the tarball and extract it to the current directory, then execute the command provided as the second argument, which by that point can run binaries from the extracted tarball.
Building the binary
An important note is that at the point the extractor runs there is nothing else in the container image, so it’ll need to be compiled statically. something like this.
CGO_ENABLED=0 go build -ldflags="-s -w" main.go
Once you’ve done that you can use a trick from the talk to ensure the binary itself doesn’t have any visible vulnerabilities. Just pack it with upx and that will fix that problem.
Getting the tarball to run
With the binary built you can just create a tarball by using docker export with any running (or stopped) container. Any Docker image should work fine for this.
Building the Docker image
Now you can create a Dockerfile like this :-
FROM scratch
COPY image.tar /
COPY main /
ENTRYPOINT ["/main","image.tar"]
An important point here is using ENTRYPOINT rather than CMD as we want to be able to pass an argument to docker run and have it be passed to main as the second argument.
With that docker file something like docker build -t obfuscator . should work fine.
Running the image
Now you can run the image like this would run the command ls in the container :-
docker run --rm obfuscator ls
Does it fool scanners?
Of course none of this is terribly useful if it doesn’t actually fool scanners. Happily a check with Trivy, Grype, Docker Scan and Docker Scout came up clean, when using our super vulnerable base image from the talk :)
Conclusion
This is a pretty silly way to bypass scanners, but it hopefully makes a useful point, which was one of our takeaways from the talk. If you can’t trust the people that built a container image (or any other artifact) then you can’t rely on tools like container vulnerability scanners to tell you any useful information about the contents of that image. You need to consider what’s in the image yourself, and make your own decisions about whether it’s safe to use! In case anyone wants to have a look at this, the code is on GitHub
