Madhu Akula - A Practical Approach to Breaking & Pwning Kubernetes Clusters $2,800 Early $2,250

DEF CON Training

Madhu Akula - A Practical Approach to Breaking & Pwning Kubernetes Clusters $2,800 Early $2,250

 EARLY BIRD $2,250 ENDS JANUARY 31ST.

Trainer bio: 

Madhu Akula is a pragmatic security leader and creator of Kubernetes Goat, an intentionally vulnerable by design Kubernetes Cluster to learn and practice Kubernetes Security. Also published author and Cloud Native Security Architect with extensive experience. Also, he is an active member of the international security, DevOps, and Cloud Native communities (null, DevSecOps, AllDayDevOps, AWS, CNCF, USENIX, etc). He holds industry certifications like CKA (Certified Kubernetes Administrator), CKS (Certified Kubernetes Security Specialist), OSCP (Offensive Security Certified Professional), etc. 
 
Madhu frequently speaks and runs training sessions at security events and conferences around the world including DEFCON 24, 26, 27, 28, 29 & 30, BlackHat 2018, 19, 21 & 22, USENIX LISA 2018, 19 & 21, SANS Cloud Security Summit 2021 & 2022, O’Reilly Velocity EU 2019, Github Satellite 2020, Appsec EU (2018, 19 & 22), All Day DevOps (2016, 17, 18, 19, 20 & 21), DevSecCon (London, Singapore, Boston), DevOpsDays India, c0c0n(2017, 18), Nullcon 2018, 19, 21 & 22, SACON, Serverless Summit, null and multiple others. 
 
His research has identified vulnerabilities in over 200+ companies and organizations including; Google, Microsoft, LinkedIn, eBay, AT&T, WordPress, NTOP, Adobe, etc. and is credited with multiple CVE’s, Acknowledgements, and rewards. He is co-author of Security Automation with Ansible2 (ISBN-13: 978-1788394512), which is listed as a technical resource by Red Hat Ansible. He is the technical reviewer for Learn Kubernetes Security, Practical Ansible2 books by Packt Pub. Also won 1st prize for building an Infrastructure Security Monitoring solution at InMobi flagship hackathon among 100+ engineering teams.
 
Trainer(s) social media links:
https://twitter.com/madhuakula
https://www.linkedin.com/in/madhuakula/
https://github.com/madhuakula

 
LINKS TO PREVIOUS TRAININGS
Blackhat

DEF CON TRAININGS LV 2022

 

TRAINING DESCRIPTION:  

 

The adoption of Kubernetes use in production has increased to 83% from a survey by CNCF. Still, most security teams struggle to understand these modern technologies.

 

 In this real-world scenario-based training, each participant will be learning Tactics, Techniques, and Procedures (TTPs) to attack and assess Kubernetes clusters environments at different layers like Supply chain, Infrastructure, Runtime, and many others. Starting from simple recon to gaining access to microservices, sensitive data, escaping containers, escalating to clusters privileges, and even its underlying cloud environments.

 

By end of the training, participants will be able to apply their knowledge to perform architecture reviews, security assessments, red team exercises, and pen-testing engagements on Kubernetes Clusters and Containersed environments successfully.  Also, the trainer will provide step by step guide (Digital Book) with resources and references to further your learning.

 

**Section-1**

* Kubernetes 101 - Fasttrack Edition

* Security Architecture review & Attack Trees using MITRE ATT&CK framework

* `kubectl` kung-fu to explore the cluster

* Attacking the supply chain by exploiting private registry

* Pwning the container images and gaining access to the cluster

* Exploiting security misconfigurations in the cluster

 

**Section-2**

* Escaping out of the container to the host system to gain more privileges

* Bypassing NSP and gaining unauthorized access to other microservices

* Lateral movement from container to node and then complete cluster access

* Escalating from ServiceAccount to more RBAC privileges (No least privileges)

* Helm with Tiller service = ClusterPwn (Complete cluster takeover)

* Gaining access to k8s volumes, logs of the services, and sensitive data

* From application vulnerability to cloud provider access (attack chain)

 

**Section-3**

* Hacker Container - The Swiss Army knife for hacking Kubernetes Clusters

* Exploiting Kubernetes Secrets and gaining access to third-party services

* DoS the services and cluster nodes by resources exemption

* Understanding Admission controller and possible attack surface around Webhooks

* Persisting in the clusters using Sidecar/Cronjob/DaemonSets

* Defense evasion techniques for Kubernetes Cluster environments

* Some useful hacks around `kubectl` (cheatsheet will be provided)

 

**Section-4**

* Tools, and techniques beyond manual exploitation and analysis

  * KubeAudit, KubeSec, popeye, trivy, dockle, rakkess, linters, and many others...

* Performing Docker & K8S CIS benchmarks to find all the possible security risks

* Auditing the cluster security posture from Code to Production running cluster

* Real-World case studies of Kubernetes Hacking, Vulnerabilities, and Exploits

* Best practices, Recommendations based on the Security Maturity

* Resources & references to further your attacks, exploitation, more learning

 

 

HOUR BY HOUR OUTLINE:  

 

**Section-1**

* Kubernetes 101 - Fasttrack Edition

 

This is a quick overview and fast-track way of teaching about What, Why, and How Kubernetes works. It is like a refresher and warmup for hackers before diving into real hacking. We focus on teaching the k8s architecture, and components (nodes, master, apiserver, controllers, scheduler, etcd, kubelet, kube-proxy, pod, services, etc.)

 

* Security Architecture Review & Attack Maps

 

Here, we will focus on understanding the attack surface and entry points at the architectural level. Also, we will be creating the attack tree maps using MITRE ATT&CK framework to perform the recon, exploitation, and beyond gaining access to cluster admin.

 

* `kubectl` kung-fu to explore the cluster

 

All participants will learn how to use the imperative commands, which is very important to learn while performing exploitation within the cluster. Also, practicing the popular useful commands like `kubectl auth can-i` and many other built-in commands to leverage the power of kubectl.

 

* Attacking the supply chain by exploiting private registry

 

In this scenario, we see an insecure private registry and a typical DevOps workflow and CI/CD systems. Where attackers use the built-in privileges of CI/CD system to gain access to the private registry and can backdoor the container images which are deployed in production systems and it can even sometimes in developer laptops in real-world.

 

* Pwning the container images and gaining access to the cluster

 

Once we got the private registry access, participants will backdoor the container image with a vulnerability to gain access to the cluster. Once the image is deployed in the cluster, the attacker ideally gains access to the cluster and has access to the Pod (least default unit in Kubernetes Cluster). From here we will see what else an attacker can do from a container within Kubernetes clusters.

 

* Exploiting security misconfigurations in the cluster

 

As we have bare minimum access to the cluster, attackers will see all the misconfigured services and resources to get more understanding of the cluster. This is like a recon phase to understand more overview of the cluster and its security posture.

 

**Section-2**

* Escaping out of the container to the host system to gain more privileges

 

In this scenario, based on the identified vulnerabilities within the cluster. The attacker uses container socket/API misconfiguration to gain access to the host system from the container by escaping outside the container. Now it's time to explore more attacks and exploitation at the host/node level to gain access to the complete cluster.

 

* Bypassing NSP and gaining unauthorized access to other microservices

 

From the recon, the attacker identifies that there is NSP (network security policies) applied in the cluster. By default, Kubernetes follows flat networking schema so hence any pod/service can talk to any pod/service within the cluster. By using this flaw, we will be gaining access to other microservices and system services which have authentication/authorization outside the cluster (due to the termination of AuthZ/AuthN at API Gateway or ingress level). For example redis, elasticsearch, application services, etc.

 

* Lateral movement from container to node and then complete cluster access

 

Now we have pretty great coverage of access like containers, services, etc. So from the previous container escape, we are at the Node level, and due to the lack of PSP (Pod Security Policies), we will be using the node service account/config to gain access to other nodes within the cluster. In the future PSP's are deprecating so we might focus on OPA(Open Policy Agent) with Admission Controller

 

* Escalating from ServiceAccount to more RBAC privileges (No least privileges)

 

Due to the complication of RBAC (Role-Based Access Controls) in Kubernetes clusters, most DevOps teams don't configure the least privileges. As an attacker, we leverage the built-in default service account attached to the pod/container to gain more privileges and access other namespaces/resources within the cluster.

 

* Helm with Tiller service = ClusterPwn (Complete cluster takeover)

 

This is one of the wildest attack entry points for Kubernetes where teams running Helm V2, We have personally exploited this in pentesting engagements in almost all the deployments we got with helm installed to date. By default, tiller service is given with Cluster-Admin access, and also it binds to 0.0.0.0 network with no NSP the attacker uses the helm chart to gain the complete cluster access and becomes cluster-admin here. If the objective of the security engagement is cluster access then it's game over here.

 

* Gaining access to k8s volumes, logs of the services, and sensitive data

 

We will be using the obtained access and privileges to gain access to other database volumes, logging resources, and even sensitive data hosted and accessed by the services. If the objective of the security engagement is data then it's game over here.

 

* From application vulnerability to cloud provider access (attack chain)

 

This is a specially crafted scenario from our real-world pentesting and assessments learning where an attacker uses a simple application vulnerability like SSRF or RCE to gain container access then pod, node, and then cluster and finally get access to the Cloud Provider.

 

**Section-3**

* Hacker Container - The Swiss Army knife for hacking Kubernetes Clusters

 

Hacker Container is specifically designed for hacking Kubernetes Clusters by myself based on real-world pen-testing engagements and while hacking many Kubernetes Cluster environments. So this scenario will showcase how to utilize Hacker Container and gain maximum benefit and power from it while performing assessments with examples, enumeration, and exploitation.

 

* Exploiting Kubernetes Secrets and gaining access to third-party services

 

In this scenario, participants will be using the obtained Kubernetes Secrets (by default these are base64 encoded and not encrypted). Still, most of the teams store the secrets in clusters. So here attackers use those credentials to access third-party resources like Sentry (A logging and Tracing system), Monitoring services, and many others.

 

* DoS the services and cluster nodes by resources exemption

 

This scenario can go as complex as based on the setup of the cluster environment. We will be seeing at different levels like App DoS, consuming the container/pod/resources limits. Even if the cluster is built using autoscaling (Most of the clusters in the cloud are designed in such a way) then we even go beyond and create more instances by consuming the resources.

 

* Understanding Admission controller and possible attack surface around Webhooks

 

The admission controller is like the final gate for performing any execution of the API call we made to the Kubernetes cluster. So, the security team in CNCF working towards moving the checks and validation to MutationWebhook and ValidationWebhook in the admission controller. In this scenario, we will be identifying the possible security risks and understanding how the Admission Controller works for further learning and exploitation.

 

* Persisting in the clusters using Sidecar/Cronjob/DaemonSets

 

This scenario is to persist as an attacker within the cluster even if the Container, Pod, and even when the Node is down. We will be using the built-in cluster resources like Cron Job, SideCar, and DaemonSets to create the persistence of the attacker to gain access to the cluster even when there is the deletion of the cluster resources.

 

* Defense evasion techniques for Kubernetes Cluster environments

 

In this scenario, we purely focus on different types of defense evasion techniques for the Kubernetes Cluster attack detection mechanism by performing logging disruption at different levels (Container, Pod, Node, Cluster). Also creating resources like a cluster-admin or other team members in different namespaces with pre-built services naming to avoid detecting by Blue teams and Defenders.

 

* Some useful hacks around `kubectl` (cheatsheet will be provided)

 

This scenario is to showcase some powerful features of kubectl in the live from the attacker's perspective like the flags of verbosity (which gives even HTTP request and response level details). Debugging the resources and services within the cluster, and many other activities which are like hidden gems :)

 

**Section-4**

* Tools, and techniques beyond manual exploitation and analysis

  * KubeAudit, KubeSec, k9s, trivy, dockle, rakkess, linters, and many others...

 

This is quite an overview of different available opensource utilities which I maintain at https://tools.tldr.run/ and helps in the automated fashion of performing information gathering, identifying vulnerabilities, exploiting them, and escalating to the next level. Most of these tools don't give them confidence but it provides an automated way to perform some mundane tasks.

 

* Performing Docker & K8S CIS benchmarks to find all the possible security risks

 

In this scenario, we will be using both Docker and Kubernetes CIS benchmarks to perform the complete cluster security risks and to identify all the possible security risks. This also helps them to get more visibility if something is missed or even if it's not a best practice.

 

* Auditing the cluster security posture from Code to Production running cluster

 

This scenario is focusing the end-to-end lifecycle of the workflow, we will be auditing the cluster from the developer writing the code to run that in the production cluster. We will see layers of the attack surface and use different utilities to identify the security risks as early as possible.

 

* Real-World case studies of Kubernetes Hacking, Vulnerabilities, and Exploits

 

This scenario showcases all the real-world security issues we identified and publicly published, known vulnerabilities and exploitation around containers and Kubernetes environments, and how they mapped to the overall training they learned and practiced over time.

 

* Best practices, Recommendations based on the Security Maturity

 

This scenario covers different best practices, recommendations for securing the cluster environments when performing the security assessments, and red teaming exercises. Sometimes these also help attackers to gain more understanding of the defense and go beyond these best practices to find more loopholes :)

 

* Resources & references to further your attacks, exploitation, more learning

 

As we all know the time of training is just not enough to learn everything. So this scenario we cover what are the resources and references to learn further to sharpen their skills to learn more about Kubernetes Security and other cloud-native infrastructure. We will provide a step by step digital guidebook as well and are always happy to help and build the participant's knowledge

 

 

Technical difficulty of the class (Beginner, Intermediate, Advanced) and any required experience or skills needed (Such as Python, knowledge of specific deep-learning algorithms, TCP dump analysis, Ghidra, etc.)  

 

Intermediate

  • Able to use Linux CLI
  • Basic understanding of system administration
  • Experience with Docker and Containers ecosystem would be useful
  • Security Experience would be plus

 

Suggested prerequisites for the class. What should the student have read or prepared in advance to get the most out of your class? This could be videos to watch, books or white papers to read, etc.  

 

My DEFCON 26 workshop on Attacking & Auditing Docker Containers Using Open Source tools and its video available at https://www.youtube.com/watch?v=ru7GicI5iyI

 

 

Items students will need to provide. What tools, systems, or equipment is required for the student to take the training?  

 

Students will need a laptop with Wi-Fi capability.

 

I will be providing students with

  • Custom built Kubernetes Cluster environment (everyone gets their own)
  • Step by Step Digital Guide book for the entire training
  • Kubectl cheatsheet, Checklist of tools, and other resources

 

DATE: April 13th-14th 2023

TIME: 8am to 5pm PDT
VENUE: Meydenbauer Center
TRAINER: Madhu Akula

- 16 hours of training with a certificate of completion.
- Note: Classes that do not meet their minimum class size by July 15 will be canceled, please register early

- 2 coffee breaks are provided per day
- Note: Food is not included.

 

Registration terms and conditions:

Trainings are refundable before March 1st, the processing fee is $250.

Trainings are non-refundable after March 10th, 2023.

Training tickets may be transferred. Please email us for specifics.

Failure to attend the Training without prior written notification, will be considered a No-Show. No refund will be given.

$2,250.00