AI SecureOps: Attacking & Defending AI Applications & Agents - DCSG2026

Name of Training: AI SecureOps: Attacking & Defending AI Applications & Agents
Trainer(s): Abhinav Singh
Dates: April 26-27, 2025
Time: TBD
Venue: Marina Bay Sands
Cost: $2,000

Short Summary:

Step into the frontline of enterprise AI security with an immersive, CTF-driven training designed to build real-world expertise in protecting AI systems. Explore the full spectrum of AI security, from offensive to defensive strategies. Master techniques to detect, exploit, and defend against AI and agentic threats, while operationalizing Responsible AI across the enterprise.

Course Description:

Can prompt injections lead to complete infrastructure takeovers? Could AI agents be exploited to compromise backend services? Can data poisoning in AI copilots impact a company's stock? Can jailbreaks create false crisis alerts in security systems? This immersive, CTF-styled training in GenAI and LLM security dives into these pressing questions. Engage in realistic attack and defense scenarios focused on real-world threats, from prompt injection and remote code execution to backend compromise. Tackle hands-on challenges with actual AI applications & agentic systems to understand vulnerabilities and develop robust defenses. You’ll learn how to create a comprehensive security pipeline, mastering AI red and blue team strategies, building resilient defenses for AI apps & agents, and handling incident response for AI-based threats. Additionally, implement a Responsible AI (RAI) program to  enforce ethical AI standards across enterprise services, fortifying your organization’s AI security foundation.

By 2026, Gartner, Inc. predicts that over 80% of enterprises will engage with AI models, up from less than 5% in 2023. This rapid adoption presents a new challenge for security professionals. To bring you up to speed from an intermediate to an advanced level, this training provides essential GenAI and LLM security skills through an immersive CTF-styled framework. Delve into sophisticated techniques for mitigating LLM threats, engineering robust defense mechanisms, and operationalizing LLM agents. Prepare to address the complex security challenges posed by the rapid expansion of AI technologies. You will be provided with access to a live playground with custom-built AI applications that replicate real-world attack scenarios. These cover use-cases defined under the OWASP LLM top 10 framework and are mapped with stages defined in MITRE ATLAS. This dense training will navigate you through areas such as red and blue team strategies, creating robust LLM defenses, incident response in LLM attacks, implementing a Responsible AI (RAI) program, and enforcing ethical AI standards across enterprise services. The focus is on improving the entire AI supply chain.

This training will also cover the completely new segments of Responsible AI (RAI), ethics, and trustworthiness in GenAI services. Unlike traditional cybersecurity verticals, these unique challenges such as bias detection, managing risky behaviors, and implementing mechanisms for tracking information are going to be the key challenges for enterprise security teams.

By the end of this training, you will be able to: 

• Exploit vulnerabilities in AI applications to achieve code and command execution, uncovering scenarios such as instruction injections, agent control bypass, remote code execution for infrastructure takeover as well as chaining multiple agents for goal hijacking. 

• Conduct AI red-teaming using adversary simulation, OWASP LLM Top 10, and MITRE ATLAS frameworks, while applying AI security and ethical principles in real-world scenarios. 

• Execute and defend against adversarial attacks, including prompt injection, jailbreaks, data poisoning, and agentic attacks. 

• Perform advanced AI red and blue teaming through multi-agent auto-prompting attacks, implementing a 3-way autonomous system consisting of attack, defend and judge models. 

• Develop LLM security scanners to detect and protect against injections, jailbreaks, manipulations, and risky behaviors, as well as defending LLMs with LLMs. 

• Build and deploy enterprise-grade LLM defenses, including custom guardrails for input/output protection, security benchmarking, and penetration testing of LLM agents. 

• Establish a comprehensive LLM SecOps process to secure the supply chain from adversarial attacks and create a robust threat model for enterprise applications. 

• Implement an incident response and risk management plan for enterprises developing or using GenAI services. 

Course Outline: 

### Introduction (1 lab)

• Introduction to LLM and GenAI  
• LLM & GenAI terminologies and architecture  
• Technology use-cases  
• Agents, multi-agents and multi-modal models  

### Elements of AI Security (1 lab)

• Understanding AI vulnerabilities with case studies on AI security breaches  
• Principles of AI ethics and safety  
• OWASP LLM Top 10 and MITRE mapping of attacks on AI supply chain  
• Threat modeling of agentic flows and RAG architectures  

### Adversarial LLM Attacks and Defenses (6 labs)

• Direct and indirect prompt injection attacks and their subtypes  
• Advanced prompt injections through obfuscation and cross-model injections  
• Breaking system prompts and their trust criteria  
• Advanced LLM red teaming: automating multi-agent conversations to prompt-inject models at scale  
• Indirect prompt injections through external input sources  
• Attack mapping through OWASP LLM Top 10 and MITRE ATLAS frameworks  

### Attacking & Defending Agentic Systems (5 labs)

• Attacking LLM agents for task manipulation and risky behavior  
• Cross-site scripting and injection attacks on AI agents for code and command execution  
• Abusing agent APIs for model extraction and data poisoning  
• Compromising backend infrastructure by abusing over-permissioning and tool usage in agentic systems  
• Defense implementation through tracing and observability  

### Responsible AI & Jailbreaking (6 labs)

• Elements of ethics and alignment in AI models  
• Jailbreaking public LLMs covering adversarial AI, offensive security, and CBRN use-cases  
• Responsible AI frameworks and benchmarks  
• Model alignment, system prompt optimization, and defense  

### AI Red & Blue Teaming (4 labs)

• Automated prompt injection and jailbreak at scale  
• Using Colab notebooks for automation of API calls and reporting  
• Jailbreak attacks and model-weight tracing for root-cause investigation  
• Implementing “LLM Judge” model to auto-evaluate attacks and refine the next stage with increasing complexity  
• Purple teaming through a 3-way LLM implementation: target, attacker, and judge  

### Building Enterprise-grade LLM Defenses (3 labs)

• Deploying LLM security scanner, adding custom rules, prompt block lists, and guardrails  
• Writing custom detection logic, trustworthiness checks, and filters  
• LLM Guard for protecting input and output  
• Building security log monitoring and alerting for models using open-source tools 
• LLM security benchmarking and continuous reporting  

### Building LLM & GenAI SecOps Process

• Summarizing the learnings into a SecOps workflow  
• Monitoring trustworthiness and safety of enterprise LLM applications  
• Implementing NIST AI Risk Management Framework (RMF) for security monitoring  

Difficulty Level:

Intermediate

Suggested Prerequisites:

Complete the simple pre-training instructions, which involve creating a paid OpenAI API key, setting up a Google Colab notebook, and reading the Introduction document. No local setup is needed. All the training materials and access to labs will be provided during the training.

What Students Should Bring:

A laptop with browser access is ideal, preferably a personal laptop without network restricting tools.

Complete the pre-training setup prior to the class which includes setting up:

• API key for OpenAI.
• Google Colab account.
• Complete the pre-training setup before the first day.

What the Trainer Will Provide:

• One year access to a live interactive playground with various exercises to practice different attack and defense scenarios for GenAI and LLM applications.
• "AI SecureOps" Metal coin for CTF players.
• Complete course guide containing 200+ pages in PDF format. It will contain step-by-step guidelines for all exercises and labs, and a detailed explanation of concepts discussed during the training.
• PDF versions of the slides that will be used during the training.
• Access to the Discord server for continued engagement, support, and development in the field of AI Security & Safety.
• Access to HuggingFace models, datasets, and transformers.

Trainer(s) Bio:

Abhinav Singh is an esteemed cybersecurity leader & researcher with over 15 years of experience across technology leaders and financial institutions, as well as an independent trainer and consultant. Author of "Metasploit Penetration Testing Cookbook" and "Instant Wireshark Starter," his contributions span patents, open-source tools, and numerous publications. Recognized in security portals and digital platforms, Abhinav is a sought-after speaker & trainer at international conferences like Black Hat, RSA, DEFCON, BruCon, and many more, where he shares his deep industry insights and innovative approaches in cybersecurity. He also leads multiple AI security groups at CSA, responsible for coming up with cutting-edge white papers and industry reports on the safety and security of AI.

Review a few examples of Abhinav's previous courses at the links below:

• Insomni’hack, Switzerland: Insomni’hack
  
• DEFCON, Las Vegas, USA: DEFCON
  
• Nsec, Montreal, Canada: Nsec
  
• LASCON, Austin, USA: LASCON
  
• OWASP AppSec, Auckland, New Zealand: OWASP AppSec

Proficiency Exam Option:

This course has the option for a proficiency certificate add-on. To earn the proficiency certificate, students will have to score at least 1400 out of 2400 on the course capture the flag (CTF). Only students who purchase the proficiency certificate will have their work evaluated by the instructor to certify mastery of the course material.

Please reach out to training@defcon.org for any questions related to the proficiency exam and certificate option.

Registration Terms and Conditions: 

Trainings are refundable before March 27, 2026, minus a non-refundable processing fee of $250.

Trainings are non-refundable after March 27, 2026.

Training tickets may be transferred. Please email us at training@defcon.org for specifics.

If a training does not reach the minimum registration requirement, it may be cancelled. In the event the training you choose is cancelled, you will be provided the option of receiving a full refund or transferring to another training (subject to availability).

Failure to attend the training without prior written notification, will be considered a no-show. No refund will be given.

By purchasing this ticket you agree to abide by the DEF CON Training Code of Conduct and the registration terms and conditions listed above.

Several breaks will be included throughout the day. Please note that food is not included.

All courses come with a certificate of completion, contingent upon attendance at all course sessions.