Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more
Editor’s note: Louis will lead an editorial roundtable on this topic at VB Transform this month. Register today.
AI models are under siege. With 77% of enterprises already hit by adversarial model attacks and 41% of those attacks exploiting prompt injections and data poisoning, attackers’ tradecraft is outpacing existing cyber defenses.
To reverse this trend, it’s critical to rethink how security is integrated into the models being built today. DevOps teams need to shift from taking a reactive defense to continuous adversarial testing at every step.
Red Teaming needs to be the core
Protecting large language models (LLMs) across DevOps cycles requires red teaming as a core component of the model-creation process. Rather than treating security as a final hurdle, which is typical in web app pipelines, continuous adversarial testing needs to be integrated into every phase of the Software Development Life Cycle (SDLC).
Gartner’s Hype Cycle emphasizes the rising importance of continuous threat exposure management (CTEM), underscoring why red teaming must integrate fully into the DevSecOps lifecycle. Source: Gartner, Hype Cycle for Security Operations, 2024
Adopting a more integrative approach to DevSecOps fundamentals is becoming necessary to mitigate the growing risks of prompt injections, data poisoning and the exposure of sensitive data. Severe attacks like these are becoming more prevalent, occurring from model design through deployment, making ongoing monitoring essential.
Microsoft’s recent guidance on planning red teaming for large language models (LLMs) and their applications provides a valuable methodology for starting an integrated process. NIST’s AI Risk Management Framework reinforces this, emphasizing the need for a more proactive, lifecycle-long approach to adversarial testing and risk mitigation. Microsoft’s recent red teaming of over 100 generative AI products underscores the need to integrate automated threat detection with expert oversight throughout model development.
As regulatory frameworks, such as the EU’s AI Act, mandate rigorous adversarial testing, integrating continuous red teaming ensures compliance and enhanced security.
OpenAI’sapproach to red teaming integrates external red teaming from early design through deployment, confirming that consistent, preemptive security testing is crucial to the success of LLM development.
Gartner’s framework shows the structured maturity path for red teaming, from foundational to advanced exercises, essential for systematically strengthening AI model defenses. Source: Gartner, Improve Cyber Resilience by Conducting Red Team Exercises
Why traditional cyber defenses fail against AI
Traditional, longstanding cybersecurity approaches fall short against AI-driven threats because they are fundamentally different from conventional attacks. As adversaries’ tradecraft surpasses traditional approaches, new techniques for red teaming are necessary. Here’s a sample of the many types of tradecraft specifically built to attack AI models throughout …