- 01AI red teaming is now a regulatory expectation (EU AI Act, NIST AI RMF, OMB AI memo). Without methodology discipline it degrades into a stream of anecdotes that cannot be compared between releases.
- 02Five layers make red-team output reproducible: a versioned threat catalog (mapped to OWASP LLM + MITRE ATLAS), a versioned test corpus, an execution harness, a scoring rubric, and comparable reporting.
- 03Automation owns coverage and regression. Humans own creative novel-attack discovery. Both deliverables are required; substituting one for the other gives you an unjustified false sense of progress.
- 04What to evaluate, minimum: prompt injection (direct + indirect), data exfiltration, system-prompt extraction, tool abuse and chained calls, output-driven injection (XSS / SQLi via model), agent scope escape, and policy-evasion (CBRN / illegal advice / etc., domain-appropriate).
AI red teaming is now an explicit regulatory expectation (EU AI Act, NIST AI RMF, the OMB AI memo). Without methodology discipline, the work degrades into a stream of anecdotes that cannot be compared between releases or between teams.
The methodology in five layers
- 01Threat catalog — versioned list of threats, mapped to OWASP LLM and MITRE ATLAS
- 02Test corpus — versioned, expanding library of attack prompts and scenarios
- 03Execution harness — automated where possible, human-driven where needed
- 04Scoring rubric — explicit criteria for severity and exploitability
- 05Reporting — comparable scores between model versions and over time
What to evaluate
Prompt injection (direct and indirect), data exfiltration via prompt manipulation, system prompt extraction, tool abuse including chained tool calls, output-driven injection (XSS, SQLi via model output), agent escape from intended scope, and policy evasion (CBRN, CSAM, illegal advice — appropriate to your domain).
Where automation helps and where it does not
Automation excels at coverage and regression. It is poor at creative novel-attack discovery. Use automated harnesses for known-attack regression and reserve human red-team time for the next-generation attacks. Both deliverables are necessary.