Red teaming

What it means

Red teaming means you ask people to attack your own system on purpose, under rules you set, so you find its weak spots before someone else does. U.S. national security rules describe a red team as "a group of people authorized and organized to emulate a potential adversary's attack or exploitation capabilities against an enterprise's security posture."¹ The same definition says the goal is to improve security "by demonstrating the impacts of successful attacks."¹

Here is an everyday example. Before a company lets an AI assistant answer customer emails, it asks a small group to try to fool the assistant: to get it to reveal private data, break its rules, or follow instructions hidden in an incoming message. Each trick that works shows the company something to fix before the assistant goes live.

Where the word comes from

The idea is older than computers. A Carnegie Mellon research team writes that earlier scholars "describe how the key concepts of red-teaming originated hundreds of years ago in warfare and religious contexts," and that "the term 'red team' was formally applied by the US military as early as the 1960s."² In that military use, the red team played the opponent and the blue team played the home side. The national security definition still pairs the red team with "the defenders (i.e., the Blue Team)."¹

The word then moved into computer security, and more recently into AI. In AI, it covers more than break-ins. It also covers getting a system to produce harmful output or behave in ways nobody intended. A 2023 U.S. executive order defined "AI red-teaming" as "a structured testing effort to find flaws and vulnerabilities in an AI system."³ That order was revoked on January 20, 2025,⁴ but the definition is still quoted. For example, OpenAI's paper on its own practice uses it.⁵

How others define it

• U.S. Committee on National Security Systems (through NIST's glossary): "A group of people authorized and organized to emulate a potential adversary's attack or exploitation capabilities against an enterprise's security posture."¹
• U.S. Executive Order 14110 (2023, since revoked): "a structured testing effort to find flaws and vulnerabilities in an AI system."³
• OpenAI researchers: "a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment."⁵
• EU AI Act: The law does not use the words "red teaming." Instead, it requires providers of the most powerful general-purpose AI models to carry out "conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks."⁶

Where they agree and differ

All of these definitions agree on the core idea: you test on purpose, with permission and structure, and you think like an opponent. They differ in what they test for. The security definition is about an attacker getting past an organization's defenses.¹ The AI definitions talk more broadly about "flaws and vulnerabilities in an AI system."³

The strongest critique comes from the Carnegie Mellon team. They argue that treating red teaming "as a panacea for every possible risk" can "verge on security theater."² OpenAI's own researchers agree on the limits. They write that "Red teaming on its own is not a panacea for risk assessment," and that risks found at one point in time "may be under-assessed or no longer reflected in an updated system."⁵ We think this critique is right, and it does not reduce the value of red teaming. It means red teaming is one test among several, and its results go out of date as the system changes.

In the news

In October 2026, Fortune reported that Gray Swan was overhauling its public red-teaming platform "to keep it relevant for the era of AI agents."⁷ The platform is described as "an online platform where participants compete for prize money by trying to trick AI models into violating their guardrails," "with $300,000 in prizes sponsored by Anthropic, Meta, Google DeepMind, and Reflection AI."⁷ One kind of attack in the contest is where "an attacker hides instructions in content an AI agent will come across while doing a task."⁷

In June 2026, Microsoft's AI Red Team reported what a year of testing AI agents had taught it. It found that "Cross-domain prompt injection delivered via external content remained the most reliable initial access vector."⁸ Prompt injection means slipping instructions to an AI inside content it reads. The team also reported that "HitL bypass was the most consistently exploited failure mode, at very high frequency"; HitL means "human in the loop," the step where a person must approve an action.⁸ In plain terms, the most common successful attacks got agents to take orders from someone other than the person they work for, or to skip the person who was supposed to sign off.

What it means for your organization

If your organization uses AI agents, the people who build or sell them have probably red-teamed them. The question is whether anyone has tested them in your setting, with your data and your approval steps. Microsoft's findings point to two places to test first: content from outside that an agent reads, and the steps where a person is supposed to approve an action.⁸ The limits noted above also apply: a test done once can go out of date when the system changes.⁵

Question to ask your team: When did someone last try, on purpose, to get one of our AI agents to follow instructions from an outsider or to skip a human approval, and what did we change afterward?