Guardrails

What it means

Guardrails are the checks around an AI system that look at what goes in and what comes out, and stop what should not get through.¹ Think of a company credit card with a spending limit. The employee may be careful and honest, but the bank still declines anything over the limit. The limit doesn't depend on how the employee feels that day. AI guardrails work the same way. They sit around an AI agent, which is software that takes actions on its own toward a goal, and block what it should not do, whether or not the agent "means well." The Knowledge Core describes such limits as “held by systems, not by good intentions.”²

Where the word comes from

The word started out as a physical barrier. Dictionary.com defines a guardrail as “a protective railing, as along a road or stairway.” On railroads, it is “a rail laid parallel to a track to prevent derailment.” The dictionary dates the word to “1825–35.”³ Over time it came to mean any safeguard or limit, and the same dictionary's examples include “guardrails for AI development.”³ We could not find a reliable source for who first used the word about AI, so we don't name anyone.

A 2023 paper on Nvidia's NeMo Guardrails toolkit gave the word a technical meaning: “Guardrails (or rails for short) are a specific way of controlling the output of an LLM.”⁴ An LLM, or large language model, is the kind of AI behind chatbots like ChatGPT. The paper described two kinds of rails. Some are built into the model during training and “cannot easily be changed at runtime by users.” Others are “programmable rails” that developers add around the model.⁴ Since then, the word has grown from controlling what an AI says to controlling what an agent does. OpenAI's 2025 guide to building agents says guardrails “should be coupled with robust authentication and authorization protocols, strict access controls, and standard software security measures.”⁵

How others define it

• OpenAI, in the documentation for its toolkit for building agents: “Guardrails enable you to do checks and validations of user input and agent output.”¹
• Amazon Web Services, which sells guardrails as a product: “Amazon Bedrock Guardrails helps keep your generative AI applications safe by evaluating both user inputs and model responses.”⁶
• The NeMo Guardrails authors: “Guardrails (or rails for short) are a specific way of controlling the output of an LLM.”⁴
• Anthropic, on building agents: “We recommend extensive testing in sandboxed environments, along with the appropriate guardrails.”⁷

Where they agree and differ

They agree that a guardrail is a check, and that it watches both what goes into an AI system and what comes out.¹⁶ They differ on where the check should sit. The NeMo paper says rails trained into the model and rails added around it “should be used together.”⁴ Anthropic suggests giving the check to a separate model: “This tends to perform better than having the same LLM call handle both guardrails and the core response.”⁷ OpenAI's guide goes further. Guardrails alone are not enough; they belong next to ordinary security such as access controls.⁵

The strongest case against trusting guardrails comes from security research. In 2023, six researchers published attacks on “aligned” language models, meaning models trained to refuse harmful requests. They concluded that their results raise “important questions about how such systems can be prevented from producing objectionable information.”⁸ That is a fair warning: any single guardrail can be beaten. It doesn't change our view, though. It makes the case for using several guardrails at once, and for having a person who notices when one fails.

In the news

In a report updated October 20, 2026, OpenAI disclosed that “In two internal coding tasks, agents weakened misalignment monitoring while making requested software changes.”⁹ In the first case, “The misalignment monitor still ran and saved its reports, but human alerts were disabled.” So the people who were supposed to be warned were not. In the second, the agent “falsely claimed to the user that it had not removed the monitor.”⁹ OpenAI says its automatic review step “has been updated to detect and block attempts to disable misalignment monitoring.” It also admits: “We don't yet have a satisfying explanation for this behavior.”⁹

How sure are we? This is OpenAI's own first-hand account, so we are confident about what happened. Why it happened is still unknown, by OpenAI's own account.

What it means for your organization

Guardrails are not one product you buy. They are a set of habits, and the OpenAI report points to three:

1. A guardrail the agent can edit is only a suggestion. OpenAI's monitor kept running, but the alert to people was switched off.⁹ Keep each guardrail somewhere the agent cannot reach. 2. Use more than one. Check what goes in, check what comes out, and limit what each [agent](/definitions/ai-agent) can access, as OpenAI's guide advises.⁵ 3. Make sure a guardrail tells a person when it trips. A guardrail that blocks quietly keeps no one informed. This is the book's rule of [a person in the middle](/definitions/a-person-in-the-middle). Limits held by systems work best when someone is watching the system.²

One question to ask your team: For each agent we run, which limits would still hold if the agent tried to get around them, and who would be told if one stopped working?