Guardrails and Governance for AI Agents: Definition and Mechanisms

Giving an AI agent the power to act on its own, on its own tools, immediately raises a question of trust: who decides what it's allowed to do, and how do you verify afterward that it didn't overstep that boundary? Guardrails and governance answer this question together, one at the technical level of each action, the other at the organizational level of the whole system.

Detailed definition

Guardrails are the concrete mechanisms that limit what an AI agent is allowed to do: precise per-action permissions, restricted access scope, quotas, human validation checkpoints on sensitive decisions. Governance is the broader framework within which these guardrails are designed, decided, and audited: who has the right to configure them, how risks get documented, how you verify after the fact that the system behaved as expected.

Two references frame the discussion on AI governance today, with different logics. NIST's AI Risk Management Framework (AI RMF 1.0), published in January 2023 in the United States, is a voluntary framework structured around four functions: Govern (the cross-cutting function that defines culture and responsibilities), Map (mapping the risks of a given system), Measure (measuring those risks), and Manage (managing them). The European Union's AI Act, by contrast, is a binding regulatory text: for systems classified as high-risk, it mandates a risk management system (Article 8), governance of training data (Article 10), technical documentation (Article 11), automatic event logging (Article 12), transparency toward users, and effective human oversight (Articles 13 and 14). The main obligations for high-risk systems apply starting December 2027 for systems covered by Annex III, and August 2028 for those covered by Annex I.

What the two frameworks share, despite their different nature (voluntary versus binding), is an insistence on traceability (logging what happens) and on human oversight as non-negotiable components of serious AI governance, regardless of jurisdiction.

How it works

An effective guardrail system operates at several levels, from the broadest to the most precise. At the broadest level, you decide which tools are connected to a system and which categories of actions are even conceivable. At the intermediate level, you define who, human or agent, is allowed to use which tool, with what scope (read-only, or read and write). At the most precise level, every individual action is checked at the moment it's requested: does the permission exist, does it actually cover this specific action, with valid arguments.

The soundest principle for building these guardrails is deny-by-default: nothing is allowed until an explicit permission has been granted, rather than starting from broad access and restricting it case by case afterward. It's more demanding to set up, but it avoids the most common security mistake, forgetting a restriction rather than forgetting an authorization.

Governance, for its part, adds a layer of accountability and verifiability: who configured a given guardrail, when, and whether an audit can reconstruct after the fact what actually happened if a question comes up.

A concrete example from Atako

At Atako, the permission model rests exactly on this deny-by-default principle. Connecting a tool (Slack, GitHub, HubSpot, or another integration) at the company level grants no agent any access until an explicit "grant" has been created. A grant links a specific agent to a specific connection, with a list of specific authorized actions (not generic access to all of GitHub, but for example only list_issues and create_issue), a scope (read-only or read-write), and an optional expiration. Even if a write action were mistakenly added to a read-only grant's list, the scope would still block its execution: it's a double check.

The decision path Atako documents follows this pattern: the agent expresses an intent to act, the platform verifies that a grant exists, that the action is on the authorized list, that the scope is sufficient, that the arguments are valid, and only then executes the call with the third-party provider and returns the result to the agent, never the access secret itself. Every failed step produces a logged denial, which feeds directly into the agent's observability.

On the revocation side, cutting off access is immediate and final: revoking a connection deletes the encrypted secret on the spot, with no grace period, and every agent that depended on it instantly loses access. It's a last-resort guardrail, built to act fast in case of doubt.

Common mistakes

A common mistake is believing that granting access to a tool is the same as giving full access to that tool. A good governance system always distinguishes access to a service (the connection) from authorization to act on it (the grant, with its specific actions and scope).

Second mistake: thinking AI governance boils down to regulatory paperwork with no practical effect. Whether through a voluntary framework like NIST's or a binding text like the EU's AI Act, the same concrete requirements come back: document risks, log actions, keep a human in the loop on sensitive decisions. These are operational mechanisms, not formalities.

Third mistake: confusing technical guardrails with human validation. Permissions and quotas apply automatically, with no human intervention every time. Human validation is a different guardrail, reserved for actions whose risk justifies deliberately slowing the process down so a person can look before it goes out.

Finally, underestimating the need for an accessible audit trail is a classic mistake. Guardrails that correctly block bad actions are useful, but without a browsable history of what was authorized, denied, or executed, it becomes impossible to calmly answer the question a customer or regulator will eventually ask: what exactly did this agent do, and why.