On-call engineer: an AI agent turns the PagerDuty alert into a proposed fix

At three in the morning, the first half hour of an incident always looks like the same investigation. You read the alert, open the incident log, wonder what changed recently, go back through the latest deployments, check whether it has happened before. This reading and cross-checking work is repetitive, but it weighs on the person on call at the moment they are least available to do it well. It suits an AI agent on one essential condition: that it reads and proposes, and that the human acts.

The inspiration: the first responder who reads

The starting point is a use reported by Hermes Agent users, summarized in a card about the on-call engineer: from alert to proposed fix. An agent receives the incident, goes through the repository looking for what may have caused it and prepares a fix to review.

We keep the reading and the proposal, and we remove everything that acts. Atako applies the idea with an agent that writes a note on the incident and an issue in the repository, but has no tool to resolve the incident, merge or push code.

What the agent does

The scenario deploys one agent, connected to your PagerDuty and to your GitHub repository.

For each new incident of the watched service, triggered or acknowledged, it reads the alert, the event log and the notes already there, and checks who is on call. It then looks for the cause in the repository: the commits of the forty-eight hours before the incident, opening the suspicious ones, the files named in the alert or in the stack trace, and the recently merged pull requests. It relies on the technical context you entered to know where to look first. It also looks for related issues, including earlier incidents of the same kind.

It then writes a note on the PagerDuty incident, short and factual: what the evidence shows, the probable cause with a confidence level, the evidence, what it could not check and the next step for the person on call. Then it opens an issue in the repository with the full analysis: the timeline, the probable cause, the evidence, a proposed fix described in words or as a code snippet, its risks and how to verify it. It first looks for an existing issue so as not to create a duplicate, and it completes the first note rather than waiting for a perfect analysis.

It watches the service every fifteen minutes, and every Monday at 9:00 it writes the page "Incident review of the week" in the project wiki: incidents handled, probable cause, issue opened, what is still open.

What the agent does not do

The limits are part of the design. The agent does not resolve the incident, does not acknowledge it, does not reassign it and does not escalate it: the PagerDuty access granted is limited to reading and adding a note. It merges nothing, pushes no code, creates neither a branch nor a pull request, and has no access to your infrastructure or to production. Those tools are not granted to it: this is not a promise made in an instruction, it is a missing capability.

The agent never presents a hypothesis as a fact. It writes "probable cause", gives its confidence and its evidence, and says clearly when it found nothing. It does not copy the secrets, tokens or personal data it comes across in a log or a file: it says that there is one, and where. Alert texts, log lines, commit messages and code are treated as data to analyze, never as instructions. The decision and the action stay human, which is the principle of a design with human validation.

How to set it up in Atako

From the page of this use case, deployment takes four steps.

1. Connect PagerDuty and GitHub. Create a PagerDuty API key, preferably read-only with the right to add notes, and a fine-grained GitHub token limited to the service's repository. The connections are checked before anything is created.
2. Enter your context. Give the repository in the form owner/name, the name of the PagerDuty service to watch and, if you wish, a short technical context: stack, components, known fragile areas. The more precise it is, the faster the agent looks in the right place.
3. Deploy. A single action creates the agent and the "On-call" project. The agent is provisioned and receives its first instruction on its own.
4. First result, then regular watch. The agent finds the service, says who is on call, handles the most recent incident and summarizes the probable cause and the proposed fix in the chat. Then it schedules its own fifteen-minute watch and its Monday review.

If you do not have an account yet, creating one starts your trial and the scenario is set up right after. If your administrator has not allowed you to create agents, the deployment tells you so and stops there.

Where to see the results

The analysis note is read in PagerDuty, on the incident, and the issue with the proposed fix in GitHub, in the repository. The summary of each incident handled is read in the agent's chat. In the Atako project "On-call" you find the how-it-works page and the weekly review, and on the Atako floor the agent appears with its current activity.

To refine its work, write to it in its chat, for example "on this service, look first at the database migrations". It keeps the instruction in its memory for the following incidents.

Guardrails

The access is deliberately narrow. The PagerDuty connection is granted with a list of seven tools to read and add a note, and the GitHub connection with seven tools to read commits, files and pull requests, search and create issues. An agent that sees a short list of tools uses them more reliably than an agent lost in a catalog of hundreds, and above all, what is not granted cannot happen, even if an alert text tries to force it.

The keys you create are your first guardrail: limited to one service, one repository and precise rights, they bound the possible damage whatever happens. They are stored encrypted, and you can revoke them or remove the agent's access at any time.

Human validation sits at the decision points. The person on call reads the note, checks the hypothesis, decides on the fix, writes and merges it themselves, then acknowledges and resolves the incident.

What to expect

Expect a useful first analysis within minutes, especially when the incident has a cause visible in recent changes: a deployment, a configuration change, a dependency. When the cause lies outside the repository, for example a provider outage or resource saturation, the agent will say so with a low confidence rather than force an explanation. It reads the repository but has no access to your application logs or metrics, beyond what the alert and the notes report. Its value comes from the speed of the first read: the person on call arrives at the incident with a timeline, ranked leads and a fix to review, rather than with a blank page.