aegis/sre
Incident response for microservices

The page fires. The rollback is already drafted.

AegisSRE pulls the traces for a failing service, works out what broke, and writes the compensating calls that undo the damage. You approve it. Temporal makes sure it finishes, even if the worker dies halfway through.

incident-INC-7F3A21C0 status: detecting
12:05:00.412alertapi-gateway 504, p99 4.8s via alertmanager
12:05:00.431detectingget_telemetry_context 2 traces
api-gateway 504 4823ms
order-service 200 93ms
payment-service 500 4710ms <-
12:05:01.210analyzinganalyze_root_cause
12:05:06.884payment-service stalls on the processor;
gateway gives up at 5s, charge + order
still commit downstream.
12:05:06.890planPOST /refundPayment {"payment_id":"pay_91c2"}
POST /cancelOrder {"order_id":"ord_4e10"}
12:05:06.891remediatingwaiting on a human, 30m timeout
12:07:42.019signalapprove_rollback
12:07:42.025compensating2 activities, in parallel
12:07:42.311/refundPayment 200 /cancelOrder 200
12:07:42.312resolved2 compensating transactions applied
01 / LIFECYCLE

One workflow, five states.

Each incident is a single Temporal workflow. It moves through the same states you see on the dashboard, and every step is recorded in its event history, so a crash resumes from the last completed step instead of starting over.

detecting

Fetches spans for the alert’s trace IDs through the MCP tool get_telemetry_context.

30s timeout, 3 attempts

analyzing

A model reads the spans and returns a root cause plus the list of compensations to run.

heartbeat every 15s

remediating

Nothing runs yet. An operator approves, rejects, or asks a question, which triggers a fresh analysis.

fails closed after 30 min

compensating

Each compensation is its own activity. They run in parallel and retry on their own.

409 / 404 count as done

resolved

The root cause and the steps that were applied get written to the incident record.

or failed, with the reason
02 / DESIGN

Opinions it holds.

Automated remediation is only useful if you can trust it at 3am. These constraints are what make that trust reasonable.

A human signs off on every change.

The agent can diagnose and propose on its own, but it can’t mutate production state until someone sends approve_rollback. No answer within 30 minutes counts as a no.

workflows/incident_response.py

Undo with sagas, not locks.

No two-phase commit across services. Each forward step has a compensating call, like /refundPayment for a charge, and those calls are idempotent, so running one twice does no harm.

activities/saga.py

Workflows never touch the network.

Workflow code is deterministic, so Temporal can replay it. Every model call, HTTP request, and mutation lives in an activity with its own timeout and retry policy.

activities/

Tools are stateless HTTP.

The agent reaches infrastructure through an MCP server where every request describes itself fully. Workers can come and go without session state getting lost.

mcp/server.py
03 / USE IT

Wire it to your alerts.

POST /incident accepts PagerDuty v3 and Alertmanager webhooks as they are, and works out which format it got. Or trigger a seeded test incident from the dashboard.

Alertmanager

Add a webhook receiver pointing at the dashboard. The first firing alert starts a workflow.

# alertmanager.yml
receivers:
  - name: aegis
    webhook_configs:
      - url: http://aegis:7080/incident

# or by hand
curl -X POST localhost:7080/incident \
  -H 'content-type: application/json' \
  -d '{"alerts":[{"labels":{"service":"payment-service"},
       "annotations":{"summary":"p99 > 4s"}}]}'

Run the stack

Python 3.11+ and Docker. One process per terminal.

  1. pip install -r requirements.txt && cp .env.example .envSet OPENAI_API_KEY for real root-cause analysis.
  2. docker run --rm -p 7233:7233 -p 8080:8080 temporalio/auto-setup
  3. python -m sre_swarm.mock_services.appOrder and payment services on :9090.
  4. python -m sre_swarm.mcp.serverTool server on :8081.
  5. python -m sre_swarm.worker
  6. python -m sre_swarm.dashboard.appOpen /dashboard and trigger an incident.