Two weeks ago I wrote The Future of AI Is CI. The factory’s product is proof, not code. I had the back half of that loop as scripts and habits.

Then I built the factory. It is called Factory, because naming things is hard.

The Itch

12 repos. 30 services. Agents writing most of the changes, fast. And one simple question nobody could answer quickly: what is actually running in prod right now?

One day it took a person and three tools to find out. The same day, a deploy caused a short outage, and nothing tied the two together until someone went digging. I had a prototype on my laptop that answered half of it. It needed to be a real thing.

The spec landed on 6 October. By 15 October, 36 PRs had merged and Factory was in production. Agents wrote most of those PRs. That is the part that still feels a little unreal.

The Flight Recorder

The main screen is a timeline called the Flight Recorder. I tried nine designs before picking it.

  • Every service gets a lane. Each lane has a track for each environment: dev, test, prod.
  • Every deploy is a bead. Promotions jump between tracks. Beads that would overlap group together.
  • The error rate runs behind it all. In dark mode it is a phosphor trace on an oscilloscope. In light mode it is ink on graph paper. When a spike follows a deploy, a bracket ties them: “+9 min”.
  • An estate pulse runs across the top. One trace for every service at once, with every prod deploy marked on it.
  • The right edge is “now”. Each lane ends with the commit running in prod, how Factory knows that, and how many commits wait to ship.

It is the screen I wanted the day of that outage. Cause, then effect, on one line.

Ship, Then Watch

Shipping is one flow. Tick the services, promote them to test, try them there, then ship to prod with one approval. Services that share commits move in the same wave. Anything with a database migration goes first. You can schedule a release for 2am, and if Factory wakes up more than 30 minutes late, it refuses to start. It never ships late.

Then the fun part. For 30 minutes after every prod deploy, Factory watches the error rate minute by minute. It compares it with the same half hour on each of the last seven days. A spike needs to clear 3 standard deviations. An error message the service has never logged before counts too. That one catches the bugs a rate alone would miss.

If something looks wrong, Factory builds the revert PR for you. It never deploys anything itself. A person merges and ships the revert, through the same flow as everything else.

Agents, With a Skeptic

Before every prod release, an agent reads the diff, the commits and the board cards, and writes a risk brief. Then a second agent reads each risk and tries to refute it. Only the risks it cannot knock down are shown. The rest stay visible, with the reason they fell.

Sonnet 5.5 writes. Haiku 5.5 argues. On 14 past prod releases, that pair found as many faults as Opus, at a quarter of the cost. Fourteen is a small sample, but I like the shape: a cheap skeptic is a better second opinion than a big model agreeing with itself.

Agents advise. Checks decide.

No agent can pass a check, fail a check, or deploy. Every check compares against the running system, never against what an agent expected. Agent output wears an “agent” label so nobody mistakes it for proof.

Ask Claude What Is in Prod

Factory is also an MCP server. From Claude Code, Claude.ai or Codex, I can ask:

  • What is waiting for prod, and how risky is it?
  • What broke in the last hour, and is any of it new?
  • Draft me the notes for this release.

The answers come from the same data as the screen. The tools are read-only, and anything that came from a repo, a log or a model is marked untrusted, so the agent on the other end treats it as data, not orders.

A coding agent is a part that can be replaced. The checks cannot. So the product of this tool is proof, not deploys.

— Factory spec, section 1

The Honest Bits

  • The health watch is report only, for now. It records a spike on the release, but it does not stop one yet. A few weeks of data first.
  • Not every service has golden replay yet. Where one is missing, the record says “not run”, never “passed”.
  • Some agents are still a plan. Deploy triage, a drift fixer and board hygiene are in the spec, not in production.
  • The front half is still people. Factory starts at the PR, not the idea.
  • Ten days is fast, not proven. The real test is the next three months of releases.

Why It Was Fun

The agents built most of Factory. Factory watches what the agents ship. That loop is the whole point of the first post, and now it has a screen, a trace and a skeptic.

The shipping part was always CI. Now I get to watch it happen.

Related Posts