The Rollback Is the Product: Feature Flags, Canaries, and Config in the Agent Era

At Vercel Ship in June 2026, Guillermo Rauch said the platform serves around six million deployments a day, more than half triggered by coding agents, against under 3% six months earlier. Previews count in that, and agents generate a lot of previews, so read it as a claim about who starts a change rather than about production traffic.

The cost shows up in two data sets. New Relic surveyed 200 engineering leaders in June 2026: 94% rate AI-generated code as better than human-written at the moment it is reviewed, and 78% report more production incidents traceable to it. Faros AI, working from telemetry across 22,000 developers, found changes merging with no review at all rose 31.3%. Neither pairs a reviewed change with its later failure, so the causal story is unproven. But a third more changes now reach production having been read by nobody, and whatever catches those is not review.

The Trigger Was Not Code. The Defect Was.

Four postmortems from the last ten months:

• Cloudflare, 18 November 2025. A database permissions change made a query return duplicate rows. That produced an oversized bot-detection file, bigger than the code had room for, and the proxy crashed instead of handling the error. Nearly six hours, and Matthew Prince called it their worst outage since 2019.
• AWS, 19 October 2025. Two copies of the same DNS automation raced. One stalled, woke, and applied an old plan; the other had already applied a newer one, then ran cleanup and deleted the record that had just gone live. DynamoDB was down nearly three hours, the knock-on across EC2 and Lambda about fourteen.
• Azure Front Door, 29 October 2025. Customer config Microsoft calls “valid and non-malicious”, applied across two versions of the control system, produced settings that woke a dormant bug. Eight and a half hours. Azure had staged rollout and a known-good snapshot. What it lacked was a test covering both versions at once.
• GitHub, February 2026. A run of incidents in one monthly report, nearly all config: a cache setting, a permissions change, a bad network config, and lost telemetry that cascaded into security rules hitting the wrong storage.

The tempting conclusion is that none of this was code. Wrong, and the postmortems say so: “latent race condition”, “latent data plane bug”, an unhandled error. Every one needed a software defect. The trigger just travelled a path the code pipeline does not govern, and woke something that had been passing tests for months.

Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input.

— Cloudflare, Cloudflare outage on November 18, 2025

Treat the files you generate yourself as carefully as the ones strangers send you. Not because they are hostile, but because they are unversioned, unbounded, and arrive faster than anything you review.

Five Levers, Not One Ladder

Tempting to rank these by how fast they undo things. They are not a hierarchy. They are five questions, and one change can answer all five.

• How does the change install? The deploy. Stripe shipped its payments APIs 5,978 times in a year and rolled back 1,100 automatically for failing their checks, with reliability above 99.999% throughout. An 18% rejection rate reads two ways, and Stripe never said how many were false alarms.
• How fast does it spread? Amazon starts on one machine, then one small region, then bigger ones, waiting hours between steps and taking most of a week to reach everywhere. Vercel steps through 5%, 10%, 25% and 50% of users at six hours a stage. Netflix compares new against old statistically rather than having someone watch a dashboard. A long wait tells you how slowly a rollout advances, not how fast you can stop it.
• Who sees it? Dark launches and flags, built into the platforms now. Vercel’s framing is the cleanest anyone has managed: “Deploying code and releasing a feature become two separate decisions.”
• Can it be switched off? The kill switch, and Cloudflare’s own fix list includes adding more. The catch nobody mentions: a switch only helps if it was wired in beforehand. Adding one mid-outage needs the deploy you are trying to avoid.
• What does the system drop under stress? Cutting off a failing dependency, shedding low-priority work, surviving a failure without reacting at all. None of these is a release mechanism. They fire whether or not you released.

What matters is how they combine. Stripe’s 1,100 rollbacks are a deploy borrowing the other four, and teams get hurt owning exactly one lever. Governance is where it goes uneven: the fastest levers are usually run outside the code pipeline, by more people, with less review. That is what the outages above keep exploiting.

Prompts Are Releases Too

If your product calls a model, you ship prompts, model versions, tool definitions and settings. Most teams route none of it through the five levers.

Anthropic published this about itself. Its 23 April 2026 postmortem traced weeks of Claude Code quality complaints to three changes: reasoning effort cut for speed, a caching bug that discarded the model’s thinking, and an instruction to be less verbose that made the code worse. On why it got through: “After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change and shipped it.”

For any change that could trade off against intelligence, we’ll add soak periods, a broader eval suite, and gradual rollouts.

— Anthropic, An update on recent Claude Code quality reports, 23 April 2026

A lab admitting it shipped prompt changes with no staged rollout, and committing to build one. OpenAI’s sycophancy postmortem shows the same shape a year earlier: tests passed, humans said it felt wrong, they shipped anyway. The transferable part is that the measurement the other levers depend on does not exist here. Showing a change to 5% of users for six hours tells you nothing when the problem only surfaces in complaints three weeks later.

The Changes You Cannot Take Back

Everything above assumes an undo exists. The expensive changes have none.

• Database migrations and backfills. A flag does not save you when the switched-off path reads a column you dropped. Add the new thing before removing the old one, and leave both working long enough to change your mind.
• Anything that left the building. Sent email, posted messages, money moved. No undo, only a correcting transaction, and that is code you write before you need it.
• Mobile app releases. No rollback at all. The original reason feature flags exist, and still the best argument for shipping risky mobile changes switched off.
• Cached and CDN content. Fixing the source does not fix what already spread.

The answer is not a faster undo. It is a tested path to fix forward, and designing the retreat before you advance. Knowing which of your changes land here is the highest-value hour in this essay, because these are the ones where every other lever is theatre.

Machines May Revert. Models May Not.

Automated release is sold as speed. What it buys is patience: nobody sits through a twelve-hour wait, a pipeline will. But automation is another thing with production access, not an escape from the problem. The AWS outage was automation racing itself, and the fix was blunt: switch it off worldwide, then repair DNS by hand.

Automatic rollback is neither new nor rare. Stripe’s 1,100 reversions. Argo Rollouts and Flagger backing changes out when metrics dip. LaunchDarkly pausing a rollout on a bad signal. Software reverts production constantly and nobody finds it remarkable.

What no vendor ships is a model allowed to make that call. PagerDuty’s SRE agent investigates, explains, and runs only pre-approved fixes. Datadog’s equivalent investigates and reports. Diagnosis automated, judgement reserved.

The line is not human against machine. It is a rule against a guess. A threshold written in advance can revert production a thousand times a night, because a person decided what counts as bad and can be held to it. A model deciding that for itself, and acting, is what nobody has shipped. Probably for ordinary reasons: it is hard to prove what caused what, some damage cannot be undone, and nobody wants to own the outcome.

What To Actually Do

• Time your undo, in stages. How long to notice, decide, execute, propagate, confirm. A flag you flip in five seconds but that takes twenty minutes to reach every server is not a five-second undo, and one average hides which stage is broken.
• Check generated files whole, and fall back rather than crash. Size, shape, version, checksum. Reject a bad one before publishing and keep serving the last good one. Do not trim it to fit: a half-loaded model file is valid and wrong, which is worse than stale because it fails quietly.
• Give every automation a stop that is safe to press. It has to pause cleanly, refuse to resume into a changed world, and leave something inspectable. Halting a job mid-write can leave worse state than letting it finish. Check the stop does not depend on the thing it stops.
• Keep a flag register, not an expiry comment. Type, owner, safe default, removal condition. Temporary flags get a date that fails the build when it passes. Kill switches and access flags are permanent by design, so a blanket expiry rule will eventually switch one off in production.
• Version prompts and models like anything else. Pin versions, version your tool definitions, and remember a revert cannot un-write what the model already saved. Pair the fast checks with a slow one, because these regressions surface over weeks.
• Write your pass/fail rule before automating the gate. What you compare against, how much traffic, what happens when data is missing, which metrics lag. Without that the gate either never fires or fires constantly, and both teach people to ignore it.
• Decide what to drop first on a calm afternoon. Uber found 28% of a six-month run of outages could have been softened by degrading gracefully, which measures how often nobody had decided in advance. Deciding is cheap. Building the thing that enforces it is the work.
• Scope agent access by damage and reversibility, not seniority analogies. Ask what an action can break, how fast you would notice, and whether you can undo it. Same question you owe your automation.

What This Doesn’t Solve

• Measurement is the real prerequisite. Per-slice comparison, alerts that fire on user harm rather than CPU, quality signals for model behaviour. Buying a flag platform creates none of them, and without them these controls are ceremony.
• The AI attribution is contested. When the Financial Times reported Amazon’s Kiro agent deleted and recreated an environment during a 13-hour outage, Amazon’s rebuttal was that it “was the result of user (AWS employee) error, specifically misconfigured access controls, not AI.” Both can be true. Most coverage picks one.
• The numbers on both sides are self-interested. Adoption percentages come from companies that benefit from the story: Anthropic’s 80% of merged code, Google’s 75% of new code. Incident percentages come from vendors selling monitoring. Nobody publishes both from the same books.

Closing

The bottleneck moved from review to release some time ago. The postmortems add something narrower: the paths that reach production fastest are the ones your change process does not cover. Config, generated data, flags, prompts, and the automation writing all of them at three in the morning.

There is no contradiction between “a human decision is the last line” and “the rollback runs without waking anyone”. Same claim, two tenses. Somebody sat down on a quiet afternoon and wrote what counts as failure, what gets dropped first, and what happens when a file arrives malformed. The machine executes that at speed, thousands of times, in the dark. That afternoon is the work. The rollback is only where it shows up.