At some point on August 17, a GitHub engineer decided the fastest way back was to make Copilot fail on purpose. They told the front door to refuse every incoming Copilot login request, waited for things to settle, then let traffic back one region at a time.
That is in the root cause analysis, stated plainly.
The Outage Splits In Two
The public number is 7 hours 47 minutes, 13:28 to 21:15 UTC, with github.com, authentication, Actions, APIs, pull requests and Copilot degraded. Error rates peaked around 20% for web and API traffic.
The recovery timeline:
- Most services recovered by 16:36 UTC, when the Central US datacentre came back.
- Actions stayed degraded until about 18:03.
- The Copilot login service was not right until 21:02.
The original fault owned about three hours. One service owned the other four and a half.
What Broke
Mundane, and worth stating so nobody mistakes this for a story about clever AI. Traffic hit a new peak. One piece of GitHub’s plumbing hit its limit and did not grow to meet the load, because the rule meant to add capacity was watching the wrong meter. That spread until the machines handling logins ran out of room, and once logins fail, everything needing a login fails too.
No bad deploy, no bad config. GitHub’s CTO is explicit that both August incidents were capacity failures.
GitHub then did the right thing and shifted failing traffic from Central US to Northern Virginia. The retry storm followed it there.
The Amplifier
— GitHub root cause analysis, incident zkxwbgr0cnmxDelayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
In plain terms: the service handing out Copilot’s login passes normally answers about 8,000 requests a second. During recovery it was being asked 70,000 to 100,000 times a second.
One slow internal service. One dormant bug in an editor on millions of desktops. A single failed login could spawn many more and settle into a loop, so every machine that failed came back harder. GitHub could not add capacity to absorb it, because the capacity was being eaten by clients whose entire response to failure was to ask again immediately.
So they cut the callers off. An emergency change to stop their own systems retrying, a blanket refusal of Copilot login requests at the front door, then a slow ramp back region by region.
Retry Storms Are Old. The Client Is New.
Nothing in the mechanism is novel. “Wait longer after each failure, and give up eventually” is decades-old practice, and GitHub’s fix list is textbook: cap the retries, budget them, vary the timeouts.
— luciana1u, Hacker News, 20 August 2026the retry loop that made the outage worse is just the internet being extra helpful. every client decided the best way to help was to ask again, louder.
What changed is who is doing the asking. A human who gets an error hits refresh two or three times, swears, and goes for coffee. That is patience enforced by boredom, and it is why bugs like this used to stay hidden: the crowd hammering the button thins out on its own.
An editor asking for a login pass does not get bored. Neither does the agent inside it, nor the build it kicked off, nor the second agent reviewing the first one’s work. Every layer has its own idea of a sensible retry count, none can see the others, all are trying to help. The crowd never thins.
This was also the second GitHub incident in August with retries as the amplifier. On August 6, build machines were handed expired jobs and got stuck asking for them repeatedly, which stopped them picking up real work.
If you run agents against any API, you are on the other side of this story. Check what yours does when a request fails or times out, and whether anything caps how many times it tries again. A retry policy invisible in normal operation is the one that shows up during someone else’s incident.
The Number Nobody Leads With
Since April, code changes saved to GitHub each month grew from 1.4 billion to 2.9 billion.
A doubling in four months, on a platform that has been the default for a decade. GitHub does not say what caused it, and I will not put words in their mouth, but there is no plausible source of an extra 1.5 billion changes a month that involves people typing.
The remediation reflects it: three million processors added, 120 petabytes of storage, Azure taken from 12% of platform load in May to roughly 58%. Every one of those builds a bigger road. One line in the whole document is about the traffic, a review of how often their systems retry. Nothing slows automated callers down, nothing treats an agent differently from a person, nothing turns arrivals away when the queue is full.
What This Isn’t
- GitHub never blames AI. They do not mention agents. Every attribution of the growth here is mine.
- The VS Code loop is a bug, not intent. The point is not that the software was clever. It was numerous, automatic and endlessly patient.
- Counting commits is a weak measure, and some of the load was plain abuse: the report names scraping attacks as a complicating factor.
The Part That Should Worry You
GitHub is the most operationally scrutinised developer platform on earth, and it still shipped a scaling rule watching the wrong limit and an editor that could tenfold its own login service under stress. Both were invisible until demand went somewhere it had never been.
Every team running agents against production has the same two bugs somewhere, without three million spare processors or the option of turning their own users away until things calm down. The fix GitHub reached for was not capacity. It was refusing traffic from its own flagship product, and the traffic that saves you by going away is increasingly traffic you generated yourself.
Related Posts
