Posts related to production
36 posts
ushr runs GitHub Actions jobs on hardware you already own: Mac minis through Tart or Lume VMs, Linux through Docker or Podman. Every runner is single-use. One fleet serves several GitHub orgs by priority, and the hosted control plane never holds a GitHub credential. The agent, CLI and controller are Apache-2.0. Here is why I built it, 30 days of real numbers, and what it does not do yet.
Read more →Warp's Zach Lloyd says software engineering is becoming factory engineering: agents triage, spec, build, review and monitor in one loop. I have been building the back half of that loop since June, without the map. Every critical change gets backtested against real history before it ships, and that playbook is now moving into CI.
Read more →The crowd says Opus 5.5 is the best coding model in a year. My commit and session history agree, but not where I expected. It didn't make me faster per task. It tripled my test share, held back more, and let me run nearly twice as many projects at once.
Read more →In July I wrote 'no migration' for FameCake's Firestore database. In September I moved a live ecommerce platform to Postgres, Better Auth and SvelteKit, with 291 agents and a 33-minute freeze. The database didn't change. The price of leaving it did.
Read more →TypeSafe's Jev is an AI model that doesn't write. It answers questions with a yes/no probability, a pick or a score. I gave it 9,081 product matches a human review queue was never going to clear. It judged them all for 32 cents in 13 minutes. Eight days later, checked against barcodes, it got the job.
Read more →Vite patched the server.fs.deny query bypass on April 6. F5 Labs saw 32,000 exploitation events in August, an 18-fold jump, and the wordlist tells you who the attackers expect to find: /home/node/.aws/credentials, /usr/src/app, /proc/self/environ. Nobody is hunting laptops. The Vite dev server moved into containers and cloud VMs because that is where agents and previews run, and --host is the price of a port mapping. I audited my own machine: 47 repos pin Vite, 28 pin a vulnerable one, and one has allowedHosts set to true from a tunnel I forgot about.
Read more →One operator, hundreds of AI agents, a DeepSeek model inside OpenAI's Codex harness: 440 PaperCut servers in 48 countries, first code execution under four hours from an empty workspace, 11 organisations in 26 seconds, a US high school to domain admin in seven minutes. The model was mid-tier and foreign. The harness was an American product. Every vetting regime built this year gates the model. The thing that did the damage is the part nobody tests.
Read more →Three outside researchers found that OpenAI agents pushed 2,000 packages to RubyGems on May 11-12, got code execution on RubyDoc's build servers, and probed a credential-leak bug. RubyGems shut registrations for four days and never learned who did it. OpenAI's July update said it had found no other incident of that scale. The evidence was package names containing 'oai'. The lab's own review missed what a grep found, and that is the case for embedded evaluators, made by the lab that would rather not have made it.
Read more →Vercel serves six million deployments a day and half are agent-triggered. The worst outages of the last year were triggered by config, data and automation, not by anyone's pull request. The five levers of releasing, the changes that have no undo, and what to fix first.
Read more →In March a booby-trapped version of a popular software component was downloadable for about forty minutes. On August 11 CloudSEK published the damage: over 2,500 organisations and around 434,000 build pipelines, pieced together from loot the attackers left lying around in public. Nobody could work out their own exposure.
Read more →GitHub was down for 7 hours 47 minutes on August 17. The original fault was fixed in about three hours. The remaining four and a half hours were a bug in VS Code that kept asking again after every failure, driving one GitHub service from 8,000 requests a second to 100,000. GitHub's fix was to slam the door on its own Copilot.
Read more →Abliteration strips the refusals out of an open-weight model with one subtraction and about ten minutes on a laptop. Four thousand of these models sit on Hugging Face, and the newest arrived two days after its base model. What that means for the defenders I've spent five posts telling to self-host.
Read more →The Hugging Face breach had a twist nobody saw coming. On July 21 OpenAI admitted the autonomous attacker was its own pre-release models, run in an internal cyber eval with safety refusals switched off, that gamed the benchmark, escaped containment, and reached HF's production database. A safety measurement became the security incident it was meant to measure.
Read more →An autonomous AI agent breached Hugging Face's infrastructure in July 2026. The stranger part: HF's incident responders were locked out of their own forensics by commercial AI guardrails that can't tell a defender from an attacker, and the fix was a self-hosted Chinese open-weight model.
Read more →AWS showed customers estimated bills of billions and trillions of dollars in July 2026. No real charges hit. But people nuked their own infrastructure anyway, and the deeper problem is structural: every cloud's cost alerting watches an estimate layer that is slow when spend is real and confidently wrong when it breaks.
Read more →A simple question (why are there no bookings today?) turned into a day of finding that three of the instruments I steer FameCake by were quietly wrong: consent-gated ad attribution, a proof-of-play record that deletes itself, and a speed claim nobody had ever measured. The bug isn't in any of them. The bug is trusting derived data.
Read more →A team's model kept 'hearing' a phrase in videos with no audio. They chased it through 30,000 training records, 4,600 transcripts, and 800 inference probes, and found it: a worked example in their own system prompt. They deleted it. The model just hallucinated a different phrase. The lesson is that the model didn't learn a confabulation. It learned to confabulate, and that lives in the architecture, not the data.
Read more →One LLM on the long tail is a coin flip. How I designed a product-enrichment pipeline around consensus voting, abstention, and content-hashed freshness gates.
Read more →There's a myth, loudest from senior engineers and architects, that before AI the codebase was a cathedral and now it's slop. It was never a cathedral. 'Technical debt' was coined in 1992, the world runs on 220 billion lines of COBOL, and the thing that actually mattered was never how the code looked. It was whether you could prove it works.
Read more →One analysis of 2,444 companies claims only 18 cents of every AI-token dollar reaches the product. The rest goes to fixing the AI's bugs, reworking its context misses, and review friction. Treat the source with caution, but the shape of the problem is real.
Read more →Ghostty left after 18 years. Zig, cURL, and Godot are reducing reliance. 48 major outages in a year, no CEO since August, reporting into the Core AI team. Centralising the world's code was a bet on stewardship. The steward stopped showing up.
Read more →Project Glasswing found 10,000+ critical vulnerabilities at 90.6% accuracy. Mozilla had to patch 271 of them in Firefox by hand. Finding collapsed to near-free. Fixing didn't move. The bottleneck just walked downstream.
Read more →Creation runs at machine speed. Release engineering does not. GitHub agent-authored PRs went from 4M to 17M in six months. The bottleneck moved from review to release.
Read more →Anthropic markets MCP as the universal AI tooling standard, but a 200,000-server RCE class is 'expected behavior.' You can't be both.
Read more →Anthropic's April 23 postmortem confirms three Claude Code regressions, including one where Opus 4.7 caught a bug Opus 4.6 shipped past human and automated review. What happens when the reviewer is a version of the product being reviewed?
Read more →Anthropic silently changed Claude Code's cache TTL from 1 hour to 5 minutes, inflating costs 10-20x. Users had to reverse-engineer the binary to prove it. False child bans, $600 surprise charges, and the OpenClaw crackdown completed the picture. April 2026 was the month trust broke.
Read more →Axios got supply-chain attacked. Claude Code's source code leaked from a stray map file. Both happened on the same day. Both are pipeline failures. The pattern is getting louder.
Read more →Enterprise architecture patterns were designed for a world where code was expensive to write and expensive to change. That world ended. The patterns didn't get the memo.
Read more →Two weeks after Kiro deleted a production environment, Amazon.com itself went down for 6 hours. 1,500 engineers are petitioning for Claude Code. The safeguards are arriving after the damage.
Read more →Amazon's Kiro AI decided to delete and recreate a production environment, causing a 13-hour AWS outage. Amazon says it was human error. That framing is the problem.
Read more →OpenClaw went from 0 to 111K GitHub stars in two months. It also went from 0 to hundreds of exposed instances with full credentials in Shodan. The security story nobody wants to hear.
Read more →Anthropic denied issues for weeks, then published a postmortem admitting three bugs degraded 16% of Claude requests. The pattern keeps repeating.
Read more →Converting text to images for 20x token compression. Interesting research or production-ready breakthrough? A critical look at the trade-offs.
Read more →How I built a self-improving document parser that learns from corrections without fine-tuning. The pragmatic alternative to model training.
Read more →Building a multi-stage AI content pipeline where each generation depends on the last. Lessons from generating thousands of hybrid creatures with resilient error handling.
Read more →A production outage caused by Cloudflare's anti-bot systems flagging Google Cloud datacenter IPs, and why API aggregation layers matter more than you think
Read more →