Posts related to production
23 posts
An autonomous AI agent breached Hugging Face's infrastructure in July 2026. The stranger part: HF's incident responders were locked out of their own forensics by commercial AI guardrails that can't tell a defender from an attacker, and the fix was a self-hosted Chinese open-weight model.
Read more →AWS showed customers estimated bills of billions and trillions of dollars in July 2026. No real charges hit. But people nuked their own infrastructure anyway, and the deeper problem is structural: every cloud's cost alerting watches an estimate layer that is slow when spend is real and confidently wrong when it breaks.
Read more →A simple question (why are there no bookings today?) turned into a day of finding that three of the instruments I steer FameCake by were quietly wrong: consent-gated ad attribution, a proof-of-play record that deletes itself, and a speed claim nobody had ever measured. The bug isn't in any of them. The bug is trusting derived data.
Read more →A team's model kept 'hearing' a phrase in videos with no audio. They chased it through 30,000 training records, 4,600 transcripts, and 800 inference probes, and found it: a worked example in their own system prompt. They deleted it. The model just hallucinated a different phrase. The lesson is that the model didn't learn a confabulation. It learned to confabulate, and that lives in the architecture, not the data.
Read more →One LLM on the long tail is a coin flip. How I designed a product-enrichment pipeline around consensus voting, abstention, and content-hashed freshness gates.
Read more →There's a myth, loudest from senior engineers and architects, that before AI the codebase was a cathedral and now it's slop. It was never a cathedral. 'Technical debt' was coined in 1992, the world runs on 220 billion lines of COBOL, and the thing that actually mattered was never how the code looked. It was whether you could prove it works.
Read more →One analysis of 2,444 companies claims only 18 cents of every AI-token dollar reaches the product. The rest goes to fixing the AI's bugs, reworking its context misses, and review friction. Treat the source with caution, but the shape of the problem is real.
Read more →Ghostty left after 18 years. Zig, cURL, and Godot are reducing reliance. 48 major outages in a year, no CEO since August, reporting into the Core AI team. Centralising the world's code was a bet on stewardship. The steward stopped showing up.
Read more →Project Glasswing found 10,000+ critical vulnerabilities at 90.6% accuracy. Mozilla had to patch 271 of them in Firefox by hand. Finding collapsed to near-free. Fixing didn't move. The bottleneck just walked downstream.
Read more →Creation runs at machine speed. Release engineering does not. GitHub agent-authored PRs went from 4M to 17M in six months. The bottleneck moved from review to release.
Read more →Anthropic markets MCP as the universal AI tooling standard, but a 200,000-server RCE class is 'expected behavior.' You can't be both.
Read more →Anthropic's April 23 postmortem confirms three Claude Code regressions, including one where Opus 4.7 caught a bug Opus 4.6 shipped past human and automated review. What happens when the reviewer is a version of the product being reviewed?
Read more →Anthropic silently changed Claude Code's cache TTL from 1 hour to 5 minutes, inflating costs 10-20x. Users had to reverse-engineer the binary to prove it. False child bans, $600 surprise charges, and the OpenClaw crackdown completed the picture. April 2026 was the month trust broke.
Read more →Axios got supply-chain attacked. Claude Code's source code leaked from a stray map file. Both happened on the same day. Both are pipeline failures. The pattern is getting louder.
Read more →Enterprise architecture patterns were designed for a world where code was expensive to write and expensive to change. That world ended. The patterns didn't get the memo.
Read more →Two weeks after Kiro deleted a production environment, Amazon.com itself went down for 6 hours. 1,500 engineers are petitioning for Claude Code. The safeguards are arriving after the damage.
Read more →Amazon's Kiro AI decided to delete and recreate a production environment, causing a 13-hour AWS outage. Amazon says it was human error. That framing is the problem.
Read more →OpenClaw went from 0 to 111K GitHub stars in two months. It also went from 0 to hundreds of exposed instances with full credentials in Shodan. The security story nobody wants to hear.
Read more →Anthropic denied issues for weeks, then published a postmortem admitting three bugs degraded 16% of Claude requests. The pattern keeps repeating.
Read more →Converting text to images for 20x token compression. Interesting research or production-ready breakthrough? A critical look at the trade-offs.
Read more →How I built a self-improving document parser that learns from corrections without fine-tuning. The pragmatic alternative to model training.
Read more →Building a multi-stage AI content pipeline where each generation depends on the last. Lessons from generating thousands of hybrid creatures with resilient error handling.
Read more →A production outage caused by Cloudflare's anti-bot systems flagging Google Cloud datacenter IPs, and why API aggregation layers matter more than you think
Read more →