PocketOS (Cursor + Claude)
Apr 2026 · Agent wiped prod DB
Coding agent found an over-scoped token, dropped the prod Postgres and every backup in 9 seconds. 30+ hours down.
Concept
A guardrail asks the model to behave. An enforcement layer makes the unsafe action impossible. The distinction is the difference between a press release and a postmortem.
The difference
Properties
If policy lives in the model's context, the model can be talked out of it. Authorization has to be decided outside the reasoning loop.
Microseconds, not milliseconds. Anything slower gets bypassed for latency reasons.
When the policy is unclear, refuse and escalate. Safety is the floor, not the goal.
Incident research
Each of these is a case where an AI agent took an action — or produced a binding output — that an independent authorization layer would have blocked, constrained, or evidenced. Figures come from court filings, SEC disclosures, tribunal decisions, government reports and first-party write-ups.
Apr 2026 · Agent wiped prod DB
Coding agent found an over-scoped token, dropped the prod Postgres and every backup in 9 seconds. 30+ hours down.
May 2026 · First Shadow-AI 8-K
Employee pasted customer SSNs into an unsanctioned GenAI tool. Bank filed the first Item 1.05 8-K in U.S. history.
May 2026 · Shareholder settlement
Settlement over AI hiring & lending discrimination — plus a board-disclosure securities-fraud overlay.
May 2026 · Cisco disclosure
One poisoned entry in CLAUDE.md steers every future session toward attacker code. Survives reboots. Invisible.
Jan 2026 · FCRA class filed
Secret resume 'fit scores' at hundreds of employers — no disclosure, no dispute path, no adverse-action notice.
Apr 2026 · AI-hallucination sanctions
Briefs contained 20+ fabricated case citations. Court found the filings were knowingly submitted.
Mar 2026 · N.D. Illinois
ChatGPT coached a claimant into 44 post-settlement filings to reopen a case the insurer had already won.
Feb 2024 · BCCRT
Airline held liable for a refund policy its chatbot invented. The precedent every 2026 case now cites.
Jul 2025 · SaaStr
Agent ignored 'code freeze', deleted 1,200+ prod records, then fabricated status reports about it.
Oct 2025 · DEWR report
Government report contained fabricated citations and a made-up federal-court quote. Final instalment refunded.
Apr 2023 · Internal
Engineers pasted source code into ChatGPT. Samsung banned public GenAI on corporate devices weeks later.
Put a deterministic enforcement layer between your AI and the action it's about to take.