Chaos Monkeys

Quality assurance is one of the clearest examples of work that exists as a department because it couldn’t exist as software. Automated tests only catch what someone thought to write a test for. Everything else — the weird click sequence, the malformed payload, the race under load, the button that silently stopped doing anything — has always required humans who poke at things suspiciously for a living.

AI agents can now operate a UI, call an API, read a log, and hold a grudge. So: can the suspicious-poking department be built as a system?

Exploratory testing resists automation because it isn’t a procedure — it’s judgment. Knowing that a page loaded is easy; knowing that it loaded wrong requires a model of what the software is for. Traditional automation has no such model, so the industry settled for scripted regression suites plus expensive human intuition on top.

There’s a second constraint that kills most attempts: false positives. An automated fault-finder that cries wolf gets ignored within a week, and then it doesn’t matter how clever it is.

Not one giant agent — a staff of specialists, the way a real QA department is organized. Separate agents hunt UI problems, API failures, regressions, edge cases, security issues, performance problems, protocol failures, and deliberately induced chaos conditions. Each specialist carries a narrow charter and a narrow definition of “wrong,” which keeps its judgment sharp.

Everything is evidence-based. A finding doesn’t exist without a reproduction: the screenshot, the request/response pair, the log excerpt, the exact steps. Suspicion is free; a filed defect must be provable.

• Agents that operate real interfaces. Current models can drive a browser, read a screen, and follow a hunch across an app — the mechanical substrate exploratory testing always lacked.
• Judgment as a callable function. “Does this look broken to you?” used to require a person. Now it’s an inference call, cheap enough to ask thousands of times a night.
• Marginal cost near zero. A human QA team scales linearly with salary. A specialist agent staff scales with compute — it works nights, weekends, and every deploy, without getting bored on the four-hundredth form.

A deployable system: an orchestrator that dispatches specialist agents against a target application, hunt bundles that define each specialist’s territory and tactics, an evidence pipeline that captures reproductions automatically, and reporting that turns a night of hunting into a ranked defect list a developer will actually read. It runs against Moonshine’s own projects first — this lab is its own test range — and has been deployed against real external targets.

• Generalist agents were mediocre at everything. “Find bugs” as a prompt produces shallow wandering; the specialist charter structure exists because the generalist version failed.
• The first evidence pipeline trusted the agents’ own descriptions of what they saw. Agents, like people, occasionally report bugs that aren’t there. Reproduction-or-it-didn’t-happen became a hard rule after the false-positive rate made early reports unusable.
• Chaos conditions found real bugs — including in the chaos system itself, which at one point was its own most productive target.

The system finds real defects unattended — the kinds of interaction, edge-case, and failure-mode bugs that scripted suites structurally cannot see — and files them with reproductions attached. It has not replaced human judgment about what matters; triage and severity still benefit from a person. But the discovery labor, the part that used to be a department’s headcount, now runs overnight.

QA is a template. Any function whose real content is “trained judgment applied repetitively at scale” — testing, review, monitoring, compliance checking, support triage — is now buildable as a staff of specialist agents plus an evidence discipline. The org chart is quietly becoming an architecture diagram.

NotebookThe hard part was never getting agents to find problems. It was making them honest.