Mayhem Monkies

Quality assurance is one of the clearest examples of work that exists as a department because it couldn’t exist as software. Automated tests only catch what someone thought to write a test for. Everything else, from the weird click sequence to the malformed payload to the race under load to the button that silently stopped doing anything, has always required humans who poke at things suspiciously for a living.

AI agents can now operate a UI, call an API, read a log, and hold a grudge. So: can the suspicious-poking department be built as a system?

Exploratory testing resists automation because it isn’t a procedure. It’s judgment. Knowing that a page loaded is easy; knowing that it loaded wrong requires a model of what the software is for. Traditional automation has no such model, so the industry settled for scripted regression suites plus expensive human intuition on top.

There’s a second constraint that kills most attempts: false positives. An automated fault-finder that cries wolf gets ignored within a week, and then it doesn’t matter how clever it is.

Not one giant agent, but a staff of specialists, the way a real QA department is organized. Separate agents hunt UI problems, API failures, regressions, edge cases, security issues, performance problems, protocol failures, and deliberately induced failure conditions. Each specialist carries a narrow charter and a narrow definition of “wrong,” which keeps its judgment sharp.

Everything is evidence-based. A finding doesn’t exist without a reproduction: the screenshot, the request/response pair, the log excerpt, the exact steps. Suspicion is free; a filed defect must be provable.

• Agents that operate real interfaces. Current models can drive a browser, read a screen, and follow a hunch across an app: the mechanical substrate exploratory testing always lacked.
• Judgment as a callable function. “Does this look broken to you?” used to require a person. Now it’s an inference call, cheap enough to ask thousands of times a night.
• Marginal cost near zero. A human QA team scales linearly with salary. A specialist agent staff scales with compute. It works nights, weekends, and every deploy, without getting bored on the four-hundredth form.

A deployable system: an orchestrator that dispatches specialist agents against a target application, hunt bundles that define each specialist’s territory and tactics, an evidence pipeline that captures reproductions automatically, and reporting that turns a night of hunting into a ranked defect list a developer will actually read. It runs against Moonshine’s own projects first (this lab is its own test range) and has been deployed against real external targets.

It has since graduated into a product you can hire: Mayhem Monkies. You post a target, the crew hunts it, and the important part is what comes back. Not a defect list to triage, but every bug delivered as a fix.

• Generalist agents were mediocre at everything. “Find bugs” as a prompt produces shallow wandering; the specialist charter structure exists because the generalist version failed.
• The first evidence pipeline trusted the agents’ own descriptions of what they saw. Agents, like people, occasionally report bugs that aren’t there. Reproduction-or-it-didn’t-happen became a hard rule after the false-positive rate made early reports unusable.
• Induced failures found real bugs, including in the fault-injection tooling itself, which at one point was its own most productive target.

The system finds real defects unattended: the kinds of interaction, edge-case, and failure-mode bugs that scripted suites structurally cannot see. It files them with reproductions attached. It has not replaced human judgment about what matters; triage and severity still benefit from a person. But the discovery labor, the part that used to be a department’s headcount, now runs overnight. And returning fixes rather than tickets moved it from “interesting output” to something a team will actually pay for, which is the difference between an experiment and a product.

QA is a template. Any function whose real content is “trained judgment applied repetitively at scale” (testing, review, monitoring, compliance checking, support triage) is now buildable as a staff of specialist agents plus an evidence discipline. The org chart is quietly becoming an architecture diagram.

Notebook The hard part was never getting agents to find problems. It was making them honest.