Every agent framework breaks in the same place: the interface. APIs rate-limit, extensions get detected, drivers need install rights. ACID CAT goes underneath all of it, a small ESP32-S3 board that plugs into USB and enumerates as an ordinary HID keyboard and mouse. No driver, no extension, nothing to integrate. To the target machine it is a person typing, which is the one integration surface no vendor can withdraw.
The device does more on one cable than it looks like it should. It runs as a composite USB device, HID keyboard,
HID mouse and a CDC serial console simultaneously, while also serving a JSON HTTP API on port 8080 over WiFi, running a
six-characteristic BLE GATT service, and driving a 240x280 touchscreen with its own menu system. Six ways in: serial,
HTTP, BLE, the on-device touch UI, a Web Bluetooth page that provisions WiFi from a browser tab with nothing installed,
and a one-line shell wrapper. HID mice only speak relative deltas, so the firmware maintains a virtual cursor and
implements absolute moveto as an interpolated stream of int8-clamped reports. The JSON parsing is hand-rolled
indexOf scanning rather than a library, no allocator, no heap fragmentation on a device meant to stay up for days.
The half nobody expects is that it can see. A pure input device is blind, so we built the eyes out-of-band: the host
takes a screenshot and reconstructs a UI tree from raw pixels, tesseract lines regrouped into elements, then classified
into buttons, inputs, links and tabs by Canny edge density and interior variance, modals found by dual-threshold contour
geometry so it works on light and dark themes, and the mouse cursor located by template-matching a pointer shape
synthesized in code. That last piece closes the loop: the mouse calibrates by slamming into the screen corner, then
verifies every move against a fresh screenshot and issues corrective deltas when the OS's pointer acceleration eats the
difference. Clicks can demand proof, screenshot before, screenshot after, retry with jitter if the screen didn't
change. On top sits a declarative JSON macro runner with wait_for, verify, click-by-label and live TOTP generation;
the reference macro logs into a 2FA-protected site end to end with zero software on the target. And there is a bongo cat
on the front that drums the correct paw for whichever half of the keyboard your current character lives on, because we
built the hardware and could.
// Technical highlights
- Composite USB device: HID keyboard + HID mouse + CDC serial console on one port, plus a WiFi HTTP API, a BLE GATT service and a touchscreen UI, 1,508 lines of firmware on one ESP32-S3.
- Screenshots become structured UI JSON: OCR lines, then buttons/inputs/links classified by Canny edge density and interior variance; modals detected by dual-threshold contour geometry in both light and dark themes.
- Absolute mouse positioning on a relative-only protocol: corner-slam calibration, interpolated int8-clamped HID reports, screenshot verification with a 20-pixel tolerance and automatic corrective deltas.
- Clicks that prove they worked: before/after screen-state diff, then up to 3 retries with ±5px jitter.
- Bongo cat animation ships only the 145x40 rectangle in which the frames actually differ, 11.6 KB per frame instead of 134 KB, a ~23x cut that keeps the paws in sync with typing at 120 ms per swap.
- WiFi setup with nothing installed on either side: Web Bluetooth page → six custom GATT characteristics → RSSI-ranked scan results → credentials written to NVS.
- Declarative JSON macros with
wait_for,verify, OCR click-by-label and live TOTP, the sample macro completes a 2FA browser login using nothing but a physical keyboard and a screenshot loop.