Seven weeks ago I pointed Opus 4.8 at the ColdCard firmware and told it to find bugs that lose people money.
The second bullet of my prompt: Weak RNG source.
To be clear I'm not claiming we could have prevented anything. But we were hunting for precisely this kind of bug.
Bugs that are easy to introduce within the C programming language, ones catastrophic in single-device settings.
And we were so fucking close to spotting this one.
For the past year we've been running LLM audits constantly on
@Frostsnap, the cryptography libraries underneath, and time lent poking around at other software which the industry leans on. Patches sometimes going to maintainers privately.
There's a lot of low hanging fruit..
Why did Opus miss?
We hadn't cloned enough of Coldcard's submodules. This bug lives between files, and some of those files weren't on my disk. What was present told a self consistent story, with convincing docstrings at the boundary: "best-quality high entropy TRNG bytes".
More significantly, this audit was conducted 4 days after the US government export controlled Fable.
I can't tell you whether things would be different if bitcoin researchers had frontier model access. The models have already changed underneath us.
When trying to run this research today, Fable's safeguards kick in and it drops back to Opus.
During the frantic investigation, researchers reproducing this bug reached for Kimi, a Chinese model, because western models refused on cybersecurity guardrails.
What I can tell you is that who gets access to frontier capability for this kind of work is now decided by governments, vendors, and big tech consortiums.
I've uploaded the original audit document, and the metadata of my transcript that survived claude code's default 30 day cleanup period :/
https://gist.github.com/nickfarrow/4e97c71c8f1acd01aedce671621081d2It's true that capable models have been available to anyone for a while, and a security researcher still didn't get there first.
Not because too many people had access, but because hardly anyone is looking.
I ran this audit out of curiosity. A lot of them turn up something real. This one nearly did. We need more skilled people doing this, with more resources.
Restricting access to these models won't stop vulnerabilities from being exploited. You'll just have fewer of us looking.
