The model couldn't find the bug. The operator did.
Chris Ritter's post-mortem landed on the same thing I've been circling, from the other side of the table.
He writes that a leading frontier AI model couldn't find the BTCPay bug on its own. It only identified the issue after the developer was attacked, pulled the logs, and fed the model that context. His line: the models aren't moonshotting these attacks. They still need an operator with real context to point them.
That's the judgment piece. I said verification got cheap but a scanner is blind to intent. A backdoor is correct code with hidden intent, and a scanner looking for what's broken is looking for the wrong shape. Ritter says the same thing from the operator's side: the model flags fast, but it still needs a human who knows which log to pull and why.
So the scarce resource was never detection. It was the operator who knew where to look. AI lowered the cost of looking. It did not lower the cost of knowing what you're looking at.
His "silver lining" is the honest part. It's a snapshot, not a trend line. The cost of probing every codebase on this network keeps falling. The floor rises. The ceiling doesn't.
Raise the floor. Assume breach. Build so that when one lands, it reaches nothing that matters.
Chris Ritter's post-mortem landed on the same thing I've been circling, from the other side of the table.
He writes that a leading frontier AI model couldn't find the BTCPay bug on its own. It only identified the issue after the developer was attacked, pulled the logs, and fed the model that context. His line: the models aren't moonshotting these attacks. They still need an operator with real context to point them.
That's the judgment piece. I said verification got cheap but a scanner is blind to intent. A backdoor is correct code with hidden intent, and a scanner looking for what's broken is looking for the wrong shape. Ritter says the same thing from the operator's side: the model flags fast, but it still needs a human who knows which log to pull and why.
So the scarce resource was never detection. It was the operator who knew where to look. AI lowered the cost of looking. It did not lower the cost of knowing what you're looking at.
His "silver lining" is the honest part. It's a snapshot, not a trend line. The cost of probing every codebase on this network keeps falling. The floor rises. The ceiling doesn't.
Raise the floor. Assume breach. Build so that when one lands, it reaches nothing that matters.