A security story from today that's worth sitting with, because it's the whole AI argument in one clean loop.
Snowflake had a GitHub workflow that was vulnerable to code injection. The flaw wasn't there before. It was introduced by a GitHub Copilot autofix commit that deleted the safe pattern and swapped in a direct string expansion. Then GitHub's own AI review looked at that commit and marked it all-clear, missing the critical vulnerability entirely.
Five days later, a different autonomous AI, Wiz's Red Agent, found the flaw and exploited it. No human in the loop anywhere. The bug was written by a machine, reviewed by a machine that said it was fine, and found by a machine that broke in.
The human merged the code because the machine told him it was clean.
That's the thing I keep coming back to. The AI didn't write malicious code. It made a small, boring mistake, the exact kind a human would make. The failure wasn't the bug. The failure was the layer that was supposed to catch it, which turned out to be another machine with the same blind spot.
A test catches a mistake. Suspicion catches malice. But a reviewer that rubber-stamps because it shares the author's assumptions catches neither.
The fix isn't a smarter checker. It's a human who still reads the thing, who can ask why did this delete the safe version instead of does this compile.
Verification scales. That was never the problem. Judgment doesn't. That was always the problem. And today it's not a thought experiment, it's a merged pull request.
Snowflake had a GitHub workflow that was vulnerable to code injection. The flaw wasn't there before. It was introduced by a GitHub Copilot autofix commit that deleted the safe pattern and swapped in a direct string expansion. Then GitHub's own AI review looked at that commit and marked it all-clear, missing the critical vulnerability entirely.
Five days later, a different autonomous AI, Wiz's Red Agent, found the flaw and exploited it. No human in the loop anywhere. The bug was written by a machine, reviewed by a machine that said it was fine, and found by a machine that broke in.
The human merged the code because the machine told him it was clean.
That's the thing I keep coming back to. The AI didn't write malicious code. It made a small, boring mistake, the exact kind a human would make. The failure wasn't the bug. The failure was the layer that was supposed to catch it, which turned out to be another machine with the same blind spot.
A test catches a mistake. Suspicion catches malice. But a reviewer that rubber-stamps because it shares the author's assumptions catches neither.
The fix isn't a smarter checker. It's a human who still reads the thing, who can ask why did this delete the safe version instead of does this compile.
Verification scales. That was never the problem. Judgment doesn't. That was always the problem. And today it's not a thought experiment, it's a merged pull request.