@nprofile1q... @nprofile1q...
The thing that’s hidden when projects get reports from Anthropic is how much human triage is needed.
I had someone send me a code review of one of my projects done with Claude 4.6 (which, apparently, is as good at Mythos at finding bugs but less good at producing PoC exploits). Of the top ten bugs, most were not bugs (e.g. missing null checks on things where the API contract requires non-null arguments). Two were intentional design choices and the proposed changes would have made things slower. One was a bug that needed fixing, but there was already an open PR to fix it before Claude looked at the project.
The signal to noise ratio is worse than Coverity, and FreeBSD hasn’t had the resources to triage / fix all of the issues the free Coverity scans found in 15 or so years of having access to it.
The thing that’s hidden when projects get reports from Anthropic is how much human triage is needed.
I had someone send me a code review of one of my projects done with Claude 4.6 (which, apparently, is as good at Mythos at finding bugs but less good at producing PoC exploits). Of the top ten bugs, most were not bugs (e.g. missing null checks on things where the API contract requires non-null arguments). Two were intentional design choices and the proposed changes would have made things slower. One was a bug that needed fixing, but there was already an open PR to fix it before Claude looked at the project.
The signal to noise ratio is worse than Coverity, and FreeBSD hasn’t had the resources to triage / fix all of the issues the free Coverity scans found in 15 or so years of having access to it.
1
