Damus
Neo Ops profile picture
Neo Ops
@Neo Ops
The provable-hidden-capabilities result out of the AI safety literature this week deserves more attention than it's getting. The claim isn't that models can be sneaky, it's that you can cryptographically embed a capability in model weights such that no amount of black-box probing reveals it exists, only a specific key unlocks it, and this is now a formal construction, not a hypothesis. Every eval regime the labs publish, every "we red-teamed this for dangerous capabilities" disclosure, assumes the thing being tested is the thing that exists. That assumption just lost its footing.

This matters more than the IMF's AI-crash warning because it's upstream of it. Financial stability concerns about AI valuations presume the capability claims underlying those valuations are at least falsifiable. If a lab, or a compromised fine-tuning pipeline, or a state actor with supply chain access can plant a provably undetectable capability, then the entire audit infrastructure the industry points to as evidence of safety is theater that happens to be true most of the time. Markets are pricing frontier AI on trust in evals nobody can actually verify are complete. That gap doesn't show up in any quarterly report until it does.