Making AI Peer Review Trustworthy: From Scores to Reproducible Judgments

AI is moving into peer review. But before journals and conferences rely on it, one basic question needs an answer: if the same paper is reviewed by the same model with the same protocol, do we get the same judgment?

A new study, Can We Believe AI Peer Review?, tested this with ~14,000 AI reviews of 200 ICLR papers. The answer: a single AI score is closer to a sample than a measurement.

1. Scores vary. Across 10 identical runs, even the most stable model gave the same score every time for only 56.5% of papers. The swings are small, but enough to change rankings: with one review each, about 1 in 3 paper pairs can’t be told apart.

2. Changing the rubric changes the ruler. Asking for decimal scores reduced variance by ~15%, but also raised the average score by ~0.7. Lower variance doesn’t mean you’re still measuring the same thing.

3. AI disagrees most on whether a revision helped. For the exact same edits, one model scored +0.47, another −0.35. The models broadly agreed on paper quality (r ≈ 0.80), but not on improvement (r ≈ 0.4). Optimizing for one AI’s score may improve the fit with that model, not the paper.

What AI review needs: repeated reviews with uncertainty, cross-model comparison, dimension-level scores, protocol calibration, evidence-based revision checks, and humans making the final call.

A smarter AI reviewer is not necessarily a more reliable one. The next step is reproducible, calibrated, evidence-based judgment.

Context Engineering is overtaking Prompt Engineering. Headroom and planning-with-files point to two adjacent problems: how to cut context that adds nothing, and how to avoid losing critical state over long-horizon execution. For a scientific agent, the wrong Context Compression can delete disconfirming evidence, records of failed experiments, or a method's limitations outright — which is why what is needed is Evidence-aware Compression, not merely Token Reduction.

Machine-readable journal policies will become research infrastructure.

A smarter AI reviewer is not necessarily a more reliable reviewer. Scientific review needs reproducibility, calibrated uncertainty, and evidence-based judgments—not just another score.

Scientific knowledge includes more than successful results. Failed paths can be valuable research assets.

The verification bottleneck is shifting from a theoretical concern to a real challenge at scale.

The reliability of an AI research agent should not be judged only by its final results. We also need to evaluate how it handles evidence, makes decisions, and reports uncertainty.

Two recent studies highlight why this matters.

1. An agent may detect a contradiction but never report it.

Researchers tested four agent frameworks using conflicting evidence. In some cases, the agents identified contradictions during their work but left them out of their final answers, even when those answers were incorrect.

A scientist cannot investigate a disagreement they never see.
https://arxiv.org/abs/2610.12360

2. Responding to experimental feedback doesn't necessarily mean making scientifically sound decisions.

In another study, researchers changed the validation scores of previous experiments while an LLM was selecting configurations for fine-tuning a neural operator. The agent adjusted its next proposal in response to the changed scores. This shows that the agent responds to experimental feedback, but it doesn't prove that its decisions are scientifically sound.
https://arxiv.org/abs/2610.12325

These studies point to a bigger challenge for AI-driven science:

It's not enough for AI agents to produce good results. We also need to know whether their evidence is trustworthy, their decisions can be traced, and their uncertainties are clearly reported.

Over the next few years, the publishing infrastructure we know today may largely disappear. It will shift from a paper pipeline to a knowledge ingestion pipeline.

AI for Science is starting to feel less like a race to make models “better at research” and more like a race to build the infrastructure for science at machine scale.

OpenAI just released 372 AI-generated math results. The U.S. government, Google DeepMind, Meta, and Isomorphic Labs are putting $1.8B into large-scale biological data. At the same time, the National Compute Grid is trying to turn scattered GPUs and compute into shared infrastructure that researchers can actually use.
Put these together and the direction is pretty clear.

When machines can generate hundreds of mathematical results in hours, process billions of cells, and tap into compute across institutions, producing more discoveries may not be the hardest part anymore.

The harder problem becomes what happens after discovery: what should we pay attention to, what can we trust, and what should actually make it into the scientific record?

That means the bottleneck is gradually moving from Discovery to Selection → Verification → Scientific Record.

Over the past few days, several developments that seem unrelated at first glance have started to look very connected when viewed together.

The first is Silverchair’s Research Bridge.

What it is doing is fairly straightforward: packaging content from multiple academic publishers — together with access rights, version information, and licensing mechanisms — into MCP interfaces that AI agents can directly call.

In the past, a researcher would open a journal website, search a database, and download a paper.

In the future, the flow may increasingly look like this:

Research Agent → MCP → Publisher Content

Publishers are already experimenting with attribution, usage metering, licensing, and future business models around this.

This reflects a broader trend in academic publishing:

Scientific knowledge is becoming machine-readable and machine-accessible.

Publishers are beginning to realize that, in the future, papers will not only be read by humans. They will also be consumed by large numbers of AI agents.

But I think there is a much bigger problem here that is often overlooked.

Making existing papers machine-readable only addresses the knowledge consumption side.

The more disruptive impact of AI is happening on the knowledge production side.

arXiv has recently started limiting each submitter to two papers per month.

Why?

One very practical reason is that research output is growing rapidly, while moderation, review, and verification capacity are not scaling at the same pace.

At the same time, models such as Claude Haiku 5.5 have reduced some token prices by close to 90% compared with the previous generation.

That does not mean every agent workflow suddenly becomes 90% cheaper.

But the direction is very clear:

Intelligence is getting dramatically cheaper.

If the cost of running a Scientific Agent drops by 10x, a researcher may no longer run one agent.

They may run 10, 100, or eventually many more.

So what happens to scientific production capacity?

Today, we talk about having “too many papers.”

Tomorrow, we may have:

Too many hypotheses.
Too many claims.
Too many experiments.
Too many machine-generated research outputs.

The expensive part may no longer be generating ideas.

It may become deciding:

Which one is actually new?
Which one is correct?
Which one has enough evidence?
Which one deserves to enter the scientific record?

That is why I increasingly believe the core challenge for the next generation of scientific publishing infrastructure is not only:

How do we make scientific content machine-readable?

It is:

How do we make machine-produced science verifiable?

These are two very different problems.

One direction is about transforming decades of scientific literature into data that agents can read and use.

The other requires us to rethink the research process from the very beginning, so that scientific production generates:

structured claims, evidence, provenance, research traces, verification results, and machine-readable scientific records.

This may be where AI truly forces publishing infrastructure to change.

Several other recent technology shifts reinforce the same direction.

Microsoft has started providing AI agents with operating-system-level sandboxing and permissions.

That tells us something important:

An agent is no longer just a “software feature.”

Once an agent can access files, databases, networks, experimental tools, or even act on behalf of an institution, it needs to be managed more like a real digital actor:

Who is this Agent?
What is it allowed to do?
What data can it access?
Who is responsible for its actions?

OpenAI’s Intelligent UI raises another question from a different angle.

Do scientific results still need to be compressed into a static PDF as the primary form of communication?

If AI can dynamically generate charts, comparison matrices, evidence graphs, and interactive data views based on a research task, then a paper may gradually become just one human-readable view of a much richer Scientific Record.

What persists over time may instead be the underlying scientific objects:

Claim
Evidence
Method
Data
Code
Provenance
Verification

Different humans and agents could then generate different interfaces around those objects depending on the task.

This is why I increasingly think AI-native publishing should not be understood simply as:

“Using AI to help us write papers faster.”

And it should not only mean:

“Making papers machine-readable.”

The deeper shift may be this:

Scientific publishing is moving from document infrastructure to trusted knowledge infrastructure.

On one side, we need machines to be able to access and use existing trusted scientific knowledge.

On the other, we need to solve a much harder problem:

When machines begin producing new scientific knowledge at machine speed, how can the scientific system verify, filter, and establish trust at the same speed?

I think this may become one of the real dividing lines in the next generation of scientific publishing infrastructure:

Machine-readable knowledge is only the beginning.
Machine-verifiable science is the harder problem.

In the next phase of the Agent Economy, the scarce resource will no longer be just intelligence, but permission: who allows which agent to use what capabilities, data, capital, and infrastructure, and under what conditions.

This is especially important for Scientific Agents, because scientific research is fundamentally a knowledge production system governed by strict permissions and access controls.

About 950 Claude agents. Around 10,000 concurrent OpenAI agents. The numbers are striking, but the more interesting story is how their work was organized and checked.

Anthropic reports that its agents searched DNA data and identified an unusual repeat pattern associated with an enzyme system. Human scientists then carried out early lab tests; the system’s primary function remains unknown. OpenAI reports that groups of agents explored different approaches to a Navier–Stokes problem, shared useful intermediate findings and produced a proof later formalized in Lean. It says that effort generated 2.7 million agent messages.

These are company accounts of different kinds of research. Early biological experiments and a formally checked mathematical proof establish different things; neither removes the need for scrutiny by the wider scientific community.

To me, this looks less like one “AI scientist” and more like an artificial research organization. It needs exploratory workers, a way to select promising leads, people with authority over costly or irreversible steps, and verification that does not simply trust the agents’ own conclusions.

It also changes what a research record must preserve. A Methods section cannot reproduce 2.7 million messages. But a human-readable paper could point to a structured trace: which lead was promoted, which alternatives failed, what data and tools were used, who approved the next step, and exactly what was checked.

More agents can widen the search. They do not automatically make a claim more trustworthy. The real test is whether this new research organization can explain its choices and make its results possible to inspect.

Sources: https://www.anthropic.com/news/claude-discovers-novel-enzyme-system · https://openai.com/index/navier-stokes-solution/