Gary King and Steven Worthington posted a paper on Oct 22: AI Article Reviews Dominate Human Peer Reviews at Their Best. Please read it. Gary King (as I can gather) is one of the most distinguished professors at Harvard and is of course all over the National Academies.
I'm a professor, a journal editor, and have research leadership roles and so I know have to be careful about this subject, because the implications are serious in both directions.
Even with those responsibilities, it is perhaps better to be honest about uncomfortable truths (with caveats) as we are woefully under-prepared as a research community. I shared my thoughts about research capabilities of AI and a 'Code Red' to my students back in April (here, and was surprised to watch it rack up 100k views in a few hours.) Again, if you do, please watch the full snippet and not just the headline.
Now, about the paper:
- The "baseline humans"aren't weak Reviewers were hand-picked from other institutions, paid, instructed to read the whole thing and write a couple of pages of constructive feedback.
- The AI was Claude Opus 4.8; Judging was by GPT-5.5, plus three former editors of leading journals
- AI "won" 100% of decided pairs under the AI judge and 87% under the human editors.
- The human editors agreed with the AI judge (87%) more often than they agreed with each other (80%).
I can predict the first reaction from the some rigorous, goalpost pushing people I know (and truly love!): “well, not in my field,” though often the caveat is that their use of AI amonuts to plugging something in to the google bar. The paper's corpus included formal mathematical modeling and statistical analysis, qualitative work, and the AI won in every grouping including novelty, research style, faculty rank, quality of the journal, etc.
.png){kind=link}
For those of us who serve as journal editors, we acknowledge that finding good reviewers is becoming almost impossible, but the uncomfortable truth is that frontier AI models (at this point of time) give better reviews than the average reviewer at top journals. Well.. not just the mean, but it may have easily gone past 1 sigma.
My [uncomfortable] guess: pre-submission AI review becomes normal, then expected, then required. Human reviewers get reserved for the hard cases/top papers which is where we were always most valuable and least replaceable anyway.
Now.. the most important part: Peer review is a small part of the bigger issue: The only thing standing between a [theoretical] claim and its acceptance is other humans understanding the argument. Once machines write the proofs and machines check them, that step becomes optional, because understanding is "expensive" and not bean countable in our bean counting world. There is thus a risk of losing control of human understanding of science... and of course.. as in the aforementioned snippet which I am posting below, training and evaluating future generations of scientists is going to be a big challenge moving forward.