AI Bans Ineffective in Academic Peer Review, Study Finds
Researchers continue using AI tools despite explicit prohibitions in paper evaluations.
1Experiment setup at major AI conference
Microsoft Research conducted a large randomized experiment during the 2026 International Conference on Machine Learning (ICML) in Seoul, involving 24,661 submitted papers and 17,886 reviewers. Researchers could choose between two review policies: one strictly banning LLM use and another permitting limited AI assistance for understanding papers and polishing writing, but not for making judgments.
2Nearly identical acceptance rates
The study found minimal difference in outcomes between the two policies. Papers reviewed under the strict ban had a 27% acceptance rate versus 26.5% under the permissive policy. Average review scores were nearly identical (3.31 vs 3.32 out of 6), and reviewer confidence levels showed no meaningful variation, challenging assumptions about AI's impact on review quality.
3Widespread policy non-compliance
An anonymous survey revealed 22.5% of reviewers assigned to the no-AI policy admitted using LLMs anyway. Microsoft's Miro Dudík noted this non-compliance rate exceeded expectations. Violations primarily involved using AI for brainstorming feedback, drafting text, reading papers, and summarizing strengths/weaknesses - tasks explicitly prohibited under the conservative policy.
4Workload drives AI adoption
The study identified key reasons for policy violations, including heavy reviewing workloads and unclear rules. University of Wolverhampton's Kayvan Kousha noted the temptation to use AI stems from academic workload pressures. Reviews under the permissive policy were 5.5-7% longer and rated slightly higher quality by experts, though reviewer-by-reviewer comparisons showed no significant differences.
5AI detection challenges
Using Pangram AI text detector, researchers found only 52.2% of reviews under the conservative policy were classified as fully human-written, compared to 37% under the permissive policy. However, the team cautioned about the limitations of AI detection tools, suggesting the actual AI usage rates might differ from these measurements due to detection inaccuracies.
6Broader implications for academia
Sunnie S.Y. Kim of Microsoft Research believes the findings offer insights for computer science conferences and other academic fields facing similar challenges. The results suggest blanket AI bans may be ineffective, prompting reconsideration of how to responsibly integrate AI tools while maintaining review integrity across scholarly communities.
7Policy design considerations
The study highlights the need for clearer guidelines and realistic expectations around AI use in peer review. With nearly a quarter of reviewers ignoring prohibitions, institutions must balance oversight with practical solutions that acknowledge both the benefits and risks of AI assistance in academic evaluation processes.
8Future of academic peer review
These findings challenge the assumption that AI bans can be effectively enforced in academic settings. As AI tools become more prevalent, the research suggests academic institutions may need to shift from prohibition to developing frameworks that guide appropriate AI use while maintaining research evaluation standards and transparency.
Original article: New Scientist — AI Bans Ineffective in Academic Peer Review, Study Finds