MediationBench: Evaluating AI Mediators in Simulated Conflict Scenarios
A new benchmark testing AI performance in resolving 62 distinct fictional disputes.
1Benchmark Design and Scope
MediationBench evaluates AI mediators through 62 authored disputes, ranging from business breakups to interpersonal conflicts. The system forces language models to play both sides of an argument, while the tested mediator decides when to speak, wait, or terminate the session. Rounds are strictly constrained within a fixed 3-to-25-round window per conflict.
2Scoring and Evaluation Mechanism
After each session, an AI judge analyzes the transcript to score five distinct metrics, resulting in a final score out of 100. The system currently uses Sol as the primary judge, though additional judges can be integrated later. Scores are averaged across all 62 conflicts to provide a comprehensive performance overview.
3Provisional Data Interpretation
The benchmark presents results as provisional, meaning small gaps between scores are not statistically significant. Because the research remains a working draft, numbers are subject to change as the paper undergoes revision. Users are advised against assuming that minor differences in rankings reflect meaningful variations in the underlying AI mediator performance.
4Scope and Technical Limitations
This framework measures performance within a closed, simulated environment rather than real-world human interactions. It does not assess human safety, legal compliance, fairness, or the long-term durability of agreements reached. The benchmark serves as a measurement tool for AI behavior, not as proof of efficacy in complex, real-life human mediation scenarios.
5Operational Workflow of Mediators
Mediators operate by selecting from three specific actions after every message: speak, wait, or end the conversation. This decision-making process is tested against 62 fictional scenarios where neither party begins with a generous disposition. By forcing these constraints, the benchmark isolates the mediator's ability to influence the trajectory of an escalating conflict.
6Transparency and Research Access
Jonathan Hendler released the working paper and full methodology on September 13, 2026. The research page provides access to earlier studies, limitations, and downloadable datasets for independent verification. By publishing the exact method and design, the project allows for community scrutiny of how AI models handle adversarial dialogue in controlled settings.
7Evolution of the Benchmark
As a working draft, the project remains in a state of flux with ongoing revisions expected until the final publication. Future updates may include additional judges to refine the scoring process and reduce potential bias from a single evaluator. These iterative improvements aim to stabilize the metrics before reaching a definitive conclusion.
8Significance of AI Mediation
Before deploying AI into sensitive human arguments, rigorous measurement is essential to understand potential outcomes. MediationBench provides a structured approach to quantifying how models navigate disputes. While currently limited to synthetic environments, this research establishes a foundation for evaluating the efficacy of automated systems in managing complex, multi-party social interactions.