# Strengthening AI Safety Through Independent Third-Party Evaluation Amazon partners with AVERI to rigorously audit internal AI safety evaluation methodologies. - **Source:** [Amazon Science](https://www.amazon.science) - **Published:** 2026-10-20 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/be2fb2ab-cbeb-4416-8dff-ca65073bed35 ## Independent Safety Review Initiative Amazon has partnered with the nonprofit AI Verification and Evaluation Research Institute to audit its internal safety evaluation datasets. This collaboration moves beyond standard output testing, focusing instead on the methodological rigor of the evaluation process itself. The initiative aims to stress-test safety boundaries through objective, external scrutiny of internal quality assurance. ## Methodology of the Audit The AVERI team, led by Miles Brundage, analyzed a random sample of prompts from Amazon’s internal safety suite. Researchers Sean McGregor, A. Feder Cooper, and George Balston assessed these prompts for diversity, methodological soundness, and difficulty calibration. This review occurred without input from Amazon’s internal team to ensure a truly independent perspective. ## Validated Evaluation Strengths The review confirmed that Amazon’s current evaluation process maintains comprehensive topical coverage across critical risk areas like cybersecurity and self-harm. Furthermore, the audit highlighted the diversity of testing approaches, including varied writing styles and conversation turns. The findings also noted a low risk of evaluation leakage due to original content creation. ## Identifying Areas for Improvement AVERI identified specific opportunities to enhance the evaluation suite, such as expanding multilingual coverage more evenly across risk categories. The audit recommended increasing the sophistication of adversarial prompts to better probe model reasoning. Additionally, the team suggested clarifying grading criteria for edge cases and addressing prompts that rely on time-limited references. ## Addressing Signal Quality Issues A key finding involved the reliability of model refusals, which can sometimes be triggered by superficial cues rather than genuine safety logic. AVERI noted that passing these tests does not always guarantee robust behavior in novel situations. This insight prompted a deeper look at how models interpret and respond to complex, adversarial input. ## Concrete Process Enhancements Following the review, Amazon implemented a tiered difficulty framework to ensure evaluations push model limits effectively. The company established a quarterly maintenance cadence to remove redundant prompts and incorporate emerging attack patterns. These updates ensure that the evaluation datasets remain fresh, challenging, and representative of real-world misuse scenarios across all categories. ## Scaling Multilingual Safety Coverage Amazon expanded its multilingual prompt sets to cover all risk categories, moving away from previous concentrations in specific areas. This adjustment ensures that safety evaluations are equally rigorous across different languages. By broadening the range of scenarios tested, the company aims to improve the consistency of safety behavior in diverse global contexts. ## Setting Industry Best Practices This collaboration establishes a model for meta-evaluation, where third parties audit the quality of the evaluation process itself. As AI safety matures, this approach complements traditional red-teaming and benchmarking efforts. Amazon plans to continue engaging independent evaluators, viewing transparency about safety processes as fundamental to building justified confidence in AI systems. ## Sources covering this story - https://www.amazon.science/blog/strengthening-ai-safety-evaluation-through-independent-third-party-review — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._