# AI Conversations Can Appear Flawless Yet Be Faulty Evaluation methods for AI agents discussed at VB Transform 2026. - **Source:** [VentureBeat](https://venturebeat.com) - **Published:** 2026-07-20 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/8039163a-c2b8-4bb7-a194-b30836a0a30f ## The Flawless Appearance of AI Agents AI agents may appear flawless on their own, but this can indicate a faulty product. This situation is driving companies to change their evaluation methods. ## Shift in Evaluation Methods Companies are moving from scoring individual traces to comparing cohorts of users against a baseline. This method provides more accurate results. ## Automated vs Human Evaluation Automated evaluation is scalable but ungrounded. Human evaluation is not scalable but more reliable. This dilemma is a challenge the industry faces. ## Contrastive Analysis Contrastive analysis detects problems by comparing user cohorts against a baseline. This method reveals issues not visible in a single trace. ## Model Selection and Cost Starting with the most capable model and moving to smaller ones reduces costs. This approach provides a more efficient evaluation process. ## The Role of Humans Using LLMs as judges does not eliminate the role of humans. Humans remain a safety net in corner cases and play a crucial role in building trust in systems. ## Sources covering this story - https://venturebeat.com/data/a-single-ai-agent-conversation-can-look-perfect-and-still-be-broken-leaders-from-langchain-conviva-and-coreweave-said-at-vb-transform-2026 — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._