Evals and Releases Announced for Fin: Evaluate Before and After Going Live
New tools introduced for continuous testing and monitoring of AI assistant Fin's performance.
1Test Fin's Behavior with Evals
Evals is designed to measure Fin's behavior across different scenarios. Using simulations generated from customer conversations, Fin's performance around specific themes (refund requests, tone, etc.) is automatically scored. Each simulation is marked pass/fail against user-defined success criteria. This system helps detect unexpected performance regressions after any changes.
2Create Simulations from Real Conversations
Simulations consist of three components: the customer's opening message and context revealed during conversation, data sources accessible to Fin, and scoring criteria from the LLM judge. Users can create simulations manually or generate them automatically from existing conversations or real email traffic. Each scenario checks whether Fin correctly followed procedures and used appropriate data sources.
3Controlled Updates with Releases
Releases provides a safe environment to test changes before deploying to production. Content updates, procedure additions, or guidance modifications can be bundled together. Team members can collaborate on the same Release and test changes with Evals. New features can be rolled out to all users immediately, gradually, or through A/B testing against Fin's current configuration.
4Continuous Monitoring with Monitors
Every live conversation is checked against set standards by Monitors, with subpar interactions flagged. These flagged conversations provide input for the next improvement cycle. This system can detect even a 1% performance regression in AI assistants like Fin that handle thousands of conversations daily.
5Probabilistic Nature and Challenges
AI assistants like Fin operate probabilistically. They may provide different responses to the same question in varying contexts. Customers also phrase identical questions countless ways, creating thousands of untestable scenarios manually. Success isn't just about correct answers but following procedures, using proper data sources, and maintaining brand voice.
6Unintended Consequences of Changes
Teams constantly update Fin's content: new product launches, policy adjustments, rewritten help articles, or added procedures. These changes carry 'drift' risk where fixes in one area may unexpectedly impact others. For large-scale operations, even a 1% regression could affect thousands of daily interactions negatively.
7Inadequacy of Traditional Solutions
Existing oversight and testing solutions weren't designed for the scale and complexity of AI assistants like Fin. This made it difficult to know exactly how Fin would handle each conversation. Evals and Releases enable support teams to conduct rigorous testing without needing developers or ML specialists.
8Eval-Driven Delivery Discipline
This system brings the 'Eval-driven delivery' discipline used by AI research teams to build and tune products to support teams. AutoDS CX AI Operations Team Lead Hila Horenshtein notes they rerun entire Eval sets after changes to immediately spot regressions. This approach enables continuous quality improvement in AI-powered customer service.