How to Evaluate LLMs Before Production
Tips for understanding how LLMs perform in real-world production scenarios.
1The Difference Between Prototype and Production
Language models may perform well on clean benchmark datasets but struggle with real-world inputs in production. Benchmark data is useful during prototyping, but as systems move closer to production, the evaluation problem changes. Real inputs are often ambiguous, and critical context may be missing. Datasets that don't reflect production distribution can be misleading.
2The GitHub Secret Scanning Example
When evaluating LLM performance in GitHub secret scanning, the primary goal was to reduce false positives. However, incorrectly suppressing real credentials can have more severe consequences than reviewing an additional alert. Therefore, precision and recall metrics were not treated equally. Precision was improved while recall was maintained as a safety constraint.
3Prioritize the Product Decision
When an LLM system doesn't meet expectations, instead of adjusting technical components, first define the product decision. Determine which mistakes are acceptable, which metrics should drive the decision, and which safety constraints must be maintained. This approach clarifies the evaluation process and ensures informed decisions.
4Organize Evaluation Criteria into Levels
Organizing evaluation criteria into three levels is beneficial: primary outcome (user benefit), safety constraint (recall), and operational guardrails (latency, cost, reliability). This distinction prevents treating all metrics equally and ensures decisions align with product goals. For example, reducing false positives without violating recall constraints is crucial.
5Treat Offline Evaluation Like Integration Testing
An LLM system continues to evolve after its initial successful evaluation, so evaluation should not be a one-time process. Any meaningful change (e.g., prompt revision, model update) should trigger a reevaluation. This helps detect regressions and unexpected shifts in system behavior, ensuring continuous improvement.
6Repeatable Evaluation Process
Ensuring the evaluation process is repeatable allows each new result to be compared against a known baseline. Recording details like the prompt, model, dataset version, and system configuration for each run helps clarify which change led to which outcome, preventing incorrect conclusions.
7Single-Variable Experiments
When designing experiments, changing one major variable at a time ensures clarity about the cause of results. For example, revising a prompt while keeping other variables constant helps isolate the impact of that change. This approach avoids attributing improvements to the wrong factors.
8Comparing Experiment Results
Comparing hypothetical experiment results helps determine which aligns better with product goals. For instance, an experiment showing a large precision improvement but a significant recall drop may not be suitable. A moderate precision improvement with maintained recall might be more aligned with the goal.
9Prioritize User Benefit
When evaluating an LLM system, defining what success means for the user is critical. Identifying which metrics improve user experience and which safety constraints must be maintained ensures decisions are user-centric. This approach generates evidence that supports product decisions.
10Consider Operational Guardrails
Evaluating an LLM system's performance must include operational guardrails like latency, cost, and reliability. A change that improves quality but makes the system too slow, expensive, or difficult to integrate isn't necessarily an improvement. Operational constraints should be part of the evaluation process.
11Production Compatibility
Assessing whether an LLM system is ready for production requires considering compatibility with the production environment. The system must integrate with existing infrastructure, meet acceptable latency and cost thresholds, and operate reliably. These factors ensure successful deployment.
12General Lessons
These experiences offer general lessons for teams developing LLM systems in areas like code analysis, developer tools, security, and data analysis. Managing the pre-production evaluation process correctly ensures the system succeeds in real-world scenarios. This approach maximizes the potential of language models.
Original article: The GitHub Blog — Üretim Öncesi Dil Modelleri Nasıl Değerlendirilir?