Livenerf: Tracking Post-Launch Capability Drift in AI Models
New benchmark detects if frontier models degrade after deployment
1Purpose of Livenerf
Livenerf is a specialized benchmark designed to detect whether frontier AI models degrade in capability after launch. It focuses specifically on Anthropic's Claude Opus 5.5 model, tracking performance changes through Claude Code subscriptions. The tool addresses widespread but unverified reports of model 'nerfing' - where models allegedly receive silent downgrades through quantization, routing changes, or reduced computational effort.
2Measurement Methodology
The benchmark uses a deterministic approach with frozen prompts, pinned CLI versions, and exact graders to isolate model performance changes. It statistically measures drift across thousands of samples using Inspect, the UK AI Security Institute's open-source evaluation framework. The statistical methods follow Anthropic's own 'Adding Error Bars to Evals' paper, ensuring methodological rigor. Output token count serves as an early indicator of potential performance changes.
3Implementation Details
Livenerf runs daily for 30 days following model launch, with days 1-10 establishing the baseline. The system requires Python 3.11+, uv, and a Claude Code installation. It uses a Max subscription by default but works with any claude -p capable setup. The benchmark maintains strict version control, pinning the CLI version and refusing to run if version mismatches occur, as updates could be mistaken for model changes.
4Statistical Approach
The primary metric tracks paired per-item score differences against baseline with clustered standard errors. A negative delta indicates performance degradation. The system requires two consecutive 10-day windows showing statistically significant change (99% interval excluding zero) with at least a 3-point effect size before declaring a change. Control arms help distinguish model changes from measurement artifacts.
5Validation Process
Before production runs, livenerf undergoes rigorous validation including positive controls and A/A checks. The design phase calibrates questions to find those the model sometimes misses, predicting minimum detectable effects. Validation proves the system can detect known degradations. All procedures are pre-registered in PREREGISTRATION.md before data collection begins, with timestamps providing auditability.
6Budget Management
The benchmark operates within subscription plan limits, expressed in weekly meter points to avoid competing with normal usage. It automatically skips runs when usage exceeds 75% of weekly or 60% of 5-hour budgets, retrying hourly until successful. Users must manually pin CLI versions and optionally back up binaries to prevent automatic updates from interfering with measurements.
7Output Analysis
Results show per-benchmark performance against baseline plus output token count changes, which often precede accuracy shifts. The system generates charts tracking each arm's performance and token usage patterns. Importantly, livenerf measures the model as served through Claude Code subscriptions, which may differ from raw API performance. Launch week serves as reference point rather than ground truth.
8Design Constraints
Tasks must be exactly gradable, fall in the 30-70% difficulty range, and be inexpensive to run daily. The benchmark maintains version control for all components - any changes to tasks, prompts or graders create new versions rather than silently replacing old ones. Frozen panel items remain private with only hashes published, while procedural generators and graders accept public contributions.
9Transparency Measures
Livenerf emphasizes transparency with public raw data, pre-registered thresholds, and pure-function graders. The project acknowledges using Claude (the measured model) in its development, highlighting the importance of methodological safeguards. All design decisions, calibration results, and validation checks are documented in markdown files within the repository for public review.
10Practical Implementation
Users implement the benchmark through cron jobs (Linux/macOS) or scheduled tasks (Windows) that run daily with hourly catch-up attempts. The system provides commands to check calibration, validate the setup, and analyze results. Inspect integration allows browsing transcripts, scores, and token counts. The design automatically generates documentation about the panel and predicted detection capabilities.
11Significance for AI Development
Livenerf addresses a critical gap in AI model evaluation by providing systematic post-launch monitoring. Its rigorous methodology offers an alternative to anecdotal reports of model degradation. The project demonstrates how open-source tools can enhance transparency in commercial AI systems, potentially influencing how companies monitor and communicate model performance changes over time.