# Using NVIDIA NeMo to Convert Agent Failures into Automated Tests

Optimize agent reliability by transforming production trace errors into actionable evaluation benchmarks.

- **Source:** [Nvidia Developer Blog](https://developer.nvidia.com/blog)
- **Published:** 2026-11-04
- **Topic:** [technology](https://brifing.si/c/technology)
- **Original article:** <https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes/>
- **On Brifing:** https://brifing.si/s/cc911e36-649a-4433-8745-d653b5f0c8c3

## Optimizing Agentic AI Reliability

Building reliable AI agents requires moving beyond generic benchmarks to domain-specific validation. Agents often fail by selecting incorrect tools, missing context, or violating operational constraints. NVIDIA NeMo provides a structured framework to analyze production traces, identify recurring failure patterns, and implement targeted improvements to prompts, tools, and system-level control flows.

## Analyzing Traces with NeMo Compass

NeMo Compass automates the inspection of large trace corpora by converting raw execution logs into a structured backlog of issues. By comparing agent actions against defined behavioral rules, such as those in an ETHOS.md file, the tool identifies anomalies and links them to specific trace IDs for efficient debugging and remediation.

## Automating Evaluation with NeMo Author

NeMo Eval Author utilizes coding agents to generate custom evaluations based on identified failures. It supports Harbor and NVIDIA NeMo Gym formats, allowing developers to create verifiers that check if agents meet specific domain requirements. This process ensures that tests evolve alongside the agent, reflecting real-world usage patterns and known failure modes.

## Iterative Harness Improvement Cycle

The optimization cycle involves updating the agent’s harness, such as modifying system prompts or middleware, to address identified flaws. After applying fixes, developers rerun evaluations on both development and held-out tasks. This rigorous verification process prevents regressions and ensures that improvements generalize effectively across diverse scenarios rather than just fitting specific examples.

## Preventing Unauthorized Tool Execution

In a practical enterprise assistant example, the agent incorrectly executed a message-sending tool without user confirmation. By analyzing the trace and updating the system prompt to explicitly require review before action, developers successfully corrected the behavior. This demonstrates how targeted harness changes can resolve critical logic errors in complex agent workflows.

## Engineering Workflow Performance Gains

Applying this loop to Dassault Systèmes engineering workflows yielded significant efficiency improvements. The optimization process reduced malformed tool calls from 142 to zero and achieved a 28% relative improvement in overall evaluation scores. Furthermore, the agent performed tasks with 35% fewer tool calls per run across six distinct engineering simulation scenarios.

## Automated Optimization at Scale

The engineering workflow optimization was achieved through an automated loop running overnight without human intervention. Over a 12-hour period, the system identified and resolved recurring issues where arguments were placed outside expected fields. This highlights the potential for autonomous systems to refine their own performance metrics and improve operational accuracy over time.

## Maintaining Evaluation Integrity

To avoid overfitting, developers must maintain a set of held-out tasks that remain hidden from the coding agents proposing changes. By accepting updates only when they improve both development and held-out sets, teams ensure robust performance. If results remain inconclusive, NeMo Compass can be rerun to generate fresh insights for further refinement.

## Future of Agentic Maintenance

The integration of trace analysis and automated evaluation represents a shift toward systematic agent maintenance. By treating failures as data for new test cases, organizations can build self-improving systems. Future efforts should focus on expanding evaluation suites to cover broader behavioral ranges, ensuring long-term reliability in increasingly complex and technical production environments.

## Sources covering this story

- https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes/ — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._