# Developing Real-Time AI Applications with NVIDIA Holoscan

How to build an endoscopic tool segmentation application using AI coding agents

- **Source:** [Nvidia Developer Blog](https://developer.nvidia.com/blog)
- **Published:** 2026-08-19
- **Topic:** [Data & AI](https://brifing.si/c/data-ai)
- **Original article:** <https://developer.nvidia.com/blog/developing-nvidia-holoscan-applications-with-cli-skills-and-ai-coding-agents/>
- **On Brifing:** https://brifing.si/s/ea3a5d33-4e6a-44c8-8c21-bc92bf9765e4

## NVIDIA Holoscan and AI Coding Agents

NVIDIA Holoscan is a platform for building real-time AI applications in edge computing. HoloHub is a repository containing reference applications and components for this platform. This blog post describes how an endoscopic tool segmentation application was developed using an AI coding agent. Engineers guided the agent through an iterative development process using Holoscan CLI and HoloHub documentation.

## Goal: Endoscopic Tool Segmentation

The project aims to create a real-time endoscopic tool segmentation application. The MONAI endoscopic tool segmentation model and Holoscan sample videos are reused. The application provides live visualization of segmentation masks along with statistical analysis. A deep learning segmentation pipeline is reused, and a comprehensive visualization and telemetry system is added.

## Engineer-Guided AI Agent

Instead of using a single large prompt, developers progress through small, verifiable engineering iterations. Each iteration focuses on a specific goal, and the results feed into the next iteration. The first iteration creates a segmentation dashboard with constraints on model and data reuse. The AI agent completes the application in 40 minutes using HoloHub examples and documentation.

## The Role of Holoscan CLI

Holoscan CLI is central to the development process. Invoked via ./holohub wrapper, the CLI provides a shared execution interface. The AI agent can discover development operations through the CLI. Engineers can also inspect and repeat the same commands. This makes the development process transparent and reproducible.

## Second Iteration: Enhanced Metrics

After reviewing the results of the first iteration, the second iteration improves visual outputs. More meaningful statistics are added: tool area, mask motion, temporal intersection-over-union (stability indicator), edge entropy, FPS, and bounding box position. Unchanging values are removed. A benchmark mode is also added to record actual latency and plot the results in Python.

## Application Modes

The developed application can run in three different modes: visual, benchmark, and export. Visual mode displays live segmentation and metrics. Benchmark mode measures application performance and plots the results. Export mode saves figures to the build folder. Thanks to Holoscan CLI, these modes are discoverable and executable.

## Performance Optimization

In each iteration, application performance is measured and optimized using benchmarks. The AI agent uses Holoscan CLI and documentation to increase throughput and reduce latency. An ablation study shows that the agent achieves the most efficient workflow with the combination of documentation and CLI. Using documentation or CLI alone is less effective.

## The Future of AI-Assisted Development

This project demonstrates the potential of AI coding agents in developing complex real-time AI applications. Engineers can increase efficiency by guiding agents with the right tools. Platforms like Holoscan standardize AI-assisted development processes, paving the way for innovative solutions in edge computing. Applications range from medical imaging to robotics.

## Sources covering this story

- https://developer.nvidia.com/blog/developing-nvidia-holoscan-applications-with-cli-skills-and-ai-coding-agents/ — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._