# Running AI Models on Edge Devices with NVIDIA TensorRT Edge-LLM Streamline embedded AI development using the DGX Spark platform and TensorRT inference runtimes. - **Source:** [Nvidia Developer Blog](https://developer.nvidia.com/blog) - **Published:** 2026-10-30 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/7b2f2416-0207-4a49-a362-22e8a7ba63ca ## Bridging Edge and Data Center NVIDIA DGX Spark 64GB provides a dedicated platform for developers to prototype, fine-tune, and test AI models intended for edge deployment. By utilizing the TensorRT Edge-LLM runtime, engineers can maintain consistent inference workflows across development on Spark and production deployment on hardware platforms like Jetson Thor, DRIVE AGX Thor, and IGX Thor. ## Simplified CLI Workflow The trtedge command-line interface streamlines the setup process by automating model downloads and building optimized TensorRT engines. Developers can initiate chat sessions with models like Nemotron-3.5 Lightning using minimal commands. This toolchain ensures that complex model preparation is handled efficiently, allowing for rapid iteration during the application prototyping phase on embedded hardware. ## Technical Environment Requirements Successful implementation requires a specific software stack, including NVIDIA CUDA 13 and TensorRT 10.16.1.11 or newer. Developers must configure a Python virtual environment, typically using versions 3.10 to 3.12, to manage dependencies. This isolation ensures that the TensorRT Edge-LLM runtime remains compatible with system-level packages while maintaining a clean development workspace. ## Local API Prototyping The trtedge serve command enables developers to host an OpenAI-compatible API server locally on the DGX Spark. This functionality allows for seamless integration testing and application development. By sending requests to the local health and chat completion endpoints, teams can validate model performance and response behavior before deploying to final edge hardware. ## DSpark Speculative Decoding DSpark speculative decoding significantly accelerates token generation by using a draft model to propose multiple tokens for verification. By configuring three draft tokens, developers can observe substantial speed improvements. This technique is essential for physical AI applications that require real-time reasoning within strict latency budgets, such as robotics or cabin monitoring systems. ## Performance Metrics and Memory Testing on DGX Spark 64GB with DSpark enabled demonstrated an increase in mean generation speed from 73.4 to 126.9 tokens per second on HumanEval prompts. Despite this performance boost, peak system memory usage remained controlled at 27.9GB, proving the efficiency of the runtime for memory-constrained edge environments during intensive inference tasks. ## Expanding Model Support Beyond text-only models like Nemotron-3.5 Lightning, the platform supports a variety of architectures, including vision-language models for image interpretation and speech recognition systems. Developers are encouraged to explore the supported models list to tailor their application's capabilities, ensuring the chosen model aligns with specific edge use cases like fleet coordination. ## Strategic Development Path The integration of TensorRT Edge-LLM on DGX Spark marks a shift toward more accessible edge AI development. By standardizing the inference runtime, NVIDIA enables developers to build sophisticated agents that perceive and act locally. Future development should focus on leveraging model-specific workflows and documentation to optimize performance for target embedded platforms. ## Sources covering this story - https://developer.nvidia.com/blog/run-ai-models-for-edge-application-development-with-just-a-few-commands/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._