Google Introduces LiteRT Tensor API for On-Device Machine Learning
New C++ API simplifies cross-platform model optimization and hardware acceleration for developers.
1Unified On-Device ML Development
Google has launched the LiteRT Tensor API, a lightweight C++ interface designed for building and optimizing machine learning models directly on-device. This tool aims to replace the traditional, labor-intensive method of writing manual C++ computations for specific backends like Metal or OpenCL, offering a more flexible approach for cross-platform model deployment.
2Overcoming Backend Coupling Challenges
Previously, developers faced significant friction because implementations were tightly coupled to specific hardware backends. This required constant code rewrites for new targets and complex manual synchronization across CPU, GPU, and NPU architectures. The new API allows developers to build computation graphs operation by operation, ensuring portability across diverse hardware without redundant engineering efforts.
3Advanced Buffer and Weight Management
The API introduces zero-copy buffer interoperability, which eliminates the need to copy accelerator buffers during pre-processing, inference, and post-processing stages. Additionally, it supports direct weight ingestion from various sources, including Hugging Face .safetensors files. These features streamline the data pipeline, significantly reducing memory overhead and improving overall execution efficiency on mobile devices.
4AI Coding Agent Integration
A core feature of the LiteRT Tensor API is its compatibility with AI coding agents. These agents can construct, modify, validate, and benchmark model graphs by generating plain C++ code. This integration allows for automated optimization, as seen in Google's demonstration where an agent successfully refactored model operations to enhance performance.
5Performance Benchmarks and Real-World Gains
Google tested the SAM 2 model using the new API across macOS, iOS, and Android devices. On a MacBook Pro, the implementation ran approximately 1.5x faster than Apple MLX. Furthermore, using a coding agent, developers achieved a performance jump from under 10 to over 30 frames per second on WebGPU platforms.
6Strategic Optimization via C++ Graphs
By defining model graphs in plain C++, the coding agent was able to move per-frame decisions from CPU code directly into the graph. It also rewrote operations that previously triggered CPU fallback on WebGPU, utilizing GPU-compatible arithmetics instead. This demonstrates how structured primitives enable agents to perform meaningful, high-level optimization work automatically.
7Expert Guidance on API Usage
Arm director of AI Laurence Moroney suggests that developers should prioritize standard model conversion first. The C++ API is best reserved for scenarios requiring granular graph-level control or when specific pre and post-processing tasks must remain on the accelerator. This tiered approach ensures developers use the right tools for their specific performance needs.
8Future Roadmap and Planned Improvements
Google is actively developing further enhancements for the LiteRT Tensor API. Planned updates include direct ML Drift backend support, additional NPU-friendly operations, and automatic full-graph quantization. The team is also working on expanding dynamic-shape support to accommodate varying image sizes, batch sizes, and sequence lengths, further increasing the API's versatility for developers.
Original article: InfoQ — Google Introduces LiteRT Tensor API for On-Device Machine Learning