# Google Introduces LiteRT Tensor API for On-Device Machine Learning New C++ API simplifies cross-platform model optimization and hardware acceleration for developers. - **Source:** [InfoQ](https://brifing.si/source/68ac18af-a4f3-4f7c-9c2c-44d4af8f89dd) - **Published:** 2026-11-03 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/8236f2cf-ddec-4f96-b481-21966d077ded ## Unified On-Device ML Development Google has launched the LiteRT Tensor API, a lightweight C++ interface designed for building and optimizing machine learning models directly on-device. This tool aims to replace the traditional, labor-intensive method of writing manual C++ computations for specific backends like Metal or OpenCL, offering a more flexible approach for cross-platform model deployment. ## Overcoming Backend Coupling Challenges Previously, developers faced significant friction because implementations were tightly coupled to specific hardware backends. This required constant code rewrites for new targets and complex manual synchronization across CPU, GPU, and NPU architectures. The new API allows developers to build computation graphs operation by operation, ensuring portability across diverse hardware without redundant engineering efforts. ## Advanced Buffer and Weight Management The API introduces zero-copy buffer interoperability, which eliminates the need to copy accelerator buffers during pre-processing, inference, and post-processing stages. Additionally, it supports direct weight ingestion from various sources, including Hugging Face .safetensors files. These features streamline the data pipeline, significantly reducing memory overhead and improving overall execution efficiency on mobile devices. ## AI Coding Agent Integration A core feature of the LiteRT Tensor API is its compatibility with AI coding agents. These agents can construct, modify, validate, and benchmark model graphs by generating plain C++ code. This integration allows for automated optimization, as seen in Google's demonstration where an agent successfully refactored model operations to enhance performance. ## Performance Benchmarks and Real-World Gains Google tested the SAM 2 model using the new API across macOS, iOS, and Android devices. On a MacBook Pro, the implementation ran approximately 1.5x faster than Apple MLX. Furthermore, using a coding agent, developers achieved a performance jump from under 10 to over 30 frames per second on WebGPU platforms. ## Strategic Optimization via C++ Graphs By defining model graphs in plain C++, the coding agent was able to move per-frame decisions from CPU code directly into the graph. It also rewrote operations that previously triggered CPU fallback on WebGPU, utilizing GPU-compatible arithmetics instead. This demonstrates how structured primitives enable agents to perform meaningful, high-level optimization work automatically. ## Expert Guidance on API Usage Arm director of AI Laurence Moroney suggests that developers should prioritize standard model conversion first. The C++ API is best reserved for scenarios requiring granular graph-level control or when specific pre and post-processing tasks must remain on the accelerator. This tiered approach ensures developers use the right tools for their specific performance needs. ## Future Roadmap and Planned Improvements Google is actively developing further enhancements for the LiteRT Tensor API. Planned updates include direct ML Drift backend support, additional NPU-friendly operations, and automatic full-graph quantization. The team is also working on expanding dynamic-shape support to accommodate varying image sizes, batch sizes, and sequence lengths, further increasing the API's versatility for developers. ## Sources covering this story - https://www.infoq.com/news/2026/11/litert-tensor-api/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._