Meta Introduces Fast On-Device AI with Muse Glimmer on ExecuTorch
ExecuTorch support boosts AI performance on NVIDIA and Apple devices.
1Muse Glimmer Announced
Meta unveiled Muse Glimmer, a 30-billion-parameter open-weight model distilled from Muse Spark. Optimized for on-device AI workflows, it handles text and image inputs with 128K+ token context. Designed for NVIDIA GPUs and Apple silicon Macs, it represents a leap in local AI capabilities. The model's architecture supports complex agentic workflows while maintaining efficiency.
2ExecuTorch Advantages
Unlike traditional local AI frameworks, ExecuTorch eliminates the need to reimplement models across backends. Researchers develop in PyTorch and export to ExecuTorch, which handles backend-specific optimization. It supports Triton on CUDA and MLX-native with custom Metal for Apple silicon. This approach significantly simplifies deployment of novel architectures and multimodal models while maintaining performance.
3DFlash Speculative Decoding
Muse Glimmer employs DFlash, a parallel diffusion-based speculative decoding algorithm for low latency. Tests showed DFlash improves performance by 52.8%, reaching 33 tokens/sec. Weight sharing allows both target and draft models in a single PTE file. The dynamic block dimension export enables runtime-selectable configurations. This innovation is particularly valuable for real-time applications requiring responsive AI.
4Prebuilt PTE Packages
Meta released verified PTE packages on Hugging Face for NVIDIA CUDA and Apple Silicon. These include text-only and text-plus-image variants, with optional DFlash support. The ready-to-use packages offer the fastest path to deployment. Each artifact bundle contains everything needed to run the model on supported hardware, significantly reducing setup complexity for developers.
5Building Custom PTEs
ExecuTorch enables users to create their own PTEs from GGUF checkpoints. The export process automatically compiles and autotunes Triton kernels for detected GPU architectures. Users can select backend, modality, context length, and DFlash usage during export. For optimal results, exporting on the target hardware architecture is recommended. This flexibility supports diverse deployment scenarios.
6Execution Examples
ExecuTorch provides CMake presets for Muse Glimmer runners. Users can build for CUDA or MLX backends with or without DFlash. Command-line examples demonstrate tasks like image description. The framework also supports agent servers, enabling integration with tools like Pi coding assistant. This architecture facilitates creation of sophisticated AI agents capable of complex, multi-step workflows.
7Performance Metrics
Benchmarks show Muse Glimmer achieves 21.6 tokens/sec on NVIDIA A100 with text-only input. DFlash boosts this to 33.0 tokens/sec (52.8% improvement). Similar gains were observed on Apple M5 Max. The model shows high acceptance rates for coding prompts. These results validate its suitability for real-world applications requiring both quality and responsiveness.
8New Era for AI Agents
The Muse Glimmer-ExecuTorch integration unlocks new possibilities for advanced on-device AI agents. It particularly enhances capabilities for complex tasks like coding, testing, and package management. The performance improvements and low latency enable real-time applications previously challenging for local models. This technology may inaugurate a new generation of AI-powered productivity tools that operate entirely on user devices.
Original article: PyTorch Blog — Meta, Muse Glimmer ile Cihaz Üzerinde Hızlı AI Desteği Sunuyor