AI runs on a $10 microcontroller in groundbreaking demo
Developer achieves 10 tokens per second on an ESP32 using quantization and offloading techniques.
1AI on a microcontroller
A developer named SlvDev successfully ran a language model on a $10 ESP32-S3 microcontroller, typically used for IoT devices. The tiny device achieved a processing speed of about 10 tokens per second. Given microcontrollers' limited memory and processing power, this breakthrough demonstrates unexpected AI potential in resource-constrained environments. The experiment used Microsoft Research's TinyStories model, showing what's possible with extreme optimization.
2Overcoming memory limits
The ESP32-S3 has only 520KB SRAM and 8MB PSRAM, while the original TinyStories model requires 60MB at 16-bit precision. The developer employed quantization to reduce memory needs by 75%, decreasing model size from 60MB to 14.9MB by lowering weight precision from 16-bit to 8-bit. This technique caused minor accuracy loss but made the model viable for microcontroller deployment, showcasing innovative approaches to hardware constraints.
3Leveraging flash storage
Even at 14.9MB, the model exceeded the ESP32's memory capacity. Inspired by Google's Gemma models, the developer used per-layer embedding (PLE) to offload about 12MB of weights to flash storage. This approach minimized performance degradation while reducing active memory requirements to just 2MB. Key components like KV cache and embeddings stayed in PSRAM, while activations used the chip's 520KB SRAM, demonstrating clever memory management.
4Performance achievements
After optimization, the ESP32-S3 processed 9.88 tokens per second - faster than average human reading speed. While TinyStories can only generate short coherent stories and another micro-model called Barista answers espresso-related questions, their operation on a microcontroller represents significant technical progress. These models show that even with severe hardware limitations, basic AI functionality is achievable through creative engineering.
5Practical applications
Current microcontroller models remain proof-of-concept demonstrations. Practical applications still require more powerful devices like Raspberry Pi or smartphones. Google's April-released Gemma 4-E2B-it model runs 5.1B parameters under 1GB memory using 4-bit quantization, enabling local chatbots and personal assistants. While microcontroller-based AI isn't ready for complex tasks, it represents important foundational work for edge computing applications.
6Technical challenges
Memory and processing constraints remain the primary barriers for LLMs on microcontrollers. Techniques like weight offloading to NVMe storage, while useful for large models, dramatically impact performance. PLE mitigates some issues by minimizing flash memory's speed disadvantage compared to DRAM. However, significant optimization breakthroughs are still needed before microcontroller AI becomes practically useful beyond niche applications.
7Industry implications
This development could enable future AI applications in embedded systems. Local AI processing on low-cost IoT devices may reduce cloud dependency, particularly valuable for energy-constrained environments and real-time applications. Research from companies like Microsoft and Google continues advancing edge AI capabilities, potentially transforming how we deploy intelligent systems in resource-limited scenarios across industries.
8Future outlook
Running AI on microcontrollers represents an important step in technology democratization. Affordable AI capabilities could accelerate innovation in developing countries and resource-constrained environments. Advances in quantization and model compression techniques will likely produce more capable micro-models in coming years, making AI more accessible worldwide. This research direction promises to expand AI's reach far beyond current data center-dependent implementations.