ESP32-S3 Cluster Runs 1.58-bit LLM via Distributed Pipeline
7-node system executes 0.5B parameter model using ternary quantization over SPI
1Distributed LLM architecture
The system distributes a 0.5B parameter language model across 7 ESP32-S3 chips, with one master node handling tokenization and embedding while six compute nodes process transformer blocks. The master node uses INT4 embeddings (14MB in flash) and communicates with compute nodes via high-speed SPI daisy-chain, passing FP32 hidden state vectors between layers. This architecture demonstrates how to partition LLM workloads across resource-constrained edge devices.
21.58-bit quantization breakthrough
The project implements BitNet's ternary quantization (1.58-bit) for all linear layers, including attention projections and MLP blocks. Each compute node handles 4 transformer blocks with 1.58-bit attention (Q,K,V,O projections) and MLP (Gate,Up,Down projections). The system uses assembly-optimized MAC operations for the ternary weights and maintains FP32 precision for RMSNorm operations and KV cache storage in PSRAM.
3Hardware communication design
The cluster uses a dual-channel SPI implementation where each compute node receives data through SPI Channel B and transmits processed results via SPI Channel A to the next node. The master node initiates the pipeline by sending token embeddings to Node 1 and receives final outputs from Node 6. This daisy-chain topology minimizes wiring complexity while maintaining sufficient bandwidth for hidden state transfers between layers.
4Memory optimization strategy
Custom partition tables allocate flash memory for specific components: 14MB INT4 embeddings on master, layer weights aligned physically for each compute node, and a dedicated 64KB 'fnorm' partition for final layer normalization. The system employs vocabulary pruning to 32K tokens and uses INT4 weight packing for embeddings. KV caching happens in PSRAM to accommodate larger context windows.
5Software toolchain components
The project includes Python tools for model preparation: vocabulary pruning scripts, embedding matrix slicers, BitNet QAT fine-tuning utilities, and binary packers that convert model weights into ESP32-flashable formats. The firmware implements assembly-optimized 1.58-bit linear layers (bitlinear_forward.S) and custom SPI DMA drivers for high-throughput inter-node communication.
6Performance characteristics
By distributing the 0.5B model across 7 chips, each ESP32-S3 handles approximately 71M parameters. The 1.58-bit quantization reduces memory footprint by ~5x compared to FP16 baseline. The system demonstrates that transformer inference can be partitioned across multiple low-power MCUs while maintaining functional accuracy through careful quantization and distributed execution strategies.
7Edge AI implications
This implementation proves that sophisticated LLMs can run on microcontroller clusters without GPUs. The SPI-based communication shows how to scale beyond single-chip memory limits. The project provides a blueprint for deploying medium-sized language models in resource-constrained environments where traditional GPU/TPU solutions are impractical due to power or cost constraints.
8Open-source contribution
Released under MIT license, the project includes complete firmware for both master and compute nodes, detailed wiring guides, architecture diagrams, and model preparation tools. This enables others to replicate the distributed inference approach or adapt it for different model architectures. The work pushes boundaries of what's possible with microcontroller-based AI systems.