Baidu Releases ERNIE 5.5 Flash Base Model
A high-efficiency sparse MoE model designed for scalable domain adaptation and research.
1New Base Model Architecture
Baidu has released ERNIE 5.5 Flash Base, a sparse Mixture-of-Experts language model featuring 280 billion total parameters. With 14 billion parameters activated per token, it was pre-trained from scratch on 33.5 trillion tokens. The model supports a 128K training context, extendable to 1 million tokens for fine-tuning and inference tasks.
2Efficiency Through Hybrid Attention
The model utilizes a hybrid attention mechanism combining Multi-head Latent Attention and Heavily Compressed Attention. By compressing distant history in most layers, the architecture significantly reduces memory usage while maintaining performance. Only six layers utilize dense attention, ensuring the model remains computationally efficient during long-context processing and multi-turn interactions.
3Advanced Multi-Router Mechanism
Traditional MoE models often suffer from poor expert selection due to single-router limitations. ERNIE 5.5 Flash replaces this with multiple independently initialized sigmoid routers that score experts jointly. This approach reduces sensitivity to noisy features and ensures that expert selection correlates more strongly with actual utility during deep-layer processing.
4Optimized Residuals and Output
The model introduces mHC residual connections, allowing layers to read and write to four parallel streams through a constrained mixing matrix. Additionally, the MultiMax output head replaces standard softmax temperature with a learnable piecewise temperature. These innovations improve token prediction accuracy and reduce pre-training loss without requiring significant extra computational overhead.
5ECHO for Future Prediction
Instead of using separate modules for multi-token prediction, ERNIE 5.5 Flash employs ECHO. This method reuses the backbone's final layer to predict future tokens, directly training the model to anticipate sequence progression. This integration enhances reasoning and coding capabilities while serving as an effective draft model for speculative decoding applications.
6Specialized Tokenization Strategy
The model features a new BPE tokenizer with a 200,000-entry vocabulary. It processes numbers right-to-left in three-digit groups, ensuring place values align across varying number lengths. This specific tokenization design is intended to improve the model's performance in arithmetic tasks and the processing of number-heavy technical documentation or datasets.
7Infrastructure and Training Tools
Baidu developed two infrastructure components to support this release: a distributed Muon optimizer and FlashMask-4. The optimizer orthogonalizes parameter updates for efficient large-scale training across thousands of GPUs. Meanwhile, FlashMask-4 optimizes attention kernels for large head dimensions, achieving up to 175% faster forward passes on SM100 hardware compared to previous standards.
8Benchmarking and Performance
ERNIE 5.5 Flash Base was evaluated against peers like DeepSeek-V4-Flash and MiMo-V2.5. It demonstrates strong results across reasoning, code generation, and exam benchmarks. Despite using sequence compression in 37 of its 43 layers, the model maintains high performance on long-context tasks, proving the effectiveness of its specialized attention architecture.
9Availability and Licensing
The model is currently available on Hugging Face in FP8 and BF16 formats. It is released under the ERNIE 5.5 Flash Base Model License Agreement v1.0. Developers can integrate the model using standard libraries like Transformers, with support for deployment via SGLang and vLLM for high-performance inference and fine-tuning workflows.
10Future Development Path
This release serves as a foundational checkpoint for the community to build upon. By providing pre-trained weights, Baidu aims to lower the barrier for domain-specific adaptation and agentic application research. Future iterations, including post-trained versions of ERNIE 5.5 Flash, are expected to further expand the model's utility in specialized fields.
Original article: Hacker News (news.ycombinator.com) — Baidu Releases ERNIE 5.5 Flash Base Model