# Baidu Releases ERNIE 5.5 Flash Base Model A high-efficiency sparse MoE model designed for scalable domain adaptation and research. - **Source:** [Hacker News (news.ycombinator.com)](https://brifing.si/source/13c923cd-6d99-442b-9b8b-1181f4d8cd91) - **Published:** 2026-11-05 - **Topic:** [technology](https://brifing.si/c/technology) - **Original article:** - **On Brifing:** https://brifing.si/s/e1b38fc3-f83b-4b68-b51b-ba3a2b1a240e ## New Base Model Architecture Baidu has released ERNIE 5.5 Flash Base, a sparse Mixture-of-Experts language model featuring 280 billion total parameters. With 14 billion parameters activated per token, it was pre-trained from scratch on 33.5 trillion tokens. The model supports a 128K training context, extendable to 1 million tokens for fine-tuning and inference tasks. ## Efficiency Through Hybrid Attention The model utilizes a hybrid attention mechanism combining Multi-head Latent Attention and Heavily Compressed Attention. By compressing distant history in most layers, the architecture significantly reduces memory usage while maintaining performance. Only six layers utilize dense attention, ensuring the model remains computationally efficient during long-context processing and multi-turn interactions. ## Advanced Multi-Router Mechanism Traditional MoE models often suffer from poor expert selection due to single-router limitations. ERNIE 5.5 Flash replaces this with multiple independently initialized sigmoid routers that score experts jointly. This approach reduces sensitivity to noisy features and ensures that expert selection correlates more strongly with actual utility during deep-layer processing. ## Optimized Residuals and Output The model introduces mHC residual connections, allowing layers to read and write to four parallel streams through a constrained mixing matrix. Additionally, the MultiMax output head replaces standard softmax temperature with a learnable piecewise temperature. These innovations improve token prediction accuracy and reduce pre-training loss without requiring significant extra computational overhead. ## ECHO for Future Prediction Instead of using separate modules for multi-token prediction, ERNIE 5.5 Flash employs ECHO. This method reuses the backbone's final layer to predict future tokens, directly training the model to anticipate sequence progression. This integration enhances reasoning and coding capabilities while serving as an effective draft model for speculative decoding applications. ## Specialized Tokenization Strategy The model features a new BPE tokenizer with a 200,000-entry vocabulary. It processes numbers right-to-left in three-digit groups, ensuring place values align across varying number lengths. This specific tokenization design is intended to improve the model's performance in arithmetic tasks and the processing of number-heavy technical documentation or datasets. ## Infrastructure and Training Tools Baidu developed two infrastructure components to support this release: a distributed Muon optimizer and FlashMask-4. The optimizer orthogonalizes parameter updates for efficient large-scale training across thousands of GPUs. Meanwhile, FlashMask-4 optimizes attention kernels for large head dimensions, achieving up to 175% faster forward passes on SM100 hardware compared to previous standards. ## Benchmarking and Performance ERNIE 5.5 Flash Base was evaluated against peers like DeepSeek-V4-Flash and MiMo-V2.5. It demonstrates strong results across reasoning, code generation, and exam benchmarks. Despite using sequence compression in 37 of its 43 layers, the model maintains high performance on long-context tasks, proving the effectiveness of its specialized attention architecture. ## Availability and Licensing The model is currently available on Hugging Face in FP8 and BF16 formats. It is released under the ERNIE 5.5 Flash Base Model License Agreement v1.0. Developers can integrate the model using standard libraries like Transformers, with support for deployment via SGLang and vLLM for high-performance inference and fine-tuning workflows. ## Future Development Path This release serves as a foundational checkpoint for the community to build upon. By providing pre-trained weights, Baidu aims to lower the barrier for domain-specific adaptation and agentic application research. Future iterations, including post-trained versions of ERNIE 5.5 Flash, are expected to further expand the model's utility in specialized fields. ## Sources covering this story - https://ernie.baidu.com/release/ernie-5-5-flash-base/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._