# SkewAdam: Memory-Efficient Optimizer Cuts MoE Training Memory by 97% 6.7B parameter MoE model now fits on a single 40GB GPU. - **Source:** [r/MachineLearning](https://www.reddit.com/r/MachineLearning) - **Published:** 2026-07-22 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/85d47e0c-2e2d-40d9-abed-5e2f41e01f19 ## The Memory Challenge in MoE Training Training Mixture-of-Experts models faces significant memory bottlenecks from optimizer states. Traditional AdamW consumes 50.6GB of state memory for a 12.6GB model, straining GPU resources. ## SkewAdam's Innovative Approach SkewAdam introduces tiered state allocation, classifying parameters into three groups: core parameters with momentum, experts with factored second moment, and router with exact second moment calculation. ## Technical Implementation The algorithm smartly allocates 5% as core, 95% as experts, and 0.01% as router parameters. Each group receives tailored optimization strategies to eliminate redundant memory usage. ## Hardware Impact Optimizer state memory drops from 50.6GB to 1.29GB (97.4% reduction). Total training memory decreases from 81.4GB to 31.3GB, enabling large models on consumer-grade 40GB GPUs. ## Model Scalability A 6.78B parameter MoE model can now run on a single A100 40GB GPU. This breakthrough particularly benefits researchers with limited computational resources. ## Industry Implications SkewAdam could democratize AI research by reducing training costs for large language models. It represents a milestone for open-source communities and small-scale AI labs. ## Sources covering this story - https://www.reddit.com/r/MachineLearning/comments/1v38k1m/skewadam_a_tiered_optimizer_that_cuts_moe_state/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._