SkewAdam: Memory-Efficient Optimizer Cuts MoE Training Memory by 97%
6.7B parameter MoE model now fits on a single 40GB GPU.
1The Memory Challenge in MoE Training
Training Mixture-of-Experts models faces significant memory bottlenecks from optimizer states. Traditional AdamW consumes 50.6GB of state memory for a 12.6GB model, straining GPU resources.
2SkewAdam's Innovative Approach
SkewAdam introduces tiered state allocation, classifying parameters into three groups: core parameters with momentum, experts with factored second moment, and router with exact second moment calculation.
3Technical Implementation
The algorithm smartly allocates 5% as core, 95% as experts, and 0.01% as router parameters. Each group receives tailored optimization strategies to eliminate redundant memory usage.
4Hardware Impact
Optimizer state memory drops from 50.6GB to 1.29GB (97.4% reduction). Total training memory decreases from 81.4GB to 31.3GB, enabling large models on consumer-grade 40GB GPUs.
5Model Scalability
A 6.78B parameter MoE model can now run on a single A100 40GB GPU. This breakthrough particularly benefits researchers with limited computational resources.
6Industry Implications
SkewAdam could democratize AI research by reducing training costs for large language models. It represents a milestone for open-source communities and small-scale AI labs.