LoRA Speedrun – Public Leaderboard for Fine-Tuning Techniques
Who can fine-tune Qwen2.5-1.5B the fastest on a single L40S GPU?
1The Speedrun Begins
A unique platform to benchmark LoRA fine-tuning techniques: fixed hardware, fixed task, and an open leaderboard. Ready to break the record for fastest training of Qwen2.5-1.5B on GSM8K dataset with ≥57% accuracy?
2Fair Competition Environment
All records are verified with 3 independent runs on identical L40S GPUs. Modal's free compute credits enable universal participation. Attempts and verifications are conducted with full transparency.
3Current Records
Current GSM8K track record: 6m 05s by @Saivineeth147. SQuAD v1.1 track remains open. Two distinct model families and task types test technique generalizability.
4Verification Process
Each submission is re-run with 3 fresh seeds. Code review, security screening, and performance verification are conducted. The entire process is transparently documented and publicly reported.
5Untapped Technique Opportunities
Many techniques like sequence packing, LoRA+, NEFTune, Unsloth kernels remain unexplored. New records can be set through data selection, learning rate tuning, and other optimizations. Ready-to-use templates available.
6Why It Matters
LoRA/QLoRA dominates real-world fine-tuning, but techniques lacked comparability. This project accelerates progress by testing approaches in standardized conditions - essentially the fine-tuning version of nanoGPT speedrunning.