LoRA Speedrun – Public Leaderboard for Fine-Tuning Techniques
How fast can you fine-tune Qwen2.5-1.5B to ≥57% on GSM8K using a single L40S GPU?
1Competitive LoRA Fine-Tuning Platform
This project enables fair comparison of LoRA-based fine-tuning techniques on real hardware. It features a leaderboard for fastest fine-tuning times on two fixed tasks (GSM8K math and SQuAD QA) using a single L40S GPU. Each record is independently verified by running 3 times under identical conditions.
2Verification Process
Participants submit PRs with their code and results. After automated security screening, submissions are re-run 3 times with fresh seeds in a network-blocked Modal sandbox. All runs must meet target accuracy, with official time being the average. Model and data integrity are protected through hash verification.
3Current Records
The GSM8K track leader is sequence packing + completion-only loss masking technique, tuning Qwen2.5-1.5B to 61.1% accuracy in 6m05s. On SQuAD v1.1, the baseline record is 77.5% accuracy in 11m08s with SmolLM2-1.7B. The dual-track system tests technique generalization across different model families and task types.
4Access & Participation
Modal's free monthly compute credits enable anyone to participate. Official timing occurs in the Modal L40S sandbox. While local experimentation is possible on 24GB+ GPUs, sandbox verification is required for leaderboard. Starter templates and setup scripts lower the entry barrier.
5Untapped Opportunities
Unclaimed techniques include 1-epoch aggressive LR schedules, data pruning, QLoRA NF4/bf16 tradeoffs, rsLoRA/DoRA/PiSSA init, LoRA+ asymmetric LR, NEFTune noise, and Unsloth kernel optimizations. Techniques proving successful on both diverse tracks demonstrate true generalization capability.
6Transparency in ML Research
By testing disparate techniques on standardized hardware and tasks, this project measures real progress where paper results are incomparable. It brings what nanoGPT speedrun did for pretraining to parameter-efficient fine-tuning. With open-source record submissions and verification reports, it establishes a trusted benchmarking environment for the ML community.