Running Gemma 4 on a 13-Year-Old Xeon Without a GPU
The unconventional story of running a modern language model on an old server.
1An Unusual Experiment
A 13-year-old HP StoreVirtual server wasn't designed to run modern language models. Yet, with dual Xeon E5-2690 v2 CPUs and DDR3 memory, it managed to run Google's Gemma 4 at 5 tokens per second. This is a significant step in making AI more accessible.
2Pushing Hardware Limits
An old Xeon with AVX1 but no AVX2 or FMA3 seemed inadequate for a modern mixture-of-experts model. However, the right optimizations overcame these limitations. This exemplifies maximizing hardware potential.
3The True Value of AI Agents
An AI agent like Claude quickly diagnosed why the code failed and suggested a solution. This shows AI can be used not just for ready-made solutions but also for solving complex technical problems.
4The Power of Optimization
The ik_llama.cpp fork includes optimizations needed for Gemma 4's MoE inference. Modifications to run without AVX2 enabled performance without loss on older hardware.
5A Real Engineering Feat
Techniques like speculative decoding, CPU-aware mixture-of-experts routing, and flash attention delivered effective results even on old hardware. This proves how optimized modern AI inference can be.
6Low-Cost AI Solutions
Running this model on hardware under $300 shows AI can be accessible to everyone. It offers a cost-effective solution, especially for batch jobs.
7AI and Hardware Evolution
A CPU from 2013 running a 2024 AI model demonstrates how far hardware and software optimizations have come. It shows older hardware can still be useful.
8Open-Source Contribution
Patch ikawrakow/ik_llama.cpp#2138 highlights the importance of community contributions. Open-source projects enable older hardware to run modern models.
9The Future of AI
This experiment shows AI isn't just for high-performance GPUs. Low-cost, accessible solutions can bring AI to broader audiences.
10Conclusion and Takeaways
Running Gemma 4 on an old Xeon is a significant step in democratizing AI technology. It underscores the importance of local models, especially when APIs are down.
Original article: Hacker News — 13 Yıllık Xeon ile GPU Olmadan Gemma 4 Çalıştırmak