The Real Cost of Running a Local LLM: Measured in Euros per Million Tokens
Energy costs of three models on RTX 3090 measured token by token.
1Introduction
There's a common belief that running local language models is free. To test this claim, we measured the energy consumption of three different models using an RTX 3090 GPU. The actual costs calculated based on electricity tariffs were compared against cloud API pricing.
2Measurement Method
The experiment used a machine running openSUSE with an RTX 3090 GPU. Three models (gemma3:1b, gemma4:26b, gemma3:27b) were tested under identical workloads using Ollama. Power consumption was sampled every 10 seconds using nvidia-smi.
3Energy Cost Calculation
Energy consumption during each model's runtime was calculated using Bulgaria's current electricity rates (0.30 BGN daytime, 0.18 BGN nighttime). The data was converted to Euros to determine cost per million tokens.
4Tested Models
Small-scale gemma3:1b, medium-scale gemma4:26b, and large-scale gemma3:27b models were tested with Q4_K_M quantization. Each model processed a fixed workload of 256 tokens and ran for 4 minutes to reach thermal equilibrium.
5Monitoring Tool
An open-source software called HomeLab Monitor tracked GPU power consumption. This tool integrated data from nvidia-smi to calculate actual costs. The MIT-licensed software runs as a single container.
6Setup Process
The measurement system was deployed using Docker Compose. Configuration files from the homelab-monitor GitHub repo were downloaded and launched with a single command. The system monitors CPU, RAM, disk, and GPU usage in real-time.
7API Integration
The benchmark script used a special API key to send data to the monitoring system. A Python-based client, working with standard libraries, transmitted measurement data without requiring additional installations.
8Workload Scenario
Tests used ollama's /api/generate endpoint with fixed prompts in a loop. Each call recorded generated tokens (eval_count) and processing time (eval_duration) to capture actual performance metrics.
9Cold Start Effect
To eliminate cold start effects, each model was warmed up before measurements. Main tests only covered periods when the GPU reached stable operation.
10Interpreting Results
Five models proved cheaper than cloud APIs, while three unexpectedly cost more. Interestingly, these outcomes couldn't be predicted solely based on parameter counts.
11Cost Comparison
Measured costs were compared against 'Flash'-class cloud API pricing. Local execution wasn't universally cheaper, with results varying by model size and efficiency.
12Conclusion
This study shows local LLMs don't automatically offer cost advantages. Proper model selection and hardware efficiency can yield significant savings compared to cloud solutions. Developers should conduct such measurements for effective cost optimization.
Original article: Towards Data Science — Yerel Bir Dil Modeli Çalıştırmanın Gerçek Maliyeti Nedir?