Real Task Cost Comparison Across GPT, Claude, Gemini and Kimi Models
10.6x cost difference found across models for 10 different tasks.
1Hidden Costs of AI Models
Tests on APIs from OpenAI, Anthropic, Gemini and Kimi revealed significantly higher costs than advertised pricing pages indicate. For 10 different realistic product tasks, total cost differences reached 10.6x.
2Invisible Token Consumption
Models are charging for 'thinking tokens' that don't appear in responses. One classification task showed a single-word answer required 197 invisible reasoning tokens.
3Lack of Cost Optimization
CostBench (ACL 2026) research found leading models routinely select non-optimal cost plans. TerminalWorld reports show failed agent attempts burn disproportionately more tokens than successful ones.
4Testing Methodology
Comparison used 10 realistic tasks including classification, RAG QA, multi-turn conversations and agentic planning. Each provider's cost-optimized tier was tested. Full methodology and raw results are published on GitHub.
5Price-Cost Discrepancy
While published rates differ by only 2x, actual task costs varied by 10.6x. This gap stems primarily from output-rate tokens that are billed but never displayed in responses.
6Industry Implications
These findings have critical cost implications for companies integrating AI models into products. Awareness of hidden costs and model selection based on real task performance is crucial for budget planning.