Rippling Tested 15 AI Models on Real Payroll Data
The cheapest model tied the most expensive one in performance.
1Rippling's AI Model Test
Rippling tested 15 different AI models on real payroll data. Each model underwent 2,100 test runs. The tests included questions like headcount by department and employee tenure distribution. Actions such as increasing salaries by 10% for employees meeting certain conditions were also tested. Each test either passed Rippling's production correctness checks or failed.
2Test Results
Three key metrics emerged from the tests: pass rate, cost, and the slowest 10% time. Pass rate indicates how often the model completed the task correctly. Cost reflects the dollar amount to run the full test set. The slowest 10% time represents the performance felt by the customer. These metrics were used to measure real-world performance.
3Top Performing Models
Opus 4.6, GPT-5.5 med, and GLM 5.2 stood out in the tests. Opus 4.6 showed the best performance with a 91% pass rate and 154 seconds in the slowest 10% time. GPT-5.5 med excelled in speed with an 89.5% pass rate and 130 seconds. GLM 5.2 was the cheapest model with an 88.7% pass rate and 243 seconds.
4Cost and Performance Balance
The tests revealed that the most expensive models do not always perform the best. For example, the Fable 5 model matched GPT-5.5 low in accuracy but was 3.3 times more expensive and twice as slow. This highlights the importance of B2B companies running their own tests before adopting new models.
5Impact of Model Tuning
Rippling spent five months optimizing settings and tools for the Opus 4.6 model. This work is estimated to have increased the model's accuracy by one or two points. This accounts for a significant portion of the 2.5-point spread among leading models, underscoring the critical role of model tuning in performance.
6Performance of Cheap Models
The tests showed that cheap models are not inexpensive because they do less work. For instance, Grok 4.5 and Opus 5 performed nearly the same amount of work, but Grok 4.5 was three times cheaper. This indicates that price lists are more influential than model performance.
7New Models Aren't Always Better
Rippling's tests demonstrated that newer models do not always perform better. For example, the Grok 4.6 model showed lower accuracy and slower performance compared to its predecessor. This emphasizes the importance of companies conducting their own tests before adopting new models.
8Considerations in Model Selection
The tests highlighted the need to balance accuracy, cost, and speed in model selection. For instance, Opus 4.6 has the highest accuracy rate, while GPT-5.5 low offers a faster and cheaper option. GLM 5.2 stands out as the cheapest model but is slower.
9Implications for B2B Companies
Rippling's tests offer significant insights for B2B companies. Firstly, the newest model may not always perform the best. Secondly, model tuning and testing processes can significantly impact performance. Lastly, cheap models are not inexpensive because they do less work; their price lists are lower.
10Future AI Model Testing
Rippling's tests set an example for future AI model testing. Companies should conduct their own tests and evaluate model performance on real-world data before adopting new models. This can lead to better decisions in terms of both cost and performance.
11Conclusion and Recommendations
Rippling's tests provide a comprehensive approach to evaluating the real-world performance of AI models. Companies should balance accuracy, cost, and speed in model selection and conduct their own tests before adopting new models. This can lead to better outcomes in both cost and performance.
Original article: SaaStr — Rippling, Gerçek Bordro Verileri Üzerinde 15 AI Modelini Test Etti