# Rippling Tested 15 AI Models on Real Payroll Data The cheapest model tied the most expensive one in performance. - **Source:** [SaaStr](https://www.saastr.com) - **Published:** 2026-08-18 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/cfb72bca-bba0-4da7-a819-32d9f486e24a ## Rippling's AI Model Test Rippling tested 15 different AI models on real payroll data. Each model underwent 2,100 test runs. The tests included questions like headcount by department and employee tenure distribution. Actions such as increasing salaries by 10% for employees meeting certain conditions were also tested. Each test either passed Rippling's production correctness checks or failed. ## Test Results Three key metrics emerged from the tests: pass rate, cost, and the slowest 10% time. Pass rate indicates how often the model completed the task correctly. Cost reflects the dollar amount to run the full test set. The slowest 10% time represents the performance felt by the customer. These metrics were used to measure real-world performance. ## Top Performing Models Opus 4.6, GPT-5.5 med, and GLM 5.2 stood out in the tests. Opus 4.6 showed the best performance with a 91% pass rate and 154 seconds in the slowest 10% time. GPT-5.5 med excelled in speed with an 89.5% pass rate and 130 seconds. GLM 5.2 was the cheapest model with an 88.7% pass rate and 243 seconds. ## Cost and Performance Balance The tests revealed that the most expensive models do not always perform the best. For example, the Fable 5 model matched GPT-5.5 low in accuracy but was 3.3 times more expensive and twice as slow. This highlights the importance of B2B companies running their own tests before adopting new models. ## Impact of Model Tuning Rippling spent five months optimizing settings and tools for the Opus 4.6 model. This work is estimated to have increased the model's accuracy by one or two points. This accounts for a significant portion of the 2.5-point spread among leading models, underscoring the critical role of model tuning in performance. ## Performance of Cheap Models The tests showed that cheap models are not inexpensive because they do less work. For instance, Grok 4.5 and Opus 5 performed nearly the same amount of work, but Grok 4.5 was three times cheaper. This indicates that price lists are more influential than model performance. ## New Models Aren't Always Better Rippling's tests demonstrated that newer models do not always perform better. For example, the Grok 4.6 model showed lower accuracy and slower performance compared to its predecessor. This emphasizes the importance of companies conducting their own tests before adopting new models. ## Considerations in Model Selection The tests highlighted the need to balance accuracy, cost, and speed in model selection. For instance, Opus 4.6 has the highest accuracy rate, while GPT-5.5 low offers a faster and cheaper option. GLM 5.2 stands out as the cheapest model but is slower. ## Implications for B2B Companies Rippling's tests offer significant insights for B2B companies. Firstly, the newest model may not always perform the best. Secondly, model tuning and testing processes can significantly impact performance. Lastly, cheap models are not inexpensive because they do less work; their price lists are lower. ## Future AI Model Testing Rippling's tests set an example for future AI model testing. Companies should conduct their own tests and evaluate model performance on real-world data before adopting new models. This can lead to better decisions in terms of both cost and performance. ## Conclusion and Recommendations Rippling's tests provide a comprehensive approach to evaluating the real-world performance of AI models. Companies should balance accuracy, cost, and speed in model selection and conduct their own tests before adopting new models. This can lead to better outcomes in both cost and performance. ## Sources covering this story - https://www.saastr.com/rippling-ran-2100-scored-agent-runs-per-model-on-real-payroll-data-the-cheapest-model-tied-the-most-expensive-one/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._