# How Shopify Achieved 98% Stability in Mobile App Testing Shopify increased mobile app test stability from 50% to 98%. - **Source:** [Shopify Engineering](https://shopify.engineering) - **Published:** 2026-08-12 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/d55ef436-2c11-4168-8a02-635673bc8ce6 ## Issues in Mobile Testing Shopify's mobile app was facing serious issues with end-to-end (E2E) testing. The tests had become so unreliable that they were blocking more good changes than bad ones in pull requests. This led to the tests being completely removed from the CI process. Test stability had dropped to 50%, severely impacting the development process. ## Problems with the Old API Shopify's mobile app E2E tests were running on Appium and WebdriverIO. However, this old API was insufficient in ensuring test reliability. Delays in screen loading caused tests to fail frequently. The tests were checking technical details of the app rather than the user experience. ## New API and Computer Vision Shopify developed a new solution by adding a strict API and computer vision technology on top of Appium. This new API made writing tests easier while also increasing their reliability. Computer vision finds elements on the screen as users see them, making tests more realistic. ## Speed and Reliability of Tests The new API significantly sped up the writing and execution of tests. Using computer vision instead of Test IDs made it easier for tests to find and interact with elements on the screen. This reduced the likelihood of test failures, increasing stability to 98%. ## Reasons for Test Failures In the new system, the reasons for test failures are easier to identify. Each test step takes a screenshot and finds elements based on these images. When tests fail, it's easy to see which element was not found or where the error occurred, allowing for quick fixes. ## CI and Remote Device Testing The new testing framework works seamlessly on local machines, CI environments, and remote device farms. This flexibility ensures that tests run reliably across different environments. ## Pre-Promotion Flakiness Gate Shopify developed a pre-promotion mechanism to check new tests before they are included in the CI process. This mechanism prevents tests with a success rate below a certain threshold from being included in CI, further increasing test reliability. ## Future Applications Shopify plans to use this new testing framework in its other mobile apps. This framework not only increases the reliability of mobile E2E tests but also speeds up the development process. With the advancement of AI technologies, the importance of such frameworks will continue to grow. ## Sources covering this story - https://shopify.engineering/mobile-e2e-testing — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._