# The Hidden Bias in Your AI Adoption Metrics Why your AI feature's success might just be a selection effect - **Source:** [Towards Data Science](https://towardsdatascience.com) - **Published:** 2026-09-13 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/5271a0bf-f14d-440d-aa89-817511c5a47c ## The Illusion of AI Impact Many companies measure AI success by comparing retention between adopters and non-adopters, often showing dramatic 15-point gaps. However, this comparison ignores that engaged accounts self-select into adoption. These accounts were already more likely to retain regardless of the AI feature. The true causal effect of the AI might be just 4 points, with the rest being selection bias from pre-existing account characteristics like engagement levels and organizational readiness. ## The Adoption Hurdle Race AI adoption requires multiple organizational steps: noticing the release, enabling it, trusting it, training teams, integrating workflows, and sustaining usage. Each step acts as a filter for account characteristics like executive sponsorship and change readiness. By the time an account shows as an adopter, it's already a high-performing organization. The AI feature's apparent impact is actually riding on these underlying qualities rather than creating them. ## Three Hidden Effects in One Metric The common adoption metric conflates three distinct effects: 1) Effect on adopters (what product teams want to know), 2) Effect on everyone (finance cares about forced rollouts), and 3) Effect at the margin (impact of changing eligibility rules). Current methods don't isolate these. A 15-point gap might reflect organizational readiness more than feature value, misleading product decisions and investment priorities. ## The Eligibility Rule Blind Spot In a simulated dataset of 40,000 B2B accounts with seat-based eligibility, the true AI effect was +4pp retention. But naive comparison showed +15.4pp - the rest being selection bias from engagement and account size. Even restricting to eligible accounts (25+ seats) still showed +13.9pp gap. The eligibility rule itself creates variation that wasn't customer-chosen, offering cleaner causal analysis opportunities. ## Engagement: The Hidden Variable The simulation includes an unobserved 'engagement' variable affecting both adoption and retention. Engaged accounts are more likely to adopt features and renew regardless. Analysts only see seat counts, tenure, adoption status, and retention outcomes. This latent variable explains why simple comparisons overstate AI impact, as adoption correlates with pre-existing engagement levels that independently improve retention. ## The Data Generation Process In the simulation: accounts have latent engagement scores (normally distributed), seat counts (skewed 3-300), and eligibility (25+ seats). Adoption probability increases with engagement. Retention depends on engagement, a smooth seat-count effect, and the true +4pp AI effect. The resulting data shows how selection effects dominate when analyzing observational adoption data without proper causal methods. ## Common Analytical Mistakes Teams often make two errors: 1) Taking adoption-retention gaps at face value, leading to overestimated ROI, and 2) Trying to 'model away' selection bias with covariates and propensity scores. The article argues for a better approach: leverage the quasi-experimental variation created by eligibility rules rather than fighting selection effects in voluntary adoption data. ## The Margin Matters Most The most actionable insights come from analyzing accounts near the eligibility cutoff (e.g., 24 vs 25 seats). These marginal accounts are most similar in engagement and characteristics, providing cleaner estimates of both eligibility effects and adoption impacts. This approach answers the most common business question: should we expand feature eligibility? ## Business Implications Misreading selection effects as treatment effects leads to poor decisions: over-investing in features that appear successful but aren't causal, misallocating resources based on skewed metrics, and misunderstanding true customer needs. Companies might roll out features broadly expecting 15pp gains when only 4pp is achievable, creating disappointment and wasted effort. ## Better Measurement Approach The article recommends focusing on eligibility rule changes as natural experiments. Since customers don't choose their eligibility status, this variation isn't confounded by engagement. Analyzing retention changes around the cutoff provides more reliable effect estimates. This method works particularly well for SaaS products with clear usage-based eligibility thresholds. ## Implementation Challenges Applying this approach requires: 1) Identifying sharp eligibility rules in your product, 2) Gathering sufficient data around cutoffs, 3) Accounting for other variables that change at the same thresholds, and 4) Communicating the more nuanced findings to stakeholders accustomed to simpler metrics. The payoff is more accurate feature evaluation and better rollout decisions. ## Beyond AI Features This selection effect problem applies to any opt-in feature, but is worse for AI due to higher adoption barriers. The methodology generalizes to other product analytics challenges where voluntary usage creates selection bias. Companies that master this causal approach gain significant competitive advantage in product development and resource allocation decisions. ## Sources covering this story - https://towardsdatascience.com/your-ai-adoption-lift-is-a-selection-effect/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._