Robust Linear Regression Methods for Outlier-Prone Data
Comparing classical and modern approaches to maintain accuracy when outliers distort ordinary least squares.
1The vulnerability of OLS
Ordinary Least Squares regression squares every residual, making it highly sensitive to outliers. A single severe outlier can disproportionately influence the fitted model, as demonstrated when 30% contaminated responses pulled the OLS estimate sharply away from the true relationship. This quadratic penalty structure means outliers dominate the objective function, undermining OLS's reliability in real-world datasets where measurement errors and anomalies are common.
2Huber's compromise solution
Huber regression (1964) reduces outlier influence by transitioning from quadratic to linear penalty for large residuals. Using a threshold δ=1.35, it assigns full weight to small residuals while progressively downweighting larger ones. Though computationally efficient at O(I_H N), Huber never fully rejects outliers - systematically biased measurements can still affect estimates. The method requires careful threshold selection and typically converges when coefficient changes fall below 10^-5.
3RANSAC's consensus approach
Random Sample Consensus (RANSAC) takes an aggressive stance by identifying inlier subsets. For linear regression, it repeatedly samples point pairs, evaluates residuals against a threshold (3σ here), and tracks the largest consensus. With adaptive trial control (p=0.999 confidence), RANSAC can terminate early when finding strong agreement. However, its O(K N) cost grows with outlier fraction, and coherent outlier clusters may falsely appear as valid consensus sets.
4GNC's continuation strategy
Graduated Non-Convexity methods (GNC-GM and GNC-TLS) gradually transform the optimization landscape. Starting from smoother surrogates, they systematically reduce continuation parameter μ to approach the target non-convex loss. GNC-GM uses Geman-McClure weights while GNC-TLS progresses toward truncated least squares. Both require setting inlier-error bounds (c̅²≈6.635 here) and carefully manage the trade-off between rejecting outliers and preserving valid high-residual observations.
5ASOR's adaptive weighting
Adaptive Selective Outlier Rejecting (ASOR, 2024) employs posterior probabilities for soft observation weighting. Unlike hard-rejection methods, ASOR maintains some influence from potentially valid high-residual points while suppressing clear outliers. Its convergence behavior varies across datasets and depends on accurate noise-scale estimation. The method demonstrates particular effectiveness when outlier characteristics (percentage, bias, distribution) are unknown beforehand.
6Computational trade-offs
The methods exhibit distinct computational profiles. OLS remains fastest at O(N) but fails with outliers. Huber regression adds modest overhead (O(I_H N)). RANSAC's cost scales with outlier prevalence (O(K N)). GNC methods require multiple solves per continuation step. ASOR's adaptive weighting may need variable iterations. Runtime comparisons must consider both statistical accuracy and computational efficiency, especially for resource-constrained applications.
7Implementation transparency
All estimators are implemented with visible weighting, rejection, and continuation mechanisms. Shared numerical solvers (like weighted_least_squares) ensure method differences stem from core algorithms rather than implementation details. The public GitHub repository contains complete weight updates, sampling rules, stopping criteria, and continuation schedules, enabling direct comparison of each method's distinguishing operations.
8Missing data vs outliers
Missing values can be excluded pre-fitting when limited and non-systematic. Outliers pose a harder challenge since they must be identified through residuals that depend on the unknown model. This creates a circular dependency - the model affects outlier identification which in turn affects model estimation. Robust methods address this by jointly estimating parameters and observation weights through iterative procedures.
9Physical interpretability advantage
Despite their simplicity, linear models retain value when coefficients require physical interpretation - unlike black-box alternatives. They're indispensable for resource-constrained deployments and low-latency applications. Even complex nonlinear relationships often behave linearly locally, making robust linear regression a versatile tool. This explains why practitioners maintain these methods alongside more sophisticated alternatives.
10Threshold sensitivity analysis
Performance varies significantly with parameter choices. Huber's δ, RANSAC's inlier threshold, and GNC's c̅² all require careful setting. ASOR depends on noise-scale estimation. The experiments use fixed, documented values to enable fair comparison, but practical applications may need cross-validation or domain knowledge to tune these parameters appropriately for specific datasets.
11Monte Carlo evaluation
All methods are tested on identical Monte Carlo realizations featuring different outlier types - varying percentages, biases, variances, distributions and structures. This comprehensive evaluation reveals how estimators perform when contamination characteristics are unknown beforehand and progressively challenging. The standardized testing protocol ensures transparent comparison of statistical accuracy across methods.
12Practical selection guidance
Choice depends on outlier characteristics and application constraints. OLS suffices for clean data. Huber offers smooth improvement when outliers are moderate. RANSAC excels with distinct inlier clusters. GNC methods handle gradual contamination well. ASOR adapts to unknown outlier statistics. For interpretable, robust modeling where computational resources allow, these methods maintain linear regression's relevance in modern machine learning workflows.
Original article: Towards Data Science — Robust Linear Regression Methods for Outlier-Prone Data