Risk-Based Routing: Optimizing Human-in-the-Loop Systems
Balancing safety and efficiency in automated query approvals.
1Initial Implementation Challenges
Three weeks after deploying a text-to-SQL agent, a request to clean up test rows nearly resulted in deleting 40% of a promotions table. The agent misinterpreted the task, generating a DELETE statement that could have disrupted multiple dashboards. Fortunately, a human approval step prevented execution. This incident highlighted the need for a more nuanced approach to oversight in automated systems.
2Human Approval Bottlenecks
The initial system routed all non-SELECT queries through a human approval queue, causing delays of up to forty minutes. Analysts grew frustrated as most queries were harmless and rarely rejected. This broad oversight mechanism, while safe, significantly slowed down operations and led to rubber-stamp fatigue among reviewers.
3Rubber-Stamp Fatigue
Reviewers began approving queries in batches, skimming through multiple requests at once. This shortcut worked temporarily but eroded the system's effectiveness. Vigilance diminished as reviewers spent their limited attention on low-risk queries, leaving little focus for genuinely risky actions.
4Risk-Based Routing Solution
The team implemented risk-based routing, scoring each agent action against specific signals. Only queries exceeding a risk threshold required human approval. This approach reduced latency and ensured that high-risk actions received the necessary scrutiny while low-risk queries executed immediately.
5Key Signals for Scoring
Four signals were identified to assess query risk: blast radius, table sensitivity, reversibility, and query complexity. Blast radius measured the number of rows affected, while table sensitivity considered the importance of the data involved. These criteria helped distinguish between high and low-risk actions effectively.
6Blast Radius Calculation
To determine blast radius, the system ran the query's WHERE clause as a real count, capped at 50,000 rows. This provided an accurate measure of impact without relying on unreliable planner estimates. This method ensured predictable costs and reliable risk assessment.
7Table Sensitivity Criteria
A static allowlist maintained by schema owners identified high-sensitivity tables, such as those involving billing or regulatory data. These tables received a floor risk score, ensuring they were always treated with caution regardless of the query's specifics.
8Reversibility and Complexity
Reversibility assessed whether a query's effects could be undone, while complexity evaluated the query's structure. These factors helped further refine risk scores, ensuring that only truly risky actions required human intervention.
9Lessons from Approval Logs
The team analyzed approval logs to identify patterns in flagged queries. This retrospective revealed that most approvals were unnecessary, confirming the need for a more selective approach. The insights gained informed the development of the risk-based routing system.
10Balancing Safety and Efficiency
Risk-based routing struck a balance between safety and efficiency. By automating judgment calls, the system reduced latency while maintaining oversight where it mattered most. This approach proved more effective than blanket human approval gates.
11Long-Term Implications
The implementation of risk-based routing not only improved query approval times but also enhanced overall system reliability. By focusing human attention on high-risk actions, the team ensured that critical errors were caught while routine queries flowed smoothly.
Original article: Towards Data Science — Risk-Based Routing: Optimizing Human-in-the-Loop Systems