AI in Incident Response: The Irony of Automation
As AI handles routine incidents, engineers risk losing critical system intuition.
1The Rise of AI in Incident Response
AI-powered tools now autonomously handle routine system incidents—inspecting alerts, forming hypotheses, querying telemetry, and implementing fixes. While this reduces mean time to resolution (MTTR) for common issues, it creates a paradox: engineers get fewer opportunities to develop critical system intuition through hands-on troubleshooting. The author, a former LinkedIn SRE, recalls building a self-healing prototype in 2012—technology that's now mainstream but carries unintended consequences.
2The Automation Paradox
Human-factors researcher Lisanne Bainbridge identified this dilemma in her 1983 paper 'The Ironies of Automation'. Automation reduces operators' practice with routine work while leaving them responsible for novel, high-severity incidents. In tech, as AI handles 90% of alerts, engineers face complex SEV0 incidents with diminished troubleshooting experience. The aviation industry shows parallels—while modern planes fly autonomously 98% of the time, pilots must master rare emergencies through rigorous simulator training.
3The TransAsia Flight 235 Case Study
This 2015 crash illustrates the automation paradox. After one engine failed (occurring <1 per 100,000 flight hours), the crew misdiagnosed the situation despite simulator training. The ATR 72 crashed 117 seconds after the first warning, killing 43. Like pilots, SREs face increasingly rare but catastrophic failure scenarios. FAA mandates 6-month simulator recertification—a model tech might emulate, as most engineers currently get zero incident practice outside production.
4Comprehension Debt in Engineering
As LLMs handle more operations, teams accumulate 'comprehension debt'—the growing gap between system complexity and human understanding. The author's experience founding Holberton School revealed this: even graduates from project-based programs lacked troubleshooting skills until given broken infrastructure to repair. Similarly, at Rootly, engineers now train via simulated e-commerce outages with AI-generated stakeholders—practicing diagnosis, coordination, and executive communication under pressure.
5Simulation vs. Observation
While AI can explain its incident response steps, passive observation can't replace active practice. The author compares this to learning tennis by watching Serena Williams versus playing matches. Tabletop exercises and chaos engineering (intentionally breaking systems) existed pre-AI, but take on new urgency. Rootly's simulations immerse engineers in realistic scenarios—diagnosing issues via observability tools while managing SLACK communications with AI-generated C-suite and customer support messages.
6The 80/20 Rule of Incident Preparedness
AI may handle 80% of incidents, but the 20% requiring human intervention become disproportionately critical. Unlike aviation where failures are mechanical and documented, software systems evolve constantly—an AI-trained on last year's incidents might not recognize new failure modes. The author predicts MTTR for routine issues will keep falling, while complex incident resolution times spike as first responders lack recent hands-on experience with system behaviors.
7Implementing Continuous Practice
Proactive solutions include: 1) Mandatory quarterly incident simulations (like FAA recertification) 2) Chaos engineering in staging environments 3) 'Fix-it Fridays' where engineers manually troubleshoot intentionally broken systems 4) AI-as-trainer scenarios that require human diagnosis before revealing solutions. The goal isn't to resist AI adoption, but to redesign skill maintenance—just as pilots train for rare events they'll likely never face.
8Balancing Automation with Human Expertise
The ultimate irony: the better AI gets at incident response, the more we must invest in human capabilities. Bainbridge's research suggests giving operators regular manual control prevents skill decay. For tech, this means treating incident response as a perishable skill requiring deliberate practice—not assuming AI explanations substitute for experience. As systems grow more complex, the human ability to navigate ambiguity becomes both rarer and more valuable.
Original article: Hacker News — AI in Incident Response: The Irony of Automation