AI in Software Development: Risks and Realities
The impact of LLMs on engineering processes and the management-engineer divide
1Insights from Future of Software Retreat
Thoughtworks report highlights verification as the new bottleneck, not code generation. As engineering evolves into a distinct discipline, the expectation gap between executives and engineers poses greater risks than technical limitations, with LLM security concerns topping the list.
2Context-Blind AI Disaster
A company using ML models trained on desert equipment for Arctic operations saved $50M on air filter replacements—until rotting mosquitoes caused $100B in fire damages. A stark reminder of context-free AI application dangers.
3Vibe-Coding and Shadow IT
Unregulated LLM use by citizen developers (vibe-coding) creates uncontrolled shadow IT systems. Untested applications with unverified data quality invite security breaches, prompting companies to develop isolated infrastructure platforms.
4Threat Modeling with Boards
Engineers conduct threat modeling sessions to educate boards about LLM risks. When AI is viewed solely as cost-cutting magic, technical limitations get overlooked. Involving legal teams helps highlight implementation risks.
5Dotcom Bubble Parallels
Participants agree current AI hype mirrors the 1990s dotcom bubble—with one key difference: Where dotcom era birthed transformative products, today's bubble prioritizes cost reduction over innovation, raising sustainability concerns.
6LLMs in Operations
AI agents analyzing observability event streams detect anomalies faster than humans. However, citizen-developed apps often lack proper observability instrumentation, creating blind spots in monitoring.
7Code-Event Trace Correlation
LLMs help ops teams understand code behavior by correlating source with event traces. This cross-referencing aids root cause analysis, especially for recurring incidents where agents can synthesize information across multiple cases.
8Executive-Engineer Expectation Gap
Management assumes LLMs that draft decent reports must also code well—a dangerous misconception. As Kelsey Hightower notes: 'The less busy work you have, the less appealing these AI tools become.'
9Rising Token Costs
Escalating LLM usage costs may temper corporate enthusiasm. As per-token expenses grow, management's cost-cutting focus could shift toward addressing engineers' security concerns about AI implementation.
10Apprenticeship Crisis
Companies face a growing skills gap as experienced engineers retire. While LLMs can partially address legacy system modernization, sustainable solutions require human-centric approaches to knowledge transfer.
11Governance Safeguards
Balancing AI-assisted development requires layered controls: deterministic data access restrictions, isolated infrastructure, and robust observability help contain the 'lethal triad' of security, compliance, and performance risks.
Original article: Martin Fowler — Yazılım Geliştirmede Yapay Zeka ve Riskler