# AI in Software Development: Risks and Realities

The impact of LLMs on engineering processes and the management-engineer divide

- **Source:** [Martin Fowler](https://martinfowler.com)
- **Published:** 2026-07-21
- **Topic:** [Data & AI](https://brifing.si/c/data-ai)
- **Original article:** <https://martinfowler.com/fragments/2026-07-21.html>
- **On Brifing:** https://brifing.si/s/0eaff382-82db-4df7-9bbb-a1a17fab25fb

## Insights from Future of Software Retreat

Thoughtworks report highlights verification as the new bottleneck, not code generation. As engineering evolves into a distinct discipline, the expectation gap between executives and engineers poses greater risks than technical limitations, with LLM security concerns topping the list.

## Context-Blind AI Disaster

A company using ML models trained on desert equipment for Arctic operations saved $50M on air filter replacements—until rotting mosquitoes caused $100B in fire damages. A stark reminder of context-free AI application dangers.

## Vibe-Coding and Shadow IT

Unregulated LLM use by citizen developers (vibe-coding) creates uncontrolled shadow IT systems. Untested applications with unverified data quality invite security breaches, prompting companies to develop isolated infrastructure platforms.

## Threat Modeling with Boards

Engineers conduct threat modeling sessions to educate boards about LLM risks. When AI is viewed solely as cost-cutting magic, technical limitations get overlooked. Involving legal teams helps highlight implementation risks.

## Dotcom Bubble Parallels

Participants agree current AI hype mirrors the 1990s dotcom bubble—with one key difference: Where dotcom era birthed transformative products, today's bubble prioritizes cost reduction over innovation, raising sustainability concerns.

## LLMs in Operations

AI agents analyzing observability event streams detect anomalies faster than humans. However, citizen-developed apps often lack proper observability instrumentation, creating blind spots in monitoring.

## Code-Event Trace Correlation

LLMs help ops teams understand code behavior by correlating source with event traces. This cross-referencing aids root cause analysis, especially for recurring incidents where agents can synthesize information across multiple cases.

## Executive-Engineer Expectation Gap

Management assumes LLMs that draft decent reports must also code well—a dangerous misconception. As Kelsey Hightower notes: 'The less busy work you have, the less appealing these AI tools become.'

## Rising Token Costs

Escalating LLM usage costs may temper corporate enthusiasm. As per-token expenses grow, management's cost-cutting focus could shift toward addressing engineers' security concerns about AI implementation.

## Apprenticeship Crisis

Companies face a growing skills gap as experienced engineers retire. While LLMs can partially address legacy system modernization, sustainable solutions require human-centric approaches to knowledge transfer.

## Governance Safeguards

Balancing AI-assisted development requires layered controls: deterministic data access restrictions, isolated infrastructure, and robust observability help contain the 'lethal triad' of security, compliance, and performance risks.

## Sources covering this story

- https://martinfowler.com/fragments/2026-07-21.html — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._