The People Using AI Should Set Its Strategy
Practitioners must own AI tool strategy while operations manages the underlying technical plumbing.
1The Evaluation Gap Problem
An evaluation gap exists between those approving AI tools and those capable of identifying output errors. Managers lacking hands-on experience often fail to spot nuanced mistakes in generated content. Practitioners who perform the actual work are the only ones qualified to judge the quality and accuracy of AI-driven results.
2Shadow Stacks as Indicators
Shadow tools represent a vote of no confidence in centralized operations. Microsoft data shows 78% of workers bring their own AI tools, while 59% of designers built custom utilities. These unofficial stacks emerge because operations teams often fail to understand the specific functional requirements of the practitioners they supposedly support.
3Historical Patterns of Adoption
The current AI shift mirrors the early web and VisiCalc eras. In 1979, managers bypassed corporate procurement to buy Apple II computers for spreadsheet access. Today, design and research teams are repeating this pattern, adopting tools that actually solve their problems while official procurement processes lag behind the rapid innovation cycle.
4The Shift to Consumption Pricing
Operations teams traditionally measure success through seats, spend, and logins. However, vendors are increasingly moving toward consumption-based token pricing. This shift makes seat-counting obsolete. Organizations must now focus on the business impact and token costs of specific tasks, requiring deeper technical knowledge than traditional license management currently provides.
5Defining Subject Matter Expertise
Subject matter experts (SMEs) possess the domain knowledge necessary to validate AI output. A 2026 study of 400,000 Claude Code sessions showed experts achieved success twice as often as novices. Domain expertise, rather than general coding skill, is the primary driver of verified success when utilizing generative artificial intelligence tools.
6Operations as Plumbing Providers
Operations should not dictate strategy but must manage the essential plumbing. This includes handling security reviews, privacy compliance, contract negotiations, and data boundaries. By securing the infrastructure, operations allows practitioners to experiment safely. This division of labor ensures that the business remains protected while the team maintains high-velocity innovation.
7The Risk of Tool Bloat
Tool bloat serves as a canary in the coal mine for failing operations. When designers report that too many tools slow them down, it indicates a lack of strategic oversight. Operations teams must cut redundant software once SMEs identify which tools provide genuine value, restoring efficiency to the daily workflow.
8Evolving the Operations Role
Many existing operations teams are struggling to adapt to the agentic AI era. Some organizations are renaming these groups to agent operations to reflect new requirements. If operations teams do not learn the tools and demonstrate hands-on competence, they risk becoming obsolete as the business reorganizes around more efficient, AI-integrated workflows.
9Practitioner Responsibilities
Owning strategy requires practitioners to define clear quality bars. Designers and researchers must defend their tool choices in budget reviews using data on work output rather than just login counts. Sharing successful tools and retiring those that stop paying off is essential for maintaining a lean, effective, and high-performing technical stack.
10Implementing Task Benchmarking
Teams should create benchmarks using 10 real-world tasks to evaluate new tools. By scoring candidate software against these trusted benchmarks, organizations can objectively measure errors, editing time, and token costs. This evidence-based approach settles disputes over tool adoption and ensures that only high-performing, reliable technology enters the corporate environment.
11Strategic Ownership for Success
Splitting decision rights along the evaluation gap is a temporary but necessary measure. As the field settles, ownership models may evolve. For now, practitioners must lead the strategy to avoid the costly mistakes of the past. By aligning authority with expertise, organizations can scale AI impact while keeping their operations safe.
Original article: UX Collective - Medium — The People Using AI Should Set Its Strategy