Building an AI Analyst for NFL Data Analysis
Developing an autonomous agent to evaluate complex sports claims using historical play-by-play data.
1The Monday Morning Agent
The project aims to build an AI agent capable of verifying common NFL fan arguments using data. By utilizing a quarter-million plays from five seasons of nflverse data, the agent processes complex queries. Each of the eight planned episodes introduces new capabilities, moving toward a fully autonomous desk-running system by the finale.
2Defining Claims Before Querying
The agent must first define ambiguous claims like 'establish the run' before accessing data. Since this phrase can imply wearing down defenses or setting up play-action, the agent writes a sentence explaining its logic. This ensures the AI interprets the specific context of the game rather than relying on broad, misleading statistics.
3Human-in-the-Loop Validation
Before the agent executes code, the developer performs the analysis manually to establish a baseline. A Markdown policy file defines the agent's scope, permitted read-only tools, and required output components. This structured approach prevents the AI from making causal claims, such as asserting that running directly causes winning, while maintaining strict operational boundaries.
4Handling Scope and Ambiguity
When asked for betting advice, the agent correctly refuses, recognizing the request falls outside its defined scope. However, user interactions can sometimes tangle instructions, requiring the developer to clarify the prompt. The process emphasizes that users must validate the generated code and answers to ensure the AI truly understands the intended task.
5The Impact of Data Filtering
Initial analysis showed teams with high run shares won 77.7% of games, but this was skewed by teams protecting leads. When filtering for first-half games within three points, the win rate for high-run teams dropped to 46.3%. This demonstrates how specific filters fundamentally alter the narrative and the resulting statistical conclusions.
6Transparency in Statistical Reporting
The agent is programmed to display filters on-screen, showing the number of rows remaining and the specific commands used. This transparency is crucial when working with smaller sample sizes, as seen in the current season's limited data. By reporting five seasons alongside current data, the agent provides necessary context for every weekly result.
7Analyzing Recent Game Trends
In Week 5, the team that ran more in the first half won only 6 of 15 games. Over the past five seasons, that team won 54.3% of the time, highlighting that recent results often fall below historical averages. Such fluctuations confirm that single-week data points are insufficient for drawing broad, definitive conclusions.
8Future Challenges and Definitions
The next episode tackles the 'Tush Push,' a play not explicitly labeled in the 261,126 play descriptions. The agent must create a permanent definition in a shared data dictionary to judge the play consistently. This underscores the necessity of human-defined parameters when data lacks standardized terminology for specific athletic maneuvers.
Original article: O'Reilly Radar — Building an AI Analyst for NFL Data Analysis