# Evaluating AI Agents on Professional Tax Assignments with DAYJOB New benchmark tests AI judgment across 50 complex, real-world tax practice environments. - **Source:** [Hacker News (news.ycombinator.com)](https://brifing.si/source/13c923cd-6d99-442b-9b8b-1181f4d8cd91) - **Published:** 2026-10-28 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/4960f38e-c1f1-4f96-8787-9705e7b85a7c ## Benchmark for Professional Judgment Surge AI introduced DAYJOB: Tax, a benchmark evaluating AI agents on 50 complex professional tax assignments. Unlike standard benchmarks focusing on isolated legal questions, this framework tests end-to-end responsibility. Agents must navigate messy records, identify overlooked client issues, and exercise professional judgment to deliver work that meets rigorous industry standards. ## Testing Beyond Simple Queries The benchmark covers eight areas of tax practice, including corporate taxation, partnerships, and multistate compliance. Assignments require agents to process client requests alongside supporting records, such as general ledgers, W-2s, and IRS notices. Success depends on the agent's ability to resolve conflicting evidence and determine when information is insufficient for filing. ## Real-World Data Complexity The dataset includes 1,789 workspace files, with a median of 30 files per assignment. These range from 1,487 PDFs and 117 Excel workbooks to handwritten receipts and CSV records. Agents must reconcile these disparate sources, such as matching item descriptions from receipts against inventory records to calculate cost of goods sold. ## Identifying Consequential Issues A key challenge involves identifying issues the client failed to mention. In one case, an agent correctly evaluated property sale dates for a tax break but missed a critical deadline that would invalidate eligibility. Professional tax advisers must recognize these hidden risks, a capability that current frontier models frequently struggle to demonstrate. ## Multistate Compliance Challenges One assignment requires agents to analyze an industrial service company's operations. While the client assumes operations are limited to Alabama, field dispatch logs reveal activity across five states. Agents must synthesize crew sizes, job locations, and site days to determine the company's actual multistate tax obligations, testing their investigative capabilities. ## Navigating Legal Ambiguity Agents were tested on dividing retirement accounts for a couple with complex marital history. While one model assumed the couple was unmarried, the evidence suggested potential common-law marriage status based on previous residency in Texas and Colorado. Recognizing such legal nuances is essential for providing accurate tax advice and compliance. ## Leaderboard Performance Metrics Claude Sonnet 5.5 leads the leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. These results are measured at a 95% rubric threshold, meaning an attempt only passes if it satisfies nearly all criteria regarding calculations, evidence, and final deliverables. ## Expert Review Methodology Each task undergoes a three-layer expert review process to ensure correctness, completeness, and solvability. Tax professionals with backgrounds in corporate, partnership, and individual taxation develop the scenarios. This rigorous creation process ensures that the benchmark reflects the actual workflows and professional standards expected in the modern tax advisory industry. ## Comparison with Existing Benchmarks DAYJOB differentiates itself from tools like TaxBench or TaxLab by focusing on the entire workflow rather than single-step problems. While other benchmarks test specific calculations or missing information retrieval, DAYJOB evaluates whether the agent can define the problem itself, handle incomplete instructions, and maintain professional accountability throughout the entire assignment. ## High-Performance Pareto Frontier The evaluation compared model performance against the average cost per trial. Claude Sonnet 5.5 and Opus 5.5 currently occupy the high-performance end of the measured Pareto frontier. These models demonstrate the strongest ability to handle complex, multi-step tax workflows, though they still face significant challenges in achieving consistent, reliable professional judgment. ## The Gap in Agentic Capability The results highlight that even when models perform individual steps correctly, exercising holistic professional judgment remains elusive. Successfully managing real-world tax workflows requires navigating conflicting evidence and taking ownership of tasks from start to finish. Bridging this gap is the primary challenge for developers aiming to create truly useful AI agents. ## Sources covering this story - https://surgehq.ai/blog/dayjob-tax — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._