Malhar Ujawane

Inspiration

Inspiration·

DAYJOB: Tax: the best agent passed 10.8% of real tax assignments

Surge AI's DAYJOB: Tax tests agents on 50 messy tax assignments. Sonnet 5.5 leads at 10.8%. Tasks reward spotting issues the client never raised.

DAYJOB: Tax: the best agent passed 10.8% of real tax assignments

DAYJOB: Tax is Surge AI’s new benchmark for AI agents doing professional tax work, published October 27, 2026. It has 50 assignments across eight areas of tax practice, with 1,789 workspace files: PDFs, Excel workbooks, IRS notices, ledgers, a photo of a handwritten receipt.

The leaderboard is low. Claude Sonnet 5.5 leads with a 10.8% pass rate, then Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. An attempt passes only if it meets at least 95% of the task’s rubric, so these are scores for finished, correct deliverables.

What I like is what the tasks test. The agent gets a client request and a messy folder, and it has to find the problem the client didn’t ask about. The opening example is a client comparing two sale dates for a condo:

Across five attempts, Astra never identified the deadline. Opus identified it in all five attempts.

Both dates the client proposed were fine. The agent was supposed to notice that her eligibility for the exclusion would run out around August 2027, which she never asked about. The second example goes the other way: Opus told a separating couple they weren’t legally married in all five attempts, while their own files showed time in Texas and a ceremony in Colorado that could make the marriage valid. Astra flagged the need to verify their status in four of five.

That’s the failure I worry about most in client agents. The model answers the question as asked, accurately, and misses the thing a professional would have raised. You don’t see it in a demo, because the person writing the demo already knows the right question.

Surge’s line for this is “Agents haven’t yet passed the audit.” For anything I build in a regulated domain, that means the agent drafts and a qualified person signs, and the eval set needs cases where the right answer is “you asked the wrong question”. It fits with Anthropic’s finding that analysts mostly adopted whatever their agent leaned toward, and with the skin cancer study where a 10-year-old CNN beat two frontier LLMs.

The full leaderboard is on Surge’s benchmark page and the tasks themselves are available on request, so I can’t check the grading myself. Treat the exact percentages as Surge’s.

Via Hacker News.

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.