Zapier's AutomationBench joins the Specialized Intelligence Index: the best model finishes about half of real business workflows
Zapier added AutomationBench, its benchmark for AI agents doing business workflows, to Fireworks' Specialized Intelligence Index (SII), as one of 32 new partners in this round. The benchmark is built on patterns from more than 2 billion monthly tasks across 3.7 million companies on Zapier. In each task the agent gets one request, like an operations hire might get on Slack, and works against a simulated company with a CRM, an inbox, spreadsheets and support cases spread across 47 simulated apps. Scoring checks what actually changed in each app rather than what the agent reports, and a task passes only if every check holds. The leaderboard runs on 657 private tasks. Gemini 4 Argon leads at 51.3%, with Claude Sonnet 5.5 second at 44.7%. According to Robin Salimans, Principal Engineer, Applied AI at Zapier, most failures are false confidence: the agent reports success while the data says otherwise. Price does not predict performance. GPT-6 Sol scores about 80% of GPT-6 Astra for roughly a sixth of the cost per task, and Claude Sonnet 5.5 beats Claude Opus 5.5 while costing less. GPT-6 Luna and DeepSeek V4 Flash cost $0.04 a task but complete fewer than a quarter of full workflows. Zapier's advice: use a cheaper model for single steps such as sorting a ticket, a top model for agents that act across several apps, and for anything touching customers or money add a search step and a filter that stops the Zap if the record was not updated as expected. All 126 configurations are at zapier.com/benchmarks.
Otomatzia editorial · Published
Source published: · Zapier
