ExpertCrowd-SWE finds four coding-agent tiers, but top pair remains unresolved

The Statistical Reality of Coding Agents For technical decision-makers, leaderboard order can look more conclusive than the underlying comparisons allow. The ExpertCrowd-SWE study , an author-reported coding-agent benchmark submitted on November 9, 2026, provides a nuanced look at seven frontier coding agents. Rather than a simple ranking, the study identifies four statistical tiers, highlighting where evidence separates models and where it remains inconclusive. Four tiers, not seven distinct places The paper evaluated seven models on 100 expert-authored software-engineering tasks drawn from 87 open-source projects and 11 languages. Each model attempted each task five times. The paper reports 3,495 scored runs. Gemini 3.8 Flash had no valid run on one assigned task, so its score covers 99 tasks.

BeCrypto News shows a short excerpt and links to the publisher. The full report belongs to Forkast.