SWE-bench

SWE-bench has earned its status as the closest thing the industry has to a standard measure of AI coding-agent competence, and for good reason: rather than testing narrow coding puzzles, it grades models on real GitHub issues from actual production codebases like Django and scikit-learn, verified against the repository's own unit tests. That grounding in real-world software engineering work, rather than synthetic problems, makes its scores far more meaningful than most coding benchmarks, and the Verified subset (built with OpenAI's Preparedness team) has become a fixture in nearly every frontier model announcement. The variant families, Lite, Multimodal, and Multilingual, extend its usefulness across different evaluation needs, and the public, continuously updated leaderboard makes cross-model comparison easy for anyone choosing a coding agent or model. As with any benchmark, there are legitimate concerns about contamination as models are increasingly trained with awareness of SWE-bench-style tasks, and topping the leaderboard doesn't guarantee performance on your specific codebase or stack. But as a free, open, well-maintained reference point for coding-agent capability, it's about as good as this category gets.