SWE-bench is a benchmark that evaluates large language models and AI agents on real-world software engineering tasks. It was created by researchers at Princeton University with Stanford collaboration, including Carlos Jimenez, John Yang, and Ofir Press, and was published at ICLR 2024 under the title "Can Language Models Resolve Real-World GitHub Issues?".
The benchmark collects over 2,000 real GitHub issues from popular open-source Python repositories such as Django, matplotlib, and scikit-learn. A model is given the codebase and issue description and must produce a code patch, which is validated against the repository's real unit tests. Variants include SWE-bench Verified, a 500-problem human-validated subset built with OpenAI's Preparedness team, as well as SWE-bench Lite, SWE-bench Multimodal, and SWE-bench Multilingual.
The public leaderboard at swebench.com tracks dozens of frontier models and agent frameworks, both open-source and proprietary, and is continuously updated with new submissions. SWE-bench has become the de facto industry standard for measuring AI coding-agent ability, with SWE-bench Verified scores quoted in nearly every frontier model launch. The team also maintains related tooling including mini-SWE-agent, SWE-smith, SWE-ReX, and the SWE-bench CLI.
SWE-bench is free, with code and data publicly available on GitHub.