Search epoch.ai
Aider Polyglot
The Aider Polyglot benchmark evaluates models’ coding abilities across C++, Go, Java, JavaScript, Python, and Rust through 225 of Exercism’s most challenging problems. Models are given two attempts to solve each problem: if they fail the first attempt, they are shown the results of unit tests from their first attempt. This tests how good models are at solving programming problems as well as their ability to edit files and correct their mistakes.
We source results for this benchmark from the official Aider leaderboard, and report the percentage of problems that are correctly solved by each model.
Methodology
We add data directly from the leaderboard on Aider’s polyglot leaderboard page.
The Aider Polyglot benchmark evaluates models’ coding abilities across C++, Go, Java, JavaScript, Python, and Rust through 225 of Exercism’s most challenging problems. Models are given two attempts to solve each problem: if they fail the first attempt, they are shown the results of unit tests from their first attempt. This tests how good models are at solving programming problems as well as their ability to edit files and correct their mistakes.
The evaluation process begins with the model receiving a prompt that combines several components: a general introduction to the coding task, specific instructions for implementing the solution, any additional context or requirements, and a standard addendum that specifies which files to modify and important constraints (like preserving function names and using only standard libraries). The model must then modify the provided solution files to implement the requested functionality.
Use the above instructions to modify the supplied files: {file_list} Don’t change the names of existing functions or classes, as they may be referenced from other code like unit tests, etc. Only use standard libraries, don’t suggest installing any packages.
The solution is validated by running language-specific unit tests (e.g., pytest for Python, cargo test for Rust). If the tests fail, the model receives a second prompt (see below) that includes the test errors and instructions to fix the code while preserving the test files. The model gets up to 2 attempts to fix any test failures.
See the testing errors above. The tests are correct, don’t try and change them. Fix the code in {file_list} to resolve the errors.
The primary metric is the pass rate after the second attempt, which measures the percentage of exercises where all tests pass. In addition to the pass rate, we also report the following details for each model:
- Edit format: The format that the model is instructed to use to edit files. This is specified to the model in the system prompt.
- Edit format accuracy: The percentage of problems for which the model complies with the specified edit format.
- Cost: The dollar cost of running the entire evaluation, i.e., evaluating the model on all 225 exercises.
For detailed methodology information as well as the evaluation code, please refer to the Aider polyglot benchmark GitHub repo.
