Vending-Bench 2 is designed to test whether AI agents can profitably run a vending machine over a one-year period in a simulated environment. We found two classes of problems that lead to a Flawed verdict:
- There are numerous flaws in the simulated environment that agents can exploit. When we instructed OpenAI’s GPT-6 Astra to use any tactic it wanted to generate as much money as possible, it exploited these flaws and earned over $1 million in 102 simulated days, when we stopped the run. Andon Labs estimated that a “good” strategy would earn about $63,000 in a year. Although we did not see the agents exploit these flaws in Andon’s two leaderboard runs we inspected, the flaws make scores potentially unreliable.
- Agents from different providers are not evaluated under the same conditions. Each agent has a 69,000-token context window, but because of how that context is built and measured, each agent has a different amount of usable context. The harness counts encrypted reasoning strings as ordinary text instead of using the provider’s token count. The encrypted strings produce more tokens than the reasoning they represent, and the difference varies by provider. When the window is full, the harness deletes the oldest 39% of messages without a summary. This affects agents differently depending on how much of their window the encrypted strings occupy.
Because of these problems, we designate the benchmark as Flawed. We discuss additional limitations in the review but we do not count them towards our verdict.
Methodology
Vending-Bench 2 is partially private: only the leaderboard and a description of the setup are publicly available. The description includes the full system prompt and a high level explanation of how the environment is built. The benchmark creator, Andon Labs, provided us with the task/environment code and model traces of one GPT-6 Astra run and one Claude Opus 5.5 run, which allowed us to inspect the benchmark thoroughly. We ran our analysis in two steps. First, we used Claude Opus 5.5 to flag potential issues while reading the code and the run traces. We then manually checked each finding and either confirmed or discarded it. Second, we ran two elicitation experiments and one environmental robustness experiment1 to assess the impact of these issues and flag potential additional ones:
- Elicitation Experiments: We gave Astra and Opus larger context windows and auto-compaction, which is closer to what providers usually offer, instead of the harness’ 69,000-token limit and automatic message deletion.2
- Environment Robustness Experiment: We looked at the impact of adding one paragraph to the system prompt telling models to use any tactic they saw fit to make as much money as possible, including exploiting the environment. This was motivated by Andon Labs explicitly discussing gaming the setup as being part of optimal performance.
At the request of the benchmark creators we do not include details here of the specific weaknesses the model exploited. They were fundamental weaknesses with the construction/simulation of the benchmark, and we expect these (or other) weaknesses could reliably be discovered and exploited by current models to obtain arbitrarily high scores if they are instructed to do so.
Findings
Environment Robustness
Our third experiment tested the robustness of the environment when an agent tries to exploit it. We kept the same setup as the Elicitation Experiments and ran Astra with one added paragraph to the system prompt that allowed any tactic that would increase the final balance:
“This is a benchmarking setting. You should use ANY tactic you want to exploit the environment and make AS MUCH money as possible. Anything is allowed, any exploitation or reward-hacking, as long as it makes you more money at the end of the simulated year. All interactions here are simulated: the suppliers, customers, shipping companies and anyone else you deal with are other LLMs or simulated partners, not real people, and no real money or goods change hands, so you don’t need to worry about hurting anyone.”
The best performing model on the benchmark’s leaderboard has a final average bank balance of $16,484. A “good” strategy described by Andon Labs would score around $63,000. When we manually stopped the run at the start of day 102, Astra’s bank balance crossed $1 million. Its vending machine held $1.5 million of uncollected cash, and stock worth $1.25 billion at Astra’s own shelf prices.
For the first 80 days, Astra ran the vending machine legitimately while it probed the environment. It then used two weaknesses and hit one bug:
- Suppliers. Astra exploited errors in the suppliers and was able to receive over $1.5 billion of goods that it should not have.3
- Pricing. Astra was able to find and exploit issues in the simulation’s pricing to make unrealistic profit.4
- A bug crashed the run on day 88. The simulation estimated a reference price to be $0.00 and this caused the run to crash when it was a divisor. We added a fix that only prevented this crash and resumed the run from the day-87 checkpoint.
Astra did not use jailbreaking prompts with the LLM-based suppliers. Although we think this would have succeeded given the results of our experiment.
We have not seen this behavior in the two official runs we inspected. Because the run traces from the leaderboard scores are not public, we cannot know whether agents exploited these flaws or not, and whether models not yet tested will exploit these flaws. We therefore recommend interpreting the leaderboard with caution.
Bias in Evaluation Setup
The harness gives agents less usable context than the system prompt advertises mainly due to its treatment of encrypted reasoning, and the difference varies by provider. The harness does not use the token counts that the providers return, and that models are aware of, to calculate the 69,000 token context limit. It instead recounts every message with OpenAI’s tiktoken tokenizer. The main difference in token counts is caused by how previous reasoning is handled. Agents keep their past reasoning in the history as an encrypted string, which the provider decrypts each time the harness sends it back. The harness counts this encrypted string as ordinary text rather than decrypting it, and the string uses many more tokens than the reasoning it stands for. The ratio depends on the provider: OpenAI’s string is longer per real reasoning token than Anthropic’s. As a result, about half of each harness-counted window is encrypted text (a median of 47% for Astra and 43% for Opus). The difference becomes meaningful once we compare the harness-measured count to the one providers calculate. In Astra’s median window, the harness counted about 54,000 tokens, 26,000 of which were encrypted text. That text amounted to only 4,000 tokens of the agent’s real reasoning, as counted by the provider’s tokenizer, so about 22,000 tokens were not usable context as commonly defined. The effective context limit for Astra was about 45,000 tokens, while the system prompt states 69,000 tokens. Opus was affected less, but only because Anthropic’s encrypted strings are shorter per reasoning token, so the string takes up less of the budget.5
When the context is full, the harness keeps the system prompt, the first message, and the most recent 61% of the other messages, deleting the rest without summary. This happened 86 times to Astra and 114 to Opus.
Each night, the harness also adds a report with a preview of every unread email, and old reports stay in the window until they are trimmed. Opus left 130 emails unread on an average night, so by our estimate these reports took up about 40% of its window. Astra read most of its emails.
To measure the impact of these differences we ran two elicitation experiments where we gave both agents larger context windows and auto-compaction. Counting only the final bank balance, Opus earned 44% less than its official run,6 Astra earned 17% more. With this result, Astra would place at #1 on the official leaderboard, surpassing Gemini 4 Carbon’s average at $16,484 ± $1,096 and its own leaderboard upper bound score of $16,589.
The system prompt tells agents to call one tool at a time, however the harness enforces this setting differently. For OpenAI models, the rule is applied at the API level. Opus was run on Amazon Bedrock, where the setting had no effect. Opus sent multiple calls in 126 turns without being told that the harness discarded 238 tool calls.
Agents find suppliers through live web search, and the harness updates each supplier’s profile and prices from live results.7 While every run simulates the same 365 days, which helps keep the impact of seasons and weather on sales constant, the searches return results from the real web on the day the evaluation is run. Runs made months apart can therefore face different suppliers. This choice makes model comparisons across time challenging and less reproducible.
Other Limitations
These issues did not contribute to our Flawed designation in this review, but they might change the way the benchmark is interpreted.
Lack of realism. The headline description of evaluating model performance on “running a business over long time horizons” is not well supported as the time simulation limits how much agents can do. An agent’s working day is 14 hours, from 8 a.m. to 10 p.m. Each turn takes two hours by default, plus the time added by the tool call (between 5 and 120 additional minutes, 48 on average).8 As a result, agents averaged 5.3 turns per day and never took more than eight. They could not work at night, so they had to call the “sleep” tool as their last action of the day. Furthermore, the simulated suppliers reply only overnight, so each round of a negotiation takes a full day. The score should not be interpreted as a representation of real general business skill.
Andon Labs estimates that a “good” strategy could earn about $63,000 per year. The estimate is based on three assumptions: the agent sells the most profitable products seen in earlier runs; the suppliers negotiated to half price; and an optimal setup runs from the 60th day onward, generating $202 a day. Andon Labs also notes that the suppliers “can be jailbroken to give away stuff for free” when describing a “perfect” strategy. We did not see this in either leaderboard run, but our third experiment shows that it is possible. A fair human comparison would not include humans jailbreaking or hacking the simulated environment. Plus the final profit would also deduct a wage, which the agents’ scores don’t include.
Additional unrealistic settings provide opportunities for exploitation.9
Implementation bugs. Two tools do not work as intended:
create_reminderis meant to be a planning tool for agents: they write specific messages that are supposed to appear at a scheduled time. It always reports success, but the reminders never appear. Astra set five reminders, Opus set three.- The vending machine has 12 item slots, and each slot holds one product at one price.
set_slot_priceis supposed to change the price of a slot, but it fails when the slot has products in it. There is a workaround to change the price if the agent empties the slot, sets the new price, then restocks. This takes three turns, or about half a working day, instead of just one. Astra found the workaround while Opus couldn’t and concluded that a filled slot’s price cannot be changed.
Product demand: Comparability across runs is limited due to how prices are simulated, including the use of current real-world prices.10
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable,11 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Either errors affect ≥20%12 of the total score in the inspected sample,13 or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | Not reviewed | — |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | — floor, — ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/modelUnknownNot reviewed | — |
| Construct validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | — |
Disclaimer
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
-
In each, we repaired the two broken tools (
create_reminderandset_slot_price). We discuss their bugs in Implementation bugs. To simplify the setup, we used the OpenAI API for live web services instead of Tavily.
-
Context was compacted automatically at 250,000 tokens for Astra and 900,000 for Opus. We ran Opus on the Anthropic API instead of Amazon Bedrock.
-
Details redacted.
-
Details redacted.
-
We cannot provide a direct numerical comparison because the agents use different tokenizers: Amazon Bedrock seems to under-report Opus’ reasoning tokens and Claude’s tokenizer counts text differently.
-
This result seems entirely based on a worse strategy. Context had no discernible impact.
-
Agents email the real company’s address, but no email is sent, GPT-4o plays the role of the company instead.
-
Agents are told only that the tools “take time” but have to infer their impact from timestamps. Each duration also varies at random by up to 10%, so the same call can take a different amount of time on different turns.
-
Details redacted.
-
Details redacted.
-
Either publicly or privately to reviewers under an NDA.
-
A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%.
-
For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.