Can AI find mistakes in photos of partially assembled IKEA furniture? We think this challenge serves as a good proxy for a number of economically important tasks that require similar visual and spatial reasoning, like fixing a car or repairing household appliances. If models can reason quickly and accurately enough in this domain, it is plausible that AI could soon provide reliable, real-time guidance for complex physical assembly and repair tasks.
Despite the importance of this capability, there are very few existing visual reasoning benchmarks that test it. We created the Furniture Assembly Benchmark (FAB) to help fill this gap. The benchmark consists of 60 images spanning three different IKEA furniture builds. Some show furniture that has been correctly assembled, and others show assemblies with (intentional) mistakes. In addition to the images, AI models are given the assembly manual and tools to inspect the images. To score well on FAB, models must identify all steps in which a mistake was made and provide a reasonable description of each mistake.
Key Take-aways
- Models have improved significantly over the last year. The best score in November 2025 was 28%, set by Anthropic’s Claude Opus 4.5. Just ten months later, OpenAI’s GPT-6 Astra now tops the leaderboard at 80%.
- GPT-6 Astra is a major improvement in speed, with a median time of 3 minutes per photo. This is the fastest of any model we evaluated and is between twice and ten times as fast as previously leading models.
- The leading model has always been closed-weight, with Chinese open-weight models lagging the frontier by at least seven months.
How we run the Furniture Assembly Benchmark
We purchased three pieces of IKEA furniture of varying assembly difficulty, as measured by the IKEA Complexity Index, which calculates complexity as the product of the number of steps and the number of pieces. This lets us examine whether models struggled with larger, more complex builds.
We then assembled the furniture, taking photos as we went. During assembly, we intentionally made mistakes. We used our best judgment to err in realistic ways and often continued building past a mistake to make it harder to identify. For each photo containing a mistake, the issue is clearly visible. An example of a mistake is shown below. Although we have not yet measured human performance on the benchmark, we expect many of these mistakes to be challenging to find for those unfamiliar with the build.
The top image is incorrect, as the laminate is facing inwards instead of outwards. The bottom image is correct.
We evaluated multiple models developed by OpenAI, Anthropic, Google, Moonshot, and Alibaba. During evaluation, we prompt a model to look at an image. If it identifies any mistakes, it must list all the steps that contain a mistake and describe the issues; otherwise, it must direct the user to the next step in the build. The model is provided with a PDF of the instructions, a zoom tool to view parts of the image in higher fidelity, and a Python interpreter. For each sample image, the model is given a budget of 80 steps in an agentic sandbox. If it approaches the budget limit, the model is given a warning, and upon reaching the limit, it is asked for a final answer. In practice, this limit was reached in less than 5% of samples.
Grading is multistep. If there is no mistake and the model correctly identifies this, then it is marked as correct. If there is a mistake, the model’s answer is checked to see if it identified the correct step(s) where the mistake(s) were made. If the model passes this too, then its description of the mistake must finally pass an LLM grader in order to confirm that the identified mistake is accurate. The LLM grader model is GPT-5.6 Sol, and it is prompted to be very lenient. So long as the model correctly identifies the mistaken step and roughly describes the issue, it gets credit.
Results
To fill out an initial leaderboard, we tested several vision-capable models released by top AI companies over the past year. The highest-scoring model is GPT-6 Astra at 80%, with the next best models, Claude Fable 5.1 and Claude Opus 5, scoring 70% and 61%, respectively.
Accuracy differences across models and items
Different AI models struggled more or less with different furniture, with no clear relationship to the IKEA Complexity Index. Because of this, we suspect other factors such as mistake subtlety, viewing angle, and how far assembly continued after an error contributed more to difficulty than the size or number of pieces used in the furniture item.
Capability progress over time
By testing models released over the last year, we can retrospectively observe how rapidly capabilities have progressed. Of the models we tested, Opus 4.5 set the frontier in November 2025 at 28%.1 Ten months later, in September 2026, Astra scored 80%.
Chinese open-weight model performance
The best Chinese model as of today, the open-weight Kimi K3, lagged the frontier by seven months at the time of its release. The gap between Kimi K3 and the frontier is wider in the FAB compared to general capabilities, where Kimi K3 trailed the frontier by only 4.4 months, as measured by the Epoch Capabilities Index (ECI). Vision capabilities are still rare in Chinese models — DeepSeek V4 Pro and Flash models lack image support, as does Z.ai’s GLM 5.3. We will continue to track progress here as these capabilities become more prevalent.
Failure Modes
Failure modes are often shared across a family of models. We see Google’s Gemini and Alibaba’s Qwen models almost always assume a mistake exists, whereas early Anthropic and OpenAI models errored in the opposite direction and too often failed to find mistakes at all.
Efficiency differences between models
In terms of efficiency, Astra is a clear standout. It is able to achieve the best accuracy in a fraction of the time and tokens that other models use.2 At a median of three minutes per task, it is still a little too slow for practical real-time feedback, but it is not far off.
Conclusion
Our results suggest that frontier models can use photos and technical instructions to reason about physical construction and recognize assembly mistakes. Performance has continually improved over the past year, with GPT-6 Astra being the most recent leap in performance and efficiency. Accuracy and slow response times are still limitations for real-time use, but the field is rapidly progressing.
We tested models on only 60 photos from three furniture builds, so it remains unclear how well these results extend to other physical tasks. Still, the ability to connect instruction manuals to real objects and identify mistakes suggests models have the spatial reasoning skills instrumental for maintenance and mechanical troubleshooting. We hope the Furniture Assembly Benchmark encourages further work on measuring models’ physical reasoning capabilities and understanding how they translate into real-world tasks.
-
It is possible the frontier at the time was somewhat higher than this. We do not have scores for GPT-5 or Gemini 3, however we would be surprised if any prior system scored significantly higher.
-
We used the official Moonshot and Alibaba API endpoints, but speed and quality for open models can vary significantly depending on the provider.
About the authors
Related work
