Where Do Vision Language Models Fail at 3D Reasoning?
★ Co-first authors · † Independent Researcher
Contact: duckhai.le@mail.utoronto.ca, nguyen-kim-hai.bui@connect.polyu.hk
★ Co-first authors · † Independent Researcher
Contact: duckhai.le@mail.utoronto.ca, nguyen-kim-hai.bui@connect.polyu.hk
TL;DR We introduce BedBench, a 13-task benchmark revealing that vision-language models can succeed at individual 3D judgments while failing to recover and consistently use exact object properties, motivating evaluations of structural recovery, cross-task consistency, and visual dependence.
Vision-language models are increasingly evaluated on spatial and 3D reasoning tasks, yet their scores do not always reveal how they reached an answer. A correct answer may depend on visual evidence, but it may also follow from language priors or cues in multiple-choice options. Conversely, a low score may reflect a model limitation, ambiguity in the available views, or a task that is difficult for humans as well.
We ask a more specific question: does success on individual judgments extend to exact answers and consistent use of object properties?
BedBench compares different questions about procedurally generated 3D objects and scenes, allowing us to examine both individual judgments and exact answers on the same objects.
We chose tasks whose answers are meant to be easier to check but harder to produce. Producing answers requires imagining the object(s) from the given inputs and then mentally rotating, decomposing, comparing, or querying.
Humans reach 93.3% on T5, the best VLM 7.2%. Fine-tuning raises T5 to 68.6%, yet the model identifies the most common colors (T7) only 35.0% of the time from images, versus 74.2% when given its own counts as text.
Averaged over tasks, humans reach 70.0%, the strongest model (Gemini-3-Flash) 51.5%, and the best-constant baseline 36.7%. The best models exceed the human interval on T2 and T6, yet on T5, where every color count must be correct, the best VLM reaches only 7.2%.
The individual counts are already unreliable: across the 12 open-weight models, per-color error is 0.80 blocks, worse than the 0.69 obtained by always predicting each color's mean count.
Asking twelve single-color questions instead of one joint answer leaves exact-count accuracy on present colors essentially unchanged (32.5% → 33.2%). Exact match falls quickly as more colors must be right, and the merged answers reach 0.0% full-distribution EM.
A matched 2D control shows the same colored blocks as separate tiles. For Qwen3.5-27B, exact match rises from 3.0% in 3D to 52.4% in 2D (+49.4 pp; 95% CI [+39.8, +59.0]), and per-color error drops from 0.50 to 0.11 blocks.
Fine-tuning Qwen3.5-27B on 800 T5 examples raises exact match from 3.0% to 68.6%, yet the model answers T7, which its counts already settle, far better from those counts written as text than from the images. Untuned open-weight models show the same gap.
Thinking raises T4 by +13.9 pp on average, yet T5 stays at 0.7–1.0%, larger budgets lower T1 and T11, and visual misreadings often persist through the reasoning.
Mean accuracy change (pp) over the 12 open-weight models at the default 1024-token budget; whiskers show 95% paired bootstrap intervals. Across all tasks the mean moves by only +0.9 pp.
Mean accuracy (%) of Qwen3.5-27B and Gemma4-31B at four reasoning budgets. T4 improves and levels off after 2048 tokens; every nonzero budget leaves T1 below budget 0.
In a manual audit, visual misinterpretations occur in 42 of the 44 annotated traces that end in incorrect answers. Models misread the scene and keep reasoning from that misreading instead of correcting it.
State-machine summary of annotated reasoning traces. Flawed steps include visual misreads; sound steps include correct perception and valid tactics. Dotted links class switches, and outer arcs trace–answer mismatches.
Gains from scale and visual cues are selective. A probe at Qwen3.5-27B's language-model input recovers exact counts on 65.0% of objects, against 3.0% for the VLM, pointing to difficulty in using available visual information.
Within Qwen3.5 (non-thinking), going from 0.8B to 27B improves T2, T4, T6, T7, T9 and T11 by 18–42 pp, while the other seven tasks change by at most 6.4 pp. Gemma4 shows the same uneven pattern, and both families stay near floor on T3 and T5.
The mixture-of-experts variants average below their dense counterparts of similar total size and remain at 0.0% on T3 and T5.
Color or coordinate-axis overlays keep the geometry fixed. Color helps most on T12 (+6.8 pp), where it can mark corresponding parts across candidate views, but complete structural answers stay wrong.
Probes on averaged frozen features recover volume, number of colors and visible floors even from the smallest deployed encoders, and this holds across encoder sizes. Complete per-color counts, however, reach at most 3.5% EM across encoders, including a depth-supervised one, when the probes are trained on the 100 benchmark objects.
Trained on 800 generated objects and tested on the 100 benchmark objects, attention probes recover exact counts from the frozen features far more often than the VLMs built on them, and scores keep rising with more training objects.
For Qwen3.5-27B, count information reaches the language-model input, pointing to difficulty in how the language model uses it. Three observations reinforce this: training only the language model raises T5 to 68.6% with the encoder and connector frozen; the model counts far more accurately when each block is a separate 2D tile; and aggregate-quantity probe scores stay flat across encoder sizes while VLM accuracy rises.
Without images, models keep some scores through constant answers, and in a fixed-layout MCQ one model reaches 100% by always choosing the third option.
Removing the images substantially reduces accuracy on several tasks while leaving others nearly unchanged. Without images, all three tested models answer 0 on every T10 instance and 3 on every T9 instance, still reaching 51% and 43%.
The candidate-selection tasks shuffle the position of the correct answer, so always choosing one position yields at most 37.3% on T11 and T13 and 51.0% on T12. As a control, posing T1 as a five-option MCQ with the correct volume always in the middle raises accuracy by +39.0 pp on average over open-ended prediction, with gains of up to +57.8 pp per model. Without images, Gemma4-31B reaches 100% by choosing the third option on every instance.
Across BedBench, we find that models get individual judgments right while the exact answer stays wrong, and recovering a property does not mean using it consistently.
For evaluation, 3D benchmarks should report exact answers with error sizes, test whether related answers agree, and check that scores depend on the image, since accuracy on individual judgments can overstate 3D understanding.
For modeling, researchers should design training objectives that improve use of visual information and consistency. For instance, models could be trained on exact labels across many tasks, stating intermediate structure before answering, with objectives that penalize disagreement between related answers.
We suggest future works to test these approaches, probe language-model layers to explain why the available information is not used reliably, and extend these diagnostics to natural and embodied scenes.
Loading the questions…
@article{leduc2026bedbench,
title = {Where Do Vision Language Models Fail at 3D Reasoning?},
author = {Le-Duc, Khai and Bui, Nguyen Kim Hai and Ngo, An and Hoang, Nam D.
and Song, Yueqi and Liu, Jiarui and Hoang, Thai Quoc and Wu, Fang
and Ngo, Chris and Zhang, Terry Jingchen and Diab, Mona T.
and Xu, Ran and Savarese, Silvio and Wu, Xiao-Ming},
journal = {arXiv preprint arXiv:2610.39639},
year = {2026}
}