Thinking with Visual Primitives

May 1, 2026

On maze solvability, a binary question, frontier models cluster around 49-50%, which is a coin flip, while DeepSeek's model gets 66.9%. On path tracing, competitors land between 24% and 47% and it reaches 56.7%. The paper behind those numbers puts the failure somewhere other than seeing the maze.

Show a frontier model a dense chart, a cluttered scene or a schematic with overlapping parts and it will describe what it sees with reasonable accuracy. What it loses is the object three steps later. The chain says "the bear in the upper left" in step 1 and refers back to it later, and those two references are strings with no location behind them. They drift. They get confused with a different bear, a different corner, a different object mentioned in between. As the chain lengthens the model starts inventing spatial relationships it no longer holds.

DeepSeek calls this the Reference Gap. Most multimodal work has been chasing the Perception Gap, meaning resolution and visual granularity, through high-resolution image cropping, better patch encoders and larger visual transformers, and that work has moved. The Reference Gap survives all of it. A model can perceive the scene and still reason wrongly, because language is a poor medium for pointing at spatial objects. You can close the Perception Gap and keep this one.

The response is a new kind of token. A spatial coordinate sits inside the chain of thought, produced while thinking, as integral as the words around it, where an annotation stapled on after the answer would sit outside the reasoning.

Take this question: "Is the red cube on the left side of the scene touching the metal sphere that's nearest to it?" Find the cube, find the spheres, pick the nearest, check adjacency. Each step needs a hold on one specific object. In language that hold is "the red cube" and "the nearest metal sphere", which are descriptions. With two cubes and several spheres in view they are ambiguous. If the wording shifts to "the cube on the left", then "the red object", then "the leftmost object", you cannot tell whether the chain still means the same thing.

People point. The computational equivalent is a bounding box or a point coordinate emitted inline. Instead of "I see a bear in the upper left", the model writes "I see a bear at coordinates (452, 23, 804, 411), clinging to a tree trunk," and the later step that needs the bear refers to the box. The paper calls boxes and points "minimal units of thought", the smallest spatial reference precise enough to stay unambiguous across a long chain.

Boxes and points

Boxes encode extent and scale, meaning where an object is and what region it occupies. Counting needs that, since you can check whether two boxes are distinct objects, and so do attribute comparison (is this box larger than that one) and identification. Points encode location and nothing about size, and suit topological questions. In a maze you track a path through a structure, so the primitive is a sequence of positions and not a set of regions.

The paper trains a specialist for each before unifying them. A box can be verified by whether it fully contains the referenced object. A point has a weaker check, since any point inside the object's boundary is technically valid, and the training signal is noisier. That is why the authors put box training ahead of point training in data curation.

Data

Pretraining starts from nearly 98,000 bounding box data sources, counted across research datasets, commercial annotation projects and web-scraped resources. A multimodal LLM reviews each dataset and rejects those with machine-generated codes, private identifiers, or labels too ambiguous to train on, which leaves 43,141. A geometric filter drops a dataset if more than 50% of annotations are missing, if boxes are severely truncated, or if boxes cover more than 90% of the image. That last case is the mega box, a classification dataset converted to box format with no localization, so the "box" is the whole frame. After the cut, 31,701 sources remain.

Category-balanced sampling from those gives approximately 40 million training examples. Box data skews toward people, cars and animals in natural scenes, and technical categories are underweight. Without balancing the model would ground common objects strongly and the rarer ones that show up in technical work poorly.

The post-training cold-start set is approximately 600,000 samples across four domains. Maze navigation takes roughly 460,000 of them because it is the hardest task and needs the densest signal. Topologies include hexagonal and circular as well as rectangular, and difficulty runs from two-node connectivity checks to problems with hundreds of chained operations. Path tracing uses entangled Bezier curves. Straight segments with clear endpoints can be solved by matching endpoints without following the path, while entangled curves demand continuity at every intersection, and the uniform-color mode removes color as a cue to make the test sharper.

Training

The base model is DeepSeek-V4-Flash, a mixture-of-experts transformer with 284 billion total parameters and 13 billion active per forward pass. Raw pixels get compressed 7,056x on the way to KV cache entries. A 756x756 image becomes 2,916 patches of 14x14, a 3x3 spatial block cuts those to roughly 324 visual tokens, and Compressed Sparse Attention shrinks the KV cache by about 4, leaving around 81 effective entries. Claude Sonnet 4.6 uses around 870 tokens for an 800x800 image. This model uses around 361, with about 90 KV cache entries, and scores higher on the benchmarks it targets. Heavy compression usually costs accuracy, and on these tasks it did not; what bound was the ability to hold a precise reference across steps.

The post-training pipeline has five stages, and the order is the careful part. Box grounding and point grounding differ enough that training them together from the start produces mixed strategies mediocre at both, so specialists come first. A box expert and a point expert go through supervised fine-tuning independently. Then GRPO (Group Relative Policy Optimization) runs on each, with a reward that scores format correctness, task-specific accuracy and quality from a separate LLM evaluator (coherent, free of internal contradiction) all at once, so a format violation is learned as a different failure from a semantic error.

Unified RFT (Reinforcement Fine-Tuning) follows: a fresh model trains from the base pretrained checkpoint on rollouts generated by the two specialists. Further fine-tuning either specialist would forget, and starting from base avoids that. On-policy distillation comes next, with the unified model minimizing reverse KL divergence against both teachers at once. Forward KL, the standard maximum-likelihood one, pushes the student to put probability wherever the teacher does and spreads mass over low-quality outputs. Reverse KL is mode-seeking: the student covers the teacher modes instead of averaging them. An average of the box and point distributions would be competent at neither.

Two reward functions avoid the obvious binary, because a terminal correct/incorrect flag teaches a long spatial chain nothing. Counting decays smoothly — if the answer is n and the model says n+k, the reward is exp(-k), so predicting 5 for 6 beats predicting 15 for 6 even though both are wrong. A binary reward flattens that and learning stalls once counts leave small numbers.

Maze navigation splits the reward into causal exploration progress, exploration completeness for unsolvable mazes (were all reachable cells visited before declaring no path?), a wall-violation penalty, and final correctness. A 200-step chain that fails at step 187 would otherwise score zero for 187 correct steps and one miss. Unsolvable mazes are in the mix on purpose. A model trained only on solvable layouts guesses a path when the obvious options run out. One trained on unsolvable layouts has to exhaust the search space before saying no path exists.

Results

Across seven benchmarks of counting and spatial reasoning the model averages 77.2%, against GPT-5.4 at 71.1%, Claude Sonnet 4.6 at 65.3%, Qwen3-VL-235B at 68.1% and Gemini-3-Flash at 76.5%. The set is the paper's own selection, and the paper says explicitly that it claims nothing about general model capability.

Maze navigation needs a precise map of visited cells across potentially hundreds of steps, and without an anchor the model cannot tell which cells it has seen, so the solvability call degrades to a guess. Path tracing, 56.7% against a competitor ceiling of 47%, has the same shape: one path through intersecting curves without losing which curve you are on, which a sequence of points emitted during reasoning can track and a verbal description cannot.

The paper lists its own limits. The 384 visual token cap makes primitives imprecise in fine-grained scenes, when objects are small relative to the image, so two closely packed parts of a technical diagram get boxes looser than their real boundaries. The mechanism also needs explicit prompt language; the model does not notice by itself that an anchor would stop a reasoning error, so a user or system prompt has to ask for visual primitives. And point grounding transfers poorly. What maze and path-tracing training build does not carry cleanly to topological problems outside that distribution, while box grounding generalizes better, probably because its data spans more object types and scene contexts.

Past the clean cases

The paper shows this where the correct process is explicit spatial tracking: counting discrete objects, navigating a maze, tracing a path. Messier spatial reasoning (a schematic, an MRI slice, a robot camera) is harder to annotate with a box. Diagnosing a circuit, finding a lesion or steering through a space from camera input is less discrete and the rewards are harder to specify, and the right primitive may not be the one built for mazes. The paper does not measure any of that. Treating coordinates as cognitive tokens would need a reward that scores accurate coordinates as well as the verbal answer, ground-truth coordinates at each reasoning step, and an eval that checks the intermediate boxes and points.