Thinking with Visual Primitives

May 1, 2026

Multimodal language models have gotten good at seeing. You can show a frontier model a dense chart, a cluttered scene, or a schematic with overlapping components, and it will describe what it sees with reasonable accuracy. What it struggles to do is reason about what it sees across multiple steps, and the reason is specific: natural language is a poor reference medium for spatial objects.

When a reasoning chain says "the bear in the upper left" in step 1 and refers back to it three steps later, those two references are strings. They are not anchored to anything. They can drift. They can be confused with a different bear, a different corner, a different object that was mentioned somewhere in between. As chains get longer, this ambiguity compounds, and the model begins hallucinating about spatial relationships it has lost track of. The paper from DeepSeek's team calls this the Reference Gap, and it is distinct from the Perception Gap that most multimodal AI research has focused on.

The Perception Gap is about resolution and visual granularity. Prior work on high-resolution image cropping, better patch encoders, and larger visual transformers all targets this gap. And the field has made real progress there. The Reference Gap is about something different: the model can perceive accurately and still reason incorrectly, because it has no mechanism to precisely anchor a verbal reference to a spatial location across multiple reasoning steps.

The paper's response is to give the model a new kind of token: a spatial coordinate embedded directly in the chain of thought. Not an output annotation added after reasoning is complete, but a first-class token produced during reasoning, as integral to the thought as the words around it.

The Reference Gap in practice

To see why this matters, consider a concrete example. A model is asked: "Is the red cube on the left side of the scene touching the metal sphere that's nearest to it?"

Answering this correctly requires identifying the red cube, identifying all metal spheres, finding the one nearest to the red cube, and then checking adjacency. Each step depends on maintaining a precise reference to a specific object across the chain. In natural language, that reference is "the red cube" and "the nearest metal sphere." These are descriptions, not pointers. If the scene has two cubes and several spheres, the description becomes ambiguous. If the model produces slightly different language at each step, "the cube on the left" versus "the red object" versus "the leftmost object," it can lose track of whether it is still talking about the same thing.

Human reasoners handling spatial tasks naturally point. The finger gesture or the gaze serves as a precise external reference that language alone cannot provide. What the paper introduces is the computational equivalent: a bounding box or point coordinate emitted inline, replacing the verbal gesture with a pixel-level anchor.

The model writes not "I see a bear in the upper left" but "I see a bear at coordinates (452, 23, 804, 411), clinging to a tree trunk." The next step that references this object can refer to its box, not to a description that might match multiple objects. The reference is precise. It does not drift.

This is what the paper means by calling bounding boxes and points "minimal units of thought." They are the smallest spatial reference that is precise enough to be unambiguous across reasoning steps.

Two primitives for two kinds of spatial reasoning

The paper defines two primitives and assigns them to different reasoning contexts based on what kind of spatial information is needed.

Bounding boxes encode extent and scale. A bounding box tells you where an object is, how large it is, and what region of the image it occupies. This is what you need for counting (you can check whether two boxes are distinct objects), for attribute comparison (you can check whether the box in question is larger or smaller than another), and for object identification (you can confirm that the object at these coordinates matches the description). Bounding boxes are more information-dense than points because they encode area rather than position alone.

Points encode location. A point tells you where something is without specifying how large it is. Points are appropriate for topological reasoning where the question is about a location or path rather than an object with extent. In maze navigation, you are not navigating around objects; you are tracking a path through a structure. The relevant spatial primitive is a sequence of positions, not a set of bounding regions.

The paper trains separate specialist models for each primitive type before unifying them. This matters because the training data for each is structurally different and the failure modes are different. Box annotations can be verified by checking whether the box fully contains the referenced object. Point annotations have a weaker verification signal, since any point within the object boundary is technically valid, which makes the training signal noisier. The authors address this by prioritizing box training over point training in the data curation stage.

Architecture: compression as a design constraint

The base model is DeepSeek-V4-Flash, a mixture-of-experts transformer with 284 billion total parameters. In a standard dense model, all parameters are active for every token. In a mixture-of-experts model, only a subset of experts activates per token, reducing the active parameter count per inference. This model has 13 billion active parameters per forward pass, which is within the range of dense models that are practical to run at scale.

The visual processing pipeline produces a 7,056x overall compression ratio from raw pixels to KV cache entries. The path: a 756x756 image is divided into 14x14 patches by the visual encoder, producing 2,916 patch tokens. A 3x3 spatial compression block reduces those to roughly 324 visual tokens passed to the language model. The Compressed Sparse Attention mechanism in the base model then compresses the KV cache by a factor of approximately 4, leaving around 81 effective entries representing the full image.

This compression has a concrete consequence for cost. Claude Sonnet 4.6 uses around 870 tokens for an 800x800 image. This model uses around 361 tokens with about 90 KV cache entries. The paper's model achieves better performance on the specific benchmarks it targets while processing fewer tokens per image. This is not always the expected tradeoff: typically, aggressive compression trades accuracy for efficiency. The paper's result suggests that for the tasks it is evaluated on, the compression is not the bottleneck.

The reason is probably that the bottleneck the paper addresses is not visual resolution. It is reasoning structure. A higher-resolution visual representation does not help if the model cannot maintain precise references across reasoning steps. Adding visual primitives addresses the actual failure mode. Compression affects a different variable.

Training pipeline: five stages

The post-training pipeline is where the paper's most careful engineering lives. The challenge is that bounding box grounding and point grounding are structurally different enough that training them simultaneously from the start causes interference. The authors address this by training specialist models first, then unifying.

Stage 1 is specialized supervised fine-tuning: a box expert and a point expert are trained independently on their respective data. This prevents the models from developing mixed strategies that are mediocre at both tasks.

Stage 2 is specialized reinforcement learning using GRPO (Group Relative Policy Optimization) applied independently to each expert. The reward models at this stage operate along three dimensions simultaneously: format correctness (is the output syntactically valid?), quality as judged by a separate LLM evaluator (is the answer coherent and free of internal contradictions?), and task-specific accuracy. Running these in parallel rather than sequentially allows the model to learn that format violations are not the same failure as semantic errors.

Stage 3 is Unified RFT (Reinforcement Fine-Tuning): a fresh unified model is trained from the base pretrained checkpoint using rollout data generated by the two specialists. This avoids the catastrophic forgetting that would result from further fine-tuning either specialist.

Stage 4 is on-policy distillation: the unified model is trained to minimize reverse KL divergence against both teacher distributions simultaneously. Reverse KL divergence is mode-seeking rather than mean-seeking, which means the student learns to cover the modes of the teacher distributions rather than averaging between them. This is appropriate here because the box and point distributions have genuinely different modes, and averaging them would produce a model that is competent at neither.

The choice of reverse KL over forward KL is a technical decision with concrete implications. Forward KL (used in standard maximum likelihood training) encourages the student to assign positive probability everywhere the teacher does, which risks spreading probability mass over low-quality outputs. Reverse KL encourages the student to focus on outputs the teacher rates highly and assign near-zero probability elsewhere. For a unified model learning from two specialist teachers, this sharpening effect is appropriate.

Reward design: counting and maze navigation

Two of the reward functions are worth examining in detail because they are not obvious choices.

For counting tasks, the reward uses smooth exponential decay rather than binary correct/incorrect. Specifically, if the correct count is n and the model predicts n+k, the reward decays as exp(-k). This means predicting 5 when the answer is 6 scores much higher than predicting 15 when the answer is 6, even though both are technically incorrect. Binary rewards create a flat landscape where the model gets no gradient signal for being closer or farther from the correct answer, which makes learning slow for any count above small numbers. Smooth decay preserves gradient direction throughout the count range.

For maze navigation, the reward decomposes into four components: causal exploration progress (how much of the solvable maze has been explored?), exploration completeness for unsolvable mazes (has the model visited all reachable cells before declaring the maze unsolvable?), wall-violation penalty (how many moves passed through walls?), and final answer correctness. The decomposition matters because maze navigation chains can be hundreds of steps long. Without intermediate reward signal, a 200-step chain that fails at step 187 receives zero reward for 187 correct steps and one incorrect one, which is not an informative training signal. Decomposing the reward provides feedback at each stage of reasoning, not just at the conclusion.

The inclusion of unsolvable mazes is a design choice that forces the model to develop genuine exploration behavior rather than pattern-matching on solvable maze structures. A model that has only seen solvable mazes will tend to guess a path when it runs out of obvious options. A model that has been trained on unsolvable mazes learns to exhaust the search space before concluding that no path exists.

Data curation at scale

The pretraining data pipeline starts from nearly 98,000 bounding box data sources, a number that reflects how many sources of labeled image data exist across research datasets, commercial annotation projects, and web-scraped resources.

The first filtering pass is semantic: a multimodal LLM reviews each dataset and rejects those containing machine-generated codes, private identifiers, or labels too ambiguous to generate useful training signal. This reduces the pool to 43,141 sources.

The second filtering pass is geometric: datasets are rejected if more than 50% of annotations are missing (miss rate threshold), if boxes are severely truncated (truncation threshold), or if boxes cover more than 90% of the image (the "mega box" case, which usually indicates that a classification dataset's labels have been converted to bounding box format without actual localization). This reduces the pool to 31,701 usable sources.

Category-balanced sampling from these 31,701 sources produces approximately 40 million training examples. The category balancing is important because bounding box data is highly skewed toward common objects in natural scenes (people, cars, animals) and underweights technical and domain-specific object categories. Without balancing, the model would develop strong box-grounding for common objects and poor grounding for the less common ones that matter in technical applications.

The cold-start data for post-training is approximately 600,000 samples across four domains. Maze navigation is the largest at roughly 460,000, reflecting that this is the hardest task and requires the most dense training signal. The maze topologies vary (rectangular, circular, hexagonal) and difficulty levels span from simple two-node connectivity checks to problems requiring hundreds of chained operations.

The path tracing data uses entangled Bezier curves as the test environment. The specific design choice of entangled curves is intentional: straight line segments with clear endpoints can be solved by endpoint-matching rather than actual path following. Bezier curves with realistic entanglement require the model to follow continuity at every intersection, which tests whether the path-following primitive is a genuine spatial reasoning skill or a surface pattern. The uniform-color mode, which removes color as a disambiguation cue, is the sharpest test of this.

Benchmark results and what they mean

Across seven benchmarks measuring counting and spatial reasoning, the model averages 77.2%. The comparison points: GPT-5.4 at 71.1%, Claude Sonnet 4.6 at 65.3%, Qwen3-VL-235B at 68.1%, Gemini-3-Flash at 76.5%.

These numbers are meaningful on their own terms but limited in scope. The benchmarks were selected for the research focus, and the paper is explicit that these scores are not indicative of overall model capability. This is not a general multimodal benchmark result.

The more informative numbers are on topological reasoning specifically. On maze navigation, frontier models cluster around 49-50%, which is approximately chance for binary solvability questions. This model scores 66.9%. On path tracing, competitors score between 24% and 47%; this model scores 56.7%.

The gap on maze navigation is particularly striking. 49-50% accuracy on a binary question means the frontier models are not reliably better than coin flips on this task. The visual primitive mechanism moves this from chance to a meaningful capability gap. The explanation is structural: maze navigation requires maintaining a precise map of visited locations across potentially hundreds of steps. Without precise spatial anchoring, the model cannot reliably track which cells it has and has not visited, and the solvability determination degrades to guessing.

Path tracing at 56.7% versus a ceiling of 47% for competitors represents similar dynamics. The task requires following a specific path from start to end through intersecting curves without losing track of which curve is being traced. The visual primitive, a sequence of points emitted during reasoning, provides the continuity tracking that verbal description cannot.

Three limits the paper identifies

The paper's own limitations section is worth taking seriously because it points at the gaps between what was demonstrated and what would be needed for general deployment.

Resolution is the first constraint. The 384 visual token cap produces imprecise primitive output in fine-grained scenarios, specifically when objects are small relative to the full image or when the spatial distinction being encoded is subtle. A model trying to distinguish two closely-packed objects in a high-resolution technical diagram will produce bounding boxes that are less precise than the actual object boundaries warrant.

Trigger dependence is the second. The mechanism requires explicit activation through specific prompt language. The model does not autonomously recognize situations where spatial anchoring would prevent reasoning errors and invoke the mechanism. This limits practical utility: a user or system prompt must explicitly signal that visual primitives should be used, rather than the model deciding this from the structure of the problem.

Cross-scenario generalization for points is the third. The point-grounding capability developed through maze and path-tracing training does not transfer cleanly to new topological problems outside the training distribution. The box-grounding capability generalizes better, likely because the training data covers a broader range of object types and scene contexts. The spatial reasoning over points appears to have been learned in a more task-specific way that does not abstract well.

What this changes about multimodal reasoning

The dominant theory of multimodal AI progress has been that capability tracks visual processing quality: better encoders, higher resolution, more visual tokens. This paper challenges that theory not by arguing against visual quality but by identifying a second bottleneck that visual quality improvements cannot address.

The bottleneck is reference precision. A model with perfect visual perception but verbal references for spatial objects will still fail at multi-step spatial reasoning tasks that require tracking specific objects across a long chain. The failure is architectural, not perceptual. Better vision does not fix it.

Treating spatial coordinates as cognitive tokens rather than output annotations is an architectural choice with consequences throughout the training and inference pipeline. It requires different reward structures (the model must be rewarded for producing accurate coordinates, not just correct verbal answers), different data (annotations that include ground-truth coordinates at each reasoning step, not just at the output), and different evaluation (checking intermediate coordinate accuracy, not just final answer correctness).

The fact that this approach produces meaningful gains on tasks like maze navigation and path tracing, where frontier models were near chance, suggests that the bottleneck identification is correct. The gains are not from better visual encoding. The encoding is comparably compressed. The gains come from giving the model a way to maintain precise spatial references across a reasoning chain.

The open question is scope. The paper demonstrates this on tasks where the correct reasoning process involves explicit spatial tracking: counting discrete objects, navigating a maze, tracing a path. These are clean experimental settings where the benefit of precise references is unambiguous. The harder question is whether the same mechanism helps for the messier spatial reasoning problems in real applications: a model diagnosing a circuit from a schematic, identifying a lesion from an MRI slice, or navigating a robot through a physical space from camera input.

These applications involve spatial reasoning that is less discrete, less well-defined, and less amenable to precise bounding box annotation. The training data for them is harder to generate, the reward signals are harder to specify, and the correct primitives may be different from the ones developed for maze navigation. The paper's contribution is to establish that the bottleneck exists and that the mechanism can address it in clean experimental settings. Whether it extends is a research question, not a given.