Part I: Shuffle the Image Tokens, and Larger VLMs Start to Overcount
Pengxiang Li
Date: October 21, 2026

Fig. 1 Same image, same image tokens, different order. Each token keeps its content and its 2D position; only the order in which the language model reads the tokens changes. Qwen3.5-9B answers 4 in the original raster order and 8 after a random shuffle. Each strip shows the 1024 image tokens in sequence order; blue marks the tokens covered by a square.
<aside> 💡
TL;DR
Qwen-family VLMs give every image token a 2D position through MRoPE, so in principle the language model does not need the order in which image tokens arrive. We shuffle the image tokens while each token keeps its content and its 2D position, and test nine models from three Qwen families.
Table of Contents
A VLM cuts an image into patches, encodes them with a ViT, and feeds the resulting tokens to the language model in raster order: left to right, top to bottom. Qwen models also give every image token a 2D position (h, w) through MRoPE [1]. The language model can therefore read where each token sits in the image directly, and in principle it does not need the order of the tokens in the sequence.
Whether models use this order matters in practice. Many methods change how image tokens are arranged: pruning or merging tokens, reordering them by content, interleaving several images. If a model reads image layout from the sequence order, such changes can break abilities that general benchmarks do not measure. An earlier study on LLaVA-1.5 [2] shuffled visual tokens and found only small drops. We test whether this still holds for current models with 2D position encoding, and at larger scale.
We shuffle the image tokens right before they enter the language model. Each token keeps its content and its 2D position; only the order in which the language model reads them changes. The effect is narrow. Spatial-relation questions and general VQA change by a few points at most. Counting and grounding drop sharply, and counting fails in a consistent direction: the model overcounts, and larger models overcount more.
This is the first post of a series. Part I describes the effect and rules out the obvious explanations. Part II locates where the model reads the order. Part III tests ways to make the model use the 2D positions instead.
After the ViT and the patch merger, an image gives $N$ tokens $(v_i, p_i)_{i=1}^{N}$, where $v_i$ is the token feature and $p_i = (h_i, w_i)$ is its MRoPE position. The language model normally receives the tokens in raster order $i = 1, \dots, N$. For a permutation $\pi$ of $\{1, \dots, N\}$, we feed instead
$$ (v_{\pi(1)}, p_{\pi(1)}),\ (v_{\pi(2)}, p_{\pi(2)}),\ \dots,\ (v_{\pi(N)}, p_{\pi(N)}). $$
The set of (feature, position) pairs does not change; only their order in the sequence does. Text tokens keep their positions. For Qwen3-VL, the DeepStack features [3] that are added to the first language-model layers are permuted with the same $\pi$.
Besides a random $\pi$, we test four structured orders: reverse order, column order, shuffling the order of rows while keeping each row intact, and shuffling within each row while keeping the order of rows. Each image uses a fixed $\pi$. With three random seeds, Qwen3.5-9B scores 25.9%, 29.5% and 27.5% on CountBench.