Pixelis: Reasoning in Pixels, from Seeing to Acting

Yunpeng Zhou

cs.CVMar 26, 2026

Pixelis: Reasoning in Pixels, from Seeing to Acting

Authors: Yunpeng Zhou

Organizations: University of Reading, Reading, UK

Abstract

Most vision-language systems are static observers: they describe pixels, do not act, and cannot safely improve under shift. This passivity limits generalizable, physically grounded visual intelligence. Learning through action, not static description, is essential beyond curated data. We present Pixelis, a pixel-space agent that operates directly on images and videos via a compact set of executable operations (zoom/crop, segment, track, OCR, temporal localization) and learns from its consequences. Pixelis trains in three phases: (1) Supervised Fine-Tuning learns a pixel-tool grammar from Chain-of-Thought-Action traces with a masked imitation loss that upweights operation/argument tokens and auxiliary heads to stabilize pixel-grounded arguments; (2) Curiosity-Coherence Reward Fine-Tuning optimizes a dual-drive objective marrying prediction-error curiosity with adjacent-step coherence and a mild efficiency prior under a KL anchor, yielding short, valid, structured toolchains; (3) Pixel Test-Time RL performs label-free adaptation by retrieving neighbors, voting over complete trajectories rather than answers, and updating toward short, high-fidelity exemplars while constraining drift with a KL-to-EMA safety control. Across six public image and video benchmarks, Pixelis yields consistent improvements: the average relative gain is +4.08% over the same 8B baseline (peaking at +6.03% on VSI-Bench), computed as (ours-baseline)/baseline, while producing shorter, auditable toolchains and maintaining in-corridor KL during test-time learning. Acting within pixels, rather than abstract tokens, grounds multimodal perception in the physical world, linking visual reasoning with actionable outcomes, and enables embodied adaptation without external feedback.

Figures & tables

Figure 1: Pixelis uses executable pixel tools to act on images and videos. SFT learns tool syntax; CC-RFT shapes exploration with curiosity, coherence, and efficiency; Pixel TTRL adapts online via retrieval and trajectory voting under KL/EMA safety, producing shorter, auditable toolchains.

Figure 2: Three-phase training of Pixelis. SFT learns a tool-use grammar; CC-RFT shapes exploration with curiosity, coherence, and a light efficiency prior; Pixel TTRL adapts at test time by retrieving neighbours and updating toward behaviourally consistent trajectories under a KL-to-EMA safety constraint, turning raw tool traces into shorter, structured pixel toolchains.

Model Metric	Size	V* Bench [ 67 ]	MMBench v1.1 en [ 35 ]	MVBench [ 30 ]	InfoVQA test [ 41 ]	Video- MMMU [ 18 ]	VSI Bench [ 70 ]
GPT-5 (minimal)	-	-	81.4	69.9	-	61.6	-
Gemini-2.5-Pro (minimal)	-	-	86.6	82.9	-	79.4	31.4
Seed-1.5-VL [ 16 ]	20B	89.5	88.0	74.3	89.3	72.1	-
Qwen3-VL-8B-Instruct [ 47 ]	8B	86.4	85.0	68.7	83.1	65.3	59.4
Qwen3-VL-30B-A3B-Instruct [ 47 ]	30B	89.5(w tools)	87.0	72.3	82.0	68.7	63.2
InternVL3.5-A3B (with tools) [ 60 ]	30B	-	84.8	72.1	-	-	63.7

Table 1: Results on public benchmarks. We compare Pixelis to answer-only self-consistency, VLM test-time adaptation methods, process-supervised baselines, and a replicated Pixel Reasoner variant for each training phase. The upper block lists proprietary, non-directly comparable models; all claims use the same 8B backbone. Scores are reported on Tool-needed subsets with matched compute, sampling budget, acceptance rate, and token-level KL;

\dagger

denotes a win on the Tool-needed subset.

Figure 3: RFT process metrics: RaPR (top) and RaCPR (bottom). We compare Answer Only, +Curiosity, +Coherence, +Curiosity+Coherence, and +Curiosity+Coherence+Penalty (Pixelis). Adding curiosity alone increases RaPR but hurts RaCPR; adding coherence and a light penalty yields the highest RaPR/RaCPR with lower variance across seeds. Bars show means over 3 seeds with 95% BCa bootstrap confidence intervals (BH-corrected).

Figure 4: Pixel TTRL: accuracy (top) and token-KL to EMA (bottom) within corridor

[0.10,0.20]

. Safe variant (value-aware retrieval, trajectory voting, EMA+KL) stays in-corridor; no-safety drifts and degrades. PID-controlled

\beta

keeps KL bounded.

Strategy	Coverage (%)	Err@Sel (%)	KL P95
Hard majority	100.0	8.7	0.29
Weighted + abstention ( $\delta{=}0.08$ )	88.9	7.1	0.19
Hard majority (adv. 30% flipped)	100.0	11.3	0.31
Weighted + abstention (adv. $\delta{=}0.10$ )	86.2	9.2	0.21

Table 4: Selective adaptation on

5{,}000

shifted queries. Coverage is the answered fraction; Err@Sel is the error on that subset. Weighted voting with abstention lowers Err@Sel and KL P95 relative to hard majority.

Figure 5: Qualitative comparison. The baseline often loops or over-zooms on irrelevant regions, while Pixelis forms shorter, more coherent toolchains that align with the queried evidence, reflected in higher RaPR/RaCPR and VisFid.

Figure 6: Qualitative behaviors. Coherence regularization prevents oscillatory tool loops and encourages consistent steps, while Pixel TTRL uses retrieved trajectories to self-correct toolchains.

Item	Setting
$L_{\text{mm}}$ position	3rd fusion block from top
Visual pooling	Masked mean; dilated $3\times 3$ if $\sum M<9$
$d$ (token dim)	1024 (projected to 768/512 by $g_{\phi}$ )
$g_{\phi}$	MLP $[2d{+}\|\mathcal{A}\|]\!\to\!768\!\to\!512$ , GELU, LN, drop 0.1
MC dropout	4 samples, $p{=}0.1$ (dynamics head only)
Dynamics head $h_{\psi}$	MLP $[2d{+}\|\mathcal{A}\|]\!\to\!1024\!\to\!(d{+}1)$

Table S1 : Implementation hyperparameters for

E_{t}

and the dynamics head.

Figure S1 : Calibration curves for Pixelis (8B) on the dev split. Reliability diagrams comparing the uncalibrated policy, temperature scaling (TS), and DS-weighted consensus. Each point aggregates a probability bin; the

x

-axis is the mean predicted probability and the

y

-axis is empirical accuracy. The diagonal corresponds to perfect calibration. Legends report expected calibration error (ECE), showing that TS reduces miscalibration and DS-weighted consensus further tightens calibration without degrading accuracy.

Figure S2 : Per-tool latency distribution for Pixelis (8B). Violin plots show the distribution of per-call latency for each tool component (SEG/ZOOM, TRK, OCR, TEMP, PROP, Retrieval+voting) over the same

1{,}500

-query eval set as Table S2 . Each violin overlays a box (IQR and median), jittered samples, and a mean marker

\hat{\mu}

. The

y

-axis is latency per tool call in seconds; the

x

-axis enumerates tools. Compared to the static medians in Table S2 , the plot makes the long tails for TRK and TEMP explicit, while PROP and Retrieval+voting stay tightly concentrated near zero.

Figure S3 : Query-level latency distribution for Pixelis (8B). Histogram of tool-side latency per query and full end-to-end latency on the same

1{,}500

-query eval split as Table S2. Tool-side latency aggregates all tool calls within a trajectory, while end-to-end latency additionally includes model forward and lightweight bookkeeping. The distributions make the tail behavior explicit beyond the static summaries in the main text (p50

\approx 5.8

s, p90

\approx 8.1

s, p95

\approx 10.2

s), and show that most queries fall well below the worst-case thresholds even under tool-heavy chains.

Figure S4 : Memory footprint vs. decisive chain length. Estimated GPU memory usage for the 8B model as a function of decisive chain length

|\tau|

on the eval workload. The base footprint corresponds to the backbone and cached features; each additional step adds buffers for pixel tools (segmentation, tracking, temporal localization) and intermediate activations. Growth with chain length in the operating regime of Pixelis is modest and approximately linear (median

|\tau|\approx 3.7

), indicating that the shorter chains induced by CC-RFT and Pixel TTRL reduce not only latency but also memory variance and peak usage.

Component	Setting	Notes
Key dims	$d_{\text{txt}}=768,\;d_{\text{pix}}=1024$	concat $\to$ 1792; OPQ $\to$ 768 (CPU)
IVF lists	$n_{\text{list}}=4096$	k-means (20 iters)
PQ	$m=64,\;b=8$	64 B/code
OPQ	1792 $\to$ 1792 rotation	200 epochs
Search	$n_{\text{probe}}=32$ , top-256	re-rank exact cosine
Scale	$N=1.25$ M items	footprint-level entries

Table S2 : Index configuration and footprint (default).

Prior	Acc. $\Delta$	RaCPR gain recovered	Notes
L2 smoothness (args)	$-0.7$ pt	$31\%$	shortest chains degrade on jumps
TV (args)	$-0.5$ pt	$36\%$	preserves edges, still argument-space
CPC on $E_{t}$	$-0.3$ pt	$54\%$	tighter but exploration contracts
Adjacency on $E_{t}$ (ours)	0.0	100%	best Acc./RaCPR at matched KL

Table S3 : Structure priors vs. our adjacency coherence (median over seeds; matched KL).

\Delta

is vs. full CC-RFT.

Tool	Model (ver.)	FT?	Source / License
SEG	SAM2	no	repo / license
TRK	BoT-SORT	no	repo / license
OCR	PP-OCR 3.0	no	repo / license
TEMP	VTG heuristics	n/a	protocol / license

Table S4 : Tool identities, earlier snapshot for ablations. Full configs in Table S8.

Tool	Model (ver./date)	FT?	Source
SEG	SAM 2 (ViT-H, 2024-09)	no	Meta SAM2
TRK	BoT-SORT (2023-11; MOT multi-object)	no	BoT-SORT repo
OCR	PP-OCR 3.0 (PaddleOCR 3.0, 2024)	no	PaddleOCR
TEMP	UniVTG heuristics / eval harness (2023-10)	n/a	UniVTG

Table S8 : Tool identities (new snapshot used in the main text; stable versions).

Item	Role	Source
TrackEval (HOTA/CLEAR - MOT)	tracker scoring	official TrackEval repo/paper
ANLS	OCR similarity	standard VQA-OCR protocol
DINOv2 ViT-L/14	fixed embeddings	GitHub (facebookresearch/dinov2)

Table S9 : Evaluator/encoder references (frozen).

Setting	Acc. (%)	RaCPR	RaPR	$\Delta$ Len
$L_{\text{mm}}$ (default)	$+$ 0.0	$+$ 0.0	$+$ 0.0	$0.00$
$L_{\text{mm}}$ (-2 blocks)	$-0.1{\pm}0.1$	$-0.5{\pm}0.3$	$+0.2{\pm}0.2$	$+0.12$
$L_{\text{mm}}$ (-1 block)	$-0.1{\pm}0.1$	$-0.3{\pm}0.2$	$+0.1{\pm}0.2$	$+0.07$
$L_{\text{mm}}$ (+1 block)	$+0.0{\pm}0.1$	$+0.2{\pm}0.2$	$-0.1{\pm}0.1$	$-0.05$
$L_{\text{mm}}$ (+2 blocks)	$+0.1{\pm}0.1$	$+0.4{\pm}0.3$	$-0.1{\pm}0.2$	$-0.09$
Gate off ( $S{=}1$ , $\beta{=}0$ )	$-0.2{\pm}0.1$	$-0.7{\pm}0.3$	$+0.3{\pm}0.2$	$+0.18$

Table S10 : Sensitivity to feature-layer choice and uncertainty gating (process-audit split; 3 seeds; mean

\pm

sd).

\Delta

Len is the change in qualified chain length vs. default.

Figure S5 : Risk–coverage under label noise and compression. Risk–coverage curves for Pixelis (8B) on the shifted eval split. The

x

-axis shows coverage (fraction of non-abstained queries) and the

y

-axis shows Err@Sel (error among selected queries). We compare clean labels, 20% label flips, and JPEG+resize perturbations, averaging over tasks with shaded 95% confidence intervals. Across coverage levels, the curve under clean labels lies lowest, JPEG+resize remains close, and 20% label flips consistently increase risk, but the ordering of the three curves stays stable, indicating that our uncertainty-weighted consensus degrades gracefully under moderate noise and compression.

Figure S6 : SFT example. “Who is asleep where”: Image_0 has $2$ woof + $1$ meow, Image_1 has $0$ woof + $2$ meow.

Figure S7 : RFT example. Multiple candidate tool-use trajectories are compared to learn preferences over answer correctness, curiosity, coherence, and penalty.

Figure S8 : RFT example. Multiple candidate tool-use trajectories are compared to learn preferences over answer correctness, curiosity, coherence, and penalty.

Figure S9 : TTRL constructs an empirical action distribution from the winning trajectories’ tool calls and nudges the policy toward this distribution under a KL-to-EMA corridor.

Figure S10 : The “vote” selects trajectories (zoom/segment/track patterns), not just answer phrases.

Tool Use in VLMs

Tool use in vision-language models (VLMs) studies how models call visual, computational, and retrieval tools to improve grounded perception and reasoning.

See all 117 papers

May 28, 2026

VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies

Recent work has begun to equip vision-language-action (VLA) policies with explicit intermediate reasoning. In embodied control, however, textual chain-of-thought is a poor fit: irrelevant or weakly textual information can interfere with action prediction, while autoregressive text decoding adds too much latency for real-time closed-loop execution. We present VISUALTHINK-VLA, a visual intermediate-reasoning framework for accurate, low-latency VLA policies. Our bootstrapping philosophy is to guide action with effective visual thinking: VISUALTHINK-VLA bootstraps action prediction through a compact visual-evidence interface that preserves spatial precision while avoiding decoding overhead. Besides, to further improve performance and efficiency, VISUALTHINK-VLA adopts a tailored selective routing mechanism to learn the visual evidence tokens, enabling low-latency inference while preserving high-capacity specialization. We also introduce VisualEvidence-Kit, a supervision-and-audit resource centered on a VisualEvidence-Agent that constructs a 754.7k VLA instructions VisualEvidence-Set for route supervision and counterfactual faithfulness tests. Across multiple benchmarks and real-robot evaluation, VISUALTHINK-VLA achieves the highest success rate on most benchmarks while reducing the multi-second latency of reasoning-augmented baselines to the sub-second regime. For example, on BridgeData V2, it reduces step latency from 8.377,s with ECoT to 0.367,s, achieving a 22.8 times speedup.

Mingjian Gao, Wenqiao Zhang, Yuqian Yuan +9

1Zhejiang University · 2Cornell University · 3National University of Singapore +1

Efficient VLM Inference Efficient VLA Model Inference Efficient VLA Models Vision-Language-Action Models Visually Grounded Reasoning VLMs for Robotics+1

Ask PDF

Jun 1, 2026

Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigation

Vision-language models (VLMs) have become a common foundation for vision-and-language navigation in continuous environments (VLN-CE). Yet most VLM-based methods cast navigation as low-level action prediction, an interface that is ambiguous, tied to short-horizon motion primitives, and inefficient due to repeated VLM querying. We propose Goal2Pixel, a pure pixel-based paradigm that reformulates VLN-CE as navigable pixel grounding. Rather than predicting actions, Goal2Pixel uses the image plane as a unified spatial interface between VLM reasoning and robot motion: the model predicts a visible navigable pixel, which is back-projected into a 3D waypoint for forward navigation. Non-forward actions are encoded by auxiliary left/right/bottom regions for turning and stopping. To support long-horizon navigation, we propose ViKeyMem, a visibility-aware keyframe memory, achieving a favorable performance-efficiency trade-off while reducing training cost by over 50%. To adapt pretrained VLMs to navigable pixel grounding, we introduce semantic embeddings and coordinate-aware auxiliary losses. Among methods trained without external data, Goal2Pixel achieves competitive state-of-the-art performance on R2R-CE and RxR-CE. On R2R-CE Val-Unseen, it achieves 54.1% SR and 52.5% SPL with only 7.75 VLM calls per episode. Compared with direct action prediction, Goal2Pixel improves SR by 21.2 percentage points while requiring 6 times fewer VLM calls. Project Page: https://baobao0926.github.io/Goal2Pixel/.

Muyi Bao, Yuxin Cai, Hang Xu +7

Carnegie Mellon University · Nanyang Technological University

Embodied Navigation VLM Adaptation Vision-Language Navigation Visually Grounded Reasoning Visual Robot Navigation Visual Grounding for Robotics

Ask PDF

Jun 30, 2026

PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seeking

This paper explores multi-turn visual reasoning and observes that MLLMs repeatedly fail to localize the target, leading to long, redundant trajectories. We attribute this failure to the entanglement of reasoning and perception within a single model, the MLLM reasons and localizes simultaneously, and inaccurate localization triggers additional reasoning turns that bloat the trajectory. To solve this problem, we propose PixelEyes, a multi-turn visual reasoning agent that explicitly decouples reasoning from perception, i.e., the reasoner decides what to look for, while a specialized perception tool answers where it is. Specifically, PixelEyes introduces 1) Mask-guided Visual Search. A referring segmentation model is invoked to provide mask-precise localization, freeing the reasoner from the need to compensate for imprecise grounding. 2) Semantic-region Breadth-first Search (BFS). To eliminate redundant loops caused by repeatedly cropping incorrect sub-regions, we organize exploration as a breadth-first search over semantic regions. To internalize these capabilities, we construct the PixelEyes-6K dataset by resynthesizing expert trajectories from existing data. This explicitly embeds our mask-guided search and BFS logic into the model. We further introduce Pinpoint-Bench, a zero-hint visual search benchmark, i.e., no location cues are provided in the question, with instance-level masks and bounding boxes that separate localization failures from reasoning failures, enabling fine-grained analysis of failure modes such as inattentional blindness. Recent state-of-the-art MLLMs and visual reasoning agents leave large headroom on Pinpoint-Bench, demonstrating its quality and difficulty. Code and models are open-sourced.

Dengxian Gong, Yuanzheng Wu, Haobo Yuan +11

Wuhan University · UC Merced · UTS +3

Vision-Language Grounding Tool-Using Vision-Language Agents VLM Reasoning Visually Grounded Reasoning Visual Search

Ask PDF

Abstract

Tool Use in VLMs

Explore similar work

VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies

Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigation

PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seeking