What Does the Encoder Actually Decide? A Controlled Comparison of Vision Backbones on Joint Tree Segmentation and Stereo Depth

Yida Lin; Bing Xue; Mengjie Zhang; Sam Schofield; Richard Green

cs.CVSep 2, 2026

What Does the Encoder Actually Decide? A Controlled Comparison of Vision Backbones on Joint Tree Segmentation and Stereo Depth

Authors: Yida Lin, Bing Xue, Mengjie Zhang, Sam Schofield, Richard Green

Organizations: Centre for Data Science and Artificial Intelligence Victoria University of Wellington, Wellington, New Zealand · Department of Computer Science and Software Engineering University of Canterbury, Canterbury, New Zealand

Abstract

A robot pruning trees needs two facts per pixel: whether it belongs to a tree, and its distance. Both are usually obtained via task heads attached to a vision backbone chosen by reputation rather than measurement. Holding dataset, decoders, losses, schedule, and evaluation fixed, we ask: how much does the encoder choice change joint semantic segmentation and stereo depth on thin vegetation? We build a hard parameter-sharing network with one encoder feeding both branches, swapping only the encoder without downstream retuning. We evaluate [N] encoders across [M] architecture families (CNNs, transformers, hybrids, MLP-mixers, state-space models) near a ~25M budget, trained from scratch. Depth is evaluated on tree pixels only; segmentation uses boundary F1 and background IoU to prevent "label-everything-tree" shortcuts. Three findings stand out. First, the strongest encoders are convolutional and hybrid, not transformers: [BestEncoder] leads with [MIoU] segmentation mIoU and [Delta] depth $δ_1$ , while [X] of [Y] plain vision transformers collapse when trained from scratch. Second, parameter count does not predict quality --- [SmallEncoder] at only [P]M parameters outranks models two orders of magnitude larger. Third, segmentation and depth rankings agree strongly (Spearman $ρ$ = [RhoValue]), showing no task conflict. Finally, [K] of [N] encoders collapse to degenerate all-tree segmentation --- exposed by boundary F1 but hidden by region IoU.

Figures & tables

Panel of Figure 1 — Fig. 1 : Hard parameter sharing. One encoder, applied to both views with shared weights, produces five feature scales; a segmentation branch and a stereo branch consume them independently, with no connection between the branches. Only the encoder is swapped — everything to the right of $f_{0}\ldots f_{4}$ is identical across all 64 runs.

#	Encoder	Grp	P ${}_{\text{e}}$	mIoU	IoU ${}_{\text{bg}}$	BF1	AbsRel	$\delta_{1}$	#	Encoder	Grp	P ${}_{\text{e}}$	mIoU	IoU ${}_{\text{bg}}$	BF1	AbsRel	$\delta_{1}$
			(M)	$\uparrow$	$\uparrow$	$\uparrow$	$\downarrow$	$\uparrow$				(M)	$\uparrow$	$\uparrow$	$\uparrow$	$\downarrow$	$\uparrow$
1	internimage_t	D	28.8	0.650	0.357	0.419	0.786	0.725	33	convmixer_512_12	H	9.9	0.577	0.215	0.114	0.562	0.525
2	wrn_40_4	A	16.1	0.649	0.358	0.444	0.465	0.699	34	zfnet_slim	A	6.4	0.590	0.237	0.231	1.750	0.499
3	swin_tiny_w4	F	27.5	0.633	0.319	0.402	0.358	0.701	35	resnet18	A	11.2	0.596	0.281	0.467	0.732	0.491
4	xception_mobile_order	A	20.8	0.634	0.321	0.389	0.843	0.687	36	cyclemlp_b1	H	14.7	0.586	0.231	0.164	0.782	0.490
5	res2next50	A	22.6	0.635	0.325	0.369	0.545	0.680	37	crossformer_tiny	F	27.4	0.569	0.199	0.103	0.667	0.486

TABLE I : Per-encoder results on the tree50 test split, sorted by the composite score (Section VI ); the ranking continues in the right-hand block. Segmentation is scored on the left image, depth on ground-truth tree pixels only. P

{}_{\text{e}}

is encoder parameters (M). Best in each column is bold;

\uparrow

\downarrow

mark higher/lower is better. mIoU is the tree-vs-background mean IoU; IoU

{}_{\text{bg}}

and BF1 are the discriminative components a “label-everything-tree” collapse cannot win, and the run of near-zero entries in those two columns is the block of collapsed encoders discussed in Section VI .

	Group	$n$	mIoU $\uparrow$	BF1 $\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	EPE $\downarrow$	Score
A	Classic CNN	15	0.596	0.255	0.732	0.564	21.26	0.554
C	Lightweight CNN	5	0.482	0.004	1.044	0.469	29.87	0.477
D	Modern CNN	8	0.585	0.183	0.764	0.597	19.32	0.590
E	Plain Transformer	4	0.466	0.002	2.884	0.193	33.11	0.329
F	Hier. Transformer	9	0.569	0.103	0.647	0.594	21.30	0.552
G	CNN-Transf. Hybrid	12	0.599	0.257	0.766	0.599	19.74	0.600

TABLE II : Median performance per architecture-family group (median, not mean, so one collapsed encoder cannot dominate). Score is the composite of Section VI . Groups are ordered A–I.

Fig. 2 : Composite score by group; each dot is one encoder, boxes show the median and quartiles. Within-group spread rivals the differences between group medians.

Encoder	Grp	P ${}_{\text{e}}$ (M)	GFLOPs	FPS	Mem(GB)	mIoU $\uparrow$	$\delta_{1}\uparrow$
internimage_t	D	28.8	2213	3.7	3.5	0.650	0.725
wrn_40_4	A	16.1	2128	9.4	3.1	0.649	0.699
swin_tiny_w4	F	27.5	2184	4.0	3.8	0.633	0.701
xception_mobile_order	A	20.8	2267	7.8	3.5	0.634	0.687
res2next50	A	22.6	2304	7.4	3.7	0.635	0.680
edgenext_xxs	G	1.2	1414	9.2	2.9	0.626	0.686

TABLE III : Accuracy versus cost for the fifteen strongest encoders by composite score. FPS and peak memory are measured end-to-end at

1920\times 1080

, batch 1, mixed precision, on a single 16 GB GPU. GFLOPs and memory cover both views and both task branches.

Fig. 3 : Composite score versus encoder parameters (log scale). The black curve is the Pareto front; compact convolutional and hybrid encoders reach it, and the largest models do not.

Fig. 4 : Rank on segmentation (mIoU) against rank on depth (

\delta_{1}

). Points cluster on the diagonal (top-left is best on both), giving a strong Spearman correlation and showing that the two tasks prefer the same encoders.

Fig. 5 : Qualitative segmentation on three held-out test scenes. Columns: left image, ground-truth tree mask, the best encoder (internimage_t) and a degenerate one (resnext50_16x4d); the two prediction columns are coloured against the ground truth. The per-panel tree IoUs are within a few points of each other, so region IoU cannot separate the columns — but resnext50_16x4d has answered “tree” almost everywhere and floods every gap in the canopy with false positives (red), while internimage_t keeps the background between the branches open. Boundary F1, in the column headers, separates them at once.

Fig. 6 : Qualitative depth on the same three scenes. Columns: left image, ground-truth depth, the best encoder (internimage_t), and the lowest-ranked encoder overall (resmlp_12). The strong encoder recovers the fine branch structure and tracks the ground truth; the weakest produces horizontally-streaked depth unrelated to the scene. Depth is drawn on each model’s own predicted tree mask, which is why a segmentation failure is visible here as well.

Stereo Depth Estimation

See all 21 papers

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
Jun 2, 2026Wei Ding, Yudong Zhang, Ruobing Xie +3VLM Evaluation Vision-Language Models
Tsinghua University Beijing, China · Tsinghua University, Tencent Beijing, China · University of Science and Technology Beijing Beijing, China
Technical Report for the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Pretraining-Diverse Ensemble of Foundation Vision Encoders for Robust Outdoor Scene Understanding
Jun 22, 2026Boyan Wang, Yongxi Huang, Wenjing Li +4Vision Foundation Model Adaptation Deep Ensembles
LION Lab, Hefei University of Technology, Hefei, China. · University of Macau, Macau, China.
Invisible Shortcuts: Why Vision Encoders Know Your Camera
Aug 5, 2026Vladan Stojnić, Ryan Ramos, Giorgos Kordopatis-Zilos +2Vision Foundation Models Shortcut Learning
VRG, FEE, Czech Technical University in Prague · The University of Osaka

Jun 2, 2026J/K move · Enter open · S save

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun+2

Tsinghua University Beijing, China · Tsinghua University, Tencent Beijing, China · University of Science and Technology Beijing Beijing, China

VLM Evaluation Vision-Language Models Large Vision-Language Models Multimodal Foundation Models

Open paper AskPDF arXiv

As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design. Yet large vision-language models (LVLMs) currently lack the tools to do so, and parameter-efficient encoder configurations remain hard to identify before training. To re-examine encoder roles under joint training, on the 16-benchmark Cambrian-1 suite we retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline (~20k GPU-hours total), and report three findings. First, retraining each subset from scratch reveals encoder rankings that differ from those obtained by masking encoders on a fixed checkpoint, including which encoder ranks first overall. Second, we decompose each encoder's contribution into two axes, Capacity, the score an encoder reaches on its own, and Necessity, the drop when it is removed from the full pool. The two axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal, while pairing a high-Capacity anchor with an adaptive complement matches the full five-encoder model. Adding further encoders beyond this pair yields only marginal gains. Third, at fixed parameter count, per-encoder pre-projector effective rank explains the residual score variation. The strongest pairs combine an anchor whose rank survives joint training with a complement whose rank expands under it, suggesting that higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder-projector interface. Together, the Capacity-Necessity decomposition and the pre-projector rank analysis, along with comprehensive evaluation through retraining, expose a methodological gap in multi-encoder LVLM design, and offer concrete primitives for closing it.

Abstract

Stereo Depth Estimation

Explore similar work

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

Technical Report for the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Pretraining-Diverse Ensemble of Foundation Vision Encoders for Robust Outdoor Scene Understanding

Invisible Shortcuts: Why Vision Encoders Know Your Camera