Lift, Associate, and Fuse: A Decision-Centric Framework for 2D-to-3D Foundation Model Transfer

Wentao Sun; Yiping Chen; John S. Zelek; Jonathan Li

cs.CVAug 21, 2026

Lift, Associate, and Fuse: A Decision-Centric Framework for 2D-to-3D Foundation Model Transfer

Authors: Wentao Sun, Yiping Chen, John S. Zelek, Jonathan Li

Organizations: Department of Systems Design Engineering, University of Waterloo, N2L 3G1 Waterloo, Canada · School of Geospatial Engineering and Science, Sun Yat-sen University, 519082 Zhuhai, China

Abstract

Methods that transfer predictions from two-dimensional foundation models into three-dimensional segmentation are commonly grouped by task or representation. Those groupings obscure the decisions that determine whether a system remains coherent across views: where image evidence is grounded, when observations become one identity, how semantic and granularity conflicts are handled, which information is fused, and what state survives for later queries. We introduce Lift, Associate, and Fuse (LAF), a decision-centric framework that represents a transfer system as five operators: Generate, Associate, Reconcile, Fuse, and Persist/Query. LAF defines an explicit contract for the persistent carrier---its spatial support, semantic state, identity state, uncertainty, provenance, and supported operations---and identifies the first stage at which discarded evidence becomes unrecoverable. We operationalize the framework as a structured audit protocol and apply it to 161 systems available through 7 August 2026, spanning point-, field-, Gaussian-, object-, graph-, and memory-based carriers. Representation, temporal, relational, and feed-forward stress tests required no additional analytical stage after the final confirmation pass. The resulting decision traces expose four recurring properties: association does not establish identity; carrier design fixes both the query interface and correction boundary; rendered-view, native-3D, and proposal-level evaluations are not interchangeable; and qualifiers such as training-free, real-time, open-vocabulary, and generalizable are meaningful only when attached to a stage and a complete cost ledger. LAF therefore supplies a representation-neutral method for comparing existing systems, diagnosing irreversible failures, and specifying revisable 3D perception for future agents.

Figures & tables

Figure 1: Decision-centric LAF framework for 2D-to-3D foundation-model transfer. A system is represented by five operators—Generate, Associate, Reconcile, Fuse, and Persist/Query—that expose where image evidence is grounded in 3D, where cross-view identity, semantic disagreement, and granularity are resolved, and what persistent carrier is constructed. The feedback path represents online updates and active observation, while the lower row shows representative irreversible error propagation.

Stage	Decision being made	Representative mechanisms	First irreversible risk
Generate	What evidence is exposed, at which scale, and with what confidence?	SAM masks and tracks, CLIP/OpenSeg features, DINO structure, generated captions ( Peng et al., 2023 ; Ye et al., 2024 ; Ding et al., 2023 )	A missing object, part, or label cannot be reconstructed after the source evidence is discarded.
Associate	Which point, voxel, ray sample, or Gaussian may receive each observation?	Calibrated projection, NeRF transmittance, Gaussian contribution weights, learned point maps ( Kerr et al., 2023 ; Shi et al., 2024 ; Gong et al., 2026 )	Pose, depth, visibility, or renderer errors move otherwise correct evidence to the wrong surface.
Reconcile	Which observations share identity, meaning, or granularity?	Tracking, overlap graphs, matching, clustering, hierarchy and scale conditioning ( Yin et al., 2024 ; Yan et al., 2024 ; He et al., 2024 )	A false merge, split, or parent relation becomes the topology inherited by later queries.
Fuse	How are repeated and conflicting observations combined?	Weighted means, robust selection, learned fields, sparse codes, external indices ( Jatavallabhula et al., 2023 ; Shi et al., 2024 ; Budimir et al., 2026 )	Averaging or compression can erase minority evidence while making the result appear confident.
Persist/Query	What state remains, and what can a later query return or revise?	Point features, neural fields, Gaussian attributes, object sets, scene graphs, temporal memories ( Peng et al., 2023 ; Wu et al., 2024 ; Xie et al., 2026 )	The stored carrier fixes the available query surface, update rule, and recovery boundary.

Table 1: Mechanism-first audit of the five transfer stages. The stages describe decisions, not mandatory software modules; one model may implement several jointly.

Figure 2: Formal LAF decision trace and persistent-carrier contract. Each operator transforms or constrains the hypotheses available to the next stage. The carrier is specified by spatial support

\mathcal{S}

, semantic state

\mathcal{Z}

, identity and topology

\mathcal{I}

, uncertainty

\mathcal{U}

, provenance

\mathcal{P}

, and supported operations

\mathcal{O}

. The lower band depicts progressive loss of recoverable alternatives, while the feedback path represents correction or additional observation.

Concept queried	Common terms encountered	Operational distinction in LAF
Connecting 2D evidence to 3D	lifting; projection; back-projection; distillation; feature or label transfer	Projection names a geometric operator; lifting is the broader association step; distillation learns a student or carrier; transfer also includes masks, identities, geometry, and labels rather than features alone.
Learning and supervision regime	training-free; optimization-free; annotation-free; zero-shot	Training-free may still optimize each scene; optimization-free may use a trained feed-forward model; annotation-free may rely on pseudo-labels; zero-shot concerns held-out concepts or tasks, not the absence of computation or training.
Vocabulary and interaction	open-vocabulary; vocabulary-free; open-set; promptable	Open-vocabulary accepts previously unspecified category text; vocabulary-free avoids a fixed enumerated label list but may still constrain language or proposals; promptable only states that user evidence can condition an output.
Evaluation domain	rendered-view; image-space; native-3D; primitive-level; proposal-level	Rendered masks score pixels under a renderer; native-3D metrics score points, surfaces, voxels, or Gaussians; proposal-level metrics also depend on the proposal source and matching protocol.
Continuous semantic state	semantic field; language field; feature field; embedding field	A semantic field may store class scores; a language field supports text-aligned queries; a feature field can contain self-supervised or task-specific descriptors without language alignment.
Retained scene state	persistent carrier; scene representation; semantic memory; map state	Carrier denotes the state actually available to later queries; scene representation can mean geometry alone; semantic memory emphasizes retention or update, and may be an external index or graph rather than the rendered scene.

Table 2: Terminology crosswalk used to operationalize decision traces. Terms grouped in one row are common neighbors or aliases, but the boundary column records why they cannot be treated as exact synonyms.

Figure 3: Four realizations of 2D-to-3D association and their characteristic failure boundaries. Hard projection uses calibrated depth and an explicit visibility rule; neural fields distribute evidence through volumetric weights; Gaussian renderers use alpha-compositing contributions over explicit primitives; and learned or temporary correspondence introduces a predicted or reconstructed intermediary. Each mechanism locates evidence on candidate 3D support, but identity, semantic agreement, and granularity still require reconciliation.

Carrier	Natural output	Main advantage	Correction boundary
Points / voxels / meshes	Native 3D labels, masks, or features	Direct geometric extraction and compatibility with standard 3D benchmarks ( Peng et al., 2023 ; Nguyen et al., 2024 )	Quantization, visibility filtering, and hard proposal topology can be difficult to undo.
Neural fields	Rendered relevance, affinity, or mask values	Continuous spatial queries and differentiable visibility ( Kerr et al., 2023 ; Kim et al., 2024 )	Native 3D extraction requires sampling and thresholds; corrections may require scene optimization.
Gaussian attributes	Rendered and primitive-level responses	Explicit editable primitives with fast differentiable rendering ( Wu et al., 2024 ; Shi et al., 2024 )	Semantics inherit reconstruction geometry, alpha compositing, and per-Gaussian storage choices.
Object sets	Ranked objects, masks, and descriptors	Compact repeated retrieval and object-level updates ( Takmaz et al., 2023 ; Lu et al., 2023 )	Objects absent or wrongly merged during construction are unavailable to later language queries.
Graphs / hierarchies	Nodes, relations, paths, and multiscale supports	Relational and parent-aware reasoning beyond unary segmentation ( Werby et al., 2024 ; Wang et al., 2025c )	One node or parent error can alter retrieval, relations, and planning simultaneously.
Temporal memories	Identity- and time-conditioned masks or states	Online updates, reappearance handling, and spatio-temporal queries ( Xu et al., 2025a ; Meng et al., 2026 )	Identity switches and stale evidence become persistent state-transition errors.

Table 3: Persistent carrier families and their query and correction contracts. “Natural output” denotes the representation returned without an additional conversion protocol.

Figure 4: Why rendered-view, native-3D, proposal-level, and downstream evaluations are not interchangeable. Rendered pixels can hide disconnected or incomplete 3D support; native-3D metrics expose geometry and topology; proposal metrics depend on instance generation and matching; and agent-level success additionally depends on the query and action policy. Cross-method comparison is valid only when geometry, views, prompts, conversions, and oracle access are controlled.

Evaluation family	Prediction unit	Compatible evidence	Required disclosure
Rendered localization	Pixels in held-out or prompted views	mIoU, boundary IoU, relevancy localization under a shared renderer and view set	Rendering backbone, test views, scale selection, and whether native geometry is evaluated.
Native semantic segmentation	Points, voxels, mesh faces, or Gaussians	mIoU or class accuracy on the same geometry, split, and label mapping	Geometry source, ignored labels, vocabulary access, and feature-to-label conversion.
Native instance / panoptic segmentation	3D proposals or instance masks	AP, AP50/AP25, PQ, recognition and segmentation quality under the same proposal protocol	Proposal source, class-agnostic versus semantic scoring, oracle masks, and overlap threshold.
Part and hierarchical segmentation	Parts, trees, or threshold-selected partitions	Part IoU, hierarchical consistency, and stability under shared object and prompt sets	Part taxonomy, orientation, granularity control, prompt visibility, and parent-child ground truth.
Online / dynamic segmentation	Frame identities, tracks, or space-time supports	Frame AP, temporal IoU, identity consistency, update and recovery measures	Sensor stream, observation order, target identity access, disappearance, and correction protocol.
Relational / downstream tasks	Graph edges, retrievals, paths, edits, or actions	Relation recall, retrieval accuracy, navigation success, or edit quality within the same task contract	Upstream oracle nodes, language model calls, map state, task policy, and whether segmentation is scored separately.

Table 4: Evaluation families that should remain separate unless the same conversion and oracle protocol is applied to every method.

Ledger stage	What must be counted	Cost scope	Common omission
Geometry	RGB-D fusion, SfM, NeRF/3DGS optimization, or feed-forward reconstruction	Per scene or per stream	Calling a method reconstruction-free when calibration or pretrained geometry is still required.
2D evidence	Mask generation, tracking, detection, captioning, and feature extraction over all views	Per image, video, or prompt	Reporting only the downstream 3D module while omitting foundation-model calls.
Association	Projection, visibility tests, inverse rendering, view selection, and temporary point-map alignment	Per observation or scene	Treating correspondence as free preprocessing.
Reconciliation	Matching, graph construction, clustering, hierarchy recovery, and identity maintenance	Per scene or update	Excluding iterative merge, tracking, or graph costs from “fusion” time.
Carrier construction	Distillation, field optimization, code learning, or cross-scene training	Per scene or amortized training	Comparing a feed-forward pass with per-scene fitting while omitting pretraining accelerator-hours.
Persistence	Carrier parameters, external indices, raw evidence, and transient peak memory	Bytes per scene and peak device memory	Reporting model size without scene state, or scene state without retained images and indices.

Table 5: Full cost ledger for separating one-time construction, persistent storage, and marginal query cost. NR should be reported when a component is not measured.

Figure 5: Failure propagation, irreversible commitment, and recovery operations in LAF. Red paths show how upstream errors can alter persistent state and downstream behavior; gray paths denote discarded evidence; and blue paths denote recovery operations supported when sufficient uncertainty and provenance remain. The first irreversible loss is system- and query-dependent and may occur at any operator where evidence needed for a later correction is discarded.

Failure source	Propagation into the carrier	Diagnostic	Recovery requirement
Geometry / visibility	Correct evidence is assigned to an occluder, floater, or wrong surface	Pose/depth perturbation, oracle geometry, and visible-only support precision/recall	Reprojection audit and local reassociation without rebuilding unrelated state.
Proposal topology	Missing, merged, or over-fragmented masks define an incomplete object inventory	Proposal-oracle versus classifier-oracle evaluation; small-object and part recall	New proposal admission and reversible split/merge operations.
Identity	Tracker switches or false cross-view matches become persistent object state	Observation-order replay, injected identity switches, disappearance and reappearance tests	Merge provenance, confidence, rollback, and contradictory-evidence handling.
Semantic conflict	Correlated teacher errors or averaging produce confident wrong labels	Per-teacher ablation, calibration, counterfactual context, and disagreement retention	Source-aware fusion, abstention, and relabeling without changing geometry.
Granularity / hierarchy	One scale or bottleneck edge changes many part and parent decisions	Scale and threshold sweeps, parent perturbation, and partition stability	Query-dependent granularity and reversible hierarchy edits.
Compression	Quantization, Top-K atoms, or hard IDs erase rare but useful evidence	Query-after-compression, rare-query recall, code occupancy, and storage-accuracy curves	Expandable codes, retained provenance, or external evidence lookup.

Table 6: Failure sources, their propagation paths, and diagnostics that identify the first irreversible information loss.

Figure 6: From passive segmentation to agentic 3D perception. A versioned carrier retains geometry, identities, semantic state, uncertainty, provenance, and relations while observations, queries, actions, and corrections form a closed loop. In this schematic, Update memory combines the Fuse and Persist operations. Uncertainty may trigger an active viewpoint, human correction, or retrieval of additional image evidence, while explicit revision operations prevent persistent identity from degenerating into an immutable mask.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 416 papers

Mar 4, 2026

PlaneCycle: Training-Free 2D-to-3D Lifting of Foundation Models Without Adapters

Large-scale 2D foundation models exhibit strong transferable representations, yet extending them to 3D volumetric data typically requires retraining, adapters, or architectural redesign. We introduce PlaneCycle, a training-free, adapter-free operator for architecture-agnostic 2D-to-3D lifting of foundation models. PlaneCycle reuses the original pretrained 2D backbone by cyclically distributing spatial aggregation across orthogonal HW, DW, and DH planes throughout network depth, enabling progressive 3D fusion while preserving pretrained inductive biases. The method introduces no additional parameters and is applicable to arbitrary 2D networks. Using pretrained DINOv3 models, we evaluate PlaneCycle on six 3D classification and three 3D segmentation benchmarks. Without any training, the lifted models exhibit intrinsic 3D fusion capability and, under linear probing, outperform slice-wise 2D baselines and strong 3D counterparts, approaching the performance of fully trained models. With full fine-tuning, PlaneCycle matches standard 3D architectures, highlighting its potential as a seamless and practical 2D-to-3D lifting operator. These results demonstrate that 3D capability can be unlocked from pretrained 2D foundation models without structural modification or retraining. Code is available at https://github.com/HINTLab/PlaneCycle.

Yinghong Yu, Guangyuan Li, Jiancheng Yang

ELLIS Institute Finland, Espoo, 02150, Finland · Aalto University, Espoo, 02150, Finland

3D Representation Learning 3D Foundation Models

Ask PDF

Dec 17, 2025

Multi-View Foundation Models

Foundation models are vital tools in various Computer Vision applications. They take as input a single RGB image and output a deep feature representation that is useful for various applications. However, in case we have multiple views of the same 3D scene, they operate on each image independently and do not always produce consistent features for the same 3D point. We propose a way to convert a Foundation Model into a Multi-View Foundation Model. Such a model takes as input a set of images and outputs a feature map for each image such that the features of corresponding points are as consistent as possible. This approach bypasses the need to build a consistent 3D model of the features and allows direct manipulation in the image space. Specifically, we show how to augment Transformers-based foundation models (i.e., DINO, SAM, CLIP) with intermediate 3D-aware attention layers that help match features across different views. As leading examples, we show surface normal estimation and multi-view segmentation tasks. Quantitative experiments show that our method improves feature matching considerably compared to current foundation models.

Leo Segre, Or Hirschorn, Shai Avidan

Tel Aviv University

Multi-View Consistency Vision Foundation Model Adaptation

Ask PDF

Sep 10, 2026

Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation

Classical classifier-combination work distinguishes score-level fusion from hard decision-level voting. We revisit this distinction where independently pretrained, frozen foundation models are composed at inference time for generalized few-shot 3D segmentation. We ask: how much useful semantic information is lost when heterogeneous sources are collapsed to a single class before they can interact? We answer with a same-input semantic-retention intervention. Dense RegionPLC and sparse cross-view SAM3 evidence, model weights, masks, geometry, vocabularies, and fusion rules are frozen; only the number of semantic alternatives retained before interaction is varied via a matched top-k ladder. On 156 held-out ScanNet200 scenes, top-1 reaches 28.47 harmonic-mean (HM) IoU while full distribution fusion reaches 34.87 HM (+6.40, 95% CI [+5.24,+7.64]). The pattern replicates on 50 ScanNet++ scenes: 23.02 vs. 26.50 HM (+3.48, 95% CI [+1.64,+5.93]). The conclusion is robust: full-distribution HM is stable across sparse-source weights 0.3--0.7; alternative operators (max, geometric pooling) also outperform top-1; and a GroundingDINO--SAM2.1 source-replacement diagnostic shows monotonic HM increase from 14.77 to 18.75 with full retention. Calibration diagnostics reveal opposite miscalibration of the two sources, yet correcting calibration does not eliminate the retention advantage. Across datasets and source stacks, most information is recovered by retaining a compact set of plausible alternatives. The contribution is a controlled diagnosis of premature semantic collapse as a repeatable information bottleneck in heterogeneous frozen-model composition.

Silas Kwabla Gah, Ebenezer Owusu

Department of Computer Science University of Ghana, Legon

3D Foundation Models Ensemble Learning Information Bottleneck 3D Semantic Segmentation Few-Shot Semantic Segmentation

Ask PDF

Abstract

Transfer Learning

Explore similar work

PlaneCycle: Training-Free 2D-to-3D Lifting of Foundation Models Without Adapters

Multi-View Foundation Models

Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation