Unified Law for Visual Tasks (The Four Laws of Computer Vision)
Organizations: Rutgers, USA · NTU, Singapore · UCF, USA · UW–Madison, USA ·
Abstract
In this paper, we propose a Unified Law for Visual Tasks, which could be further divided into Four Laws of Computer Vision, with the goal of enabling a single framework or method to uniformly train on and perform nearly all visual tasks. Our starting point is an inspiration and hypothesis: if the world information is encoded into image, which we call World Snapshots, then image processing is world processing, and image generation is World Snapshot generation. Furthermore, if all visual and visualization tasks can be reformulated as generation tasks, then a unified method, World Snapshot Models (WSMs), can perform the vast majority of visual tasks, much as large language models have largely unified natural language processing. This paper validates these laws across more than 1,000 trained and evaluated visual tasks (defined by distinct input-output mappings) and presents a unified training framework for future visual-task research and applications.
Figures & tables
| Comparison Aspect | WSMs | LLMs |
|---|---|---|
| Domain | Vision | NLP |
| Methodology | Encode Information into Image | Predict Next Token |
| Main Laws/Principles Realized | Unified Laws : all vision tasks are generative forms | Scaling Laws : emergent intelligence, prompt-answer pairs |
| Can Unify All Corresponding Tasks | Yes | Yes |
| Can Subsume Other Domains | Yes | Yes |
| Can Scale Domains and Tasks Multiplicatively | Yes | Yes |
| TunnelingNet | Interface | Condition target | Attention | RGB evidence route |
|---|---|---|---|---|
| 003 | Estimation/reconstruction | RGB c Normal c | MTA | RGB c condition |
| 270 | Narrow generation | Text + Normal c + RGB 1 RGB c | NJA | Native reference |
| 275 | Conditional control | Normal c Edge c | MTA | None |
| 313 | Inverse rendering | RGB 1 + Normal c Edge c | MTA | RGB 1 condition |
| Objective | 003 Normal | 275 Edge | Mean |
|---|---|---|---|
| Plain rectified flow | 0.6805 | 0.5961 | 0.6383 |
| Single-step clean latent | 0.4308 | 0.4186 | 0.4247 |
| High-noise anchor | 0.3263 | 0.4062 | 0.3662 |
| Multi-resolution noise | 0.6505 | 0.5917 | 0.6211 |
| Interface | Heads | Inputs | Targets | Evaluations |
|---|---|---|---|---|
| RGB estimation / reconstruction | 38 | 1 | 38 | 692 |
| Text + WSG narrow generation | 38 | 38 | 1 | 704 |
| WSG conditional control | 536 | 37 | 37 | 8,411 |
| RGB-assisted inverse rendering | 536 | 37 | 37 | 8,384 |
| Total | 1,148 | – | 39 | 18,191 |
| Interface | Error reduction |
|---|---|
| RGB estimation / reconstruction | 57.2% |
| Text + WSG narrow generation | 78.3% |
| WSG conditional control | 33.5% |
| RGB-assisted inverse rendering | 46.3% |
| All interfaces | 41.4% |
| Level | The question | What we did | What it shows |
|---|---|---|---|
| I: broad use | Was the law fitted only to the four tasks that first revealed it? | We established the four interfaces with four initial Tunnels, then kept the shared backbone frozen and trained the other 1,144 task routes. | The same rule realizes 1,148 trained and evaluated tasks, covering 39 output groups and all four interfaces. |
| II: real input–output learning | Did each route learn the requested relation for each example, or only a typical output for its modality? | We keep every prediction fixed and deliberately pair it with the wrong held-out target. A generic output would be equally close before and after this shift. | Correct pairs have 41.4% lower error overall, with a 51.9% median reduction per TunnelingNet. The routes therefore follow the individual input and target. |
| III: why representation matters | Does a frozen generator learn the same information more easily when it is written in a representation the generator already knows? | We write the same surface-normal target in five lossless coordinate systems: its native VAE coordinates and four reversible channel rearrangements. Every version reconstructs exactly the same normal image; only its familiarity to the backbone changes. | Native coordinates lower checkpoint-averaged error by 20.1%–26.6%, produce the lowest final error, and win 59 of 64 final sample–transformation comparisons. |
| Method | Venue | Parameters | Training Sample Budget (Updates Global Batch) | Relative Sample Budget | Training Time / Hardware | MAE | RMSE | PSNR | Acc | Acc | Acc | Pearson | Spearman | Inference Steps | Total Inference Time (s) | Inference Speed (FPS) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Marigold [ 55 ] | CVPR 2024 Best Paper Award Candidate | B | K | A few days 1 GPU | 0.247 | 0.291 | 11.22 | 0.183 | 0.293 | 0.441 | 0.890 | 0.875 | 105.47 | 0.2181 | ||
| DMP [ 56 ] | CVPR 2024 | B+LoRA | K | N/R | 0.277 | 0.324 | 10.14 | 0.119 | 0.211 | 0.386 | 0.736 | 0.693 | 5 | 24.38 | 0.9434 | |
| GeoWizard [ 57 ] | ECCV 2024 | B | K | K | Time N/R 8 GPUs | 0.308 | 0.384 | 8.72 | 0.237 | 0.345 | 0.440 | 0.818 | 0.795 | 7834.00 | 0.0029 | |
| Scaling Diffusion Perception [ 16 ] | CVPR 2025 | B ( B active) | Mixed-task phase N/R + K MoE updates | N/R | N/R | 0.232 | 0.274 | 12.22 | 0.235 | 0.344 | 0.508 | 0.868 | 0.825 | 833.92 | 0.0276 | |
| VGGT [ 58 ] | CVPR 2025 Best Paper Award | B | K frames | N/R | 9 days/64 A100s 13.8K A100-h | 0.243 | 0.296 | 10.98 | 0.193 | 0.297 | 0.453 | 0.719 | 0.698 | 1 pass | 4.83 | 4.7619 |
| MODUS [ 15 ] | ICML 2026 | B ( B active) | K packed seq. | K | 88.5 h/64 GH200s 5,664 GH200-h | 0.167 | 0.216 | 14.04 | 0.335 | 0.479 | 0.666 | 0.867 | 0.799 | 50 | 9025.88 | 0.0025 |
| Method | Venue | Parameters | Training Sample Budget (Updates Global Batch) | Relative Sample Budget | Training Time / Hardware | RMSE | Acc | Pearson | Spearman | Inference Steps | Total Inference Time (s) | Inference Speed (FPS) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Marigold [ 55 ] | CVPR 2024 Best Paper Award Candidate | B | K | A few days 1 GPU | 0.474 | 0.224 | 0.627 | 0.814 | 105.47 | 0.2181 | ||
| DMP [ 56 ] | CVPR 2024 | B+LoRA | K | N/R | 0.427 | 0.279 | 0.593 | 0.698 | 5 | 24.38 | 0.9434 | |
| GeoWizard [ 57 ] | ECCV 2024 | B | K | K | Time N/R 8 GPUs | 0.508 | 0.304 | 0.484 | 0.569 | 7834.00 | 0.0029 | |
| Scaling Diffusion Perception [ 16 ] | CVPR 2025 | B ( B active) | Mixed-task phase N/R + K MoE updates | N/R | N/R | 0.370 | 0.359 | 0.736 | 0.741 | 833.92 | 0.0276 | |
| VGGT [ 58 ] | CVPR 2025 Best Paper Award | B | K frames | N/R | 9 days/64 A100s 13.8K A100-h | 0.316 | 0.420 | 0.737 | 0.695 | 1 pass | 4.83 | 4.7619 |
| MODUS [ 15 ] | ICML 2026 | B ( B active) | K packed seq. | K | 88.5 h/64 GH200s 5,664 GH200-h | 0.225 | 0.632 | 0.858 | 0.826 | 50 | 9025.88 | 0.0025 |
| Method | Venue | Parameters | Training Sample Budget (Updates Global Batch) | Relative Sample Budget | Training Time / Hardware | Boundary Precision @2 px | Boundary Precision @3 px | Boundary FPR | Gradient MAE | Gradient RMSE | Inference Steps | Total Inference Time (s) | Inference Speed (FPS) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Marigold [ 55 ] | CVPR 2024 Best Paper Award Candidate | B | K | A few days 1 GPU | 0.736 | 0.768 | 0.0148 | 0.00575 | 0.0153 | 105.47 | 0.2181 | ||
| DMP [ 56 ] | CVPR 2024 | B+LoRA | K | N/R | 0.548 | 0.581 | 0.0183 | 0.00631 | 0.0169 | 5 | 24.38 | 0.9434 | |
| GeoWizard [ 57 ] | ECCV 2024 | B | K | K | Time N/R 8 GPUs | 0.203 | 0.234 | 0.2588 | 0.02707 | 0.0381 | 7834.00 | 0.0029 | |
| Scaling Diffusion Perception [ 16 ] | CVPR 2025 | B ( B active) | Mixed-task phase N/R + K MoE updates | N/R | N/R | 0.660 | 0.690 | 0.0164 | 0.00531 | 0.0139 | 833.92 | 0.0276 | |
| VGGT [ 58 ] | CVPR 2025 Best Paper Award | B | K frames | N/R | 9 days/64 A100s 13.8K A100-h | 0.531 | 0.604 | 0.0183 | 0.00655 | 0.0149 | 1 pass | 4.83 | 4.7619 |
| MODUS [ 15 ] | ICML 2026 | B ( B active) | K packed seq. | K | 88.5 h/64 GH200s 5,664 GH200-h | 0.703 | 0.723 | 0.0255 | 0.00582 | 0.0173 | 50 | 9025.88 | 0.0025 |
| Target | Name | Heads | Samples | Strict | DAcc | mIoU |
|---|---|---|---|---|---|---|
| 111 | Object category | 39 | 497 | 0.250 | 0.288 | 0.255 |
| 131 | Material class | 39 | 460 | 0.311 | 0.298 | 0.298 |
| 135 | Container flag | 39 | 429 | 0.915 | 0.925 | 0.493 |
| 136 | Object class | 39 | 460 | 0.141 | 0.085 | 0.085 |
| 195 | Semantic segmentation | 22 | 528 | 0.341 | 0.401 | 0.107 |
| TunnelingNet | Mapping | Native metrics | Result |
|---|---|---|---|
| 000000 | RGB crop SMPL pose and shape | SMPL shape-coefficient MAE | 0.0373 |
| 000001 | RGB person boxes | Precision / Recall / F1 | 0.820 / 0.770 / 0.794 |
| 000002 | RGB crop DWPose | Median XY error / [email protected] / [email protected] | 0.0925 / 0.526 / 0.725 |
| 000003 | RGB crop MediaPipe pose | Median XY error / [email protected] / [email protected] | 0.0852 / 0.568 / 0.797 |
| 000005 | Audio + RGB source box | Mean IoU / Macro-category IoU / [email protected] | 0.325 / 0.325 / 0.351 |
| 000006 | Audio semantic category | Top-1 / Balanced accuracy | 0.671 / 0.602 |
| Difficulty assessment | Tasks | Archive share |
|---|---|---|
| Difficulty A: structured development via WSM sparse mode | 705 | 36.2% |
| Difficulty B: minor interface, loss, label, or decoder adaptation | 286 | 14.7% |
| Difficulty C: aligned-modality data and training | 255 | 13.1% |
| Difficulty D: directly compatible application or scenario | 231 | 11.9% |
| Non-visual archive cross-listing | 470 | 24.1% |
| Total archived CV entries | 1,947 | 100.0% |
| Historical-overlap judgment | Tasks | Share of 1,072 |
|---|---|---|
| No historical full-contract match (unambiguously new) | 655 | 61.1% |
| Historical umbrella is underspecified (conservative ambiguity) | 401 | 37.4% |
| Clear historical full-contract overlap | 16 | 1.5% |
| Archived task | Level | Archived description | Manual interface judgment |
|---|---|---|---|
| Semantic Segmentation | B | N/A | Semantic Segmentation is compatible with existing WSG dense fields or pose representations; only labels, data adapters, or a small output-side… |
| Object Detection | A | N/A | Object Detection requires language, ranked retrieval, tracks/instances, navigation/trajectory, a mesh/graph, or temporal/other sparse structured… |
| Image Classification | B | Image Classification is a fundamental task in vision recognition that aims to understand and categorize an image as a whole under a specific label. Unlike object detection | Image Classification is compatible with existing WSG dense fields or pose representations; only labels, data adapters, or a small output-side… |
| Visual Question Answering (VQA) | A | Visual Question Answering (VQA) is a task in computer vision that involves answering questions about an image. The goal of VQA is to teach machines to understand the content of an image and answer… | Visual Question Answering (VQA) requires language, ranked retrieval, tracks/instances, navigation/trajectory, a mesh/graph, or temporal/other… |
| Named Entity Recognition (NER) | X | Named Entity Recognition (NER) is a task of Natural Language Processing (NLP) that involves identifying and classifying named entities in a text into predefined categories such as person names, or… | Named Entity Recognition (NER) is nonvisual in this archive entry, with no intrinsic visual input, output, or visual auxiliary role. |
| Pose Estimation | B | Pose Estimation is a computer vision task where the goal is to detect the position and orientation of a person or an object. Usually, this is done by predicting the location of specific keypoints … | Pose Estimation is compatible with existing WSG dense fields or pose representations; only labels, data adapters, or a small output-side… |
| Dimension | ControlNet | LoRA | MoE | IP-Adapter | IC-Control style | TunnelingNet / WSM |
|---|---|---|---|---|---|---|
| Method level | Large auxiliary control architecture | Parameter-efficient weight update | Conditional-computation and capacity architecture | External image-conditioning adapter | Joint visual-context organization | Condition–target task architecture spanning representation, routing, trajectory deflection, training, and deployment |
| Intuition | Copy a large control body beside the backbone | Attach a small numerical patch to selected weights | Add experts and route tokens among them | Attach an image-prompt branch | Place text, conditions, references, and target in one context | Open a small target tunnel inside a shared backbone and steer its generation path repeatedly through depth |
| Main modification site | Copied multi-layer network with injections into the backbone | Weight-side branches of selected linear, attention, or feed-forward layers | Expert layers and routers | Image encoder, projector, and adapter attention | Input sequence, joint attention, and associated fine-tuning layers | Target-side residual paths plus a dedicated condition-to-target evidence route |
| Added structure | Near-backbone-scale control body and injection links | Low-rank matrices and | Expert networks and router | Image encoder, projector, and adapter attention | Context organization, usually with fine-tuning or LoRA | Task tunnel, task key, condition–target roles, interface type, and evidence routing |
| Backbone treatment | Usually frozen | Original weights usually frozen | Implementation dependent; experts and routing are coordinated | Usually frozen | Shared model or LoRA is commonly fine-tuned | Generative backbone, text encoder, and VAE are frozen |
| How conditions enter | A copied control body extracts features and injects them layer by layer | Uses condition interfaces already present in the host architecture | Router assigns tokens to experts | Independent image-conditioning branch enters attention | Condition and target tokens interact in a common context | Clean evidence acts on the noised target through a declared evidence route |
| Extension paradigm | Relative scale |
|---|---|
| TunnelingNet | |
| IP-Adapter | |
| IC-Control style | |
| LoRA | typically – |
| ControlNet | |
| MoE | – |
| Family (named variants) | Motivation and variants tested | Observed result | What this taught us |
|---|---|---|---|
| Expanded RGB VAE (1) | We expected the pretrained RGB latent to contain spare capacity for world information, so we enlarged its encoder and decoder boundaries for additional WSG channels. | Additional channels changed the latent distribution immediately; preserving RGB and learning precise new fields competed for the same limited code. | Simply widening a mature RGB codec was not a scalable solution: every new field altered the shared latent and risked degrading all fields already stored there. |
| Frozen RGB path (1) | Because the expanded codec disturbed the mature RGB prior, we froze its RGB path and trained new layers around the encoder and decoder. | RGB survived better, while the additional channels had weak influence and decoded toward smooth averages. | Freezing protected RGB but starved the new fields of influence. Preserving an old capability and giving a new capability sufficient representational freedom require separate mechanisms. |
| Progressive TinyVAE expansion (6) | We froze the pretrained RGB path, widened hidden layers in 16-channel stages, expanded the latent from four through eight channels, and trained each new stage on full 256-channel WSG samples. | Dense fields began to reconstruct, sparse pose did not, and processing the complete WSG tensor reduced throughput from the original TinyVAE by a large margin. | Progressive growth did not remove the mismatch among dense and sparse fields, and its full-tensor cost grew immediately. More codec capacity alone could not solve heterogeneity or scaling. |
| Offset-routing schemes D–F (7) | Starting from , we compared single- and multi-depth direct offset injection, decoder-entrance versus later latent injection, four-scale encoder–decoder jump links, decoder-predicted and fixed-ratio offset recovery, deeper decoders, LCPE, dropout controls, and complete restoration. | Direct injection at several decoder depths and a deeper decoder helped most. Four-scale jump links could bypass the compact latent but transferred poorly to latent-only generation; decoder-predicted offsets were difficult to optimize, fixed-ratio recovery did not improve the result, and complete restoration failed. | The location changed what the model learned: early injection disturbed the shared RGB computation, late injection left less capacity for the new field, and jump links improved access to encoder evidence without making the latent self-contained. Initialization and information path mattered more than simply adding parameters. |
| LayerDiffuse-style auxiliary path (3) | We hypothesized that a separate lane could carry new fields without rewriting RGB, so we added an auxiliary encoder–decoder interaction modeled on transparency encoding. | It carried extra information and produced the strongest result in this family, but precise dense fields and sparse structures still exceeded the small protected channel. | A protected auxiliary lane worked for a small amount of added information, but its averaged outputs and limited bandwidth ruled it out as the carrier for a growing world representation. |
| Independent channel VAE (3) | Shared-codec interference suggested that each variable should first reconstruct in its natural space, so we trained channel-specific continuous VAEs. | Individual dense fields became easier to reconstruct, but every modality acquired its own coordinate system and distribution. | Separate codecs made individual fields reconstructable, but created mutually incompatible latent coordinates. This result directly motivated the later bridge experiments. |
| Family | Variant and key change | Why we tried it | What happened | What we learned |
|---|---|---|---|---|
| Direct map | Direct “God Bridge” Feed the source latent to one compact CNN and regress the paired target latent in a single pass. | The most optimistic hypothesis was that a compact endpoint mapper could solve the paired relation in one pass. | The bridge was trainable, but its loss initially showed little meaningful decline and its outputs lacked fidelity. | A small one-pass network could reduce latent error, but it could not reproduce the target images faithfully. We therefore stopped treating the entire cross-space conversion as ordinary pointwise regression. |
| Direct map | Attention-augmented CNN Insert attention blocks into the compact CNN so every latent location can use distant locations. | Convolution might miss global relationships, so attention blocks were added while the bridge was kept lightweight. | Global interaction did not remove the endpoint mismatch. | Because attention did not solve the mismatch, the failure was not caused mainly by a limited receptive field. Global communication alone could not translate coordinates whose meanings differed across the two latent spaces. |
| Measurement | Cached paired latents Encode every paired RGB/target sample once, save both latent tensors, and train the bridge only from this cache. | Image loading and repeated VAE encoding obscured the mapping experiment, so paired RGB and target latents were precomputed once and streamed during bridge training. | Training reached roughly thousands of updates per hour and became much easier to inspect, while decoded failures remained. | After removing repeated encoding from the loop, training became fast while the same decoded errors remained. This ruled out the data pipeline as the main bottleneck and isolated latent-space translation as the problem. |
| Alignment | Whitening Measure each space’s mean and scale, standardize both before the bridge, and undo target standardization before decoding. | Mean and scale mismatch might dominate optimization, so each latent population was standardized with measured statistics and inverted after prediction. | Numerical conditioning improved, while semantic errors remained after first- and second-order scale alignment. | Standardizing both latent populations made the optimizer behave better, but the same samples still decoded to the wrong content. Similar means and scales therefore did not imply corresponding latent meanings. |
| Direct map | Wider residual CNN Enlarge each directional residual CNN to roughly 3M parameters while keeping the paired data and objective fixed. | Insufficient bridge capacity was a simple explanation for the plateau, so each directional CNN was enlarged to roughly three million parameters. | Training loss improved more than decoded quality. | The larger mapper fit the paired coordinates better without producing comparably better images. The plateau was therefore not simply caused by too few bridge parameters. |
| Measurement | FP16 versus FP32 cache Store and decode the same cached latent pairs in FP16 and FP32, then compare the resulting images. | Blurred decoding might have come from quantization in the cached latent pairs, so the same pairs were checked and decoded under both cache precisions. | The precision change did not explain the large visual gap. | Both cache precisions produced the same large visual gap. Quantization during latent storage was therefore ruled out as the source of the bridge failure. |
| Experimental family | Formal knowledge |
|---|---|
| Expanded RGB VAE | |
| Frozen RGB path | |
| Progressive TinyVAE expansion | |
| Offset-routing schemes D–F | |
| LayerDiffuse-style auxiliary path | |
| Independent channel VAE |
| ID | Family | Variant | Direct observation and knowledge retained | Formal link |
|---|---|---|---|---|
| 1 | Expanded RGB VAE | Direct expanded RGB input/output boundary | Directly widening the mature RGB boundary disturbed its latent; added information and preserved RGB competed. | |
| 2 | Frozen RGB path | Frozen native RGB path with trainable outer layers | Freezing preserved RGB but left the added field too little influence. | |
| 3 | Progressive TinyVAE expansion | Hidden-width expansion in 16-channel stages | Extra codec capacity helped dense reconstruction but did not solve sparse structure or full-tensor scaling. | |
| 4 | Progressive TinyVAE expansion | Five-channel latent stage | Extra codec capacity helped dense reconstruction but did not solve sparse structure or full-tensor scaling. | |
| 5 | Progressive TinyVAE expansion | Six-channel latent stage | Extra codec capacity helped dense reconstruction but did not solve sparse structure or full-tensor scaling. | |
| 6 | Progressive TinyVAE expansion | Seven-channel latent stage | Extra codec capacity helped dense reconstruction but did not solve sparse structure or full-tensor scaling. |
Vision Foundation Models
Vision foundation models are large pretrained visual systems adapted to recognition, segmentation, geometry, and video tasks.
See all 152 papers