Taskonomy: Disentangling Task Transfer Learning

Amir Zamir; Alexander Sax; William Shen; Leonidas Guibas; Jitendra Malik; Silvio Savarese

cs.CVApr 23, 2018

Taskonomy: Disentangling Task Transfer Learning

Authors: Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, Silvio Savarese

Organizations: Stanford University · University of California, Berkeley

Abstract

Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.

Figures & tables

Figure 1: A sample task structure discovered by the computational task taxonomy ( taskonomy ) . It found that, for instance, by combining the learned features of a surface normal estimator and occlusion edge detector, good networks for reshading and point matching can be rapidly trained with little labeled data.

Figure 2: Computational modeling of task relations and creating the taxonomy. From left to right: I. Train task-specific networks. II. Train (first order and higher) transfer functions among tasks in a latent space. III. Get normalized transfer affinities using AHP (Analytic Hierarchy Process). IV. Find global transfer taxonomy using BIP (Binary Integer Program).

Figure 3: Task Dictionary. Outputs of 24 (of 26) task-specific networks for a query (top left). See results of applying frame-wise on a video here .

Figure 4: Transfer Function . A small readout function is trained to map representations of source task’s frozen encoder to target task’s labels. If order $>1$ , transfer function receives representations from multiple sources.

Figure 5: Transfer results to normals (upper) and 2.5D Segmentation (lower) from 5 different source tasks. The spread in transferability among different sources is apparent, with reshading among top-performing ones in this case. Task-specific networks were trained on 60x more data. “Scratch” was trained from scratch without transfer learning.

Figure 6: Higher-Order Transfers. Representations can contain complementary information. E.g. by transferring simultaneously from 3D Edges and Curvature individual stairs were brought out. See our publicly available interactive transfer visualization page for more examples.

Figure 7: First-order task affinity matrix before (left) and after (right) Analytic Hierarchy Process (AHP) normalization. Lower means better transfered. For visualization, we use standard affinity-distance method $\textit{dist}=e^{-\beta\cdot P}$ (where $\beta=20$ and $e$ is element-wise matrix exponential). See supplementary material for the full matrix with higher-order transfers.

Figure 8: Computed taxonomies for solving 22 tasks given various supervision budgets (x-axes), and maximum allowed transfer orders (y-axes). One is magnified for better visibility. Nodes with incoming edges are target tasks, and the number of their incoming edges is the order of their chosen transfer function. Still transferring to some targets when tge budget is 26 (full budget) means certain transfers started performing better than their fully supervised task-specific counterpart. See the interactive solver website for color coding of the nodes by Gain and Quality metrics. Dimmed nodes are the source-only tasks, and thus, only participate in the taxonomy if found worthwhile by the BIP optimization to be one of the sources.

Task	$avg$	$rand$	Task	$avg$	$rand$	Task	$avg$	$rand$
Denoising	100	99.9	Layout	99.6	89.1	Scene Class.	97.0	93.4
Autoenc.	100	99.8	2D Edges	100	99.9	Occ. Edges	100	95.4
Reshading	94.9	95.2	Pose (fix)	76.3	79.5	Pose (nonfix)	60.2	61.9
Inpainting	99.9	-	2D Segm.	97.7	95.7	2.5D Segm.	94.2	89.4
Curvature	78.7	93.4	Matching	86.8	84.6	Egomotion	67.5	72.3
Normals	99.4	99.5	Vanishing	99.5	96.4	2D Keypnt.	99.8	99.4

Table 1: Task-Specific Networks’ Sanity: Win rates vs. random (Gaussian) network representation readout and statistically informed guess

avg

Figure 9: Evaluation of taxonomy computed for solving the full task dictionary . Gain (left) and Quality (right) values for each task using the policy suggested by the computed taxonomy, as the supervision budget increases(

\rightarrow

). Shown for transfer orders 1 and 4.

Figure 10: Generalization to Novel Tasks . Each row shows a novel test task. Left: Gain and Quality values using the devised “all-for-one” transfer policies for novel tasks for orders 1-4. Right: Win rates (%) of the transfer policy over various self-supervised methods, ImageNet features, and scratch are shown in the colored rows. Note the large margin of win by taxonomy. The uncolored rows show corresponding loss values.

Figure 11: Structure Significance. Our taxonomy compared with random transfer policies (random feasible taxonomies that use the maximum allowable supervision budget). Y-axis shows Quality or Gain , and X-axis is the supervision budget. Green and gray represent our taxonomy and random connectivities, respectively. Error bars denote 5 th –95 th percentiles.

Figure 12: Evaluating the discovered structure on other datasets: ImageNet [ 78 ] (left) for object classification and MIT Places [ 157 ] (right) for scene classification. Y-axis shows accuracy on the external benchmark while bars on x-axis are ordered by taxonomy’s predicted performance based on our dataset. A monotonically decreasing plot corresponds to preserving identical orders and perfect generalization.

Figure 13: Task Similarity Tree. Agglomerative clustering of tasks based on their transferring-out patterns (i.e. using columns of normalized affinity matrix as task features). 3D , 2D , low dimensional geometric , and semantic tasks clustered together using a fully computational approach.

Figure 1: Selected policies. For tasks in the table form Figure 10 of the main paper, we show the best policies for transferring to that task. The center node of each cluster is the target task, and the surrounding nodes at the tail of each arrow are the suggested source tasks.

Figure 2: Higher Order Distances. Calculated task distances for all 16k transfers. The columns are sorted by transfer order and the orders are shown along the x-axis. Each row is a target task.

Figure 3: Low data transfers. Expected loss after training transfer functions with 1k samples (shown before AHP normalization). This is a low-data version of the transfer loss matrix in the main paper.

Figure 4: Feature extraction. Performance of readout functions trained on MIT Places. We compare our networks to common baselines. The x-axis is ordered according to transfer performance on our dataset, and the y-axis shows Top-1 and Top-5 accuracy on MIT Places. The sources on the left were trained on our dataset, while the right portion contains external baselines.

Figure 5: Drop-k performance. Expected performance of transferring to a task after dropping

k

tasks from our dictionary. The grey bars show the standard deviation of the performance.

Figure 6: Transitivity. Loss after transitive transfer is on the y-axis and loss after direct ( $A\rightarrow C$ ) transfer is on the x-axis. The line x=y is shown. Red dots are transfers which transferred well to $C$ but not to $B$ in first-order. Blue dots transferred well to the target, $C$ , but not to $B$ . Green dots transferred well to both.

Figure 7: Model Specificity. (left) Affinity matrices. (right) Standard Deviation of Affinities.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

Sep 29, 2026

OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?

Training a model to generate visual content can encourage it to learn rich perceptual capabilities related to geometry, spatial relationships, and objectness; yet, its benefits for visual understanding remain unclear. We ask: when and how does visual generation supervision improve visual understanding? We study controlled pairs of image-to-image (I2I) generation and image-to-text (I2T) understanding tasks that express the same underlying problem in different output modalities. We find that under the correct recipe, I2I training improves downstream I2T performance, with larger gains as the amount of I2I training data increases. We next ask which generation tasks benefit which understanding capabilities. To study transfer beyond paired tasks, we introduce OmniTaskonomy, a unified taxonomy spanning 19 I2I generation tasks and 25 I2T understanding capabilities. The resulting transfer map reveals selective, task-dependent benefits. Some follow intuitive correspondences, e.g., depth prediction improving metric 3D reasoning, object pointing improving counting, and jigsaw reconstruction improving 2D ordering. Interestingly, we also uncover surprising connections: 2.5D segmentation improving category recognition and Z-depth prediction improving localization. To probe these patterns, we analyze gradient alignment between generation and understanding tasks and find that stronger alignment is associated with larger downstream transfer gains. Together, our results highlight visual generation as a rich source of supervision for visual understanding and provide a roadmap for unlocking its benefits through the right training curriculum and task selection. Project page: https://omni-taskonomy.github.io/.

Jiaxin Ge, Yiming Qin, Ji Xie +13

University of California, Berkeley · Duke University · Impossible Research +3

Image-to-Image Translation Visual Representation Learning Multi-Task Learning Transfer Learning

Ask PDF

May 22, 2026

VisAnalog: A Diagnostic Suite for Visual Concept Transfer on Natural Images

A useful test of visual concept learning is not just whether a model can recognize a concept in a single image, but whether it can preserve and manipulate concept-level properties under transformation and transfer them to new scenes. We introduce VisAnalog, a controlled suite for this setting on natural images. Each example instantiates

A\!:\!B::C\!:\,?

: images

B

and a hidden target image

D

are produced by applying the same deterministic transformation sequence to source images

A

and

C

. Given

A

B

, and

C

, a model must answer a multiple-choice question about

D

. The benchmark contains 617 human-validated questions spanning one- to four-step transformations such as zoom, quadrant swap, rotation, flip, and hue rotation. Across strong proprietary and open-source VLMs, end-to-end accuracy is substantially lower than oracle accuracy when

D

is directly shown, and degrades sharply as transformation depth increases, while human performance remains near the ceiling. A program-conditioned evaluation further separates failures of relation inference from failures of transformation application, showing that inferring the visual relation from

A \rightarrow B

is the dominant bottleneck, with additional application errors emerging on harder multi-step cases. The dataset is publicly available at https://huggingface.co/datasets/zli99/VisAnalog.

Zhaonan Li, Kyle R. Chickering, Bangzheng Li +13

Arizona State University · Luma AI · UC Davis

VLM Evaluation Analogical Reasoning Visual Reasoning Relational Reasoning

Ask PDF

May 19, 2026

Sample Complexity of Transfer Learning: An Optimal Transport Approach

Transfer learning is an essential technique for many machine learning/AI models of complex structures such as large language models and generative AI. The essence of transfer learning is to leverage knowledge from resolved source tasks for a new target task, especially when the sample size

m

of the training data for the latter is low. In this work, we rigorously analyze the potential benefit of transfer learning in terms of sample efficiency. Specifically, taking an optimal transport viewpoint of transfer learning, we find that when the data dimension

d

is higher than

3

, the sample complexity for transfer learning is

O(m^{-(α+1)/d})

, with

α

indicating the smoothness of the data distribution, as opposed to the

O(m^{-p/d})

sample complexity for direct learning with

p

indicating the smoothness of the optimal target model. Our finding theoretically supports a better sample efficiency for transfer learning, when the target task is optimizing over a family of not-so-smooth models (i.e., highly complex networks with the possible use of non-smooth activation functions). Using image classification as an example, we numerically demonstrate the sample efficiency for transfer learning, that is, in the data hungry regime, the model performance can be significantly improved by transfer learning.

Haoyang Cao, Xin Guo, Wenpin Tang +1

Department of Applied Mathematics and Statistics, John Hopkins University, Baltimore, MD, USA · Department of Industrial Engineering and Operations Research, University of California, Berkeley, CA, USA · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY, USA +1

Sample Complexity Transfer Learning Statistical Learning Theory Data-Efficient Learning

Ask PDF

Abstract

Transfer Learning

Explore similar work

OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?

VisAnalog: A Diagnostic Suite for Visual Concept Transfer on Natural Images

Sample Complexity of Transfer Learning: An Optimal Transport Approach