cs.CVApr 23, 2018
SaveTaskonomy: Disentangling Task Transfer Learning
Organizations: Stanford University · University of California, Berkeley
Abstract
Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.
Figures & tables
Figure 1: A sample task structure discovered by the computational task taxonomy ( taskonomy ) . It found that, for instance, by combining the learned features of a surface normal estimator and occlusion edge detector, good networks for reshading and point matching can be rapidly trained with little labeled data.
Figure 2: Computational modeling of task relations and creating the taxonomy. From left to right: I. Train task-specific networks. II. Train (first order and higher) transfer functions among tasks in a latent space. III. Get normalized transfer affinities using AHP (Analytic Hierarchy Process). IV. Find global transfer taxonomy using BIP (Binary Integer Program).
Figure 3: Task Dictionary. Outputs of 24 (of 26) task-specific networks for a query (top left). See results of applying frame-wise on a video here .
Figure 4: Transfer Function . A small readout function is trained to map representations of source task’s frozen encoder to target task’s labels. If order , transfer function receives representations from multiple sources.
Figure 5: Transfer results to normals (upper) and 2.5D Segmentation (lower) from 5 different source tasks. The spread in transferability among different sources is apparent, with reshading among top-performing ones in this case. Task-specific networks were trained on 60x more data. “Scratch” was trained from scratch without transfer learning.
Figure 6: Higher-Order Transfers. Representations can contain complementary information. E.g. by transferring simultaneously from 3D Edges and Curvature individual stairs were brought out. See our publicly available interactive transfer visualization page for more examples.
Figure 7: First-order task affinity matrix before (left) and after (right) Analytic Hierarchy Process (AHP) normalization. Lower means better transfered. For visualization, we use standard affinity-distance method (where and is element-wise matrix exponential). See supplementary material for the full matrix with higher-order transfers.
Figure 8: Computed taxonomies for solving 22 tasks given various supervision budgets (x-axes), and maximum allowed transfer orders (y-axes). One is magnified for better visibility. Nodes with incoming edges are target tasks, and the number of their incoming edges is the order of their chosen transfer function. Still transferring to some targets when tge budget is 26 (full budget) means certain transfers started performing better than their fully supervised task-specific counterpart. See the interactive solver website for color coding of the nodes by Gain and Quality metrics. Dimmed nodes are the source-only tasks, and thus, only participate in the taxonomy if found worthwhile by the BIP optimization to be one of the sources.
| Task | Task | Task | ||||||
|---|---|---|---|---|---|---|---|---|
| Denoising | 100 | 99.9 | Layout | 99.6 | 89.1 | Scene Class. | 97.0 | 93.4 |
| Autoenc. | 100 | 99.8 | 2D Edges | 100 | 99.9 | Occ. Edges | 100 | 95.4 |
| Reshading | 94.9 | 95.2 | Pose (fix) | 76.3 | 79.5 | Pose (nonfix) | 60.2 | 61.9 |
| Inpainting | 99.9 | - | 2D Segm. | 97.7 | 95.7 | 2.5D Segm. | 94.2 | 89.4 |
| Curvature | 78.7 | 93.4 | Matching | 86.8 | 84.6 | Egomotion | 67.5 | 72.3 |
| Normals | 99.4 | 99.5 | Vanishing | 99.5 | 96.4 | 2D Keypnt. | 99.8 | 99.4 |
Table 1: Task-Specific Networks’ Sanity: Win rates vs. random (Gaussian) network representation readout and statistically informed guess .
Figure 9: Evaluation of taxonomy computed for solving the full task dictionary . Gain (left) and Quality (right) values for each task using the policy suggested by the computed taxonomy, as the supervision budget increases( ). Shown for transfer orders 1 and 4.
Figure 10: Generalization to Novel Tasks . Each row shows a novel test task. Left: Gain and Quality values using the devised “all-for-one” transfer policies for novel tasks for orders 1-4. Right: Win rates (%) of the transfer policy over various self-supervised methods, ImageNet features, and scratch are shown in the colored rows. Note the large margin of win by taxonomy. The uncolored rows show corresponding loss values.
Figure 11: Structure Significance. Our taxonomy compared with random transfer policies (random feasible taxonomies that use the maximum allowable supervision budget). Y-axis shows Quality or Gain , and X-axis is the supervision budget. Green and gray represent our taxonomy and random connectivities, respectively. Error bars denote 5 th –95 th percentiles.
Figure 12: Evaluating the discovered structure on other datasets: ImageNet [ 78 ] (left) for object classification and MIT Places [ 157 ] (right) for scene classification. Y-axis shows accuracy on the external benchmark while bars on x-axis are ordered by taxonomy’s predicted performance based on our dataset. A monotonically decreasing plot corresponds to preserving identical orders and perfect generalization.
Figure 13: Task Similarity Tree. Agglomerative clustering of tasks based on their transferring-out patterns (i.e. using columns of normalized affinity matrix as task features). 3D , 2D , low dimensional geometric , and semantic tasks clustered together using a fully computational approach.
Figure 1: Selected policies. For tasks in the table form Figure 10 of the main paper, we show the best policies for transferring to that task. The center node of each cluster is the target task, and the surrounding nodes at the tail of each arrow are the suggested source tasks.
Figure 2: Higher Order Distances. Calculated task distances for all 16k transfers. The columns are sorted by transfer order and the orders are shown along the x-axis. Each row is a target task.
Figure 3: Low data transfers. Expected loss after training transfer functions with 1k samples (shown before AHP normalization). This is a low-data version of the transfer loss matrix in the main paper.
Figure 4: Feature extraction. Performance of readout functions trained on MIT Places. We compare our networks to common baselines. The x-axis is ordered according to transfer performance on our dataset, and the y-axis shows Top-1 and Top-5 accuracy on MIT Places. The sources on the left were trained on our dataset, while the right portion contains external baselines.
Figure 5: Drop-k performance. Expected performance of transferring to a task after dropping tasks from our dictionary. The grey bars show the standard deviation of the performance.
Figure 6: Transitivity. Loss after transitive transfer is on the y-axis and loss after direct ( ) transfer is on the x-axis. The line x=y is shown. Red dots are transfers which transferred well to but not to in first-order. Blue dots transferred well to the target, , but not to . Green dots transferred well to both.
Figure 7: Model Specificity. (left) Affinity matrices. (right) Standard Deviation of Affinities.
More in this topic
Transfer Learning
Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.
See all 417 papers