§15.2Transfer Learning and Domain Adaptation

§15.2 Learning in one setting, generalizing in another

Transfer learning transfer learning Initialize a network with the first k layers of a net trained on a related task/dataset, then jointly fine-tune on the target task (often with fewer examples); §15.2. defined in ch. 8 — open in glossary and domain adaptation domain adaptation Transfer where the TASK stays the same but the input distribution shifts (sentiment trained on book reviews, deployed on electronics reviews); denoising-autoencoder pretraining has worked well here. defined in ch. 15 — open in glossary name the situation where what was learned in one setting — a distribution $P_1$ — is exploited to improve generalization in another, $P_2$. The previous section was already an instance (unsupervised task → supervised task, same inputs); here the idea goes general:

The gradual case has its own name: concept drift concept drift Gradual change in the data distribution over time — a form of transfer learning, and like it, a special case of multi-task learning. defined in ch. 15 — open in glossary , where the distribution shifts slowly over time — viewable as a form of multi-task learning across successive time steps. In all cases the core representation-learning bet is the same: the same representation may be useful in both settings, letting it benefit from both settings’ training data. The 2011 transfer competitions made the payoff concrete: participants learned a feature space purely unsupervised from $P_1$, then a linear classifier was trained on very few labeled $P_2$ examples — and the deeper the unsupervised representation, the better the transfer learning curve, with fewer labels needed to reach the apparent asymptote.

Which layers to share? Usually the lower ones — most visual tasks share edges and shapes. But sometimes what is shared is the semantics of the output: a speech recognizer must emit valid sentences no matter who speaks, while its early layers must cope with wildly different renditions of the same phonemes per speaker. Then share the upper layers and give each task (or speaker) its own preprocessing:

One label, or none at all

The extremes of transfer are one-shot learning one-shot learning Learning a transfer task from a SINGLE labeled example — possible when the learned representation already separates the underlying classes cleanly, so one label tags an entire cluster in representation space. defined in ch. 15 — open in glossary (a single labeled example of the new task) and zero-shot learning zero-shot learning Learning with NO labeled examples of the target task, by modeling p(y | x, T) with a task description T represented so that generalization is possible (learned embeddings, not one-hot task codes) — read that cats have four legs and pointy ears, then recognize one. defined in ch. 15 — open in glossary (none). One-shot works because the representation learned in stage one already separates the classes: the lone label names a cluster, and test examples clustering around the same point inherit it. Zero-shot sounds impossible until the task itself gets a representation:

The same three-ingredient recipe powers multi-modal learning multi-modal learning Learning a representation for each of two modalities (fx, fy) plus the relationship between the two representation spaces, so concepts anchor across modalities and the model generalizes to never-seen pairs — the mechanism behind fig 15.3's zero-shot transfer. defined in ch. 15 — open in glossary and embedding-based translation: learn a representation for each modality (or language), and learn the relationship between the two spaces from paired examples. Concepts anchored in one space transfer through the map to the other — even for pairs never observed together:

Next: the hypothesis underneath all of this — the best representations disentangle the causes of the data — semi-supervised disentangling of causal factors.