Transfer Learning and Catastrophic Forgetting: Reusing Model Knowledge Without Losing It

Transfer learning is the machine learning technique of taking knowledge a model acquired while learning one task and reusing it as the starting point for a different but related task. Instead of training a new model from a blank slate, you begin with a model that already understands general patterns — language structure, visual features, domain vocabulary — and adapt it with a comparatively small amount of additional training. The entire foundation model paradigm rests on this idea, and its main side effect, catastrophic forgetting, is what happens when the adaptation goes too far.

What transfer learning actually transfers

During training, a model doesn’t memorize its dataset — it learns internal representations: layers of features that capture progressively more abstract structure. An image model trained on product photos learns edges, textures, shapes, and object parts. A language model trained on news articles learns grammar, entity relationships, and how summaries compress meaning.

Those representations are not welded to the original task. The visual features a retailer’s model learned recognizing its own product photos are equally useful for spotting damaged packaging — edges, dents, and tears are made of the same low-level visual vocabulary. So rather than gathering a massive damaged-packaging dataset and training from zero, the retailer initializes a new model from the existing one’s learned features and trains briefly on a modest set of packaging examples. The transferred knowledge does most of the work; the new training only teaches the delta.

The same logic applies to language. A model trained to summarize news articles can be redirected to summarize internal meeting notes with only minor additional training, because the core skill — compressing long text into its key points — carries over. The task changed; the underlying competence transferred.

Why transfer learning underpins foundation models

A foundation model is transfer learning at industrial scale. The expensive pre-training phase — weeks of compute over internet-scale data — produces broad general knowledge exactly once. Every downstream adaptation (prompting, fine-tuning, domain specialization) is a transfer of that knowledge to a narrower purpose. This is why one pre-trained model can power a support chatbot, a document summarizer, and a code assistant: each application transfers the same base competence to a different task. For the fuller picture of how pre-training creates this reusable base, see what a foundation model is.

The practical benefits that exam scenarios highlight:

• Speed. Adapting a pre-trained model takes hours or days, not the weeks required to train from scratch.
• Data efficiency. The related task needs far less labeled data, since general competence is already in place.
• Cost. You pay for a small adaptation run instead of a full training run.
• Quality floor. The pre-trained representations are usually better than anything a small from-scratch dataset could produce.

	Training from scratch	Transfer learning
Starting point	Randomly initialized model	Model pre-trained on a related task
Data required	Very large task-specific dataset	Small-to-moderate task dataset
Compute and time	Weeks; full training cost	Hours to days; fraction of the cost
Typical result on niche tasks	Poor unless data is abundant	Strong, thanks to inherited representations
Key risk	Underfitting from insufficient data	Catastrophic forgetting of prior abilities

Catastrophic forgetting: the price of aggressive adaptation

Adaptation updates the model’s weights — and the weights being updated are the same ones storing the original knowledge. Fine-tune a foundation model heavily on a narrow task, such as a specialized legal-document workload, and the updates can overwrite the representations that supported everything else. The result: the model becomes excellent at legal documents while its general conversational ability, which it previously handled well, measurably degrades. That degradation of previously learned capabilities during new training is catastrophic forgetting.

The mechanics are intuitive once you picture weights as shared real estate. Pre-training distributed general knowledge across billions of parameters. Narrow fine-tuning keeps pushing those parameters toward whatever minimizes error on the narrow dataset, with nothing anchoring them to their old values. The longer and more aggressive the fine-tuning — high learning rates, many epochs, a dataset with no diversity — the more of the original capability gets bulldozed.

Mitigation strategies you should recognize at the AIF-C01 level:

• Gentler tuning: lower learning rates and fewer epochs limit how far weights drift.
• Mixed data: blending some general-domain examples into the fine-tuning set reminds the model of its old skills.
• Parameter-efficient fine-tuning (PEFT): techniques like LoRA (Low-Rank Adaptation) freeze the original weights and train small add-on parameters, structurally preventing the base knowledge from being overwritten.
• Evaluation on general benchmarks: measuring the tuned model on broad tasks, not just the target task, catches forgetting before deployment.

Note the distinction the exam draws: catastrophic forgetting is not overfitting. Overfitting is failing to generalize from training data to new data on the same task; forgetting is losing performance on previous tasks after training on a new one. A model can be perfectly fit to its legal corpus and still have forgotten how to chat.

How the AIF-C01 exam tests this

• Name the concept. A definitional question asks which term describes reusing knowledge from one task to learn a related task faster. The answer is transfer learning — distractors include incremental learning, federated learning, and reinforcement learning, none of which mean knowledge reuse across related tasks.
• Recognize it in a scenario. A model trained for one purpose (product recognition, news summarization) is reused with light additional training for a related purpose (damage detection, meeting-note summarization). The exam expects you to label this transfer learning even when the word never appears in the stem.
• Diagnose the failure mode. A model is fine-tuned intensively on a narrow domain and afterward performs worse on general tasks it used to handle. The risk being illustrated is catastrophic forgetting — not overfitting, not underfitting, not data drift.

Transfer learning shows up in both the ML-fundamentals and foundation-model domains — the full AIF-C01 study guide maps the overlap, and AI Practitioner practice exams will hit it from both angles.

Quick reference

• Transfer learning reuses representations learned on one task as the starting point for a related task.
• It cuts training time, compute cost, and labeled-data requirements versus training from scratch.
• Foundation models institutionalize transfer learning: pre-train broadly once, adapt cheaply many times.
• Fine-tuning is the most common transfer mechanism for foundation models.
• Catastrophic forgetting = losing previously learned general abilities after heavy narrow fine-tuning.
• Forgetting ≠ overfitting: forgetting harms old tasks; overfitting harms generalization on the new task.
• Mitigations: lower learning rates, mixed general/target training data, PEFT methods like LoRA, and broad post-tuning evaluation.