ModelRefs / Transfer Learning — Tutorial
Transfer Learning — Tutorial
Use models pretrained on millions of images — then fine-tune them for your task in minutes. Covers Why transfer learning works.
Overview
Use models pretrained on millions of images — then fine-tune them for your task in minutes
Level: Advanced. Estimated reading time: 35 minutes.
Why transfer learning works
A model trained on ImageNet (1.2M images, 1000 classes) learns to detect edges, textures, shapes, and object parts in its convolutional layers. These visual concepts are universally useful — whether you're classifying medical images, satellite photos, or product photos.
The key insight: early layers learn general features (edges, colours), late layers learn task-specific features (dog breeds, car models). When you transfer to a new task, replace only the final classification head while keeping the general feature extractor.
Transfer learning advantages: reach competitive accuracy with 100× less data, train in minutes instead of days, require far less compute. A ResNet50 trained from scratch on 10k images might hit 60% accuracy. The same architecture fine-tuned from ImageNet weights typically hits 85%+.
Feature extraction vs fine-tuning
Two transfer learning strategies:
Feature extraction (frozen base): freeze all pretrained weights, add a new head, train only the head. Fast — only the head's parameters are updated. Best when your dataset is small (<1k images) or very similar to the pretraining domain.
Fine-tuning: first train the head (frozen base), then unfreeze all or part of the base and continue training at a low learning rate (1e-5 to 1e-4). The low LR prevents catastrophic forgetting — overwriting the pretrained representations.
Common recipe: 1. Load pretrained model, replace classification head 2. Freeze base, train for 5–10 epochs at lr=1e-3 3. Unfreeze top N layers (or all), train at lr=1e-5 for 10–20 more epochs 4. Use cosine decay and early stopping
The dividing line for how many layers to unfreeze: the more dissimilar your domain from ImageNet (medical scans vs natural photos), the more layers you should fine-tune.
Beyond vision: transfer learning in NLP
The same principle applies to language. BERT and its variants are pretrained on billions of tokens of text, learning general language representations. Fine-tuning on a small labelled dataset (few thousand examples) typically matches or beats training from scratch on millions.
For text classification: load bert-base-uncased, replace the [CLS] head with a linear layer of size (768, num_classes), fine-tune at lr=2e-5 for 3–5 epochs. This is the standard recipe that works across most NLP tasks.
For image-text tasks: CLIP (OpenAI) learns joint image-text embeddings pretrained on 400M image-caption pairs. Fine-tuning CLIP on small domain-specific datasets gives strong zero-shot and few-shot performance on novel visual categories.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Transfer Learning — Tutorial.