Pretraining vs fine-tuning vs prompt-based learning — differences?hard

Type
conceptual
Topic
pretraining-vs-fine-tuning-vs-prompt-based-learning-differ
Frequency
common
Tags
deep-learning, pretraining, fine, tuning, prompt, based
Answer

Pretraining: train on massive corpus to learn general representations (expensive, done once).

Explanation

Pretraining: train on massive corpus to learn general representations (expensive, done once). Fine-tuning: update weights on task-specific labeled data — strong performance, needs labels. Prompt-based: craft inputs to steer a frozen LLM — zero/few-shot, no gradient updates. In production: prompt engineering first (fast, cheap), fine-tune if quality insufficient, pretrain only if domain is radically out-of-distribution.

Follow-upWhen would you choose one approach over the other?