L1 vs L2 regularization — when do you use each?medium
Answer
L1 (Lasso) adds sum of |weights| — drives some weights to exactly zero, giving sparse models.
Explanation
L1 (Lasso) adds sum of |weights| — drives some weights to exactly zero, giving sparse models. Good for feature selection. L2 (Ridge) adds sum of weights² — shrinks all weights small but nonzero. Better when all features contribute. ElasticNet combines both. In LLM training, L2-like weight decay is standard.
Follow-upWhen would you choose one approach over the other?