What is LoRA fine-tuning and how does it reduce memory cost?

What is LoRA fine-tuning and how does it reduce memory cost?hard

Type: conceptual
Topic: is-lora-fine-tuning-and-how-does-it-reduce-memory-cost
Frequency: common
Tags: llms, what, lora, fine, tuning, and

Answer

LoRA freezes original model weights and adds small trainable rank-decomposition matrices (A, B where r << d) to attention layers: W' = W + BA.

Explanation

LoRA freezes original model weights and adds small trainable rank-decomposition matrices (A, B where r << d) to attention layers: W' = W + BA. Only A and B are trained — ~0.1% of parameters. Memory savings: no optimizer states for full weights. LoRA weights can be merged at inference (no latency cost). QLoRA adds 4-bit quantization for further memory reduction.

Follow-upCan you give a production example?