What is softmax and what does temperature do to it?medium
Answer
Softmax converts raw logits to a probability distribution: softmax(z_i) = e^z_i / Σe^z_j.
Explanation
Softmax converts raw logits to a probability distribution: softmax(z_i) = e^z_i / Σe^z_j. Temperature T scales logits before softmax: softmax(z_i/T). T<1 (e.g., 0.3): sharpens → more deterministic. T>1 (e.g., 1.5): flattens → more random/creative. T→0 → argmax (greedy). Use low temperature for extraction tasks (a document extraction pipeline), higher for creative generation.
Follow-upCan you give a production example?