NLP

NLP is best understood as a progression of representation learning: sparse hand-built features (one-hot, bag-of-words, TF-IDF) gave way to dense static embeddings (Word2Vec), then contextual embeddings (ELMo), then attention and self-attention, then transformers — the architecture that BERT, GPT, T5, and every modern LLM is built on. Each step solved a concrete limitation of the one before it.

Tokenization: splitting raw text into meaningful units — words, subwords (BPE, WordPiece), or characters. Subword tokenization handles out-of-vocabulary words by breaking them into known pieces.

From sparse to dense: one-hot vectors and TF-IDF treat every pair of different words as equally unrelated — king and queen look as distant as king and apple. Embeddings fix this by mapping words to dense vectors where nearby points share meaning, so relations like king − man + woman ≈ queen emerge from training rather than being hand-coded.

Word2Vec: learns static embeddings by predicting a word from its context (CBOW) or context from a word (skip-gram). It captures semantic similarity well, but gives each word exactly one vector — so bank has the same embedding in "river bank" and "bank account." That inability to handle polysemy is its core limitation.

Contextual embeddings (ELMo): instead of one fixed vector per word, ELMo runs a bidirectional LSTM over the whole sentence so a word's embedding depends on what surrounds it. "Bank" gets a different vector depending on context — the first real step from static to contextual meaning.

Attention: early encoder-decoder models compressed an entire input sequence into one fixed-size vector, which lost information on long sequences. Attention removed that bottleneck by letting the decoder dynamically weight every input token. Modern attention formalizes this with three vectors per token — Query, Key, and Value — where Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V: queries are compared against keys to get relevance scores, and values are combined using those scores as weights.

Self-attention: applies attention within a single sequence — every token compares itself against every other token to build a context-aware representation. In "The cat drank the milk because it was hungry," self-attention lets it attend strongly to cat, updating its representation with that context.

Multi-head attention: runs several self-attention operations in parallel, each with its own learned Q/K/V projections. One head might track subject-verb relationships, another pronoun reference, another semantic similarity — the outputs are concatenated into one richer representation.

Why self-attention beats RNNs: RNNs pass information step by step, so a signal from token 1 has to survive many hidden-state updates to influence token 100 — this is the vanishing-gradient problem, and it also forces strictly sequential (slow) training. Self-attention lets any two tokens interact directly regardless of distance, and every token can be processed in parallel, which is why transformers scale so much better with data and compute.

The transformer block: stacks multi-head self-attention, a position-wise feed-forward network, residual connections, and layer normalization. Attention decides which tokens should exchange information; the feed-forward layer then transforms each token's representation independently. Residuals and normalization are what make it possible to stack dozens of these blocks without training collapsing.

Positional encoding: self-attention has no built-in sense of order — without it, "dog bites man" and "man bites dog" look identical. Positional encoding adds order information to each token's embedding, via fixed sinusoidal signals (original Transformer), learned position embeddings, or rotary position embeddings (RoPE), which most modern LLMs use.

Encoder-only vs decoder-only vs encoder-decoder: encoder-only models (BERT) use bidirectional attention and excel at understanding tasks — classification, NER, embeddings. Decoder-only models (GPT, LLaMA) use causal (masked) self-attention, where each token only attends to previous tokens, which is what enables left-to-right generation. Encoder-decoder models (T5, BART) combine both for sequence-to-sequence tasks like translation and summarization.

Classic NLP tasks: POS tagging, named entity recognition (NER), dependency parsing, coreference resolution, sentiment analysis, text classification, machine translation, and summarization.

Evaluation: BLEU (translation), ROUGE (summarization), F1 for NER/classification, perplexity for language model quality, and human evaluation for open-ended generation.

From transformers to LLMs: transformers scale unusually well — parallel training lets them absorb far more data and compute than RNNs ever could. Pretraining a decoder-only transformer on next-token prediction over massive corpora, then applying fine-tuning and instruction tuning, is how a simple next-word objective turns into the general-purpose language ability behind modern LLMs.

Language is the primary medium of human knowledge. Every email, support ticket, research paper, contract, and social media post is unstructured text. NLP transforms this into structured signals for search, classification, extraction, and generation. Modern LLMs are NLP systems at scale — understanding the embeddings → attention → transformer progression explains why prompting works, where models fail, and how to design applications around language model capabilities and limitations.

NLP questions test breadth and depth: text preprocessing decisions, representation tradeoffs (sparse vs static vs contextual embeddings), and evaluation methodology. Mid-to-senior roles lean heavily on architecture: why RNNs struggle with long text, what Query/Key/Value actually compute, why self-attention parallelizes better than recurrence, what positional encoding solves, and when to reach for an encoder-only, decoder-only, or encoder-decoder model. Senior roles add handling multilingual text, domain adaptation of pretrained models, latency vs quality tradeoffs in production NLP, and knowing when a fine-tuned smaller model beats a general LLM for constrained classification.