Everything they can
reasonably ask, derived.

You cannot learn ML tonight. You can make what you already know retrievable under stress, and close the two or three gaps most likely to be probed.

Interviewers don't test linear algebra for its own sake. They test the four or five results that keep reappearing inside models — and they test whether you can connect the symbol to the thing.

Linear algebra: the five facts

1. A matrix is a function. y = Ax maps ℝⁿ → ℝᵐ. Rank = dimension of the output space actually reachable. If rank < n, the map destroys information and is not invertible — this is exactly why collinear features break ordinary least squares.

2. Eigen-decomposition. For symmetric A, A = QΛQᵀ with Q orthonormal. Eigenvectors are directions the matrix only stretches; eigenvalues are the stretch. Covariance matrices are symmetric PSD, so they always decompose this way — that is the whole of PCA.

3. SVD generalises this to any matrix: A = UΣVᵀ. Always exists. Truncating to the top-k singular values gives the best rank-k approximation in Frobenius norm (Eckart–Young). This single fact underlies PCA, LSA, recommender factorization, and LoRA.

4. Norms. L2 = √Σxᵢ², smooth, penalises large values quadratically. L1 = Σ|xᵢ|, non-differentiable at 0, and that kink is precisely why it produces exact zeros. L∞ = max|xᵢ|.

5. Gradients of the things you'll write on a whiteboard:

Probability: the parts that bite

• Expectation / variance: 𝔼[aX+b] = a𝔼[X]+b; Var(aX+b) = a²Var(X). Var(X) = 𝔼[X²] − 𝔼[X]².
• Independence vs uncorrelated: independent ⟹ uncorrelated; the converse fails (X ~ N(0,1), Y = X² has zero correlation, total dependence). Interviewers love this one.
• CLT: the mean of n i.i.d. samples → Normal with variance σ²/n, whatever the source distribution. This is why standard errors shrink as 1/√n, and why you need 4× the data to halve a confidence interval.
• Covariance matrix: Σ = 𝔼[(x−μ)(x−μ)ᵀ]. Diagonal = variances, off-diagonal = covariances. Always symmetric PSD.

Information theory (you need this for losses)

The core question of ML: why should performance on data you've seen say anything about data you haven't? Every regularizer, every validation scheme, every architectural prior is an answer to it.

The setup, stated properly

There's an unknown joint distribution P(x, y). You want the hypothesis f minimising risk, R(f) = 𝔼₍ₓ,ᵧ₎~P [L(f(x), y)]. You can't compute it, so you minimise empirical risk over your n samples, R̂(f) = (1/n)Σ L(f(xᵢ), yᵢ). Everything interesting is in the gap R(f) − R̂(f).

That gap grows with how expressive your hypothesis class is (it can chase noise) and shrinks with n. Formally VC dimension / Rademacher complexity bound it; practically, you control it with regularization, data, and inductive bias.

Bias–variance, derived

For squared error, with y = f(x) + ε, Var(ε) = σ², and f̂ trained on a random dataset:

Regularization: five distinct mechanisms

Validation done right

• k-fold CV: split into k folds, train on k−1, validate on 1, rotate, average. Lower-variance estimate than one holdout. k=5 or 10 by convention; k=n is leave-one-out (nearly unbiased, high variance, expensive).
• Stratified k-fold preserves class ratios per fold — mandatory for imbalanced data.
• Time-series: never shuffle. Use forward-chaining (train on [0,t], validate on [t,t+Δ], roll forward). Random CV on temporal data leaks the future and is the single most common résumé-project mistake.
• Grouped CV: if you have multiple rows per patient/user/session, all rows for a group must be in the same fold, or the model memorises the group.
• Nested CV: when you tune hyperparameters and want an unbiased performance estimate, you need an inner loop for tuning and an outer loop for evaluation. Reporting the best inner-loop score as your final number is optimistically biased — a subtle point that impresses.

Choosing a metric is choosing what kind of mistake you're willing to make. Interviewers use metric questions to check whether you think about the product, not just the model.

ROC-AUC vs PR-AUC — the distinction they're fishing for

ROC plots TPR against FPR as you sweep the threshold. AUC = probability that a randomly chosen positive is ranked above a randomly chosen negative. It's threshold-free and invariant to class balance.

That invariance is both its strength and its flaw. FPR has TN in the denominator, and with 99.9% negatives, TN is enormous — so hundreds of false positives barely move FPR. ROC-AUC can look excellent while your fraud alerts are 95% noise.

PR curve plots precision against recall. Precision has no TN term, so it feels every false positive. On heavily imbalanced problems where you care about the positive class, PR-AUC is the honest metric. Note the PR baseline is the positive rate (0.001), not 0.5.

Calibration — your home turf

A model is calibrated if, among all predictions with confidence 0.8, exactly 80% are correct. Accuracy and calibration are independent: a model can rank perfectly (AUC 1.0) and be catastrophically overconfident.

• Fixes: temperature scaling (single parameter T fitted on a held-out set, divides logits — preserves ranking so accuracy is unchanged), Platt scaling (logistic fit), isotonic regression (non-parametric, more flexible, needs more data, can overfit).
• Why it matters: anything that thresholds on confidence — abstention, routing, early-exit, self-consistency stopping, cascades — is only as good as the calibration underneath.
• Critical: the temperature must be fitted on data disjoint from both training and test. This is the leak-free requirement your own work is about; expect to be asked why it matters.

Each of these is a set of assumptions plus an optimisation. Know the assumptions and you can answer any follow-up; memorise the algorithm and you can't.

Linear regression

Model: ŷ = Xw. Loss: ‖Xw − y‖². Set the gradient 2Xᵀ(Xw − y) to zero:

Assumptions: linearity in parameters, independent errors, homoscedasticity (constant error variance), no perfect collinearity, and for inference, normally distributed errors. Complexity: O(np² + p³) closed-form — so use gradient descent when p is large.

Logistic regression — derive this tonight

It's a linear model of the log-odds:

Likelihood over the dataset, taking y ∈ {0,1}: ∏ pᵢ^yᵢ (1−pᵢ)^(1−yᵢ). Negative log-likelihood:

Derivation detail worth knowing: σ′(z) = σ(z)(1 − σ(z)). In the chain rule, ∂L/∂p · ∂p/∂z, the 1/(p(1−p)) from the log-loss cancels the p(1−p) from the sigmoid, leaving (p − y). That cancellation is why cross-entropy avoids the vanishing-gradient problem that MSE-plus-sigmoid has.

No closed form — the NLL is convex but transcendental, so you use gradient descent or Newton's method (IRLS). Convexity means one global optimum. On perfectly separable data the weights diverge to infinity to push σ toward exactly 0 or 1; regularization is what keeps it finite, and that's a great detail to volunteer.

Support vector machines

Idea: among all separating hyperplanes, pick the one with the largest margin, because a wide margin is a more robust decision boundary.

The dual replaces w with Σ αᵢ yᵢ xᵢ and the objective depends on data only through inner products xᵢᵀxⱼ. That's the opening for the kernel trick.

Support vectors are the points with αᵢ > 0 — only those on or inside the margin. The solution depends on nothing else, which makes SVMs sparse in examples (unlike Lasso, sparse in features) and robust to distant outliers. Scaling is mandatory — RBF's distance metric is meaningless with mismatched feature scales.

Cost: training is roughly O(n²)–O(n³), so SVMs don't scale past ~100k rows. That's the honest reason they lost to boosted trees and neural nets in practice.

Decision trees

Greedily choose the (feature, threshold) split that most reduces impurity, recurse, stop on a depth/leaf-size/gain criterion.

• Strengths: no scaling needed, handles mixed types, captures interactions automatically, interpretable, invariant to monotone feature transforms.
• Weaknesses: extremely high variance — change a few rows and the top split flips, restructuring the whole tree. Greedy and myopic (can't see a split that only pays off two levels down). Axis-aligned, so diagonal boundaries need staircases. Cannot extrapolate beyond the training range — a regression tree predicts a constant outside it.
• Bias in split selection: impurity-based importance favours high-cardinality and continuous features, because they offer more thresholds to try. Use permutation importance or SHAP instead when it matters.

k-NN and Naive Bayes

The most reliable performance gain in classical ML, and the most common source of "explain the difference" questions. Anchor everything to the bias–variance decomposition and you can't get lost.

Bagging attacks variance

Train M models on bootstrap resamples (sample n rows with replacement), average their predictions. If the models were independent with variance σ², the average has variance σ²/M. They aren't independent — they share data — so the real result is:

Random forest = bagging + a second decorrelation trick: at each split, consider only a random subset of m features (typically √p for classification, p/3 for regression). This stops every tree from choosing the same dominant feature at the root. Bias goes up slightly; ρ drops a lot; variance drops more.

Out-of-bag estimate: each bootstrap sample omits about (1 − 1/n)ⁿ → 1/e ≈ 36.8% of rows. Predict each row using only the trees that didn't see it — a free, nearly unbiased validation estimate with no separate holdout. Deriving the 36.8% is a favourite question.

Boosting attacks bias

Train models sequentially, each one correcting the previous ensemble's errors. Base learners are deliberately weak and high-bias (shallow trees, "stumps" or depth 3–6).

XGBoost / LightGBM / CatBoost — what each actually adds

• XGBoost: second-order approximation of the loss (uses gradient and Hessian, so the leaf weight is −G/(H+λ)), explicit regularization on leaf count and leaf weights in the objective, sparsity-aware split finding with a learned default direction for missing values, approximate quantile sketch for candidate thresholds, and column subsampling.
• LightGBM: histogram binning of features (O(#bins) instead of O(#rows) per split), leaf-wise growth instead of level-wise (grows the leaf with max loss reduction — faster convergence, more overfitting risk, control with num_leaves), GOSS sampling, and EFB feature bundling. Usually the fastest on large data.
• CatBoost: ordered target statistics for categorical features that avoid target leakage, and ordered boosting to correct the prediction-shift bias. Best out-of-the-box on categorical-heavy tabular data.

Less commonly asked than supervised learning, which is exactly why a confident PCA derivation or a clean EM explanation stands out.

k-means

Minimise within-cluster sum of squares: J = Σₖ Σ_{x∈Cₖ} ‖x − μₖ‖². Alternate two steps:

1. Assign each point to its nearest centroid (holding μ fixed, this is the optimal assignment).
2. Update each centroid to the mean of its points (holding assignments fixed, the mean provably minimises squared distance).

Each step is non-increasing in J and there are finitely many assignments, so it converges — but to a local minimum, depending on initialisation. Hence k-means++: pick the first centroid at random, then pick each subsequent one with probability proportional to D(x)² (squared distance to the nearest existing centroid), which spreads the seeds out and gives an O(log k) approximation guarantee in expectation.

• Assumes spherical, similar-sized, similar-density clusters — because Euclidean distance to a centroid defines a sphere. It will slice a crescent moon or two elongated clusters straight down the middle.
• Choosing k: elbow on the inertia curve (subjective), silhouette score (mean of (b−a)/max(a,b), ranges −1 to 1, higher is better), gap statistic, or a downstream task metric. Say the elbow is a heuristic, not a criterion.
• Complexity O(nkdi). Scale features first — otherwise the largest-range feature dominates the distance.

GMM and the EM algorithm

Model the data as a mixture: p(x) = Σₖ πₖ N(x | μₖ, Σₖ). Unlike k-means, assignments are soft (responsibilities), and clusters can be elliptical and differently sized because each has its own covariance.

PCA — derive it two ways

Setup: centre the data (subtract the mean — non-negotiable, or the first component just points at the mean). Compute Σ = (1/n)XᵀX.

Derivation 1 — maximum variance. Find the unit vector w maximising the variance of the projection, wᵀΣw subject to wᵀw = 1. Lagrangian: wᵀΣw − λ(wᵀw − 1). Differentiate, set to zero:

Derivation 2 — minimum reconstruction error. Find the k-dimensional subspace minimising Σ‖xᵢ − x̂ᵢ‖². Because total variance is fixed, minimising the residual is the same as maximising the retained variance. The two objectives are equivalent — say this, it shows real understanding.

In practice use the SVD of the centred X directly (X = UΣVᵀ, principal directions are columns of V, and forming XᵀX squares the condition number and loses precision).

• Components are orthogonal and ordered by variance. Standardise features first if units differ, otherwise PCA just finds the largest-variance unit.
• Limits: linear only; variance ≠ discriminative information (the class-separating direction can be a low-variance one — that's what LDA optimises instead); components are dense combinations, so interpretability suffers.

If they hand you a marker, this is what they hand it to you for. Be able to do the two-layer backprop with shapes, out loud, without hesitating.

Forward pass

Backprop, term by term

Why the softmax + cross-entropy gradient is just (ŷ − y): the Jacobian of softmax is ∂pᵢ/∂zⱼ = pᵢ(δᵢⱼ − pⱼ). Multiply by ∂L/∂pᵢ = −yᵢ/pᵢ and sum over i; the pᵢ cancel and, using Σy = 1, you're left with p − y. Libraries fuse these two ops for exactly this reason (numerical stability plus the clean gradient).

Activations — and why each was invented

Initialisation — the failure they'll ask about

Initialise all weights to zero and every neuron in a layer computes the same thing, receives the same gradient, and stays identical forever — symmetry is never broken and the layer has an effective width of 1. Initialise too large and activations explode; too small and the signal dies out over depth.

Optimisers, normalization, and the debugging story. The debugging question — "your loss isn't going down, what do you do" — is asked in nearly every ML interview and rewards a systematic answer.

Optimisers, built up in order

AdamW — the one to name for transformers. In Adam, an L2 penalty added to the loss gets divided by √v̂ along with everything else, so parameters with large historical gradients get less decay — the opposite of what regularization should do. AdamW decouples it, applying w ← w − ηλw directly, separate from the adaptive step. This measurably improves generalisation and is the default for every modern LLM.

Why SGD+momentum still wins sometimes: on vision tasks it often generalises slightly better than Adam. The common explanation is that Adam's per-parameter scaling steers toward sharper minima; SGD's isotropic noise biases toward flatter ones, which generalise better.

Learning rate — the single most important hyperparameter

• Warmup: start near zero and ramp up over the first few thousand steps. Essential for transformers — at initialisation, Adam's variance estimates are unreliable and large early steps destabilise LayerNorm statistics and attention patterns permanently.
• Cosine decay from the peak to ~10% of it is the standard LLM schedule. Also: step decay, linear decay, one-cycle.
• LR range test: exponentially increase the LR over a few hundred steps and plot loss; pick roughly an order of magnitude below where it diverges.
• Batch size interaction: larger batches give lower-variance gradients, permitting a larger LR. Linear scaling rule: multiply batch by k, multiply LR by k (with warmup). The √k rule is the alternative and often more stable for Adam.

Normalization

Why transformers use LayerNorm, not BatchNorm: sequences have variable length and the batch statistics of a padded token position are meaningless; BatchNorm makes each example's output depend on the others in its batch, which breaks autoregressive inference (batch size 1 at serving time); and NLP batch statistics are far noisier than vision's. LayerNorm needs no running statistics and behaves identically at train and test time.

Pre-LN vs Post-LN: the original transformer put LayerNorm after the residual add (Post-LN) and needed careful warmup to train at all. Pre-LN — x + Attn(LN(x)) — keeps a clean identity path through the whole network and trains stably without warmup. Every modern LLM uses Pre-LN. RMSNorm drops the mean subtraction (empirically the re-centring wasn't doing the work; the re-scaling was) for a ~10-15% speedup; used in LLaMA and most current models.

BatchNorm's train/test asymmetry is a classic bug source: at train time it uses batch statistics, at test time the running averages. Forget model.eval() and your inference silently changes. Also: BN's regularizing effect comes from the noise in batch statistics, which is why very small batches hurt it badly (use GroupNorm below batch size ~8).

Less central than they were, but the reasoning — inductive bias, parameter sharing, receptive field — is what transformer questions build on.

Convolutions

Two inductive biases: locality (nearby pixels are related, so a small kernel suffices) and translation equivariance (shift the input, the feature map shifts identically). Pooling adds approximate translation invariance. These priors are why CNNs beat MLPs on images with far less data — and why ViTs need much more data or heavy augmentation to match them, since they must learn the priors instead of assuming them.

• Receptive field grows linearly with depth for stacked 3×3 convs (layer L sees 2L+1 pixels), and exponentially with dilation. Two stacked 3×3 convs have the same receptive field as one 5×5 but use 18C² instead of 25C² parameters and insert an extra nonlinearity — the VGG argument.
• 1×1 convolutions don't mix spatially; they mix channels and change channel depth cheaply. That's the bottleneck block in ResNet and the core of Inception.
• Depthwise separable (MobileNet): one spatial conv per channel, then a 1×1 to mix — cuts cost by roughly 1/k² + 1/C_out.
• ResNet: y = x + F(x). Solves degradation (deeper plain nets did worse even on training error, so it wasn't overfitting) by making the identity easy to represent — a block only has to learn the residual. The gradient consequence is in §07.

RNNs, LSTMs, GRUs

RNN: h_t = tanh(W_hh h_{t−1} + W_xh x_t). Backprop through time unrolls it, and the gradient across k steps carries W_hhᵏ — so if the largest eigenvalue is <1 gradients vanish, >1 they explode. In practice an RNN can't learn dependencies beyond ~10 steps.

Why transformers replaced them: the recurrence is inherently sequential — you cannot compute h_t before h_{t−1}, so training can't be parallelised over the time axis. Attention computes all positions simultaneously, giving O(1) sequential operations instead of O(n), and a constant path length between any two tokens instead of O(n). You trade O(n²) compute for parallelism and short gradient paths, and on modern hardware that trade is overwhelmingly worth it.

Given your résumé, assume this is where the interview spends its time. Know the shapes, the complexity, and the reasons behind every design choice.

Scaled dot-product attention

Why divide by √d_k — the question that separates memorisers from understanders. If q and k have i.i.d. components with mean 0 and variance 1, their dot product Σᵢ qᵢkᵢ is a sum of d_k such terms, so it has variance d_k and typical magnitude √d_k. Feed logits of that scale into softmax and it saturates — the output approaches a one-hot vector, and softmax's gradient p(1−p) goes to zero. Dividing by √d_k returns the logits to unit variance, keeping the softmax in its responsive region and gradients alive.

The rest of the block

• FFN: FFN(x) = W₂ · act(W₁x + b₁) + b₂, with hidden dimension 4× d_model by convention. Applied position-wise and identically at every position. It holds roughly ⅔ of the parameters — and mechanistic interpretability work suggests it acts as a key-value memory storing factual associations.
• Residual + norm: Pre-LN, as in §08: x = x + Attn(LN(x)), then x = x + FFN(LN(x)). The residual stream is the model's working memory that every block reads from and writes to.
• Causal mask: set the upper triangle of the score matrix to −∞ before softmax, so position i can't see j > i. This is what makes training parallel and autoregressive — every position's next-token prediction is computed in one forward pass without leaking the future.

Positional information

Attention is a permutation-equivariant set operation — shuffle the tokens and you get the same outputs, shuffled. Position must be injected explicitly.

Complexity, and how it's attacked

• FlashAttention — the important one. It doesn't change the math at all; it changes the memory access pattern. Attention is memory-bandwidth bound, not compute-bound: writing the n×n score matrix to HBM and reading it back dominates. FlashAttention tiles the computation, keeps blocks in SRAM, and uses the online-softmax trick to compute the exact result without ever materialising the full matrix. Result: exact attention, linear memory in n, 2–4× faster.
• Sparse / sliding-window (Longformer, Mistral): each token attends to a local window plus a few global tokens. Stacking layers still propagates information globally, like a receptive field.
• Linear attention: replace softmax with a kernel feature map so you can reassociate (QKᵀ)V → Q(KᵀV) and get O(n·d²). Cheaper, generally worse quality. Related family: state-space models (Mamba), which recover recurrence with a parallel scan.
• GQA / MQA: share K and V heads across multiple Q heads. Barely affects quality and cuts the KV cache by the sharing factor — the dominant memory cost at inference. Nearly universal now.

The pipeline end to end, plus the parts your own research touches. Expect the follow-ups here to go deeper than anywhere else in the interview.

The training stages

DPO removes the reward model and the RL loop entirely. The insight: for the KL-constrained RLHF objective the optimal policy has a closed form, which can be inverted to express the implicit reward in terms of the policy itself. Substituting into the Bradley-Terry loss gives a plain supervised objective:

Parameter-efficient fine-tuning

• Scaling factor α/r multiplies the update; keeping α/r constant lets you change r without re-tuning the learning rate.
• QLoRA: base weights frozen in 4-bit NF4, LoRA adapters in bf16, plus double quantization and paged optimizers. Puts a 70B fine-tune on a single 48GB GPU.
• Which modules: originally just Q and V projections; current practice targets all linear layers (Q,K,V,O and the FFN), which is usually worth the extra parameters.
• Limits: LoRA adapts style, format and task behaviour well; it's a poor tool for injecting large amounts of new knowledge — full fine-tuning or retrieval is better for that.

Decoding and test-time compute

• Greedy — argmax each step. Deterministic, repetitive, and myopic: locally optimal tokens can lead to globally poor sequences.
• Beam search — keep k partial hypotheses ranked by cumulative log-prob, with length normalization. Good for translation and summarisation where there's a single right answer; produces bland, degenerate text in open-ended generation.
• Temperature — divide logits by T before softmax. T<1 sharpens, T>1 flattens, T→0 is greedy. Note this is the same operation as calibration temperature scaling, used for a different purpose.
• Top-k — sample from the k most likely tokens. Top-p / nucleus — sample from the smallest set whose cumulative probability exceeds p; adaptive, so it uses a wide set when the model is uncertain and a narrow one when it's confident. Usually better than top-k for that reason. Min-p scales the cutoff by the top token's probability.
• Self-consistency — sample n chains of thought, take the majority answer. Reliably improves accuracy on reasoning benchmarks at n× the cost.
• Best-of-n / verifier reranking — sample n, score with a reward or process model, keep the best. Process reward models score intermediate steps rather than only the final answer, giving denser signal.

Serving and efficiency

• Quantization: INT8/INT4 weights. Post-training (GPTQ uses second-order info to minimise layer-wise error; AWQ protects salient channels identified by activation magnitude) vs quantization-aware training (better, costlier). Weight-only quantization helps most because decode is memory-bound. Outlier activation channels are the main difficulty — hence per-channel scaling and mixed precision.
• Distillation: train a student on the teacher's soft output distribution. The "dark knowledge" is in the relative probabilities of the wrong classes — that a 7 looks somewhat like a 1 — which carries far more information per example than a one-hot label. Use temperature T on both sides and scale the gradient by T². When the teacher is API-only and gives no logits, you fall back to sequence-level distillation on sampled outputs, optionally with rationales.
• Speculative decoding: a small draft model proposes k tokens; the large model verifies them in one forward pass; a rejection-sampling rule guarantees the output distribution is identical to sampling from the large model alone. Pure latency win with no quality cost.
• Batching: continuous/in-flight batching (vLLM, TGI) plus PagedAttention, which stores the KV cache in non-contiguous blocks like OS virtual memory to eliminate fragmentation.
• MoE: replace the FFN with N experts and route each token to the top-k (usually 2). Parameter count scales without proportional FLOPs. Requires a load-balancing auxiliary loss or a few experts get all the traffic; and all experts must be in memory even though only k are active, so it trades memory for compute.

RAG and agents

RAG: embed the corpus, retrieve top-k chunks by vector similarity for a query, put them in the prompt. Use it when knowledge is dynamic, private, verifiable, or too large to fit; fine-tune instead when you need a behaviour, format, or style. The honest failure modes: retrieval quality caps everything (garbage retrieved, garbage generated); chunking destroys context across boundaries; pure dense retrieval misses exact terms, so hybrid BM25 + dense with a cross-encoder reranker is the strong default; and "lost in the middle" — models attend poorly to content in the centre of a long context, so put the highest-ranked chunk first or last.

Agents: an LLM in a loop with tools, memory, and a termination condition. ReAct interleaves reasoning traces with tool calls. Practical concerns you should raise unprompted: error compounding over steps (95% per-step reliability is 60% over 10 steps), the need for schema-constrained output to make tool calls parseable, deterministic guardrails around irreversible actions rather than trusting the model, and evaluation being genuinely hard — trajectory-level judging, not just final-answer accuracy.

The round where they find out whether you've shipped anything. The scoring rubric is almost always: did the candidate clarify requirements before designing, and did they mention data before models?

The framework — use it verbatim

1. Clarify (2–3 min, always). What's the business objective, and what proxy metric represents it? What's the scale — QPS, users, items? Latency budget? Is it online or batch? What data exists already? Any fairness or regulatory constraints?
2. Frame as an ML problem. What's the input, what's the output, what's a training example, and where does the label come from? Is ML even necessary — would a heuristic get 80% of the value?
3. Data. Sources, volume, labelling strategy, the train/val/test split (and why it's split that way), class balance, leakage risks.
4. Features. Include a baseline. Say which are computed offline vs at request time, and how you'd avoid train/serve skew.
5. Model. Start with the simplest thing that works — logistic regression or gradient boosting — and justify complexity only against a measured gap.
6. Evaluation. Offline metric, online metric, and the guardrail metric you'd refuse to regress.
7. Serving. Latency, batching, caching, precomputation, model size.
8. Monitoring. Drift, staleness, retraining trigger, rollback plan.

Imbalanced data

Order of preference: (1) change the metric and threshold — often nothing else is needed; (2) class weights in the loss — cheap, principled, no data distortion; (3) resampling — undersample the majority if you have plenty of data, oversample or SMOTE the minority if you don't (SMOTE interpolates between minority neighbours; it's unreliable in high dimensions and can create points inside the majority region); (4) reframe as anomaly detection if the positive rate is extreme (<0.1%). Critical detail: resample only the training fold, never the validation or test set, or your metrics describe a distribution that doesn't exist.

Drift and monitoring

• Data/covariate drift: P(x) changes, P(y|x) doesn't. Detect with PSI, KS test, or KL between reference and live feature distributions. Often survivable.
• Concept drift: P(y|x) changes — the relationship itself moved. Requires retraining. Detectable only once labels arrive, which is why proxy metrics matter.
• Label lag: if ground truth takes 30 days, monitor prediction distribution and feature drift as leading indicators.
• Training–serving skew: the top cause of "great offline, bad online". Same feature code path for both, or a feature store, ideally with logging of the exact features used at inference.

The implementations most often asked for on a shared screen. Read them until the shapes feel obvious — then close the page and write one from memory.

Numerically stable softmax + cross-entropy

# The max subtraction is the whole point of the question.
def softmax(z):                          # z: (N, C)
    z = z - z.max(axis=1, keepdims=True)   # shift-invariant: same result, no overflow
    e = np.exp(z)
    return e / e.sum(axis=1, keepdims=True)

def cross_entropy(logits, y):              # y: (N,) integer labels
    p = softmax(logits)
    N = logits.shape[0]
    return -np.log(p[np.arange(N), y] + 1e-12).mean()

def ce_grad(logits, y):                    # dL/dlogits — the (p - y)/N result
    p = softmax(logits); N = logits.shape[0]
    p[np.arange(N), y] -= 1.0
    return p / N

Linear regression by gradient descent

def linreg(X, y, lr=0.01, epochs=1000, l2=0.0):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(epochs):
        pred = X @ w + b
        err  = pred - y                       # (n,)
        dw   = (X.T @ err) / n + l2 * w       # (d,)  ridge term
        db   = err.mean()
        w -= lr * dw; b -= lr * db
    return w, b

# Closed form, for comparison: w = np.linalg.solve(X.T@X + l2*np.eye(d), X.T@y)
# Use solve(), never inv() — better conditioned and ~2x faster.

Logistic regression

def sigmoid(z):
    return np.where(z >= 0, 1/(1+np.exp(-z)),
                          np.exp(z)/(1+np.exp(z)))   # stable both tails

def logreg(X, y, lr=0.1, epochs=2000, l2=0.0):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(epochs):
        p  = sigmoid(X @ w + b)
        g  = p - y                            # identical form to linreg — GLM
        w -= lr * ((X.T @ g)/n + l2*w)
        b -= lr * g.mean()
    return w, b

k-means with k-means++ init

def kmeans(X, k, iters=100, seed=0):
    rng = np.random.default_rng(seed)
    C = [X[rng.integers(len(X))]]
    for _ in range(k-1):                            # k-means++ seeding
        d2 = ((X[:,None,:] - np.array(C)[None,:,:])**2).sum(-1).min(1)
        C.append(X[rng.choice(len(X), p=d2/d2.sum())])
    C = np.array(C)
    for _ in range(iters):
        d2  = ((X[:,None,:] - C[None,:,:])**2).sum(-1)   # (n, k)
        lab = d2.argmin(1)
        newC = np.array([X[lab==j].mean(0) if (lab==j).any() else C[j]
                         for j in range(k)])
        if np.allclose(newC, C): break             # converged
        C = newC
    return C, lab

Two-layer network, full forward + backward

def train_step(X, y, W1, b1, W2, b2, lr):
    N = X.shape[0]
    # ---- forward
    z1 = X @ W1 + b1;  a1 = np.maximum(0, z1)      # ReLU
    z2 = a1 @ W2 + b2
    loss = cross_entropy(z2, y)
    # ---- backward
    d2 = ce_grad(z2, y)                             # (N, C)
    dW2 = a1.T @ d2;      db2 = d2.sum(0)           # (h,C), (C,)
    d1 = (d2 @ W2.T) * (z1 > 0)                     # ReLU' is the mask
    dW1 = X.T @ d1;       db1 = d1.sum(0)           # (d,h), (h,)
    # ---- update
    for p, g in ((W1,dW1),(b1,db1),(W2,dW2),(b2,db2)):
        p -= lr * g
    return loss

Self-attention with causal mask

def attention(X, Wq, Wk, Wv, causal=True):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv          # each (n, d_k)
    d_k = Q.shape[-1]
    S = Q @ K.T / np.sqrt(d_k)                # (n, n) — the sqrt matters
    if causal:
        n = S.shape[0]
        S = np.where(np.tril(np.ones((n,n))) == 0, -1e9, S)
    A = softmax(S)                            # rows sum to 1
    return A @ V, A                           # (n, d_v)

def multi_head(X, Wq, Wk, Wv, Wo, h):      # Wq: (d, d)
    n, d = X.shape; dk = d // h
    Q = (X@Wq).reshape(n,h,dk).transpose(1,0,2)   # (h, n, dk)
    K = (X@Wk).reshape(n,h,dk).transpose(1,0,2)
    V = (X@Wv).reshape(n,h,dk).transpose(1,0,2)
    S = Q @ K.transpose(0,2,1) / np.sqrt(dk)      # (h, n, n)
    S = np.where(np.tril(np.ones((n,n))) == 0, -1e9, S)
    A = np.exp(S - S.max(-1, keepdims=True))
    A = A / A.sum(-1, keepdims=True)
    O = (A @ V).transpose(1,0,2).reshape(n, d)    # concat heads
    return O @ Wo

LayerNorm and RMSNorm

def layer_norm(x, g, b, eps=1e-5):         # normalise the FEATURE axis
    mu  = x.mean(-1, keepdims=True)
    var = x.var(-1, keepdims=True)
    return g * (x - mu) / np.sqrt(var + eps) + b

def rms_norm(x, g, eps=1e-6):             # no mean subtraction, no bias
    return g * x / np.sqrt((x**2).mean(-1, keepdims=True) + eps)

ROC-AUC without sklearn (the rank trick)

def auc(y, s):
    # AUC = P(score of a random positive > score of a random negative)
    order = np.argsort(s)
    ranks = np.empty(len(s)); ranks[order] = np.arange(1, len(s)+1)
    n_pos, n_neg = y.sum(), len(y) - y.sum()
    return (ranks[y==1].sum() - n_pos*(n_pos+1)/2) / (n_pos*n_neg)
# This is the Mann-Whitney U statistic. Handles ties poorly — use
# average ranks (scipy.stats.rankdata) if ties are common.

Best-of-n with a confidence threshold

# Adaptive test-time compute: stop early when calibrated confidence clears tau.
def adaptive_sample(model, prompt, n_max=16, tau=0.9, T=1.0):
    answers, budget = [], 0
    for i in range(n_max):
        out, logits = model.generate(prompt)
        conf = softmax(logits / T).max()        # T fitted on a HELD-OUT split
        answers.append(out); budget += 1
        agree = sum(a == out for a in answers) / len(answers)
        if i >= 2 and conf > tau and agree > tau:
            break                               # the stopping rule — only as good
    return majority(answers), budget            # as the calibration underneath

Short questions, short answers. Read the question, answer in your head, then open. Anything you fumble goes on the revisit list.

Fundamentals

Deep learning

LLMs

Highest-leverage section on the page. Everything above is shared with every other candidate. This part isn't — and interviewers spend more time here than on any single technical topic.

The 90-second version of the calibration paper

Structure it as problem → why it's hard → what you did → what you found → what's still open. Something like:

Then stop. Let them ask. Talking for four minutes unprompted is the most common self-inflicted wound in a research interview.

The questions they will actually ask

The Samsung project — have these numbers ready

The interviewer wants your decisions, not the org chart. Lead with the design choice and the reason for it.

• Why a knowledge graph rather than dumping state into the prompt? Implicit commands ("it's too dark in here") need resolution against device capabilities and current state. The two-layer split — a static ontology of what devices can do, plus a dynamic layer of what's currently true — keeps the slow-changing schema separate from fast-changing state, and lets you query rather than re-serialise everything into context each turn. Have the KG-vs-JSON ablation ready as evidence.
• Why four modules instead of one call? Device → function → condition → NLG separates concerns that fail differently, so you can evaluate and fix them independently, and each sub-task gets a constrained output schema, which is what makes the output parseable at all.
• The safety gates — grounding, range/type, precondition, impact tier, self-consistency confidence — are deterministic, not model-judged. That's the defensible design position: never let a probabilistic system be the last check before an irreversible action. The 0% false-HIGH-impact rate across the 38-case suite is the number to quote.
• Distillation under a black-box teacher. No logits from an API model, so no classical soft-target KD — you're doing sequence-level distillation on sampled outputs plus rationales. Knowing exactly why the standard method doesn't apply is a strong signal.
• The benchmark: 5 models × 6 categories × 3 context modes, ~4,050 LLM calls, 1,800 scored comparisons, with Gemini as both benchmark and judge. Volunteer the flaw before they find it — using the same model as both a compared system and the judge is a bias you should name, along with how you'd control for it (swap judges, human-validate a subsample, position-swap).
• Be straight about scope: the SFT and GRPO scripts were delivered but not run by the final review, so there are no student checkpoints. Say that plainly if asked. Overstating it is the one thing that can actually cost you the offer.

Your other projects, in one line each

Read this one in the morning. Nothing new here — just the things that are easy to forget while nervous.

Ask them something real

• How do you decide when a model is good enough to ship — what's the bar, and who sets it?
• What does the evaluation setup look like day to day? Offline suites, human review, online?
• Where does the team spend most of its time — data, modelling, or infrastructure?
• What's a recent result that surprised the team, or changed your approach?
• If I joined, what would the first project be?

Questions about evaluation and data always land well, because they signal you've worked on real systems rather than notebooks.