Glossary

Every term the book defines — 510 of them — with the chapter that introduced it. These are the same definitions behind the dotted-underline tooltips throughout the site.

510 shown

absolute value rectification ch. 6: A ReLU generalization with slope α = −1 for z < 0, giving g(z) = |z|; used for polarity-invariant image features.
activation function ch. 6: The fixed element-wise nonlinearity g applied after an affine map: h = g(Wᵀx + b); its choice mostly distinguishes hidden-unit types.
active path ch. 16: A path whose intermediate variables leave its endpoints dependent; blocked/'inactive' paths do not. Observing a variable can block a path — or, at a collider, unblock it.
adagrad ch. 8: Scale each parameter's learning rate ∝ 1/√(sum of its historical squared gradients), so steep directions slow fast and gentle ones stay quick; strong in convex settings, but the full-history accumulation can shrink the rate too early for deep nets.
adaline ch. 1: Adaptive linear element: predicts real values with f(x); trained by a special case of stochastic gradient descent.
adam ch. 8: "Adaptive moments": momentum (a bias-corrected 1st-moment estimate of the gradient) + RMSProp (a bias-corrected 2nd-moment estimate). The bias corrections undo the moments' initialization at 0; fairly robust (defaults ρ₁=0.9, ρ₂=0.999, ε=0.001).
adaptive learning rate ch. 8: Use a separate, automatically-adjusted learning rate per parameter (assuming axis-aligned sensitivity); AdaGrad, RMSProp and Adam are examples. Delta-bar-delta was an early full-batch heuristic.
adversarial example ch. 7: An input x′ = x + ε·sign(∇_x J), imperceptibly close to x, that a high-accuracy network misclassifies (panda→gibbon, fig 7.8) — caused by excessive local linearity.
adversarial training ch. 7: Training on adversarially perturbed examples to enforce a local-constancy prior, which reduces error on the clean i.i.d. test set too.
ancestral sampling ch. 16: Draw a sample from a directed model by sampling the variables in topological (parents-first) order from their local conditionals; fast and exact, but directed-only.
annealed importance sampling (ais) ch. 18: Estimate Z₁/Z₀ by bridging p₀ to p₁ with intermediate distributions p_η ∝ p₁^η p₀^{1−η} and chaining their partition-function ratios; the standard method for RBM/DBN partition functions.
approximate inference ch. 19: Techniques for the intractable posterior p(h|v) (or expectations under it) needed for learning; usually cast as approximately maximizing a lower bound on log p(v).
artificial intelligence ch. 1: The field of building software that solves tasks requiring intelligence.
asynchronous sgd ch. 12: Lock-free parallel SGD: several cores read shared parameters, compute a gradient, and update without locks. Some updates overwrite each other, but the higher step rate speeds learning overall. Scaled across machines via a parameter server (Dean 2012).
attention (mechanism) ch. 10: A mechanism that produces a soft, weighted focus over a set of elements (a context sequence, or memory cells). Its addressing form is identical to soft memory addressing; it fixes the fixed-context bottleneck of encoder-decoder models (§12.4.5).
auto-regressive network ch. 20: A directed model with no latents that factorizes p(x)=Πᵢ p(xᵢ|x₁,…,xᵢ₋₁) via the chain rule, each conditional a neural net; e.g. FVBN, NADE.
autoencoder ch. 1: An encoder plus decoder trained to preserve information while giving the representation useful properties.
back-propagation ch. 6: (Rumelhart 1986) the method that flows information from the cost backward to compute the gradient — a table-filling (dynamic-programming) application of the chain rule; NOT the whole learning algorithm. (First named ch01.)
back-propagation through time (bptt) ch. 10: Ordinary back-propagation applied to the unfolded computational graph of an RNN. Cost and memory are O(τ) and the forward/backward passes are inherently sequential (not parallelizable across time).
bagging (bootstrap aggregating) ch. 7: Train k models on datasets bootstrap-resampled with replacement from the training set and average their votes; reduces variance because the models make partly independent errors.
batch / deterministic gradient method ch. 8: An optimizer that uses the ENTIRE training set for each update (full-batch). Note the clash: "batch gradient descent" means full-set, but "batch size" separately means the minibatch size.
batch normalization ch. 8: An adaptive REPARAMETRIZATION (not an optimizer): replace a layer's minibatch activations H with (H−µ)/σ (back-propagating THROUGH µ,σ), then γĤ+β to keep expressiveness. Removes cross-layer update coordination; test-time uses running averages.
bayes error ch. 5: The error an oracle knowing the true p(x,y) still makes — the irreducible noise floor.
bayes rule ch. 3: P(x|y) = P(x)P(y|x)/P(y), with P(y) = Σ_x P(y|x)P(x) — invert a conditional using the prior.
bayesian inference ch. 5: Treat θ as a random variable: prior p(θ) → posterior via Bayes; predict by integrating over ALL θ — uncertainty preserved, overfitting resisted, costly for large m.
bayesian network ch. 16: A directed graphical model (belief network): a DAG plus local conditionals p(xᵢ | Pa(xᵢ)); factorizes p(x)=Πᵢ p(xᵢ | Pa(xᵢ)). Best when causality flows one way.
bayesian probability ch. 3: Probability as a degree of belief (0 = certainly false, 1 = certainly true); obeys the same axioms as frequentist probability.
bernoulli distribution ch. 3: Single binary variable with P(x=1) = φ; mean φ, variance φ(1−φ).
bfgs / l-bfgs ch. 8: Quasi-Newton: iteratively approximate H⁻¹ with a matrix Mₜ refined by low-rank updates (BFGS, O(n²) memory), tolerant of loose line searches; L-BFGS starts M as the identity each step and stores only O(n) recent vectors — the practical variant for large models.
bias-variance trade-off ch. 5: MSE = Bias² + Variance: capacity ↑ typically bias ↓ variance ↑ — the U-curve again; negotiated by cross-validation.
biased importance sampling ch. 17: An importance-sampling estimator using unnormalized p̃,q̃ (needs no normalizers); biased but asymptotically unbiased as n→∞.
bidirectional rnn ch. 10: An RNN combining a forward sub-RNN (state h) and a backward sub-RNN (state g), so the output o^(t) depends on the whole input sequence — both past and future — useful for speech and handwriting.
block coordinate descent ch. 8: Coordinate descent over a subset (block) of variables at once — e.g. alternating between the dictionary W and the codes H in sparse coding, each subproblem convex while the other block is fixed.
block gibbs sampling ch. 16: Gibbs sampling that resamples a whole conditionally-independent group at once (all h given v, then all v given h in an RBM), exploiting the model's factorial conditionals.
boltzmann distribution ch. 16: Any distribution of the form p̃(x)=exp(−E(x)); models of this form are often called Boltzmann machines.
boltzmann machine ch. 16: An energy-based model, today usually one WITH latent variables (an EBM without latents is more often called a Markov random field / log-linear model).
boosting ch. 7: An ensemble method that INCREASES capacity by incrementally adding members (models, or even hidden units to one net) — the opposite intent to baggings variance reduction.
bridge sampling ch. 18: Estimate Z₁/Z₀ using a single bridge distribution p_* overlapping both p₀ and p₁ (optimal p_* ∝ p̃₀p̃₁/(r p̃₀+p̃₁)) instead of AIS's chain of intermediates.
broadcasting ch. 2: DL shorthand: C = A + b adds vector b to every row of A — implicit copying.
burn-in ch. 17: The initial run of a Markov chain, discarded, needed to reach the equilibrium distribution before its samples are usable.
calculus of variations ch. 19: Optimizing over a space of FUNCTIONS (rather than vectors); the origin of the 'variational' in variational inference. Used to derive the functional form of q.
capacity ch. 5: A model's ability to fit a wide variety of functions (hypothesis space size/identity); best matched to task complexity and data amount; generalization error is U-shaped in it.
cascade (of classifiers) ch. 12: A sequence of classifiers for detecting rare objects: cheap high-recall early stages reject easy negatives, and an expensive high-precision final stage confirms — high-confidence detection without paying full cost per example (e.g. Viola-Jones face detector).
catastrophic forgetting ch. 6: When a network forgets tasks it was previously trained on; maxout's filter redundancy helps resist it.
cell state (self-loop) ch. 10: The LSTM’s internal memory s^(t), updated by s^(t)=f^(t)s^(t-1)+g^(t)·(input). Its linear, gated self-loop is the path along which gradients can flow across many time steps.
centered deep boltzmann machine ch. 20: A DBM reparametrized by subtracting a fixed µ from every state, E(x)=−(x−µ)ᵀU(x−µ)−(x−µ)ᵀb, which better-conditions the Hessian so the model can be trained jointly without greedy layer-wise pretraining.
centered difference ch. 11: A more accurate finite-difference derivative estimate, (f(x+ε/2)−f(x−ε/2))/ε, with O(ε²) truncation error instead of O(ε) for the one-sided forward difference.
central limit theorem ch. 3: Sums of many independent random variables are approximately normally distributed.
chain rule of calculus ch. 6: dz/dx = (dz/dy)(dy/dx); vector form ∇_x z = (∂y/∂x)ᵀ∇_y z — backprop performs one Jacobian-gradient product per operation. (NOT the chain rule of probability.)
chain rule of probability ch. 3: Any joint factors into one-variable conditionals: P(x⁽¹⁾…x⁽ⁿ⁾) = P(x⁽¹⁾) Π P(x⁽ⁱ⁾|x⁽¹⁾…x⁽ⁱ⁻¹⁾).
cliff / exploding gradient ch. 8: A very steep region of the loss (from multiplying several large weights) where a single gradient step catapults the parameters far, discarding progress; common in RNNs (one factor per time step).
clique potential ch. 16: φ(C): a non-negative factor over a clique C measuring the affinity of its variables for each joint state; the building block of an undirected model's unnormalized p̃(x).
cold-start problem ch. 12: The difficulty a collaborative-filtering recommender faces with a brand-new user or item: with no rating history, similarity to others cant be estimated. Addressed by content-based recommenders that use user/item features.
collaborative filtering ch. 12: Recommending items by exploiting similarity between users (or items) preference patterns — if users share tastes on some items, ones preferences predict the others. Often uses learned user/item embeddings with a bilinear score.
complex cell ch. 9: A V1 cell responding like a simple cell but invariant to small position shifts — the inspiration for pooling units; ≈ the L² norm of a quadrature pair of simple cells.
complex-step differentiation ch. 11: A finite-difference-free numerical derivative using complex inputs: f(x) ≈ imag(f(x+iε))/ε with O(ε²) error and NO cancellation, so ε can be as tiny as 1e-150. Requires the function to accept complex numbers.
computational graph ch. 6: A DAG whose nodes are variables (any type) and whose edges are operations (simple single-output functions) — the precise language for backprop.
concept drift ch. 15: Gradual change in the data distribution over time — a form of transfer learning, and like it, a special case of multi-task learning.
condition number ch. 4: How sensitive a function's output is to input perturbation; for a matrix, maxᵢⱼ|λᵢ/λⱼ| — large means error amplification, an intrinsic property.
conditional computation ch. 12: Dynamic structure inside a network: computing only the subset of features (hidden units) an input actually needs, so rarely-relevant components are skipped for speed.
conditional independence ch. 3: x⊥y|z: the joint over x,y factorizes given every value of z.
conditional probability ch. 3: P(y|x) = P(y,x)/P(x), defined when P(x) > 0; NOT the same as computing an intervention's consequences.
conditional rnn (context) ch. 10: An RNN whose generation is conditioned on an extra input: a fixed vector x can enter as an extra per-step input (weight matrix R), as the initial state h^(0), or both (e.g. image captioning), or the condition can itself be a sequence.
conjugate directions ch. 8: Two directions dₜ, dₜ₋₁ are conjugate if dₜᵀHdₜ₋₁ = 0 (H-orthogonal); descending a conjugate direction does not undo the progress made along earlier directions.
conjugate gradients ch. 8: Avoid inverting H by descending H-conjugate directions dₜ = ∇J + βₜdₜ₋₁ (βₜ via Fletcher–Reeves or Polak–Ribière), eliminating steepest descent's zig-zag; ≤ k line searches minimize a k-dim quadratic. Nonlinear CG adds occasional resets.
connectionism ch. 1: The second wave (1980s–90s): intelligent behavior from many networked simple computational units.
connectionist temporal classification (ctc) ch. 12: A framework for training a sequence model (e.g. a deep LSTM RNN) end-to-end without a pre-specified frame-to-label alignment — key to removing the HMM from speech recognition (Graves 2006/2013).
consistency ch. 5: plim θ̂m = θ as m → ∞: the estimate converges to truth; implies asymptotic unbiasedness but not conversely.
constrained optimization ch. 4: Optimizing f over x within a feasible set S rather than all x — by projection, re-parametrization, or the KKT machinery.
content-based addressing ch. 10: Choosing a memory cell by how well its (vector-valued) content matches a query pattern — like recalling a song from a few of its words. Contrasts with location-based addressing (‘read slot 347’).
context vector ch. 10: The fixed-length summary C that an encoder emits from an input sequence and passes to a decoder. When too small to capture a long input, it motivates the attention mechanism.
context-specific independence ch. 16: An independence that holds only for certain values of some variable; it cannot be expressed by graph structure alone.
contextual bandit ch. 12: A reinforcement-learning special case where a learner, given a context, takes a single action and observes only that actions reward (not the others) — the model for online recommendation/advertising. Requires trading off exploration and exploitation.
continuation method ch. 8: Solve a sequence of increasingly hard objectives J⁽⁰⁾…J⁽ⁿ⁾ (e.g. progressively less-blurred J⁽ⁱ⁾ = E_{θ'~N}J(θ')), each solution initializing the next, to reach a good region of the true cost J.
contractive autoencoder (cae) ch. 14: An autoencoder penalizing its encoder's Jacobian, Ω(h) = λ‖∂f(x)/∂x‖²_F, so input neighborhoods contract; the few directions with singular values above 1 become the learned manifold tangents. Weights of f and g must be tied to block the ε-scaling cheat.
contrastive divergence (cd) ch. 18: CD-k: approximate the negative phase by initializing the Gibbs chains from DATA each step and running k steps; cheap but biased, and fails to suppress spurious modes.
convex optimization ch. 4: Optimization restricted to functions with positive semidefinite Hessian everywhere: no saddles, every local minimum global; in DL mostly a subroutine/analysis tool.
convolution ch. 9: A specialized linear operation, s(t)=∫x(a)w(t−a)da = (x∗w)(t); a CNN uses it in place of general matrix multiplication in ≥1 layer. Discrete 2-D: S(i,j)=Σₘₙ I(m,n)K(i−m,j−n).
coordinate descent ch. 8: Minimize with respect to one variable (or a BLOCK — block coordinate descent) at a time, cycling; effective when the variable groups are separable (e.g. sparse coding W vs H, each subproblem convex), poor when they interact strongly.
covariance ch. 3: E[(f(x)−E f)(g(y)−E g)]: linear co-variation and scale; correlation normalizes the scale away.
covariance matrix ch. 3: For random vector x ∈ ℝⁿ: Cov(x)_{i,j} = Cov(xᵢ, xⱼ); variances on the diagonal.
coverage ch. 11: The fraction of examples for which a system chooses to produce a response (rather than refuse/defer to a human). Traded against accuracy: refusing every example gives 100% accuracy at 0% coverage. Street View targets 98% accuracy at ≥95% coverage.
critical point ch. 4: (Stationary point) where the derivative/gradient is zero: local minimum, local maximum, or saddle point.
cross-correlation ch. 9: Convolution WITHOUT flipping the kernel: S(i,j)=Σₘₙ I(i+m,j+n)K(m,n). Most ML libraries implement this but call it convolution — the learned kernel just comes out flipped.
cross-entropy ch. 3: H(P,Q) = H(P) + D_KL(P‖Q) = −E_{x∼P} log Q(x); minimizing over Q ≡ minimizing the KL.
cross-entropy cost ch. 6: J = −E_{x,y∼p̂} log p_model(y|x): the default neural-net cost from maximum likelihood; its log undoes the exp of saturating output units.
cross-validation ch. 5: k-fold: split data into k parts, train on k−1, test on 1, rotate — all data contributes to the error estimate (algorithm 5.1).
curriculum learning ch. 8: A continuation method that presents easy examples first and harder ones later (a STOCHASTIC curriculum mixing both, with the hard fraction rising, works best); mirrors how humans and animals are taught.
curse of dimensionality ch. 5: Configurations grow exponentially with dimensions: typical grid cells contain NO training example, defeating purely local generalization.
curvature ch. 4: What second derivatives measure: whether a gradient step under/over-delivers vs the linear prediction (negative: better; positive: worse, can overshoot).
cybernetics ch. 1: The first wave of neural-network research (1940s–60s): linear models of biological learning.
d-separation ch. 16: The directed-graph analogue of separation ('d' = dependence); the active-path rules are more subtle (chain, common cause, collider, descendant).
damping ch. 19: A heuristic block update for mean field: solve each ĥᵢ optimally, then move all of them a small step in that direction (not guaranteed to increase L, but works in practice).
data parallelism ch. 12: Distributing computation by sending different examples to different machines. Trivial for inference (each example is independent); for training it means larger minibatches, with less-than-linear returns.
dataset augmentation ch. 7: Enlarging the training set with label-preserving transforms of inputs (translate/rotate/scale images) — never class-changing ones (b↔d, 6↔9). Input noise counts too; controlled benchmarks must fix the scheme.
decision tree ch. 5: Axis-aligned recursive splits of input space, one output per leaf region; struggles with diagonal boundaries.
deep belief network (dbn) ch. 20: A hybrid generative model: the top two layers form an undirected RBM, all lower layers are directed downward; trained greedily layer-wise. Started the 2006 deep-learning renaissance.
deep boltzmann machine (dbm) ch. 20: A fully undirected generative model with several hidden layers (no intra-layer edges); bipartite even/odd structure → two-block Gibbs and proper mean-field inference with top-down feedback.
deep learning ch. 1: Machine learning via representations expressed in terms of simpler representations — a deep concept hierarchy.
deep rnn ch. 10: An RNN with added depth inside one or more of its three transformation blocks (input→hidden, hidden→hidden, hidden→output). Depth adds capacity but lengthens the shortest time-step-to-time-step path, often mitigated by skip connections.
default baseline model ch. 11: The first sensible end-to-end system to build for a task, chosen by data structure: MLP for fixed vectors, CNN for grids/images, gated RNN (LSTM/GRU) for sequences — with ReLUs, SGD+momentum or Adam, early stopping, and dropout.
denoising autoencoder (dae) ch. 14: An autoencoder fed a corrupted x̃ ~ C(x̃|x) and trained to reconstruct the CLEAN x — it must undo corruption rather than copy. With Gaussian corruption and squared error, its reconstruction-minus-input field g(f(x)) − x estimates the score ∇ₓ log p(x).
denoising score matching ch. 18: Score-matching a noise-smoothed data distribution p_smoothed = ∫ p_data(y) q(x|y) dy; equivalent to several autoencoder training objectives (§14.5.1).
depth ch. 6: The length of the composed-function chain — the source of the name 'deep learning'. (First introduced ch01.)
design matrix ch. 2: X ∈ ℝ^{m×n} stacking one example per row: X_{i,:} = (x⁽ⁱ⁾)ᵀ.
detector stage ch. 9: The nonlinearity stage (e.g. ReLU) of a conv layer, between the convolution (affine) stage and the pooling stage; emulates V1 simple cells.
determinant ch. 2: det(A) = product of eigenvalues; |det| measures how much A expands or contracts volume.
diagonal matrix ch. 2: Nonzero entries only on the main diagonal; diag(v)x = v⊙x — multiply and invert cheaply.
differentiable generator network ch. 20: A neural net g(z;θ) that maps latent samples z to data samples x (or to p(x|z)); the shared core of VAEs, GANs, and standalone generators.
directed model ch. 3: Graphical model with directed edges: p(x) = Π p(xᵢ | Pa(xᵢ)) — factors are conditionals given parents.
discrete convolution ch. 9: Convolution over integer indices, s(t)=Σₐ x(a)w(t−a); implemented as a finite sum since the kernel and input are zero outside a finite set of points.
discriminator ch. 20: The GAN network d(x) that estimates the probability x is a real training example rather than a generator sample; its adversary is the generator.
disentangled representation ch. 15: A representation whose separate features or directions correspond to SEPARATE underlying causal factors of the data (one direction = gender, another = wearing glasses) — the chapter's answer to what makes one representation better than another.
distributed representation ch. 1: Each input represented by many features, each feature involved in many possible inputs.
domain adaptation ch. 15: Transfer where the TASK stays the same but the input distribution shifts (sentiment trained on book reviews, deployed on electronics reviews); denoising-autoencoder pretraining has worked well here.
dot product ch. 2: xᵀy — matrix product of two same-length vectors; equals ‖x‖‖y‖cos θ.
double backprop ch. 7: Regularizing the Jacobian ∇ₓf to be small in ALL input directions; adversarial training is its non-infinitesimal counterpart (as dataset augmentation is to tangent prop).
doubly block circulant matrix ch. 9: The (sparse) matrix corresponding to 2-D convolution when convolution is written as matrix multiplication.
dropconnect ch. 7: A dropout variant (Wan 2013) that drops individual weight×state products (each scalar weight) rather than whole units.
dropout ch. 7: A cheap, powerful regularizer that approximates BAGGING over the exponentially many sub-networks formed by multiplying units by a random binary mask µ (keep-prob ≈0.8 input, 0.5 hidden). Training minimizes E_µ J(θ,µ); the sub-networks share parameters.
dropout mask ch. 7: The per-example binary vector µ (one entry per input/hidden unit, each sampled independently with its keep-probability) that selects which units are kept in that sub-network; multiplying a unit by 0 removes it.
e-step ch. 19: The expectation step of EM: set q(h|v) = p(h|v; θ⁰) using the current parameters (exact inference), making the ELBO tight.
early stopping ch. 7: Return the parameters from the training step with the lowest VALIDATION error (stop after 'patience' worsening checks). The most common DL regularizer — simple, unobtrusive, and equivalent to L² under a quadratic model with τ ≈ 1/(εα).
echo state network (esn) ch. 10: A recurrent network whose input and recurrent weights are fixed (not learned) so the hidden units form a rich reservoir of temporal features; only the output weights are trained, often via a convex criterion.
effective capacity (of training time) ch. 7: The product of training steps τ and learning rate ε bounds the volume of parameter space reachable from the init θ₀, so it behaves like model capacity — and 1/(τε) plays the role of a weight-decay coefficient.
eigendecomposition ch. 2: A = V diag(λ)V⁻¹; real symmetric matrices give A = QΛQᵀ with orthogonal Q — scaling space by λᵢ along eigendirection i.
eigenvalue ch. 2: The scale factor λ paired with an eigenvector.
eigenvector ch. 2: Nonzero v with Av = λv — multiplication by A only rescales it.
embedding ch. 14: A representation of a data point on (or near) a manifold: a low-dimensional vector with fewer dimensions than the ambient input space. Non-parametric manifold methods learn one per example; encoders map any point to its embedding.
empirical distribution ch. 3: p̂(x) = (1/m) Σ δ(x − x⁽ⁱ⁾): mass 1/m on each training point — what training samples from; maximizes training likelihood.
empirical risk minimization ch. 8: Minimizing the training-set average loss (1/m)Σ L(f(x⁽ⁱ⁾;θ),y⁽ⁱ⁾) as a stand-in for the true risk E_{p_data}[L]; prone to overfitting and blocked when the true loss (e.g. 0-1) has useless gradients — so deep learning rarely uses it directly.
encoder-decoder (sequence-to-sequence) ch. 10: An architecture where an encoder RNN reads the input to a context C (typically its final state) and a decoder RNN generates the output conditioned on C. The input and output lengths may differ — the key advance for machine translation.
energy barrier ch. 17: A region of low probability / high energy separating two modes; transitions across a high barrier are exponentially unlikely, causing slow mixing.
energy function ch. 16: E(x): the exponent in an EBM, p̃(x)=exp(−E(x)); its additive terms correspond to cliques/factors, so an EBM is a product of experts.
energy-based model ch. 16: An undirected model written p̃(x)=exp(−E(x)), guaranteeing p̃>0 and allowing unconstrained learning of the energy function E.
equivariance to translation ch. 9: f(g(x)) = g(f(x)): shift the input and the feature map shifts identically. Convolution has this (from parameter sharing) but is NOT equivariant to scale or rotation.
estimator bias ch. 5: E[θ̂] − θ: systematic deviation. Sample mean: unbiased; sample variance: biased (fix: divide by m−1).
euler-lagrange equation ch. 19: The general functional-derivative condition allowing the functional to depend on f and its derivatives; the full form of the optimality condition for functionals.
evidence lower bound (elbo) ch. 19: L(v,θ,q) = log p(v) − D_KL(q(h|v)‖p(h|v)) = E_{h∼q}[log p(h,v)] + H(q): a tractable lower bound on log p(v), tight iff q = p(h|v). Inference = maximize it over q.
expectation ch. 3: E_{x∼P}[f(x)] = Σ P(x)f(x) (or integral): the average of f under P; linear.
expectation maximization (em) ch. 19: Coordinate ascent on the ELBO: E-step sets q to the exact posterior at the current θ; M-step maximizes L over θ with q fixed. Learning with an approximate posterior, not approximate inference.
explaining away ch. 16: At a V-structure (a, b both parents of c), observing c (or a descendant) makes the two causes DEPENDENT: learning one cause explains the effect and lowers the other's probability.
explicit constraint & reprojection ch. 7: Take an SGD step then PROJECT θ back onto Ω(θ) < k. Avoids the dead-unit local minima and runaway-weight feedback that soft penalties can cause; used for Hinton column-norm constraints.
explicit memory ch. 10: An addressable external memory (a set of cells) coupled to a task network that learns to read from and write to specific addresses — giving the model a working memory for facts, which plain RNNs store poorly.
exploration vs exploitation ch. 11: The tradeoff in model-based hyperparameter search between trying uncertain settings that might be much better (exploration) and trying settings the model is confident are good (exploitation).
f-score ch. 11: A single-number summary of precision p and recall r: F = 2pr/(p+r) (the harmonic mean). High only when BOTH precision and recall are high; used when a rare-event detector must be summarized by one metric.
factor analysis ch. 13: A linear factor model with prior h~N(0,I) and DIAGONAL (per-variable) Gaussian noise ψ=diag(σ²), so x~N(b, WWᵀ+ψ). The latents capture dependencies between the observed variables.
factor graph ch. 16: A bipartite drawing of an undirected model with explicit square factor nodes joined to their variable (circle) nodes, resolving the ambiguity of which cliques carry which factors.
factorial prior ch. 13: A prior over latent factors that fully factorizes, p(h)=∏ᵢ p(hᵢ), so the factors are independent and easy to sample — the standard assumption in linear factor models.
factors of variation ch. 1: The separate sources of influence that explain the observed data, often unobserved.
fantasy particles ch. 18: The model's own samples used in the negative phase ('hallucinations'): points the model believes in strongly, whose probability the negative phase pushes down.
fast dropout ch. 7: An analytical approximation to the dropout marginalization (Wang & Manning) that removes the training stochasticity, giving faster convergence.
fast pcd (fpcd) ch. 18: Split θ = θ_slow + θ_fast, training the fast copy with a large learning rate (and weight decay) to force the negative-phase chain to mix rapidly during learning.
feature ch. 1: One piece of information included in the representation of the data given to a learner.
feature map ch. 9: The output of a convolution — a map of where a feature (kernel response) appears across the input.
feedforward network ch. 6: (MLP) the quintessential deep model: y = f(x;θ) as a composed chain f³∘f²∘f¹ with NO feedback; approximates a target f*.
fit-a-tiny-dataset test ch. 11: A debugging test: if a model cannot even overfit a very small dataset (e.g. a single example), the problem is a software defect blocking optimization, not genuine underfitting.
fixed-point equations ch. 19: Iterated updates solving ∂L/∂ĥᵢ = 0 one variable at a time to find the mean-field q quickly in a learning inner loop; the mean-field update is itself a recurrent inference network.
forget gate ch. 10: The LSTM sigmoid unit f^(t) that sets the weight of the cell’s self-loop between 0 and 1 — deciding how much of the previous cell state to keep. Empirically the most crucial LSTM gate (bias it toward 1).
forward mode accumulation ch. 6: Automatic differentiation multiplying Jacobians from the input forward — cheaper than backprop when #outputs > #inputs.
forward propagation ch. 6: The forward flow x → hidden units → ŷ → scalar cost J through a network.
free energy ch. 16: F(x) = −log Σₕ exp(−E(x,h)): the energy of x after marginalizing the latent variables h in an EBM.
frequentist probability ch. 3: Probability as long-run event rates over repeatable experiments.
frobenius norm ch. 2: The matrix analogue of L²: ‖A‖_F = √Σ A²_{i,j} = √Tr(AAᵀ).
full convolution ch. 9: Enough zero-padding that every input pixel is visited k times in each direction; output grows to m+k−1 (border outputs depend on fewer inputs).
fully-visible belief network (fvbn) ch. 20: An auto-regressive network whose conditionals are simple (log-linear) models; the simplest auto-regressive generative model.
functional ch. 6: A mapping from functions to real numbers; viewing the cost as a functional lets calculus of variations show which statistic each loss recovers.
functional derivative ch. 19: δJ/δf(x): the variational derivative of a functional w.r.t. the value of f at a point x; set to zero at every x to optimize a functional (generalizes ∇J = 0).
gabor function ch. 9: A Gaussian gate × oriented cosine (eq 9.16) describing V1 simple-cell weights: it responds to a specific spatial frequency of brightness in a specific direction at a specific location; countless learning algorithms learn Gabor-like first-layer filters.
gated recurrent unit (gru) ch. 10: A simpler gated RNN than the LSTM: a single update gate u jointly controls forgetting and updating, and a reset gate r selects which parts of the past state feed the new target state. Comparable in effectiveness to the LSTM.
gated rnn ch. 10: The most effective sequence models in practice (LSTM, GRU). Like leaky units they create paths whose gradient neither vanishes nor explodes, but with connection weights (gates) that are learned and change at every time step.
gaussian distribution ch. 3: N(x; μ, σ²), the bell curve — default for reals via the central limit theorem and max entropy for fixed variance; multivariate form uses covariance Σ or precision matrix.
gaussian mixture model ch. 3: Mixture with Gaussian components (means μ⁽ⁱ⁾, covariances Σ⁽ⁱ⁾, priors αᵢ); universal approximator of smooth densities.
gaussian-bernoulli rbm ch. 20: An RBM with real-valued visible units and binary hidden units, p(v|h)=N(Wh, β⁻¹); models only the conditional MEAN of the data.
generalization ch. 5: Performing well on new, unseen inputs — the gap between machine learning and mere optimization; measured as test/generalization error.
generalized lagrangian ch. 4: L(x, λ, α) = f(x) + Σλᵢg⁽ⁱ⁾(x) + Σαⱼh⁽ʲ⁾(x): converts constrained problems into unconstrained min-max over KKT multipliers.
generalized pseudolikelihood ch. 18: Pseudolikelihood with larger left-of-bar index sets S⁽ⁱ⁾, interpolating between pseudolikelihood (singletons) and the full log-likelihood.
generative adversarial network (gan) ch. 20: A generator g(z) trained against a discriminator d(x) in a minimax game v=E_data log d + E_model log(1−d); at equilibrium samples are indistinguishable and d→½. No inference or Z-gradient needed, but training can fail to converge.
generative moment matching network ch. 20: A differentiable generator trained (without a discriminator) to match the moments of the data via the maximum mean discrepancy cost.
generative stochastic network (gsn) ch. 20: A generalization of the denoising autoencoder that adds latent variables to the generative Markov chain; parametrizes the generative PROCESS, with the joint defined only implicitly as the chain's stationary distribution.
gibbs distribution ch. 16: The distribution obtained by normalizing a product of clique potentials, p(x)=p̃(x)/Z.
gibbs sampling ch. 16: Sample an undirected model by repeatedly resampling each xᵢ from p(xᵢ | its neighbors); a multi-pass process whose distribution only converges to p(x) asymptotically (ch17).
global contrast normalization (gcn) ch. 12: Preprocessing that removes per-image contrast variation: subtract the image mean, then rescale so the pixel standard deviation equals a constant s (with a regularizer λ to avoid amplifying noise in near-constant images). Maps examples onto a spherical shell.
gmm-hmm ch. 12: The dominant pre-2012 speech-recognition model family: a hidden Markov model generates a phoneme/sub-phoneme sequence and a Gaussian mixture model turns each symbol into a segment of audio. Later replaced by deep neural nets for the acoustic model.
gradient ch. 4: ∇ₓf(x): the vector of all partial derivatives; points directly uphill, so −∇f points directly downhill.
gradient checking ch. 11: A debugging test that compares back-propagated (analytic) derivatives against numerical ones (finite differences or complex-step); for vector functions, use random projections uᵀg(vx) to reduce cost to one input/output.
gradient clipping ch. 8: Thresholding the gradient magnitude before a step so a cliff cannot catapult the parameters — recalling that the gradient gives a direction, not a safe step size (§10.11.1).
gradient descent ch. 4: (Steepest descent, Cauchy 1847) iterate x′ = x − ε∇ₓf(x): small steps against the gradient until it (nearly) vanishes.
grandmother cell ch. 9: A neuron that responds to a specific concept invariant to many transformations (the 'Halle Berry neuron' in the medial temporal lobe) — the endpoint of stacked detection→pooling.
graphical model ch. 16: Synonym for a structured probabilistic model: a distribution described by a graph whose nodes are random variables and whose edges are direct interactions (missing edges = independence assumptions).
greedy layer-wise pretraining ch. 8: Pretrain each hidden layer as part of a shallow supervised MLP on the previous layer's output, then stack the layers and jointly fine-tune (Bengio et al. 2007, fig 8.7) — a greedy strategy that gives good intermediate-layer guidance.
grid search ch. 11: A hyperparameter search that trains a model for every combination in the Cartesian product of a small value set per hyperparameter (usually log-scaled, repeated/zoomed). Cost grows exponentially, O(nᵐ), with the number of hyperparameters.
hadamard product ch. 2: Element-wise matrix product A⊙B, distinct from the standard matrix product.
hebbian learning ch. 20: 'Fire together, wire together': a local learning rule strengthening a connection between two units that frequently co-activate; the Boltzmann-machine positive phase is an instance.
hessian matrix ch. 4: H(f)ᵢⱼ = ∂²f/∂xᵢ∂xⱼ — the Jacobian of the gradient; symmetric (a.e.), so real eigenvalues + orthogonal eigenbasis; dᵀHd is directional curvature.
heteroscedastic model ch. 6: A model whose predicted output variance depends on the input x (rather than being a fixed constant).
hidden layer ch. 6: An intermediate layer whose targets the data does NOT specify — the learning algorithm decides what it computes. (First introduced ch01.)
hierarchical softmax ch. 12: A way to make a softmax over a huge vocabulary cheap by arranging words as leaves of a tree of nested categories: a words probability is the product of binary decisions along its root-to-leaf path, costing O(log|V|) instead of O(|V|).
hyperbolic tangent ch. 6: tanh(z) = 2σ(2z) − 1: a sigmoidal activation preferred over the logistic sigmoid as a hidden unit because tanh(0) = 0 resembles the identity near 0.
hyperparameter ch. 5: A setting not adapted by the learning algorithm itself (degree, λ) — tuned on the validation set, never the test set.
hypothesis space ch. 5: The set of functions the learning algorithm may pick as its solution (e.g. all linear functions; all degree-9 polynomials).
identity matrix ch. 2: Iₙ: ones on the main diagonal, zeros elsewhere; Iₙx = x for all x.
iid assumptions ch. 5: Examples are independent; train and test are identically distributed from the shared data-generating distribution p_data.
ill-conditioning (of the hessian) ch. 8: When high curvature forces a tiny learning rate: a GD step of −εg adds ½ε²gᵀHg − εgᵀg to the cost, so learning stalls (even small steps raise the cost) once the ½ε²gᵀHg curvature term outgrows εgᵀg, despite a strong gradient.
immorality ch. 16: A directed sub-structure — two unconnected co-parents of a child — that no undirected graph can represent perfectly; converting to undirected requires 'moralizing' it.
importance sampling ch. 12: Estimating an expectation (here the softmax negative-phase gradient) by sampling from a cheap proposal distribution q instead of the model, correcting the bias with importance weights — avoids computing scores for the whole vocabulary. Biased importance sampling normalizes the weights.
independence ch. 3: x⊥y: the joint factorizes p(x,y) = p(x)p(y); stronger than zero covariance (excludes nonlinear relations too).
independent component analysis (ica) ch. 13: A linear factor model that separates an observed signal into fully INDEPENDENT (not just decorrelated) underlying signals — e.g. the cocktail-party problem or removing heart/eye artifacts from EEG. Requires a non-Gaussian prior p(h) for W to be identifiable; typically learns sparse features.
independent subspace analysis ch. 13: An ICA generalization that learns non-overlapping GROUPS of features, allowing statistical dependence within a group but discouraging it between groups.
inexact gradients ch. 8: In practice we have only noisy or biased gradient/Hessian estimates — from minibatch sampling, or from approximating intractable objectives (e.g. contrastive divergence for Boltzmann machines).
inferotemporal cortex (it) ch. 9: The high-level object-recognition brain area (retina→LGN→V1→V2→V4→IT); its first-~100ms feedforward firing rates are well predicted by a CNN's last feature layer.
infinitely strong prior ch. 9: A prior placing zero probability on some parameters (forbidding them). Convolution ↔ weights zero outside a shared, shifted receptive field; pooling ↔ each unit invariant to small translations. A wrong such prior causes underfitting.
input (of a convolution) ch. 9: The first argument of the convolution (the data grid); in a deep net it is the previous layer's multi-channel feature maps.
input gate ch. 10: The LSTM sigmoid unit g^(t) that controls how much of the newly computed candidate value is accumulated into the cell state.
invariance to translation ch. 9: Small input shifts leave most pooled outputs UNCHANGED — useful when we care whether a feature is present, not exactly where. (Contrast equivariance, where the output shifts WITH the input.)
jacobian matrix ch. 4: All partial derivatives of a vector→vector function: Jᵢⱼ = ∂f(x)ᵢ/∂xⱼ.
k-means clustering ch. 5: Alternate assigning points to nearest centroid and re-averaging centroids; yields one-hot (extreme sparse) codes; clustering is inherently ill-posed.
k-nearest neighbors ch. 5: Non-parametric: output the (average) label of the k closest training points; 1-NN approaches 2× Bayes error asymptotically; can't learn feature relevance.
kernel ch. 9: The (usually small, learned) array of parameters convolved with the input — the second argument of the convolution.
kernel trick ch. 5: Rewrite an algorithm in dot products, then substitute k(x, x⁽ⁱ⁾) = φ(x)·φ(x⁽ⁱ⁾): nonlinear in x, still convex, sometimes infinite-dimensional φ with cheap k (Gaussian/RBF = template matching).
kkt conditions ch. 4: Necessary optimality conditions: ∇L = 0, all constraints satisfied, complementary slackness α⊙h(x) = 0 (each inequality active or its multiplier zero).
kkt multiplier ch. 7: The coefficient α ≥ 0 multiplying a constraint term in the generalized Lagrangian L = J + α(Ω(θ) − k); a parameter norm penalty is exactly such a term with α fixed.
kl divergence direction ch. 20: The direction of KL you minimize picks the failure mode. Forward D_KL(p_data‖p_model) (maximum likelihood) is mode-COVERING — spreads mass to cover every data mode, risking probability on unrealistic points. Reverse D_KL(p_model‖p_data) (variational) is mode-SEEKING — locks onto some modes, rarely emitting unrealistic points but dropping others. §20.14: the right metric depends on the intended use (fig 3.6).
knowledge base ch. 12: A database conveying commonsense or expert knowledge to an AI, often as (subject, verb, object) relation triplets (e.g. Freebase, WordNet). Entity and relation embeddings can be learned from its triplets.
knowledge base approach ch. 1: Hard-coding world knowledge in formal languages for automatic logical inference (e.g. Cyc).
krylov methods ch. 6: Iterative techniques (inversion, eigenvectors) using only matrix-vector products — how deep learning works with the Hessian via Hessian-vector products Hv = ∇((∇f)ᵀv) instead of forming the full n×n Hessian.
kullback-leibler divergence ch. 3: D_KL(P‖Q) = E_{x∼P}[log P − log Q] ≥ 0, zero iff P = Q; asymmetric — NOT a distance.
l1 regularization ch. 7: Penalty α‖w‖₁ = αΣ|wᵢ|. Its constant-magnitude sub-gradient α·sign(w) soft-thresholds weights to EXACTLY zero, inducing sparsity — unlike L², which only shrinks proportionally.
l2 regularization ch. 7: Weight decay: Ω = ½‖w‖₂². Shrinks each Hessian-eigencomponent of w* by λᵢ/(λᵢ+α), decaying low-curvature (unimportant) directions while preserving high-curvature ones.
label smoothing ch. 7: Replace hard 0/1 softmax targets with soft ε/(k−1) and 1−ε, so maximum likelihood need not chase unreachable extreme probabilities (which would grow weights forever); accounts for label mistakes.
language model ch. 12: A model defining a probability distribution over sequences of tokens (words, characters, or bytes) in a natural language.
laplace distribution ch. 3: Laplace(x; μ, γ): a sharp probability peak at an arbitrary point μ (exponential distribution: sharp point at 0).
laplace prior (map view) ch. 7: L¹ regularization equals MAP inference with an isotropic Laplace prior on the weights — just as L² equals MAP with a Gaussian prior.
las vegas algorithm ch. 17: A randomized algorithm that always returns the exact answer (or reports failure) but consumes a random amount of time/memory.
lasso ch. 7: Least Absolute Shrinkage and Selection Operator (Tibshirani 1995): an L¹ penalty + linear model + least-squares cost; the zeroed weights perform automatic feature selection.
latent variable ch. 3: A random variable that is never observed directly, like the mixture's component identity c.
leaky relu ch. 6: ReLU with a small fixed negative slope (α ≈ 0.01) so units keep receiving gradient when inactive.
leaky unit ch. 10: A hidden unit with a linear self-connection µ^(t) ← αµ^(t-1) + (1-α)v^(t) acting as a running average. α near 1 remembers the distant past; α near 0 forgets quickly — a smooth, adaptable way to hold information over time.
learned approximate inference ch. 19: Replace the slow iterative optimization q* = argmax_q L(v,q) with a neural network f̂(v;θ) trained to output the approximate posterior in one pass (e.g. the VAE encoder).
learning algorithm ch. 5: Mitchell: a program that improves at tasks T, measured by P, with experience E — task, measure, and data define the problem.
learning rate ch. 4: ε: the positive scalar step size of gradient descent — constant, solved-for, or chosen by line search.
learning-rate schedule ch. 8: A rule for decreasing the SGD learning rate over time (e.g. linear decay εₖ=(1−α)ε₀+αε_τ, α=k/τ, then constant). Required because SGD sampling noise never vanishes; convergence needs Σεₖ=∞ and Σεₖ²<∞.
line search ch. 4: Evaluate f(x − ε∇f) for several ε and keep the best — step-size selection by trial.
linear combination ch. 2: Σᵢ cᵢ v⁽ⁱ⁾ — scale each vector by a coefficient and add.
linear factor model ch. 13: A simple latent-variable probabilistic model with a stochastic LINEAR decoder: sample factors h from a factorial prior, then generate x = Wh + b + noise. Probabilistic PCA, factor analysis, ICA, and sparse coding differ only in the noise and prior choices.
linear independence ch. 2: No vector in the set is a linear combination of the others; redundant vectors add nothing to the span.
linear least squares ch. 4: Minimize ½‖Ax − b‖₂²: gradient AᵀAx − Aᵀb; solvable by GD (alg 4.1), one Newton step, or the pseudoinverse.
linear model ch. 1: A model based on f(x,w)=x·w; widely used but unable to learn XOR.
linear output unit ch. 6: An output unit that is a pure affine map ŷ = Wᵀh + b (no nonlinearity); used as the mean of a conditional Gaussian, so ML = MSE. Does not saturate.
link prediction ch. 12: Predicting missing arcs (facts) in a knowledge graph from known ones — a form of generalization to new facts; evaluated with metrics like precision-at-10% over corrupted-fact rankings.
lipschitz continuous ch. 4: |f(x) − f(y)| ≤ L‖x−y‖₂: rate of change bounded by L — the weak guarantee most DL objectives can be given.
local constancy prior ch. 7: The preference — encoded by adversarial training — that the function change little under small input perturbations near the data, resisting the fragility of locally-linear behavior.
local contrast normalization (lcn) ch. 12: Like GCN but per small window: each pixel is normalized by the mean and std of nearby pixels (rectangular or Gaussian-weighted), highlighting edges. Implementable via separable convolution; a differentiable op usable as a layer.
locally connected layer / unshared convolution ch. 9: Same sparse connectivity as convolution but EVERY connection has its own weight (a 6-D tensor, eq 9.9) — no parameter sharing. Use when a feature belongs to a specific location (e.g. a mouth in the lower half of a face).
log-uniform distribution ch. 11: A distribution for sampling a positive hyperparameter uniformly on a logarithmic scale: sample log₁₀(v) ~ u(a,b), then v = 10^(that). Used in random search for learning rates, hidden-unit counts, etc.
logistic regression ch. 5: Classification via p(y=1|x) = σ(θᵀx): squash a linear score into (0,1); no closed form — gradient descent on the NLL.
logistic sigmoid ch. 3: σ(x) = 1/(1+e⁻ˣ) ∈ (0,1): produces Bernoulli φ; saturates at both extremes; σ' = σ(1−σ).
logit ch. 6: The pre-sigmoid/pre-softmax score z = log P̃(y): an unnormalized log-probability that the output activation exponentiates and normalizes.
long short-term memory ch. 1: Gated recurrent network (1997) resolving the difficulty of learning long-term dependencies in sequences; still widely used.
long short-term memory (lstm) ch. 10: A gated RNN whose cell has a linear self-loop state s^(t) controlled by a forget gate, with an input gate and output gate. The self-loop lets gradients flow for long durations; the gates let the integration time scale adapt per input.
long-term dependencies ch. 10: Dependencies spanning many time steps. Learning them is hard because gradients propagated over many stages tend to vanish (usually) or explode (rarely), so long-range signals are exponentially weaker than short-range ones.
loopy belief propagation ch. 16: A popular approximate-inference algorithm for sparsely-connected traditional graphical models; almost never used in deep learning, whose distributed representations make graphs too dense.
m-step ch. 19: The maximization step of EM: maximize Σ L(v⁽ⁱ⁾,θ,q) over θ holding q fixed (possibly a large or analytic step).
machine learning ch. 1: Acquiring knowledge by extracting patterns from raw data instead of hard-coding it.
manifold hypothesis ch. 5: Real data (images, text, sound) concentrates near low-dimensional connected manifolds in the embedding space — so learn manifold coordinates, not ℝⁿ coordinates.
manifold interpretation (of pca) ch. 13: Viewing probabilistic PCA / linear autoencoders as learning a flat pancake Gaussian aligned with a low-dimensional linear manifold: narrow (noise) orthogonal to the manifold, wide (signal) along it; reconstruction error equals the sum of the discarded eigenvalues Σλᵢ.
manifold tangent classifier ch. 7: Tangent prop WITHOUT hand-specified tangents: an autoencoder first learns the manifold tangent vectors (ch14), which then regularize the classifier — capturing object-specific factors beyond translation/rotation/scaling.
map estimation ch. 5: Maximum a posteriori point estimate: arg max log p(x|θ) + log p(θ) — keeps the prior's pull; Gaussian prior ⇒ weight decay.
map inference ch. 19: Compute the single most likely latent value h* = argmax_h p(h|v) instead of the whole posterior; equivalent to restricting q to a Dirac δ(h−µ) in the ELBO. The workhorse for sparse coding.
marginal probability ch. 3: The distribution of a subset of variables: P(x) = Σ_y P(x,y) (sum rule) or the integral analogue.
markov chain ch. 17: A state x repeatedly updated by a transition operator T(x′|x); used to approximately sample distributions (e.g. EBMs) that have no tractable exact sampler.
markov chain monte carlo (mcmc) ch. 17: The family of algorithms that run a Markov chain whose stationary distribution is the target p, using its states as (approximate) Monte Carlo samples.
markov random field ch. 16: An undirected graphical model (Markov network): non-negative clique factors φ(C) give p̃(x)=Π φ(C), normalized by the partition function Z. Best for symmetric interactions.
matrix ch. 2: A 2-D array of numbers, bold uppercase (A ∈ ℝ^{m×n}); A_{i,:} is row i, A_{:,i} column i.
matrix inverse ch. 2: A⁻¹ with A⁻¹A = I; solves Ax = b analytically as x = A⁻¹b — a theoretical tool, not a numerical one.
matrix product ch. 2: C = AB with C_{i,j} = Σₖ A_{i,k}B_{k,j}; each entry is a row·column dot product; not commutative.
max pooling ch. 9: Pooling that reports the MAXIMUM activation in a rectangular neighborhood (Zhou & Chellappa 1988); insensitive to the exact location of the maximum → translation invariance.
maximum likelihood ch. 5: θ_ML = arg max Σ log p_model(x⁽ⁱ⁾; θ) — equivalently minimize D_KL(p̂_data‖p_model) = cross-entropy; consistent and efficient (Cramér-Rao).
maximum mean discrepancy (mmd) ch. 20: A kernel-based cost measuring the difference in (infinitely many) moments between two distributions; zero iff the distributions are equal. Trains GMMNs.
maxout unit ch. 6: Outputs the max over a group of k linear pieces — learns a piecewise-linear convex activation itself; needs more regularization; resists catastrophic forgetting.
mean absolute error ch. 6: The L¹ loss E‖y − f(x)‖₁; its minimizer predicts the MEDIAN of y|x (MSE predicts the mean).
mean field ch. 19: The variational approximation with a fully factorial q(h|v) = Πᵢ q(hᵢ|v); each latent is independent given v.
mean-covariance rbm (mcrbm) ch. 20: A real-valued RBM whose hidden layer has separate MEAN units (a Gaussian RBM) and COVARIANCE units (a cRBM), so it models both the conditional mean and covariance; training needs Hamiltonian Monte Carlo.
measure zero ch. 3: A set occupying no volume (points, lines in ℝ²); 'almost everywhere' = except on such a set.
memory network ch. 10: A network (Weston et al. 2014) augmented with a set of memory cells accessed by an addressing mechanism; the original version required supervision about how to use its memory.
metropolis-hastings ch. 17: A widely-used MCMC transition-construction algorithm (common outside deep learning, where Gibbs sampling dominates).
minibatch ch. 5: The small uniformly-sampled example set (size m′) whose average loss gradient estimates the true gradient in SGD.
minibatch stochastic method ch. 8: An optimizer using a small random subset (minibatch, typically a power of 2 from 32–256) per update; gives an unbiased gradient estimate with standard error σ/√n, so large batches have diminishing returns. Now usually just called "stochastic".
mixing time ch. 17: The number of steps a Markov chain needs to reach equilibrium; governed by the second-largest eigenvalue of A, but generally unknowable in practice.
mixture density network ch. 6: A net whose output parametrizes a Gaussian mixture p(y|x) = Σ p(c=i|x)N(y; μ⁽ⁱ⁾, Σ⁽ⁱ⁾) — for multimodal regression; softmax mixing weights, unconstrained means, diagonal covariances, latent c.
mixture distribution ch. 3: P(x) = Σᵢ P(c=i) P(x|c=i): a multinoulli picks which component generates each sample.
mixture of experts ch. 12: A model where a gater network softmax-weights the outputs of several expert networks. A HARD mixture picks a single expert per input (saving computation); a soft mixture averages all experts (no compute saving).
mode (of a distribution) ch. 17: A region of high probability / low energy (e.g. a connected manifold of similar images) around which a Markov chain tends to random-walk.
model averaging / ensemble method ch. 7: Combining several models predictions; the ensemble squared error is (1/k)v + ((k−1)/k)c, falling to (1/k)v when member errors (variance v, covariance c) are uncorrelated.
model compression ch. 12: Replacing an expensive model (often a large regularized net or an ensemble) with a smaller model trained to mimic its function f(x) on sampled/generated inputs — cutting inference cost. Copying the posterior over classes is called distillation.
model identifiability ch. 8: Whether enough data can pin down a single parameter setting. Neural nets are NON-identifiable, so their cost surfaces have many equivalent local minima (all equal in cost — not a real optimization problem).
model parallelism ch. 12: Distributing a single datapoint across machines, each running a different part of the model — feasible for both inference and training.
model-based hyperparameter optimization ch. 11: Casting hyperparameter search as optimization: fit a (usually Bayesian) model of validation error vs hyperparameters, then propose new settings balancing exploration and exploitation (Spearmint, TPE, SMAC). Promising but not yet a reliable default.
momentum ch. 8: Accumulate an exponentially-decayed average of past gradients as a velocity: v ← αv − εg, θ ← θ + v. It accelerates along consistent directions (terminal speed ε‖g‖/(1−α)) and averages out the zig-zag across an ill-conditioned canyon.
monte carlo algorithm ch. 17: A randomized algorithm returning an answer with random error that shrinks as more compute is spent; gives an approximate answer for any fixed budget (contrast Las Vegas).
monte carlo sampling ch. 17: Approximating a sum/integral s=E_p[f(x)] by the empirical average ŝ_n=(1/n)Σf(x⁽ⁱ⁾) of n samples from p; unbiased, with error ∝ 1/√n.
moore-penrose pseudoinverse ch. 2: A⁺ = VD⁺Uᵀ: minimal-norm solution for wide A, least-squares closest solution for tall A.
moralized graph ch. 16: The undirected graph obtained from a directed one by connecting all co-parents ('marrying' them) and dropping arrow directions; may add many edges and thus lose independences.
multi-channel convolution ch. 9: Many parallel convolutions over vector-valued (multi-channel) input, using a 4-D kernel K_{i,j,k,l} (output channel i, input channel j, k/l offsets); input/output are 3-D tensors (channel + 2 spatial), 4-D with a batch axis.
multi-modal learning ch. 15: Learning a representation for each of two modalities (fx, fy) plus the relationship between the two representation spaces, so concepts anchor across modalities and the model generalizes to never-seen pairs — the mechanism behind fig 15.3's zero-shot transfer.
multi-prediction deep boltzmann machine (mp-dbm) ch. 20: A jointly-trainable DBM that views the mean-field equations as a family of recurrent networks and back-propagates through the inference graph to predict held-out variables; a good classifier but with a defective p(v).
multi-task learning ch. 7: Improve generalization by sharing lower-layer parameters across several tasks (a common factor pool) while keeping task-specific upper layers; pooled data constrains the shared part toward values that generalize.
multilayer perceptron ch. 1: A function composed of many simpler functions; the quintessential deep model (feedforward deep network).
multinoulli distribution ch. 3: (Categorical) one discrete variable with k enumerable states, parametrized by p ∈ [0,1]^{k−1}.
n-gram ch. 12: A fixed-length sequence of n tokens. An n-gram language model defines P(sequence) as a product of conditionals P(xₜ | previous n−1 tokens), estimated by counting; unigram/bigram/trigram for n=1/2/3.
nade ch. 20: The neural autoregressive density estimator: an auto-regressive net with a weight-sharing scheme (W′_{j,k,i}=W_{k,i}) that mimics one step of RBM mean-field inference.
negative log-likelihood ch. 5: The standard cost: NLL = −E log p_model. ANY NLL is a cross-entropy — MSE is the cross-entropy of a Gaussian model.
negative phase ch. 18: The −∇θ log Z = −E_{x∼p}[∇θ log p̃(x)] term: lower the unnormalized probability (raise the energy) on samples drawn from the model; the difficult term.
neocognitron ch. 9: Fukushima's 1980 model with most of the modern CNN's architecture (parameter sharing across locations) but trained by layer-wise unsupervised clustering, not back-propagation.
nesterov momentum ch. 8: A momentum variant (Sutskever et al.) that evaluates the gradient AFTER applying the current velocity — at the interim point θ+αv — a look-ahead correction. Improves the convex batch rate O(1/k)→O(1/k²); no help stochastically.
neural language model ch. 12: A language model that overcomes the curse of dimensionality by learning distributed word embeddings, letting it share statistical strength between a word and other similar words (Bengio 2001).
neural turing machine (ntm) ch. 10: An explicit-memory model (Graves et al. 2014) that learns to read from and write to memory cells without supervision, using differentiable soft (softmax-weighted) addressing so the whole system trains by gradient descent.
newtons method ch. 4: Jump to the critical point of the local second-order Taylor: x* = x⁽⁰⁾ − H⁻¹∇f; exact in one step for positive definite quadratics; attracted to saddles.
no free lunch theorem ch. 5: Averaged over ALL data-generating distributions, every classifier has the same error on unseen points — so seek algorithms for the distributions we actually meet.
noise injection ch. 7: Adding random noise to inputs (≈ a weight-norm penalty), to weights (≈ flat-minima / Bayesian uncertainty), or to hidden units (dropout) as a form of regularization / augmentation.
noise robustness ch. 7: Training with injected noise so the learned function is insensitive to small perturbations. Input noise ≈ a weight-norm penalty; weight noise adds an η·E‖∇_W ŷ‖² term that pushes toward FLAT minima (robust to weight perturbation).
noise-contrastive estimation ch. 12: A training objective that learns a model by discriminating real data from noise samples, avoiding an explicit normalized softmax over a large vocabulary; applied to neural language models (Mnih & Teh 2012).
noise-contrastive estimation (nce) ch. 18: Learn log p_model = log p̃ + c (c a learned −log Z) by training a binary classifier to tell data from a noise distribution: p_joint(y=1|x)=σ(log p_model − log p_noise).
non-distributed representation ch. 15: A representation spending one parameter set (symbol, template, tree leaf, cluster) per region of input space — k-means, k-NN, decision trees, Gaussian mixtures, local-kernel machines, n-grams. n parameters carve only n regions, so generalization rests entirely on the smoothness prior.
nonlinear ica ch. 13: A generative generalization of ICA using a nonlinear encoder/decoder; NICE (Dinh 2014) stacks invertible transforms whose Jacobian determinants are cheap, enabling exact likelihood and easy sampling.
norm ch. 2: A size measure for vectors: f(x)=0 ⟹ x=0, triangle inequality, |α|-homogeneity; the Lᵖ family (‖x‖ₚ).
norm penalty as constrained optimization ch. 7: A penalty αΩ(θ) is the KKT/Lagrangian form of the hard constraint Ω(θ) < k: L² ↔ confining w to an L² ball, L¹ ↔ a bounded-L¹ region; larger α ⇒ smaller region.
normalized / xavier–glorot initialization ch. 8: W_{i,j} ~ U(−√(6/(m+n)), √(6/(m+n))) for a layer with m inputs and n outputs — a compromise that keeps both forward-activation variance and backward-gradient variance roughly equal across layers (derived for a linear chain).
normalizing constant ch. 3: Z: the sum/integral of the product of clique factors — divides the unnormalized product into a distribution (ch18's partition function).
numerical gradient (finite differences) ch. 11: Approximating a derivative by evaluating the function at nearby points: (f(x+ε)−f(x))/ε. Used in gradient checking to verify a hand-implemented back-prop; ε must be large enough to avoid rounding but small enough to limit truncation error.
objective function ch. 4: The function being minimized/maximized (= criterion; when minimized also called cost, loss, or error function); optimum written x* = arg min f(x).
one-shot learning ch. 15: Learning a transfer task from a SINGLE labeled example — possible when the learned representation already separates the underlying classes cleanly, so one label tags an entire cluster in representation space.
online learning ch. 8: Learning from a stream of never-repeated examples drawn from p_data; since each example is a fresh fair sample, SGD then follows the gradient of the true generalization error.
optimal importance sampling ch. 17: The minimum-variance choice q*∝p(x)|f(x)| (eq 17.13); gives zero variance when f has constant sign, but computing it solves the original problem.
orthogonal initialization ch. 8: Initialize weights to random orthogonal matrices with a nonlinearity-dependent gain g (Saxe et al.); makes the iterations-to-converge independent of depth and can train 1000-layer nets.
orthogonal matching pursuit (omp-k) ch. 7: Encode x with the representation h minimizing ‖x − Wh‖² subject to at most k non-zero entries (‖h‖₀ < k); efficient when W is orthogonal. OMP-1 is an effective deep-net feature extractor.
orthogonal matrix ch. 2: Square with mutually orthonormal rows AND columns: AᵀA = AAᵀ = I, so A⁻¹ = Aᵀ.
output gate ch. 10: The LSTM sigmoid unit q^(t) that controls how much of the (tanh of the) cell state is exposed as the cell’s output h^(t).
output layer ch. 6: The final layer of a feedforward net, driven directly by the training targets.
output recurrence ch. 10: An RNN design (fig 10.4) whose only feedback is from the previous output o^(t-1) to the hidden state — less powerful than hidden-to-hidden recurrence, but trainable in parallel via teacher forcing.
overfitting ch. 5: The gap between training and test error is too large — the model memorizes training quirks.
overflow ch. 4: Large-magnitude numbers approximated as ±∞; further arithmetic turns them into NaN.
parallel tempering ch. 17: Running many chains at different temperatures simultaneously and stochastically swapping states, so high-temperature chains that mix freely feed the temperature-1 chain.
parameter norm penalty ch. 7: A term αΩ(θ) added to the objective (J̃ = J + αΩ) that limits capacity by penalizing weight size; α∈[0,∞) sets the strength. Only weights are penalized, not biases.
parameter server ch. 12: A machine (or set of machines) that holds the shared model parameters in distributed asynchronous SGD; workers read parameters, compute gradients, and push updates back.
parameter sharing ch. 7: FORCING sets of parameters to be equal (e.g. the same weights at every image location in a CNN); stores only the unique set, cutting memory and encoding invariance directly.
parameter tying ch. 7: Regularizing sets of parameters to be CLOSE via a penalty ‖w^(A) − w^(B)‖² (e.g. tying a supervised model to an unsupervised one that shares structure).
parametric relu ch. 6: (PReLU) leaky-ReLU whose negative slope α is a learned parameter.
partition function ch. 16: Z = Σ/∫ p̃(x): the normalizer of an undirected model (term borrowed from statistical physics); usually intractable in deep learning (ch18) and can fail to exist if the integral diverges.
patience ch. 7: In early stopping, the number of successive validation-error worsenings tolerated before halting; larger patience trains longer before giving up.
perceptron ch. 1: The first model that could learn its category-defining weights from labeled examples (Rosenblatt, 1958).
persistent contrastive divergence (pcd) ch. 18: The deep-learning name for SML: persistent negative-phase Markov chains carried across gradient steps (PCD-k).
point estimator ch. 5: Any function of the data θ̂ = g(x⁽¹⁾…x⁽ᵐ⁾); good ones land near the true θ.
polyak averaging ch. 8: Average the parameter trajectory θ̂ = (1/t)Σθ⁽ⁱ⁾ (an exponentially-decayed running average for non-convex problems) to land near a valley floor the optimizer merely oscillated across.
pooling ch. 9: Replace an output with a summary statistic of its neighborhood (max, average, L², weighted). The third stage of a conv layer; confers approximate invariance to small translations.
positive definite ch. 2: All eigenvalues > 0; guarantees xᵀAx > 0 for x ≠ 0 (semidefinite allows zeros).
positive phase ch. 18: The ∇θ log p̃(x) term of the undirected log-likelihood gradient: raise the unnormalized probability (lower the energy) on training examples.
posterior probability ch. 3: The updated belief P(c|x) AFTER observing x.
pr curve ch. 11: A curve of precision (y) vs recall (x) traced by sweeping the classifiers decision threshold; summarizes the precision/recall tradeoff. Often reduced to a single number via the F-score or the area under the curve (PR-AUC).
precision ch. 3: Inverse variance β = 1/σ² (or matrix β = Σ⁻¹): the cheap-to-evaluate parametrization of a Gaussian.
precision (classification) ch. 11: Of all the detections a model reports, the fraction that are correct: TP/(TP+FP). (Distinct from the ch03 statistical precision = inverse variance.) Paired with recall for rare-event tasks where accuracy is uninformative.
predictive sparse decomposition (psd) ch. 14: A hybrid of sparse coding and parametric autoencoders trained on ‖x − g(h)‖² + λ|h|₁ + γ‖h − f(x)‖²: h is optimized during training (warm-started by f), while deployment uses only the fast parametric encoder f. An instance of learned approximate inference (§19.5).
principal components analysis ch. 2: Lossy linear compression: encode c = Dᵀx, decode Dc; optimal D = top-l eigenvectors of XᵀX.
prior probability ch. 3: The model's belief P(c) BEFORE observing the data x.
probabilistic max pooling ch. 20: A way to include max-pooling in an energy-based (convolutional Boltzmann) model by constraining at most one detector unit per region to be active.
probabilistic pca ch. 13: A linear factor model with prior h~N(0,I) and ISOTROPIC Gaussian noise, giving x~N(b, WWᵀ+σ²I). Fit by EM; as σ→0 it becomes ordinary PCA (the posterior mean of h is the orthogonal projection onto Ws columns).
probability density function ch. 3: p(x) for continuous variables: p ≥ 0 (may exceed 1!), ∫p = 1; region probabilities come from integrating p.
probability mass function ch. 3: P(x) for discrete variables: domain = all states, 0 ≤ P ≤ 1, Σ P = 1 (normalized).
product of experts ch. 16: Hinton's view of an EBM: each energy term is an 'expert' enforcing one soft constraint on a low-dimensional projection; multiplying their probabilities enforces a complex joint constraint.
pseudolikelihood ch. 18: Train by maximizing Σᵢ log p(xᵢ | x₋ᵢ); each conditional is a Z-free ratio (Z cancels), costing k·n instead of kⁿ evaluations; asymptotically consistent.
quadrature pair ch. 9: Two simple cells with identical parameters except a quarter-cycle (90°) phase difference; a complex cell taking the L² norm of the pair becomes phase- (small-translation-) invariant.
quasi-newton method ch. 8: A second-order method that APPROXIMATES the inverse Hessian rather than computing it exactly (BFGS is the prominent example), gaining Newton-like directions without the O(k³) inversion.
random features (in cnns) ch. 9: Convolution kernels obtained without supervised training — set randomly, hand-designed, or learned unsupervised (e.g. k-means patches). Random filters work surprisingly well and give a cheap way to compare architectures.
random search ch. 11: A hyperparameter search that samples each hyperparameter from a marginal distribution (e.g. log-uniform) instead of a fixed grid. Exponentially more efficient than grid search when only a few hyperparameters matter, because no two trials are wasted on equivalent settings.
random variable ch. 3: A variable taking values randomly (plain typeface x, values x); meaningful only when coupled with a distribution; discrete or continuous.
ratio matching ch. 18: The discrete/binary analogue of score matching, using bit-flip probability ratios (Z cancels); pushes down fantasy states within Hamming distance 1 of the data.
recall ch. 11: Of all the true positive events, the fraction the model detects: TP/(TP+FN). A detector that flags everyone has perfect recall but poor precision. Traded against precision by moving the decision threshold.
receptive field ch. 9: The set of input units that affect a given output unit; it GROWS with depth, so deep units indirectly see most of the input even though each connection is sparse.
recirculation ch. 14: An autoencoder-specific learning algorithm comparing the network's activations on the original input with those on the reconstruction; regarded as more biologically plausible than back-propagation but rarely used.
rectified linear unit ch. 6: g(z) = max{0, z}: the default hidden unit — near-linear, large consistent gradients, easy to optimize; dead where its activation is 0. (First introduced ch01.)
recurrent convolutional network ch. 9: Iteratively refining a structured output (e.g. pixel labels) by re-applying the SAME convolutions with weights shared across steps (fig 9.17) — a recurrent network (ch10).
recurrent neural network (rnn) ch. 10: A family of neural networks for processing a sequence of values x^(1),…,x^(τ). Its power comes from sharing the same parameters across every time step, letting it handle variable-length sequences and generalize across positions.
recursive neural network ch. 10: A generalization of an RNN whose computational graph is a deep tree rather than a chain, reducing depth for a length-τ sequence from τ to O(log τ). Not abbreviated ‘RNN’ to avoid confusion with recurrent nets.
regularization ch. 5: Any modification intended to reduce generalization error but not training error — preferences among hypothesis-space members (e.g. weight decay λwᵀw).
regularized autoencoder ch. 14: An autoencoder whose loss demands properties beyond copying — sparse codes, small derivatives, robustness to noise — so that even an overcomplete, high-capacity model learns data structure instead of the identity function.
reinforce ch. 20: A gradient estimator for DISCRETE stochastic operations: ∂E[J]/∂ω = E[J·∂log p(y)/∂ω], estimated by Monte Carlo; high variance, reduced with a baseline b(ω).
reinforcement learning ch. 1: Learning to perform a task by trial and error, from rewards rather than labeled examples, without guidance from a human operator.
reparametrization trick ch. 20: Rewrite a sample y∼N(µ,σ²) as y=µ+σz with z∼N(0,1) external noise, turning sampling into a differentiable function so gradients flow through it (stochastic back-propagation). Continuous variables only.
representation learning ch. 1: Using learning to discover not just representation→output but the representation itself.
representational regularization ch. 7: Adding a norm penalty on the representation h (J̃ = J + αΩ(h)) rather than on the parameters — e.g. an L1 penalty Ω(h)=‖h‖₁ that drives activations to zero.
reservoir computing ch. 10: The umbrella term (covering echo state networks and liquid state machines) for models that map a sequence into a fixed-length recurrent state — a fixed ‘reservoir’ of features — on which a simple linear predictor is trained.
reset gate ch. 10: The GRU sigmoid gate r that controls which parts of the previous state are used to compute the next candidate (target) state, adding a nonlinear interaction between past and future state.
restricted boltzmann machine (rbm) ch. 16: An energy-based model with binary v, h and energy −bᵀv − cᵀh − vᵀWh, with NO intra-layer edges → factorial p(h|v) & p(v|h) with sigmoid conditionals → efficient block Gibbs sampling; the canonical deep-learning graphical model.
reverse correlation ch. 9: Estimating a neuron's linear weights by showing white-noise images and fitting a linear model to its responses (used to recover V1 Gabor weights when the weights are not directly accessible).
reverse mode accumulation ch. 6: The automatic-differentiation family backprop belongs to: multiply Jacobians from the output backward — cheap when outputs are few and inputs many.
ridge regression / tikhonov regularization ch. 7: Other-community names for L² parameter regularization (weight decay).
risk ch. 8: The expected generalization error E_{(x,y)∼p_data}[L(f(x;θ),y)] taken over the TRUE data-generating distribution — the quantity we actually want to minimize but cannot compute, hence empirical risk.
rmsprop ch. 8: AdaGrad with the squared-gradient accumulation replaced by an exponentially-decayed moving average (decay ρ), so it stops shrinking the rate once it reaches a good bowl — a go-to deep-learning optimizer.
rnn as a directed graphical model ch. 10: The view of an RNN over outputs y as a graphical model whose hidden state h^(t) gives an efficient O(1)-parameter, stationary parametrization of dependencies that a tabular model would need O(k^τ) parameters to represent.
rnn-rbm ch. 20: A sequence model where an RNN emits all of an RBM's parameters at each time step; trained by back-propagating a CD gradient through time.
saddle point ch. 4: A critical point that is a minimum along some cross-sections and a maximum along others (Hessian has both signs of eigenvalues).
saddle-free newton ch. 8: A modified second-order method (Dauphin et al.) that avoids Newton's attraction to the saddle points that proliferate in high-dimensional non-convex loss surfaces.
same convolution ch. 9: Just enough zero-padding to keep the output the same size as the input, allowing arbitrarily deep conv stacks (border pixels influence fewer outputs).
scalar ch. 2: A single number, written in italics (s ∈ ℝ), in contrast to the arrays of linear algebra.
score ch. 14: The gradient field ∇ₓ log p(x) of a log-density. Learning it is one way to learn p's structure: denoising autoencoders estimate it (arrows point uphill toward the data manifold, vanishing at density maxima); score matching proper appears in §18.4.
score matching ch. 18: Train by matching the model's score ∇_x log p_model to the data's score ∇_x log p_data; since ∇_x Z = 0, Z drops out; needs 1st/2nd x-derivatives (continuous data only).
second derivative test ch. 4: At a critical point: H positive definite → local min; negative definite → local max; mixed eigenvalue signs → saddle; a zero eigenvalue → inconclusive.
self-contrastive estimation ch. 18: NCE with the noise distribution set to the current model before each step; its expected gradient equals the maximum-likelihood gradient.
self-information ch. 3: I(x) = −log P(x): rare events carry more information; units nats (base e) or bits (base 2).
semantic hashing ch. 14: Information retrieval via dimensionality reduction to BINARY codes: saturate final-layer sigmoids (by annealed additive noise before the nonlinearity) so entries hash to buckets; flipping individual code bits searches slightly-less-similar entries.
semi-supervised learning ch. 7: Use both unlabeled examples from P(x) and labeled examples from P(x,y) to estimate P(y|x) — e.g. share parameters between a generative model of P(x) and a discriminative model of P(y|x), so unlabeled data shapes the representation.
separable kernel ch. 9: A d-D kernel expressible as the outer product of d vectors; then convolution decomposes into d 1-D convolutions, costing O(w·d) runtime/params instead of O(w^d). Not every kernel is separable.
separation ch. 16: Conditional independence implied by an UNDIRECTED graph: A⊥B|S iff every path from A to B passes through an observed (inactive) variable in S.
sequence ch. 10: An ordered list of values x^(1),…,x^(τ) (e.g. words, audio frames, or measurements over time). Sequences are what RNNs specialize in, as grids are what CNNs specialize in.
shannon entropy ch. 3: H(x) = E[I(x)] = −E[log P(x)]: expected information — the uncertainty of a whole distribution; max for uniform.
shared vs task-specific parameters ch. 7: In multi-task learning, generic parameters shared across all tasks (lower layers, benefit from pooled data) vs task-specific parameters (upper layers, benefit only from their own task).
sigmoid belief network ch. 20: A directed generative model of binary units where each unit's probability is a sigmoid of its ancestors, p(sᵢ)=σ(Σⱼ<ᵢ Wⱼᵢsⱼ+bᵢ); efficient to sample but with intractable inference.
sigmoid output unit ch. 6: ŷ = σ(wᵀh + b) for a Bernoulli output; paired with NLL loss J = ζ((1−2y)z) that saturates only when the model is already correct.
simple cell ch. 9: A V1 cell whose response is ≈ a linear function of the image in a small, localized receptive field — the inspiration for a CNN's detector (convolution) units; its weights ≈ a Gabor function.
singular matrix ch. 2: A square matrix with linearly dependent columns; not invertible; equivalently, some eigenvalue is zero.
singular value decomposition ch. 2: A = UDVᵀ for ANY real matrix: orthogonal U, V and rectangular-diagonal D of singular values.
skip connection ch. 6: A connection from layer i to layer i+2 or beyond, easing gradient flow from output toward input.
skip connections through time ch. 10: Recurrent connections with a delay d>1 that link a unit to one d steps in the past, so gradients decay over τ/d rather than τ steps — one way to capture coarser (longer) time scales.
slow feature analysis (sfa) ch. 13: A linear factor model applying the slowness principle in closed form: learn features that vary as slowly as possible over a time sequence, subject to zero mean, unit variance, and mutual decorrelation. Features come ordered slowest-first; nonlinear via a basis expansion.
slow mixing ch. 17: When successive Markov-chain samples stay highly correlated and fail to visit modes in proportion to probability; the chain gets stuck in one mode instead of exploring.
slowness principle ch. 13: The idea that important characteristics of a scene change SLOWLY compared with raw measurements (a zebras stripes flicker a pixel fast, but zebra present does not) — so regularize features to change slowly over time (penalty λΣₜ L(f(x⁽ᵗ⁺¹⁾),f(x⁽ᵗ⁾))).
smoothing (language models) ch. 12: Techniques that shift probability mass from observed to unobserved-but-similar token tuples, so an n-gram model does not assign zero probability (and −∞ log-likelihood) to sequences unseen in training; includes add-mass, back-off, and mixtures.
smoothness prior ch. 5: f(x) ≈ f(x+ε): the implicit prior of k-NN/local kernels/trees — needs O(k) examples for O(k) regions and can't extrapolate a checkerboard; DL's composition assumption gets O(2ᵏ).
soft-thresholding ch. 7: The L¹ analytical solution wᵢ = sign(wᵢ*)·max(|wᵢ*| − α/Hᵢᵢ, 0): shift each weight toward zero by α/Hᵢᵢ and clamp at 0 (a dead zone where small weights vanish).
softmax ch. 4: softmax(x)ᵢ = exp(xᵢ)/Σⱼexp(xⱼ): turns a real vector into multinoulli probabilities; stabilized by subtracting maxᵢxᵢ (shift-invariant).
softmax output unit ch. 6: softmax(z)ᵢ = exp(zᵢ)/Σⱼexp(zⱼ) for a multinoulli output; log-softmax's non-saturating first term keeps ML learning alive and penalizes the most-active wrong class.
softplus ch. 3: ζ(x) = log(1+eˣ) ∈ (0,∞): softened max(0,x); produces σ/β params; ζ' = σ.
span ch. 2: All points reachable by linear combination of a set of vectors; the span of A's columns = its column space.
sparse autoencoder ch. 14: An autoencoder trained on L(x, g(f(x))) + Ω(h) with Ω(h) = λΣ|hᵢ|; equivalent to approximate maximum-likelihood training of a generative latent-variable model with a Laplace prior on h. Rectified linear codes give actual zeros.
sparse coding ch. 13: A linear factor model whose encoder is an OPTIMIZATION, not a parametric map: h* = argmin λ‖h‖₁ + β‖x−Wh‖² (a sharply-peaked prior like Laplace + an L1 penalty giving a sparse code). No encoder generalization error, but slow iterative inference and poor samples.
sparse initialization ch. 8: Give each unit exactly k nonzero weights (Martens) so its total input is independent of fan-in without shrinking individual weights; adds unit diversity but imposes a strong prior on the large values.
sparse interactions ch. 9: (a.k.a. sparse connectivity / sparse weights) Making the kernel smaller than the input, so each output depends on only k inputs — k×n params and O(k×n) runtime vs a dense layer's m×n.
sparse representation ch. 7: A learned code h with many zero (or near-zero) entries, obtained by penalizing/constraining the ACTIVATIONS (L1, Student-t, KL, or a hard OMP-k constraint) rather than the weights.
sparsity (parameter vs representational) ch. 7: Parameter sparsity = many weights are exactly zero (from an L¹ penalty on w); representational sparsity = many activations hᵢ are zero (from a penalty on h). §7.10 covers the latter.
spectral radius ch. 10: The maximum absolute value among a matrix’s eigenvalues. For an RNN’s state-to-state Jacobian it governs whether small perturbations (and gradients) grow or shrink over time; reservoir methods tune it near the edge of stability.
spike and slab ch. 13: A sparse-coding prior that mixes a spike at exactly zero with a broad slab, so samples from the prior actually contain true zeros — unlike a Laplace prior, under which an exact zero is a probability-zero event.
spike-and-slab rbm (ssrbm) ch. 20: A real-valued RBM whose binary spike unit hᵢ gates a real slab unit sᵢ, letting it model conditional COVARIANCE without matrix inversion (CD/PCD apply).
spurious modes ch. 18: Modes with high probability under the model but absent from the data; CD's short data-initialized chains rarely visit them, so it can't push them down.
standard error ch. 5: √Var of an estimator across resampled datasets; SE(μ̂) = σ/√m powers the 95% CI μ̂ ± 1.96·SE used to compare algorithms.
state (of an rnn) ch. 10: The hidden vector h^(t) carried from one time step to the next. It is a lossy summary of the task-relevant aspects of the past sequence x^(1),…,x^(t).
stationary distribution ch. 17: The equilibrium distribution v=Av (eq 17.23): the eigenvector of the transition matrix with eigenvalue 1, to which the chain converges.
stationary transition ch. 10: The assumption that the conditional distribution over variables at one time step given the previous ones is the same at every step — what lets an RNN share one transition function across all of time.
stochastic encoder/decoder ch. 14: The generalization of encoder/decoder functions to distributions p_encoder(h|x) and p_decoder(x|h); any latent-variable model p_model(h, x) defines both. Trained by minimizing −log p_decoder(x|h) using the standard output-unit menu (Gaussian→MSE, Bernoulli→sigmoid, softmax).
stochastic gradient descent ch. 1: The dominant training algorithm: the gradient is an EXPECTATION, so estimate it on a small minibatch and step θ ← θ − εg — cost per update O(1) in dataset size. ADALINE's 1960 rule was a special case.
stochastic matrix ch. 17: A matrix whose columns are probability distributions (the transition matrix A); by Perron–Frobenius its largest eigenvalue is 1.
stochastic maximum likelihood (sml) ch. 18: Approximate the negative phase by initializing each step's Gibbs chains from the PREVIOUS step's chains (persistent chains); finds all modes and trains deep models. = persistent contrastive divergence.
stride (strided convolution) ch. 9: Sampling the convolution output only every s pixels (downsampling), reducing cost: Zᵢⱼₖ = c(K,V,s). Equivalent to unit-stride convolution followed by downsampling.
structure learning ch. 16: Searching (usually greedily) for a good graph structure by proposing edge add/removes and scoring accuracy vs complexity; deep learning avoids it by using latent variables instead.
structured output ch. 9: A CNN emitting a high-dimensional tensor (e.g. Sᵢⱼₖ = probability pixel (j,k) is class i) rather than a single label — enabling pixel-wise labeling / segmentation masks.
structured probabilistic model ch. 3: (Graphical model) a factorization of a joint distribution described by a graph: nodes = RVs, edges = direct interaction.
structured variational inference ch. 19: Variational inference with any chosen graphical-model structure on q (richer than mean field's full factorization).
sum-product network (spn) ch. 15: A probabilistic model computing a distribution with a polynomial circuit of sums and products; certain distributions require a minimum SPN depth to avoid exponentially large models — the exponential advantage of depth, in probabilistic form.
supervised pretraining ch. 8: Train simpler models or subtasks first, then the full model: greedy layer-wise pretraining, FitNets teacher→student hints, or transfer learning. Often followed by joint fine-tuning; aids both optimization and generalization.
support vector machine ch. 5: Linear score wᵀx + b, class-only output; famous for the kernel trick; predictions need only the support vectors (nonzero αᵢ).
surrogate loss function ch. 8: An efficiently-optimizable proxy for the loss we truly care about (e.g. negative log-likelihood as a surrogate for 0-1 loss); it has good gradients and can keep improving after the true loss saturates, by pushing classes further apart.
symmetric matrix ch. 2: A = Aᵀ; arises from order-independent two-argument functions like distances.
symmetry breaking ch. 8: The one initialization requirement known for certain: units sharing inputs and activation must get DIFFERENT initial parameters (via random init), or a deterministic algorithm updates them identically forever and they compute the same function.
tangent distance ch. 7: A non-parametric nearest-neighbor algorithm whose metric is the distance between the class MANIFOLDS (approximated by their tangent planes at each point) rather than the generic Euclidean distance — invariant to local factors of variation.
tangent plane ch. 14: At a point x on a d-dimensional manifold, the plane spanned by the d basis vectors of local variation — the directions in which x can move infinitesimally while staying on the manifold.
tangent prop ch. 7: Train a net with a penalty Ω(f)=Σᵢ(∇ₓf·v⁽ⁱ⁾)² that keeps each output locally invariant along known manifold tangents v⁽ⁱ⁾ (translation, rotation, scaling). The infinitesimal cousin of dataset augmentation.
teacher forcing ch. 10: A training procedure for models with output-to-hidden recurrence that feeds the ground-truth y^(t) (not the model’s own prediction) as the next step’s input. It enables parallel, BPTT-free training but can cause a train/test mismatch, mitigated by mixing in free-running inputs.
tempering ch. 17: Flattening a distribution via p_β(x)∝exp(−βE(x)) with β<1 (β=1/temperature) so a chain can mix between modes more easily; e.g. tempered transitions, parallel tempering.
tensor ch. 2: An array with any number of axes on a regular grid, written sans-serif bold.
tied weights ch. 9: A synonym for parameter sharing: the same kernel weight is applied at many input locations, so the weights are tied together.
tiled convolution ch. 9: A compromise (Gregor & LeCun): cycle through t different kernels across space (eq 9.10). t=1 is ordinary convolution; t = output width is a locally connected layer; memory grows only ×t.
time-delay neural network (tdnn) ch. 9: A 1-D convolutional network applied to time series (Lang & Hinton 1988) — the first CNNs trained with back-propagation, preceding LeCun's 2-D image CNNs.
toeplitz matrix ch. 9: The structured matrix of a 1-D discrete convolution: each row equals the row above shifted by one element (2-D convolution ↔ a doubly block circulant matrix).
topographic ica ch. 13: An ICA variant giving hidden units spatial coordinates and forming overlapping groups of neighbors, so nearby features share orientation/location/frequency; on images it learns Gabor filters and pooling gives translation invariance.
trace ch. 2: Tr(A) = Σᵢ A_{i,i}; transpose-invariant and cyclic-permutation-invariant — the workhorse of matrix calculus.
transfer learning ch. 8: Initialize a network with the first k layers of a net trained on a related task/dataset, then jointly fine-tune on the target task (often with fewer examples); §15.2.
transition operator ch. 17: T(x′|x): the conditional distribution giving the probability a Markov-chain update moves to x′ from x; as a matrix, A_{i,j}=T(i|j).
transpose ch. 2: Mirror of a matrix across its main diagonal: (Aᵀ)_{i,j} = A_{j,i}.
transpose convolution ch. 9: Multiplication by the transpose of the convolution matrix — the operation to back-propagate error to the inputs (and to reconstruct visibles in autoencoders/RBMs). One of the three ops (conv, kernel-gradient g, input-gradient h) needed to train a CNN.
triangulated (chordal) graph ch. 16: An undirected graph whose loops of length >3 all have a chord; a loop must be triangulated (chords added) before its undirected model can be converted to a directed one.
under-constrained problem ch. 7: A problem where XᵀX is singular or the optimum diverges (e.g. logistic regression on linearly separable data); regularization (invert XᵀX+αI, or weight decay) makes it well-defined and convergent.
undercomplete autoencoder ch. 14: An autoencoder whose code dimension is smaller than its input dimension, forcing the copy task to keep only the most salient features of the data; with a linear decoder and MSE it learns PCA's principal subspace.
underfitting ch. 5: Training error itself is too high — the model can't fit the training set.
underflow ch. 4: Numbers near zero rounding to exactly zero — then division and log blow up (÷0, log 0 = −∞).
undirected model ch. 3: Graphical model with undirected edges: p(x) = (1/Z) Π φ⁽ⁱ⁾(C⁽ⁱ⁾) — non-negative clique factors, normalized by Z.
unfolding (a computational graph) ch. 10: Expanding a recurrent definition s^(t)=f(s^(t-1);θ) into a repeated, acyclic computational graph — one copy of the shared function f per time step. Unfolding gives a fixed input size and one reusable transition model regardless of sequence length.
unit vector ch. 2: A vector with ‖x‖₂ = 1.
universal approximation theorem ch. 6: (Hornik/Cybenko 1989) one hidden layer with a squashing (or ReLU) activation can approximate any Borel-measurable function to any nonzero error given enough units — representation, not learnability, and with no size bound.
unsupervised pretraining ch. 15: The greedy layer-wise protocol (Algorithm 15.1): train each layer with an unsupervised learner (RBM, autoencoder, sparse coding) on the previous layer's output, freeze, repeat; optionally fine-tune the whole stack with a supervised criterion. Launched the 2006 deep learning renaissance; survives today mainly as NLP word-vector pretraining.
update gate ch. 10: The GRU sigmoid gate u that acts as a conditional leaky integrator — choosing per dimension whether to copy the old state or replace it with the new target state (combining the LSTM’s forget and input roles).
v-structure (collider) ch. 16: Two variables that are both parents of a common child with no direct edge between them; the only path type activated (not blocked) by observing the child.
v1 / primary visual cortex ch. 9: The first cortical area doing advanced visual processing; a spatial map of simple and complex cells that inspired the conv layer's detector→pooling structure (light flows retina→LGN→V1).
valid convolution ch. 9: No zero-padding: the kernel visits only positions where it lies entirely within the input; output shrinks to width m−k+1.
validation set ch. 5: A split of the TRAINING data (~20%) held out to select hyperparameters; the test set stays untouched until the end.
vanishing and exploding gradients ch. 8: Repeated multiplication by W gives W^t = V diag(λ)^t V⁻¹, so eigenvalues not near |1| vanish (<1) or explode (>1); plagues deep/recurrent graphs (same W each step), making learning stuck or unstable.
variance ch. 3: E[(f(x) − E[f(x)])²]: spread around the mean; its square root is the standard deviation.
variational autoencoder ch. 20: A deep generative model whose encoder network performs learned approximate inference: the inference net simply defines the ELBO L, and its parameters are trained to increase L — no explicit inference targets needed. Full treatment in §20.10.3.
variational free energy ch. 19: The negative of the ELBO; the 'free energy' name from statistical physics for the same quantity.
variational inference ch. 19: Maximize the ELBO over a RESTRICTED family of q (specifying only how q factorizes); the optimization derives q's optimal form. Minimizes the reverse KL D_KL(q‖p).
vector ch. 2: An ordered 1-D array of numbers, written bold lowercase (x ∈ ℝⁿ); a point in space, one coordinate per axis.
virtual adversarial example ch. 7: An adversarial example generated against the model OWN predicted label (not the true label), enabling semi-supervised adversarial training along the data manifold.
wake-sleep ch. 19: Train an inference network by sampling (h,v) from the model (ancestral sampling) and learning the reverse map v→h; supplies the missing supervised targets, but only trains on model samples. A candidate account of dreaming.
walk-back ch. 20: A training procedure for denoising-autoencoder generative chains that performs several stochastic encode-decode steps from a data point, removing spurious modes far from the data more efficiently.
weight decay ch. 5: The L² regularizer: J = MSE + λwᵀw; trades data fit against small weights; = MAP with a Gaussian prior; = ch4's constrained optimization.
weight-scaling inference rule ch. 7: Approximate the dropout ensemble in ONE forward pass by multiplying each unit outgoing weights by its keep-probability (≈ ÷2). Makes the expected test-time input to a unit match training; EXACT for softmax/linear nets, an approximation for deep nonlinear ones.
weight-space symmetry ch. 8: A source of non-identifiability: permuting the m×n hidden units (n!^m ways) or, for ReLU/maxout, scaling a unit's incoming weights by α and outgoing by 1/α gives an equivalent network — so equivalent minima abound.
whitening (sphering) ch. 12: Rescaling the principal components of data to equal variance, so a fitted Gaussian has spherical contours. Distinct from GCN (which maps examples onto a sphere); also called sphering.
width ch. 6: The dimensionality of a hidden layer — how many parallel unit/neuron activations it holds.
word embedding ch. 12: A learned distributed representation that maps each word (originally a one-hot vector, all words equidistant) to a point in a lower-dimensional feature space where words appearing in similar contexts are close — enabling generalization across semantically similar words.
zero-padding ch. 9: Implicitly padding the input with zeros so the output size and kernel width can be chosen independently; without it the representation shrinks by k−1 each layer.
zero-shot learning ch. 15: Learning with NO labeled examples of the target task, by modeling p(y | x, T) with a task description T represented so that generalization is possible (learned embeddings, not one-hot task codes) — read that cats have four legs and pointy ears, then recognize one.