Reader’s guide

What this book covers¶

Bayesian deep learning (BDL) asks how the machinery of Bayesian inference—priors, posteriors, predictive distributions, marginal likelihoods, and the decisions they license—can be brought to bear on models with millions or billions of parameters, for which none of the classical guarantees or algorithms apply unmodified. This book covers that question in ten parts. Five of them develop the major families of approximate inference used in practice: Monte Carlo sampling (Part I), Laplace approximations (Part II), variational inference (Part III), ensembles (Part IV), and kernel and Gaussian-process methods (Part V). Two cover the modelling questions that are assumed as prerequisites by the aforementioned algorithms: what a prior over a neural network actually means (Part VI), and what the symmetries and non-identifiability of neural parameterisations do to inference and interpretation (Part VII). One is devoted to making all of this run at modern scale (Part VIII), one to worked applications and software (Part IX), and the last to topics at the current research frontier, from diffusion models and singular learning theory to causal, credal, and reinforcement-learning extensions (Part X). The parts are largely parallel rather than sequential: they are alternative and complementary answers to the same problem, and comparing them is a large part of what the book is for.

How the book is organised¶

Several deliberate choices make non-linear reading possible, and it is worth knowing about them before you start.

Parts have entry chapters. Most parts open with an introductory chapter that sets up the problem, fixes the notation for that family of methods, and previews the chapters that follow. These entry chapters are short, self-contained, and written for readers arriving cold. If you want to know what a part is about without committing to it, read only its entry chapter.

Notation is shared across the whole book. A single set of symbols is used throughout, defined by shared macros and summarised in the mathematical glossary in the front matter. Whenever a chapter writes $\theta$ for network parameters, $f(x,\theta)$ for the network, or $\mathcal{D}$ for the training data, it means the same thing as every other chapter. If an unfamiliar symbol appears, the glossary is the first place to look.

Chapters are cross-referenced, not chained. Where a chapter uses a result developed elsewhere, it says so and points to the specific chapter. Treat those pointers as hyperlinks rather than as instructions: follow them when you need the derivation, and ignore them when you are happy to take the result on trust.

Coverage is broad by design. The book is a collaborative work by many authors, and different chapters are pitched at different levels—some are tutorial, some survey a literature, some present a specific recent framework in depth. This is a feature for a reference work. It does mean that reading order should follow your interests rather than the page numbers.

Six chapters that unlock the rest¶

If you are new to the field, the fastest way in is not to start at Chapter 1 and read on. It is to read the following six entry chapters, which between them establish essentially all of the vocabulary the remaining fifty chapters rely on.

• Chapter 12, on the evidence lower bound and inference as optimisation. This is the single most widely reused chapter in the book.

• Chapter 1, on Monte Carlo, Markov chain Monte Carlo, Hamiltonian Monte Carlo, and convergence diagnostics.

• Chapter 7, on the Laplace approximation and why a Gaussian around a trained network is the cheapest useful posterior.

• Chapter 19, on deep ensembles and their relation to Bayesian model averaging.

• Chapter 23, on kernels, reproducing kernel Hilbert spaces, and Gaussian processes—the function-space view used throughout.

• Chapter 28, on what a prior over a neural network is and what it does.

With these six in hand, any other chapter in the book becomes accessible, and the per-chapter prerequisites in Tables 1 and 2 tell you what else, if anything, a given chapter expects.

The ten parts at a glance¶

Part I, Sampling methods (Chapters 1–6). Asymptotically exact posterior inference by simulation. Chapter 1 covers Monte Carlo, rejection and importance sampling, Metropolis–Hastings, Gibbs, MALA, HMC, diagnostics, and sequential Monte Carlo; Chapter 2 makes MCMC compatible with mini-batching (SGLD, SG-HMC, cyclical schedules); Chapter 3 treats likelihood-free inference for black-box simulators; Chapter 4 develops multilevel and parallel SMC samplers; Chapter 5 asks what low-precision arithmetic does to a Markov chain; and Chapter 6 covers gradient-based samplers on discrete spaces. Start at Chapter 1; the remaining five chapters can then be read in any order, and Chapter 3 is nearly independent of the rest.

Part II, Laplace approximations (Chapters 7–11). The cheapest route from an already-trained network to a posterior: a Gaussian centred at the optimum with curvature-derived covariance. Chapter 7 motivates and derives it; Chapter 8 turns it into a Gaussian process over functions and introduces the generalised Gauss–Newton matrix and its scalable approximations; Chapter 9 views the same object through the lens of Riemannian and information geometry; Chapter 10 extracts closed-form predictive and decision-theoretic quantities from it; and Chapter 11 uses the resulting marginal likelihood for model selection. Read the first two in order, then whichever of the last three you need.

Part III, Variational inference (Chapters 12–18). Inference recast as optimisation over a family of distributions. Chapter 12 derives the evidence lower bound; Chapter 13 develops the Gaussian mean-field workhorse, coordinate ascent, and the reparameterisation gradient; Chapter 14 shows that dropout at test time is a variational approximation; Chapter 15 exploits the geometry of distribution space and derives practical optimisers such as VON, VOGN, and IVON; Chapter 16 replaces the parametric family with interacting particles (SVGD and gradient flows); Chapter 17 moves the variational problem into function space; and Chapter 18 handles streaming and non-stationary data through filtering. Read Chapters 12–13 in order; the remaining five are independent branches.

Part IV, Ensemble methods (Chapters 19–22). The most widely deployed approach in practice, and the one with the lightest prerequisites. Chapter 19 explains why retraining from different initialisations approximates Bayesian model averaging; Chapter 20 combines ensembling with sampling to explore multimodal posteriors; Chapter 21 surveys applications from mixtures of experts to weather forecasting; and Chapter 22 gives PAC-Bayesian generalisation bounds for weighted majority votes. Readers who want to see something work before studying inference machinery can begin here.

Part V, Kernel methods (Chapters 23–27). The function-space tradition, and the bridge between deep networks and exact Bayesian inference. Chapter 23 covers feature maps, kernels, RKHS, and Gaussian processes; Chapter 24 learns the kernel with a neural network (deep kernel learning); Chapter 25 composes Gaussian processes into deep Gaussian processes; Chapter 26 develops deep kernel processes and machines together with the infinite-width limit; and Chapter 27 meta-learns deep kernels for small-data tasks. Chapter 23 first; the rest branch from it.

Part VI, Priors (Chapters 28–31). What a prior over weights actually says about functions. Chapter 28 surveys the main families and the role of the prior predictive; Chapter 29 studies their statistical properties, including tail behaviour, infinite-width limits, and posterior contraction; Chapter 30 constructs informative, data-driven priors; and Chapter 31 specifies priors directly in function space and matches weight-space priors to them. This part is short and unusually high-leverage: it explains a great many empirical observations reported elsewhere in the book.

Part VII, Identifiability and symmetries (Chapters 32–36). Neural parameterisations are massively redundant, and this has consequences. Chapter 32 catalogues weight-space symmetries and mode connectivity; Chapter 33 works out what they do to samplers and to convergence diagnostics; Chapter 34 turns the question around and learns invariances and inductive biases from data; Chapter 35 constructs deep generative models that are identifiable by design; and Chapter 36 analyses Bayesian mixtures of experts with softmax gating. Start at Chapter 32; the last three chapters are independent of one another.

Part VIII, Scalability (Chapters 37–42). Making the preceding methods affordable. Chapter 37 covers methods that produce uncertainty from one forward pass; Chapter 38 covers sparse, inducing-point, and spectral approximations for Gaussian processes; Chapter 39 restricts inference to the final layer; Chapter 40 scales the Laplace approximation through structured curvature, function-space duality, and iterative solvers; and Chapters 41 and 42 form a self-contained pair on Bayesian data compression, with and without quantisation. This is the part practitioners tend to need soonest.

Part IX, Applications (Chapters 43–47). Worked end-to-end problems. Chapter 43 covers Bayesian treatment of large language models, including Bayesian fine-tuning and model merging; Chapter 44 evaluates uncertainty quantification methods head-to-head through selective classification; Chapter 45 treats missing biomedical data with deep generative models; Chapter 46 develops hybrid Bayesian hierarchical and deep models for wildfire extremes and spread; and Chapter 47 surveys the software ecosystem. Any of these can be read early, as motivation. If you intend to implement something, Chapter 47 is worth reading first.

Part X, Topical developments (Chapters 48–56). Nine largely independent chapters at the research frontier: low-dimensional subspace inference (Chapter 48); a behavioural characterisation of what makes a predictor Bayesian (Chapter 49); diffusion models (Chapter 50); prior-data fitted networks and amortised in-context prediction (Chapter 51); singular learning theory and what it corrects about the standard asymptotic picture (Chapter 52); active learning and Bayesian experimental design (Chapter 53); causal-aware BDL (Chapter 54); credal sets and imprecise probability for separating aleatoric from epistemic uncertainty (Chapter 55); and Bayesian reinforcement learning (Chapter 56). Read whichever interests you; none of them requires another.

A map of the parts¶

Figure 1 shows how the parts depend on one another. It is a map, not a timetable: an arrow records that the target part will make more sense if you have seen the source, not that you are forbidden from starting elsewhere. Three parts have no incoming arrows and are therefore legitimate places to begin—sampling, variational inference, and kernel methods—and each corresponds to one of the three distinct ways this book views a posterior: as something to simulate, as something to optimise, and as something to specify directly over functions. Choosing a starting point among them is largely a matter of taste.

Figure 1. Recommended dependences between the ten parts. Solid arrows mark background the target part assumes; the dashed arrow marks material that enriches it without being required. The entry parts—Sampling methods, Variational inference, and Kernel methods—have no prerequisites and can be read first.

Chapter-level prerequisites¶

Tables 1 and 2 give, for every chapter, the small number of chapters worth reading first and a second list of chapters that deepen it. The first list is deliberately short: it names what a chapter genuinely leans on, not everything related to it. Chapters with an empty first column need nothing beyond a standard background in probability, linear algebra, and deep learning. Used together with the map in Figure 1, these tables let you assemble your own route: pick the chapter you actually want, take the transitive closure of its prerequisites, and read that—usually two or three chapters rather than forty.

Suggested routes through the book¶

The following itineraries are ways of reading the book that we have found coherent. Each is self-contained in the sense that its chapters supply one another’s prerequisites.

A first pass through the field (about twelve chapters). Chapters 12, 13, 1, 2, 7, 8, 19, 23, 24, 28, 39, 44. This covers all five families of approximate inference treated in the book, the prior, one scalable method, and one empirical comparison of the lot. It is also a reasonable skeleton for a one-semester graduate course, with Chapter 47 added for practical sessions.

I have a trained network and need calibrated uncertainty. Chapters 7, 8, 40, 39, 19, 37, 44, and 47. This is the shortest path from a point estimate to a usable posterior predictive, and it avoids retraining throughout.

Sampling and Monte Carlo. Chapters 1, 2, 4, 5, 6, 20, 16, and 33. The last two are the payoff: particle methods sit between sampling and optimisation, and the symmetry chapter explains why diagnostics on neural posteriors can mislead.

Variational and optimisation-based inference. Chapters 12, 13, 14, 15, 16, 17, 18, then 43 as a large-scale application.

The function-space view. Chapters 23, 24, 25, 26, 38, 31, 17, and 27. This route never leaves function space, and is the natural one for readers coming from Gaussian processes or spatial statistics.

Priors, model selection, and inductive bias. Chapters 28, 29, 30, 31, 11, and 34. The question running through all six is what we are entitled to assume before seeing data, and how much of it can be learned instead.

Theory and foundations. Chapters 52, 29, 32, 33, 35, 22, 49, and 55. These chapters ask what Bayesian inference means for singular, over-parameterised, non-identifiable models, and what guarantees survive.

Large models and modern systems. Chapters 43, 15, 39, 37, 41, 42, 51, and 47.

Uncertainty for decisions. Chapters 10, 44, 53, 55, and 56. The unifying theme is that a posterior earns its cost only through the decisions it improves: abstention, data acquisition, robustness under ambiguity, and exploration. \makeatletter \renewcommand{\thetable}{\thechapter.@arabic\c@table} \makeatother