Bayesian Inference

Definition

A statistical framework that treats model parameters as random variables and updates a prior belief distribution with observed data to produce a posterior distribution using Bayes’ theorem.

Intuition

Frequentist inference asks “what does this data say about a fixed unknown parameter?” Bayesian inference asks “after seeing this data, what should I believe about the parameter?” The prior encodes what is known before the data; the likelihood encodes what the data says; Bayes’ theorem combines them into the posterior.

Formal Description

Bayes’ theorem for inference:

$$\underset{\text{posterior}}{\underbrace{P(\theta \mid \mathcal{D})}}=\frac{\underset{\text{likelihood}}{\underbrace{P(\mathcal{D}\mid \theta )}}\cdot \underset{\text{prior}}{\underbrace{P(\theta )}}}{\underset{\text{evidence}}{\underbrace{P(\mathcal{D})}}}$$

The evidence $P(\mathcal{D})=\int P(\mathcal{D}\mid \theta )P(\theta )d\theta$ normalises the posterior but is often intractable.

Maximum Likelihood Estimation (MLE): finds the $\theta$ that maximises the likelihood, ignoring the prior:

$${\hat{\theta }}_{\mathrm{MLE}}=\arg \underset{\theta }{\max }P(\mathcal{D}\mid \theta )=\arg \underset{\theta }{\max }\sum _i\log p(x_i\mid \theta )$$

Maximum A Posteriori (MAP): finds the mode of the posterior:

$${\hat{\theta }}_{\mathrm{MAP}}=\arg \underset{\theta }{\max }\log P(\theta \mid \mathcal{D})=\arg \underset{\theta }{\max }\left[\log P(\mathcal{D}\mid \theta )+\log P(\theta )\right]$$

MAP with a Gaussian prior is equivalent to L2-regularised MLE (the prior acts as a regulariser).

Posterior predictive distribution:

$$P(\tilde{x}\mid \mathcal{D})=\int P(\tilde{x}\mid \theta )P(\theta \mid \mathcal{D})d\theta$$

Integrates over parameter uncertainty — more principled than plugging in a point estimate.

Conjugate Priors

A prior $P(\theta )$ is conjugate to a likelihood $P(\mathcal{D}\mid \theta )$ if the posterior is in the same distributional family as the prior. This makes updates analytical:

Likelihood	Conjugate prior	Posterior
Bernoulli($p$)	Beta($\alpha ,\beta$)	Beta($\alpha +k,\beta +n-k$)
Poisson($\lambda$)	Gamma($\alpha ,\beta$)	Gamma($\alpha +\sum x_i,\beta +n$)
Gaussian (known $\sigma$)	Gaussian(${\mu }_0,{\sigma }_0^2$)	Gaussian
Categorical($\mathbf{\pi }$)	Dirichlet($\mathbf{\alpha }$)	Dirichlet($\mathbf{\alpha }+\mathbf{c}$)

Beta-Binomial example: after observing $k$ successes in $n$ trials with prior Beta($\alpha ,\beta$):

$$P(p\mid k,n)=\mathrm{Beta}(\alpha +k,\beta +n-k)$$

The posterior mean is $(\alpha +k)/(\alpha +\beta +n)$, which interpolates between the prior mean and the MLE as $n$ grows.

Approximate Inference

When the posterior is intractable:

• MCMC (Markov Chain Monte Carlo): sample from the posterior (Metropolis-Hastings, Hamiltonian MC via Stan/PyMC).
• Variational inference (VI): approximate the posterior with a tractable family $q(\theta )$ by minimising $\mathrm{KL}(q\Vert p)$.
• Laplace approximation: fit a Gaussian at the MAP estimate.

Applications

• Naive Bayes classifier: uses class-conditional independence to compute $P(y\mid \mathbf{x})\propto P(y){\prod }_{j}P(x_j\mid y)$.
• Bayesian linear regression: places a Gaussian prior on weights; posterior is analytical and gives uncertainty estimates over predictions.
• Topic models (LDA): Dirichlet-Multinomial hierarchy.
• Bayesian hyperparameter optimisation (Gaussian processes): prior over functions, update with evaluated points.
• Online learning: sequential Bayesian updates as new data arrives.

Trade-offs

Approach	Advantage	Limitation
Full Bayesian	Calibrated uncertainty, principled	Expensive (MCMC/VI), prior choice matters
MAP	Cheap, equivalent to regularised MLE	Point estimate, no uncertainty
MLE	Simplest, unbiased in large $n$	Overconfident, no regularisation

• Prior choice can dominate with small datasets; with large datasets the likelihood overwhelms the prior and results converge to MLE.

Links

• Probability Theory
• Probability Distributions
• Hypothesis Testing
• Bias–Variance Analysis
• Probabilistic Models