Probability Distributions

Definition

A probability distribution specifies how probability mass (discrete) or density (continuous) is assigned over the possible values of a random variable.

Intuition

Distributions are the vocabulary of probabilistic modelling. Different processes generate data with characteristic shapes: coin flips are Bernoulli, counts of rare events are Poisson, natural measurements often cluster near a mean (Gaussian), and proportions live on the unit interval (Beta).

Formal Description

Discrete Distributions

Bernoulli — single binary trial with success probability $p$:

$$P(X=k)=p^k(1-p{)}^{1-k},k\in \{0,1\},\mathbb{E}[X]=p,\mathrm{Var}(X)=p(1-p)$$

Binomial — number of successes in $n$ independent Bernoulli($p$) trials:

$$P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k},\mathbb{E}[X]=np,\mathrm{Var}(X)=np(1-p)$$

Poisson — number of events in a fixed interval at rate $\lambda$:

$$P(X=k)=\frac{{\lambda }^k{e}^{-\lambda }}{k!},\mathbb{E}[X]=\lambda ,\mathrm{Var}(X)=\lambda$$

Categorical — generalisation of Bernoulli to $K$ categories with probabilities $({\pi }_1,\dots ,{\pi }_K)$, ${\sum }_k{\pi }_k=1$.

Continuous Distributions

Uniform $\mathrm{Unif}(a,b)$:

$$f(x)=\frac{1}{b-a},\mathbb{E}[X]=\frac{a+b}{2},\mathrm{Var}(X)=\frac{(b-a{)}^2}{12}$$

Gaussian (Normal) $\mathcal{N}(\mu ,{\sigma }^2)$ — the most important distribution in statistics:

$$f(x)=\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right),\mathbb{E}[X]=\mu ,\mathrm{Var}(X)={\sigma }^2$$

Standard normal: $Z=(X-\mu )/\sigma \sim \mathcal{N}(0,1)$.

Multivariate Gaussian $\mathcal{N}(\mathbf{\mu },\mathbf{\Sigma })$:

$$f(\mathbf{x})=\frac{1}{(2\pi {)}^{d/2}|\mathbf{\Sigma }{|}^{1/2}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^{\top }{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })\right)$$

$\mathbf{\Sigma }$ is the covariance matrix; marginals and conditionals of joint Gaussians are Gaussian.

Exponential $\mathrm{Exp}(\lambda )$ — time until first event at rate $\lambda$, memoryless:

$$f(x)=\lambda e^{-\lambda x},x\ge 0,\mathbb{E}[X]=1/\lambda ,\mathrm{Var}(X)=1/{\lambda }^2$$

Gamma $\mathrm{Gamma}(\alpha ,\beta )$ — sum of $\alpha$ independent $\mathrm{Exp}(\beta )$ variables; conjugate prior for Poisson rate:

$$f(x)=\frac{{\beta }^{\alpha }}{\Gamma (\alpha )}{x}^{\alpha -1}{e}^{-\beta x},x>0$$

Beta $\mathrm{Beta}(\alpha ,\beta )$ — distribution over $[0,1]$; conjugate prior for Bernoulli/Binomial $p$:

$$f(x)=\frac{x^{\alpha -1}(1-x{)}^{\beta -1}}{B(\alpha ,\beta )},x\in [0,1]$$

Symmetric when $\alpha =\beta$; $\alpha =\beta =1$ is Uniform; concentrates near extremes when $\alpha ,\beta <1$.

Dirichlet $\mathrm{Dir}(\mathbf{\alpha })$ — multivariate generalisation of Beta; conjugate prior for Categorical/Multinomial:

$$f(\mathbf{x})\propto \prod _{k=1}^K{x}_k^{{\alpha }_k-1},\sum _k{x}_k=1,x_k\ge 0$$

Central Limit Theorem

For i.i.d. $X_1,\dots ,X_n$ with mean $\mu$ and variance ${\sigma }^2<\infty$:

$$\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}\stackrel{d}{\to }\mathcal{N}(0,1)\text{as }n\to \infty$$

Justifies Gaussian approximations in large-sample inference.

Exponential Family

Many distributions share the form:

$$f(x\mid \mathbf{\eta })=h(x)\exp \left({\mathbf{\eta }}^{\top }T(x)-A(\mathbf{\eta })\right)$$

where $\mathbf{\eta }$ are natural parameters, $T(x)$ are sufficient statistics, and $A(\mathbf{\eta })$ is the log-partition function. Conjugate priors exist for exponential family likelihoods. Examples: Gaussian, Bernoulli, Poisson, Gamma, Beta, Dirichlet.

Applications

• Gaussian: linear regression errors, natural measurements, approximate posteriors
• Bernoulli/Binomial: classification, A/B testing
• Poisson: count data in insurance (claims frequency), NLP (word counts)
• Beta/Dirichlet: Bayesian priors for probabilities and topic models (LDA)
• Exponential/Gamma: survival analysis, insurance claim severity

Trade-offs

• The Gaussian assumption is often violated in practice (heavy tails, skewness); always check empirically before assuming normality.
• Conjugate priors give analytical posteriors but may not match domain knowledge; non-conjugate priors require MCMC or variational inference.

Links

• Probability Theory
• Bayesian Inference
• Evaluation Metrics
• Probabilistic Models
• Loss Functions