Common distributions, and where they come from in deep learning

A quick reference, not a course: for each distribution, its pmf/pdf, mean and variance, and the one-line reason it is the default choice for a particular kind of output (from Probability notation and random variables).

Bernoulli($p$)

$$ p(x) = p^x(1-p)^{1-x}, \ x\in\{0,1\}, \qquad \mathbb{E}[X]=p, \qquad \operatorname{Var}(X)=p(1-p). $$

Intuition: Bernoulli is the distribution of a single yes/no outcome with no other structure, which is exactly a binary label; a classifier's sigmoid output is interpreted as the $p$ of a Bernoulli over $y$.

Categorical($p_1,\dots,p_K$)

$$ p(x=k) = p_k, \ \textstyle\sum_{k=1}^K p_k=1, \qquad \mathbb{E}[\mathbb{1}[X=k]]=p_k. $$

A softmax layer $p_k=\dfrac{e^{z_k}}{\sum_j e^{z_j}}$ turns arbitrary real logits $z\in\mathbb{R}^K$ into a valid Categorical parameter vector.

Intuition: Categorical is Bernoulli generalized from 2 outcomes to $K$; softmax maps unconstrained logits onto the probability simplex.

Gaussian $\mathcal{N}(\mu,\sigma^2)$

$$ p(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \qquad \mathbb{E}[X]=\mu, \qquad \operatorname{Var}(X)=\sigma^2. $$

Shows up as: regression targets, weight initialization, the VAE latent prior, diffusion noise.

Intuition: the Gaussian is the max-entropy distribution for a fixed mean and variance: the least you can assume beyond "this average, this spread."

Multivariate Gaussian $\mathcal{N}(\mu,\Sigma)$

$$ p(x) = \frac{1}{\sqrt{(2\pi)^d\det\Sigma}}\exp\!\left(-\tfrac12(x-\mu)^T\Sigma^{-1}(x-\mu)\right), \qquad \underbrace{\mu}_{d\times1}, \ \underbrace{\Sigma}_{d\times d}\text{ (symmetric, PSD)}. $$

$(x-\mu)^T\Sigma^{-1}(x-\mu)$ is the quadratic form from Quadratic forms and positive definiteness; $\Sigma$'s eigenvectors/eigenvalues are its principal axes/variances (Principal component analysis). Usually restricted to diagonal $\Sigma$ ($d$ numbers, not $\tfrac{d(d+1)}2$):

$$ \Sigma = \operatorname{diag}(\sigma_1^2,\dots,\sigma_d^2) \ \ \Longleftrightarrow\ \ p(x) = \prod_{i=1}^d p(x_i). $$

Intuition: a multivariate Gaussian is $d$ single Gaussians coupled by a covariance matrix; dropping the off-diagonal terms is the independence assumption, traded for tractability.

A couple more, briefly

$$ \textbf{Laplace}(\mu,b):\ p(x)=\tfrac1{2b}e^{-|x-\mu|/b}, \qquad \textbf{Uniform}(a,b):\ p(x)=\tfrac1{b-a} \text{ on } [a,b]. $$

Laplace: sharper peak, heavier tails than Gaussian; maximizing its log-likelihood minimizes $|x-\mu|$, so Laplace noise $\to$ L1 loss, the way Gaussian noise $\to$ L2 (MSE). Uniform: weight initialization ranges, augmentation parameters: every value a priori equally likely.

Intuition: every distribution here is a noise/shape assumption baked into a loss, made exact in Supervised learning as KL minimization.