Lebesgue measure, densities, and KL divergence

The measurable sets: a σ-algebra

A measure cannot consistently assign a size to every subset of $\Omega$ (the Vitali set, built with the axiom of choice, has no well-defined length). So fix in advance which subsets are allowed to be measured: a σ-algebra $\mathcal{F}$ on $\Omega$ is a collection of subsets with:

$$ \Omega\in\mathcal{F}, \qquad A\in\mathcal{F}\Rightarrow A^c\in\mathcal{F}, \qquad A_1,A_2,\dots\in\mathcal{F}\Rightarrow\bigcup_{i=1}^\infty A_i\in\mathcal{F}. $$

Elements of $\mathcal{F}$ are the events, the only sets $P$ or $\lambda$ ever measure. On $\mathbb{R}$, the standard choice is the Borel σ-algebra $\mathcal{B}(\mathbb{R})$: the smallest σ-algebra containing every open interval.

Intuition: not "can this set be measured" but "is it even eligible": closure under complement and countable union is exactly what countable additivity requires.

A measure is $\mu:\mathcal{F}\to[0,\infty]$ with $\mu(\varnothing)=0$ and countable additivity on disjoint sets; the triple $(\Omega,\mathcal{F},\mu)$ (or $(\Omega,\mathcal{F},P)$, a probability space) is a measure space. $\lambda$ lives on (the completion of) $\mathcal{B}(\mathbb{R})$, never on all of $2^{\mathbb{R}}$.

Why measure theory at all

Discrete: a mass $p(x)\geq0$ per point, $\sum_x p(x)=1$, $P(A)=\sum_{x\in A}p(x)$.

Continuous: uncountably many points, so any positive per-point mass sums to $\infty$. Fix: weigh sets by extent, not points. The Lebesgue measure $\lambda$ gives $[a,b]$ the measure $b-a$, and:

$$ \lambda(\{x\}) = 0 \quad \text{for every single point } x. $$

Intuition: $\lambda$ measures extent, and a point has none.

A probability measure is a density $p$ relative to $\lambda$:

$$ P(A) = \int_A p(x)\,d\lambda(x), \qquad p = \frac{dP}{d\lambda} \ \ \text{(the Radon-Nikodym derivative).} $$

Worked example: the Dirichlet function

$\mathbb{Q}$ is countable, $\mathbb{Q}=\{q_1,q_2,\dots\}$, so countable additivity gives:

$$ \lambda(\mathbb{Q}) = \sum_{i=1}^\infty\lambda(\{q_i\}) = \sum_{i=1}^\infty 0 = 0. $$

Take $D(x)=\mathbb{1}[x\in\mathbb{Q}]$, the Dirichlet function (nowhere continuous: every interval holds both rationals and irrationals). Its Lebesgue integral is immediate:

$$ \int_0^1 D(x)\,d\lambda(x) = 1\cdot\lambda(\mathbb{Q}\cap[0,1]) + 0\cdot\lambda(\mathbb{Q}^c\cap[0,1]) = 0. $$

Its Riemann integral does not exist: every upper sum is $1$, every lower sum is $0$, and refining the partition never closes that gap.

Intuition: a dense set can still weigh nothing: Lebesgue sees straight through a pathology Riemann cannot handle.

$P(X=x)=0$, yet $p(x)>0$

Same mechanism as $\lambda(\mathbb{Q})=0$, applied to one point:

$$ P(X=x) = \int_{\{x\}}p\,d\lambda = p(x)\cdot\lambda(\{x\}) = p(x)\cdot0 = 0, $$

regardless of how large $p(x)$ is: it can exceed $1$ (e.g. $\mathcal{N}(0,0.01)$ at $x=0$, Common distributions).

Intuition: $P$ is mass, $p$ is a rate; zero width always gives zero mass, whatever the rate.

One formula, two cases

Discrete and continuous differ only in the base measure: $\lambda$ for continuous, counting measure $\nu$ for discrete ($\nu(\{x\})=1$):

$$ P(A) = \int_A p\,d\mu \ \ \Longrightarrow\ \ \begin{cases} \sum_{x\in A}p(x), & \mu=\nu \ \text{(discrete)} \\[2pt] \int_A p(x)\,dx, & \mu=\lambda \ \text{(continuous)} \end{cases} $$

Intuition: $\int\!\cdot\,d\mu$ refuses to commit to discrete or continuous; swap $\mu$, get the right formula for free.

KL divergence

$$ KL(p\|q) = \mathbb{E}_{x\sim p}\!\left[\log\frac{p(x)}{q(x)}\right] = \int p(x)\log\frac{p(x)}{q(x)}\,d\mu(x). $$

Non-negative, by Jensen on the convex $-\log$:

$$ KL(p\|q) = \mathbb{E}_{x\sim p}\!\left[-\log\frac{q(x)}{p(x)}\right] \geq -\log\mathbb{E}_{x\sim p}\!\left[\frac{q(x)}{p(x)}\right] = -\log\!\int q\,d\mu = -\log 1 = 0. $$

$KL(p\|q)\neq KL(q\|p)$ and no triangle inequality: a divergence, not a distance.

Intuition: averaging $\log(p/q)$ under $p$ makes $p$'s own frequent places dominate the score.

Forward vs. reverse KL

• Forward, $KL(p\|q)$, mode-covering. Averaged under $p$: $q(x)\to0$ where $p(x)>0$ blows up the integrand, so $q$ must cover every mode of $p$.
• Reverse, $KL(q\|p)$, mode-seeking. Averaged under $q$: ignoring a mode of $p$ costs nothing, but mass where $p(x)\to0$ is punished, so $q$ locks onto one mode.

Intuition: forward KL fears zeros in $q$; reverse KL fears its own mass landing where $p$ is zero.

Forward KL is MLE (Supervised learning as KL minimization); reverse KL drives variational methods (Generative modeling).