Generative modeling: approximating pθ

Ketan Suhaas Saichandran

information-theorydeep-learninggenerative-modelingkl-divergencemaximum-likelihood

Same objective, one level up

Now $x$ itself (image, sentence, molecule) is drawn from $p_{\text{data}}(x)$, and the model is a distribution $p_\theta(x)$. Swap $y\mid x\to x$ in Supervised learning as KL minimization; forward KL is still MLE:

$$ \min_\theta\ KL\big(p_{\text{data}}(x)\,\|\,p_\theta(x)\big) \ \Longleftrightarrow\ \max_\theta\ \mathbb{E}_{x\sim p_{\text{data}}}\big[\log p_\theta(x)\big]. $$

Intuition: the objective was never the hard part; $p_\theta(x)$ being intractable to evaluate or sample is. Every method below is a different trick for making $\max_\theta\mathbb{E}[\log p_\theta(x)]$, or a stand-in for it, computable.

A map of the methods

Autoregressive models (e.g. language models): factor $p_\theta(x)=\prod_i p_\theta(x_i\mid x_{<i})$ by the chain rule (Probability notation and random variables). Exact likelihood, directly optimized by forward KL/MLE; no approximation needed, at the cost of sequential generation.
Normalizing flows: an invertible $x=f_\theta(z)$ from a simple base density; change of variables gives $p_\theta(x)$ exactly via a Jacobian determinant (Gradients, Jacobians, and Hessians). Also exact likelihood, but the architecture must stay invertible.
Variational autoencoders (VAEs): $p_\theta(x)=\int p_\theta(x\mid z)p(z)\,dz$ is intractable, so introduce $q_\phi(z\mid x)$ and optimize the ELBO, a tractable lower bound on $\log p_\theta(x)$ built around $KL(q_\phi(z\mid x)\,\|\,p(z))$. Approximates the marginal likelihood via a bound; that inner KL is a reverse-KL, variational-inference term (Lebesgue measure and KL divergence), and training needs a gradient through the sampling step $z\sim q_\phi(z\mid x)$ (The reparameterization trick).
GANs: no explicit $p_\theta(x)$ at all, just a sampler, trained adversarially against a discriminator. At the optimal discriminator the implicit objective relates to the Jensen–Shannon divergence; still "match $p_\theta$ to $p_{\text{data}}$", but with likelihood evaluation traded for sample quality and a different (symmetrized) divergence than KL.
Diffusion / score-based models: sidestep $p_\theta(x)$ by learning the score $\nabla_x\log p_\theta(x)$ instead (equivalently, denoising at many noise levels). Training reduces to a tractable regression loss that is provably bounded in relation to the log-likelihood; approximate MLE without ever forming $p_\theta(x)$ itself.

Exact factorization, an invertible change of variables, a variational bound, a different divergence, a score instead of a density: five disguises for the same project, fitting $p_\theta$ to $p_{\text{data}}$.