Variance decomposition and the eigenbasis

Variance is not one number per dataset; it is one number per direction. This note: what $\operatorname{Var}(u^Tx)$ means, how it relates to eigenvalues, and why $\sum_i\operatorname{Var}(x_i)$ is fixed regardless of basis.

Variance along a direction is a quadratic form

Let $x\in\mathbb{R}^d$ be centered data ($\mathbb{E}[x]=0$) with covariance $\Sigma=\mathbb{E}[xx^T]$. Pick any unit direction $u$ ($\|u\|=1$) and project each point onto it: $z=u^Tx$, a scalar. The variance of that scalar is:

$$ \operatorname{Var}(u^Tx) = \mathbb{E}[(u^Tx)^2] = \mathbb{E}[u^Txx^Tu] = u^T\mathbb{E}[xx^T]u = u^T\Sigma u. $$

Intuition: $\Sigma$ encodes the spread of the data in every direction at once; squeezing it between $u^T$ and $u$ reads off the spread in one specific direction, exactly as in Quadratic forms and positive definiteness.

Re-expressing a direction in the eigenbasis

By the spectral theorem, $\Sigma=Q\Lambda Q^T$ with orthonormal eigenvectors $q_1,\dots,q_d$ and eigenvalues $\lambda_1\geq\cdots\geq\lambda_d\geq0$. Substitute and let $v=Q^Tu$, i.e. $u$'s own coordinates in the eigenbasis:

$$ u^T\Sigma u = u^T(Q\Lambda Q^T)u = (Q^Tu)^T\Lambda(Q^Tu) = v^T\Lambda v = \sum_{i=1}^d\lambda_iv_i^2. $$

Because $Q$ is a rotation, it preserves length: $\|v\|^2=v^Tv=u^TQQ^Tu=u^Tu=1$. So the weights $v_i^2$ sum to $1$, and $v_i=\cos\theta_i$ is exactly the cosine of the angle between $u$ and eigenvector $q_i$. The variance formula becomes:

$$ \operatorname{Var}(u^Tx) = \sum_{i=1}^d\lambda_i\cos^2\theta_i. $$

Intuition: every direction's variance is a weighted average of the eigenvalues, weighted by how much that direction aligns with each eigenvector.

Example

Take the running $\Sigma=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, eigenvalues $\lambda_1=3,\lambda_2=1$ with eigenvectors $q_1=\tfrac1{\sqrt2}(1,1)$, $q_2=\tfrac1{\sqrt2}(1,-1)$. For $u=(1,0)$, the angle to $q_1$ is $45^\circ$, so $\cos^2\theta_1=\cos^2\theta_2=\tfrac12$:

$$ \operatorname{Var}(u^Tx) = 3\cdot\tfrac12 + 1\cdot\tfrac12 = 2, $$

matching $u^T\Sigma u=\Sigma_{11}=2$ directly. As $u$ sweeps around the circle, $\operatorname{Var}(u^Tx)$ oscillates smoothly between $1$ and $3$, never outside that range:

The Rayleigh–Ritz bounds

Since $u^T\Sigma u=\sum_i\lambda_iv_i^2$ is a weighted mean of the eigenvalues with nonnegative weights $v_i^2$ summing to $1$, it can never fall outside the range spanned by the eigenvalues themselves. This is the Rayleigh–Ritz bound (the quantity $u^T\Sigma u$ for unit $u$ is called the Rayleigh quotient of $\Sigma$):

$$ \lambda_{\min} \ \leq\ u^T\Sigma u \ \leq\ \lambda_{\max} \qquad \text{for every unit vector } u. $$

The bounds are attained, not just approached. Setting $u=q_1$ collapses the weights to $v_1^2=1,\,v_{i>1}=0$, giving the largest possible mean, $\lambda_{\max}$; setting $u=q_d$ (the eigenvector of the smallest eigenvalue) gives $\lambda_{\min}$ the same way. Every other direction is a genuine compromise, an interpolation across the principal axes governed entirely by the direction cosines $\cos^2\theta_i$, strictly between the two extremes unless it happens to coincide with an edge eigenvector.

Intuition: $q_1,q_d$ realize the best and worst case, which is exactly why the top eigenvector solves PCA's "maximize variance" problem (Principal component analysis).

Example

For our running $\Sigma$, the bound reads $1\leq u^T\Sigma u\leq3$ for every unit $u$. The direction $u=(1,0)$ scored $2$, comfortably inside; $u=q_1=\tfrac1{\sqrt2}(1,1)$ scores exactly $3$; $u=q_2=\tfrac1{\sqrt2}(1,-1)$ scores exactly $1$. No direction can do better or worse than these two.

Rotating the data itself: the covariance diagonalizes

The previous section rotated the direction $u$ into eigen-coordinates. Rotating the data the same way, $y=Q^Tx$, has a cleaner consequence: the cross terms vanish entirely. The covariance of $y$ is:

$$ \Sigma_y = \mathbb{E}[yy^T] = \mathbb{E}[Q^Txx^TQ] = Q^T\mathbb{E}[xx^T]Q = Q^T\Sigma Q = Q^T(Q\Lambda Q^T)Q = \Lambda, $$

using $Q^TQ=I$. So in the eigenbasis, the covariance matrix is exactly diagonal: $\operatorname{Var}(y_i)=\lambda_i$ and $\operatorname{Cov}(y_i,y_j)=0$ for $i\neq j$.

Intuition: the eigenbasis is the one coordinate system where spread splits cleanly into independent pieces, one number per axis.

Total variance never changes

Summing variance over all original dimensions, $\sum_i\operatorname{Var}(x_i)=\operatorname{tr}(\Sigma)$. Summing over the eigenbasis instead, $\sum_i\operatorname{Var}(y_i)=\operatorname{tr}(\Lambda)=\sum_i\lambda_i$. These are the same number, by the cyclic property of the trace:

$$ \operatorname{tr}(\Sigma) = \operatorname{tr}(Q\Lambda Q^T) = \operatorname{tr}(\Lambda Q^TQ) = \operatorname{tr}(\Lambda) = \sum_{i=1}^d\lambda_i. $$

Intuition: rotating axes redistributes variance but cannot create or destroy it: the eigenbasis just splits that fixed total with no cross terms.

Example

$\operatorname{tr}(\Sigma)=2+2=4$; $\lambda_1+\lambda_2=3+1=4$. ✓ Same total, however you slice it.

Putting it together

$$ \underbrace{\operatorname{tr}(\Sigma)}_{\text{conserved total}} = \sum_i\lambda_i, \qquad \underbrace{\lambda_{\min}\leq u^T\Sigma u\leq\lambda_{\max}}_{\text{any direction}}, \qquad \underbrace{\Sigma_y=\Lambda}_{\text{eigenbasis: zero cross-terms}}. $$

PCA asks which direction claims the largest share of that fixed total (Principal component analysis).