Quadratic forms and positive definiteness

Ketan Suhaas Saichandran

linear-algebraoptimizationquadratic-formspositive-definitenesshessianeigenvalues

Quadratic forms

$$ q(x) = x^TAx, \quad A \text{ symmetric (WLOG: the antisymmetric part of } A \text{ always contributes } 0). $$

Shapes

$$ q(x)=\underbrace{x^T}_{1\times n}\ \underbrace{A}_{n\times n}\ \underbrace{x}_{n\times1}=\underbrace{q(x)}_{1\times1} =\sum_{i,j}A_{ij}\,x_ix_j. $$

Squeezed between a row and a column, the $n\times n$ matrix collapses to a single number: $Ax$ is a vector ($n\times1$), and $x^T$ dots it with $x$ to give a scalar.

Intuition: $q$ traces a bowl, dome, or saddle; $A$'s eigenvalues are the curvature along each principal axis.

Example

Reusing $A = \begin{pmatrix}2&1\\1&2\end{pmatrix}$:

$$ q(x) = 2x_1^2 + 2x_1x_2 + 2x_2^2, \quad q(1,1) = 2+2+2 = 6. $$

Diagonalizing a quadratic form

Using the spectral theorem $A=Q\Lambda Q^T$, substitute $y = Q^Tx$ (coordinates in the eigenbasis):

$$ q(x) = x^TQ\Lambda Q^Tx = y^T\Lambda y = \sum_i \lambda_iy_i^2. $$

Shapes

$$ \underbrace{y}_{n\times1}=\underbrace{Q^T}_{n\times n}\underbrace{x}_{n\times1}, \qquad q=\underbrace{y^T}_{1\times n}\underbrace{\begin{pmatrix}\lambda_1&&\\&\ddots&\\&&\lambda_n\end{pmatrix}}_{\Lambda,\ n\times n}\underbrace{y}_{n\times1}. $$

Intuition: in the eigenbasis, a quadratic form is just a weighted sum of squares, no cross terms.

Example

For $x=(1,1)$: $y = Q^Tx = (\sqrt2, 0)$, so $q(x) = 3(\sqrt2)^2 + 1(0)^2 = 6$. ✓

Positive definiteness

$A$ is positive definite if $q(x)>0$ for every $x\neq0$. From the sum-of-squares form, that holds iff every $\lambda_i>0$ (semidefinite allows $\lambda_i\geq0$; indefinite means mixed signs, a saddle).

Quick test without eigenvalues, for $A=\begin{pmatrix}a&b\\b&d\end{pmatrix}$: positive definite iff $a>0$ and $ad-b^2>0$.

Intuition: positive definite means every direction curves upward, a genuine bowl with one bottom.

Example

Our $A$: $a=2>0$, $ad-b^2=4-1=3>0$, positive definite, matching eigenvalues $3,1>0$.

Contrast $B=\begin{pmatrix}0&1\\1&0\end{pmatrix}$: $a=0$ fails immediately; eigenvalues are $\pm1$, indefinite (a saddle). Note $B$ shares $A$’s eigenvectors, since $A=2I+B$.

Level sets of $q(x)=x^TAx$ for our running $A$ are ellipses aligned with $q_1,q_2$; the semi-axis along $q_i$ has length $\propto 1/\sqrt{\lambda_i}$, so the curve is tightest along the largest eigenvalue.

For the indefinite $B=\begin{pmatrix}0&1\\1&0\end{pmatrix}$ (same eigenvectors as $A$, eigenvalues $+1,-1$), level sets are hyperbolas opening along $q_1$ for $c>0$ and along $q_2$ for $c<0$: a saddle, not a bowl.

The Hessian and convexity

Near a critical point $x_0$ (where $\nabla f(x_0)=0$), a smooth $f$ looks like its own quadratic form:

$$ f(x_0+\Delta x) \approx f(x_0) + \tfrac12 \Delta x^TH\Delta x, \quad H = \nabla^2f(x_0) \text{ (the Hessian)}. $$

Shapes

$$ \underbrace{\Delta x^T}_{1\times n}\ \underbrace{H}_{n\times n}\ \underbrace{\Delta x}_{n\times1} = \underbrace{\text{scalar}}_{1\times1}, \qquad H = H^T \ (n\times n\text{, symmetric}). $$

Intuition: the Hessian is just another $A$ in $x^TAx$, built from an $n\times n$ grid of second partial derivatives, one feature pair at a time.

$H$ positive definite: strict local minimum. $H$ negative definite: local maximum. $H$ indefinite: saddle point. $f$ is convex everywhere iff $H(x)$ is positive semidefinite everywhere.

Intuition: close to a critical point, the Hessian's definiteness is the only thing that decides what kind of point it is.

Example

$f(x_1,x_2) = x_1^2+x_1x_2+x_2^2$ has Hessian $H=\begin{pmatrix}2&1\\1&2\end{pmatrix}=A$, our running example, positive definite. So $x=0$ is a strict (global) minimum of $f$.