Gradients, Jacobians, and Hessians

Ketan Suhaas Saichandran

linear-algebradeep-learningoptimizationgradientshessiancalculus

The gradient

For a scalar function $f:\mathbb{R}^n\to\mathbb{R}$, the gradient collects all $n$ partial derivatives into a vector, same shape as $x$:

$$ \nabla f(x) = \begin{pmatrix}\partial f/\partial x_1\\ \vdots \\ \partial f/\partial x_n\end{pmatrix} \in \mathbb{R}^n, \quad f(x+\epsilon v) \approx f(x) + \epsilon\,\nabla f(x)^Tv. $$

Intuition: the gradient is "every partial derivative, stacked as a vector"; it points in the direction $f$ increases fastest.

Shapes

$$ \underbrace{\nabla f(x)}_{n\times1}\in\mathbb{R}^n, \qquad \underbrace{\nabla f(x)^T}_{1\times n}\,\underbrace{v}_{n\times1}=\underbrace{\nabla f(x)^Tv}_{1\times1}\in\mathbb{R}. $$

Intuition: the gradient is a column, so it lives in the same space as $x$; transposed into a row it eats a direction $v$ and returns a single number, how fast $f$ changes that way.

Example

$f(x_1,x_2)=x_1^2+x_1x_2+x_2^2$ (from Quadratic forms): $\nabla f = (2x_1+x_2,\ x_1+2x_2)$. At $(1,1)$: $\nabla f=(3,3)$.

The gradient at $x$ is perpendicular to the level curve through $x$ ($c_1<c_2<c_3$) and points toward increasing $f$.

The Jacobian

For a vector-valued $g:\mathbb{R}^n\to\mathbb{R}^m$, the Jacobian stacks one gradient per output, as rows:

$$ J \in \mathbb{R}^{m\times n}, \quad J_{ij} = \partial g_i/\partial x_j, \quad g(x+\Delta x) \approx g(x) + J\Delta x. $$

Shape rule: rows are outputs, columns are inputs. A scalar function is the case $m=1$: its Jacobian is a $1\times n$ row vector, and $\nabla f := J^T$ by convention, a column, so it lives in the same space as $x$ (handy for $x \leftarrow x-\eta\nabla f$).

Intuition: $J$ is the single matrix that best approximates $g$ linearly near $x$, exactly like a linear map's matrix.

Example

$g(x_1,x_2) = (x_1+x_2,\ x_1x_2,\ x_1^2)$, a map $\mathbb{R}^2\to\mathbb{R}^3$:

$$ J = \begin{pmatrix}1&1\\x_2&x_1\\2x_1&0\end{pmatrix} \in \mathbb{R}^{3\times2}, \quad J(1,1) = \begin{pmatrix}1&1\\1&1\\2&0\end{pmatrix}. $$

Shapes

$$ \underbrace{J}_{m\times n}\,\underbrace{\Delta x}_{n\times1}=\underbrace{J\Delta x}_{m\times1}\in\mathbb{R}^m. $$

Intuition: $J$ eats an $n$-dimensional nudge in the input and returns an $m$-dimensional nudge in the output; the shared inner $n$ is the input dimension disappearing, exactly like any matrix-vector product.

Near $x$, a smooth $g$ acts like the linear map $J$: a small disk maps to a stretched, rotated ellipse, the local-linearization idea used throughout linear algebra. (Shown for a square $J$, $m=n=2$.)

Directional derivatives, JVPs, and VJPs

The directional derivative of $f$ at $x$ along a unit vector $v$ reduces an $n$-dimensional problem to ordinary single-variable calculus along the line $x+hv$, and it's exactly the earlier gradient-dot-direction formula:

$$ D_vf(x) := \lim_{h\to0}\frac{f(x+hv)-f(x)}{h} = \nabla f(x)^Tv. $$

Intuition: this is the actual reason gradient descent moves along $-\nabla f$, it is provably the single steepest downhill direction, not just a reasonable-sounding one.

For vector-valued $g$, the same object $J v$ is the directional derivative of every output of $g$ at once, called a Jacobian-vector product (JVP):

Shapes

$$ \underbrace{u^T}_{1\times m}\,\underbrace{J}_{m\times n}=\underbrace{u^TJ}_{1\times n}, \qquad\text{vs.}\qquad \underbrace{J}_{m\times n}\,\underbrace{v}_{n\times1}=\underbrace{Jv}_{m\times1}. $$

$Jv$ (a JVP) is what forward-mode autodiff propagates layer by layer, $J_k v = J_h(J_gv)$, cheap when there are few inputs to differentiate against. $u^TJ$ (a vector-Jacobian product, VJP) is the dual object, $u^TJ_k=(u^TJ_h)J_g$, propagated backward from a scalar loss ($u=1$ at the output); this row-vector-backward pass is exactly backpropagation from the Chain rule and shapes section above, now named.

Intuition: a JVP asks "if the input moves this way, how does the output move?"; a VJP asks "if I care about the output this way, how should I adjust the input?" Forward-mode computes the first, reverse-mode (backprop) computes the second; deep learning almost always wants the second, since a scalar loss has exactly one output to seed from.

The Hessian

$\nabla f$ is itself a map $\mathbb{R}^n\to\mathbb{R}^n$ (gradient in, gradient out), so it has its own Jacobian:

$$ H = \nabla^2f(x) := J_{\nabla f} \in \mathbb{R}^{n\times n}, \quad H_{ij} = \partial^2f/\partial x_i\partial x_j. $$

Symmetric for smooth $f$ (mixed partials commute), which is exactly the symmetric $A$ from Quadratic forms.

Intuition: the Hessian isn't a new object, it's just the Jacobian of the gradient map.

Example

Same $f$ as above, $\nabla f = (2x_1+x_2,\ x_1+2x_2)$: taking its Jacobian,

$$ H = \begin{pmatrix}2&1\\1&2\end{pmatrix} = A, \quad \text{our running matrix, exactly as in Quadratic forms.} $$

Shapes

$$ \underbrace{v^T}_{1\times n}\,\underbrace{H}_{n\times n}\,\underbrace{v}_{n\times1}=\underbrace{v^THv}_{1\times1}\in\mathbb{R}. $$

Intuition: squeezed between a vector and its transpose, $H$ turns a direction $v$ into the curvature of $f$ along that direction, the exact quadratic form from Quadratic forms, built here from second derivatives.

When $H\succ0$ the level curves close up around a local minimum; when $H$ is indefinite they open up into a saddle, curving up along one eigendirection and down along another.

The shape rule

One rule covers all three: a derivative's shape is always (number of outputs) $\times$ (number of inputs).

$f:\mathbb{R}^n\to\mathbb{R}$: gradient, $n\times1$ (or its transpose, $1\times n$)
$g:\mathbb{R}^n\to\mathbb{R}^m$: Jacobian, $m\times n$
$\nabla f:\mathbb{R}^n\to\mathbb{R}^n$: Hessian, $n\times n$

Side by side, with the outputs down the rows and the inputs across the columns:

$$ \underbrace{\nabla f}_{n\times1}=\begin{pmatrix}\partial_1f\\\vdots\\\partial_nf\end{pmatrix}, \qquad \underbrace{J}_{m\times n}=\begin{pmatrix}-\,\nabla g_1^T-\\\vdots\\-\,\nabla g_m^T-\end{pmatrix}=\begin{pmatrix}\partial_1g_1&\cdots&\partial_ng_1\\\vdots&&\vdots\\\partial_1g_m&\cdots&\partial_ng_m\end{pmatrix}, $$

$$ \underbrace{H}_{n\times n}=\begin{pmatrix}\partial_1\partial_1f&\cdots&\partial_1\partial_nf\\\vdots&\ddots&\vdots\\\partial_n\partial_1f&\cdots&\partial_n\partial_nf\end{pmatrix}. $$

Intuition: don't memorize three shapes, memorize one rule and read off outputs and inputs.

Chain rule and shapes

For $k = h\circ g$ with $g:\mathbb{R}^n\to\mathbb{R}^m$ and $h:\mathbb{R}^m\to\mathbb{R}^p$, Jacobians compose by matrix multiplication, in the same order:

$$ J_k = J_h J_g, \quad (p\times m)(m\times n) = p\times n. $$

$$ \underbrace{J_k}_{p\times n}=\underbrace{\begin{pmatrix}\ast&\cdots&\ast\\\vdots&&\vdots\\\ast&\cdots&\ast\end{pmatrix}}_{J_h,\ p\times m}\underbrace{\begin{pmatrix}\ast&\cdots&\ast\\\vdots&&\vdots\\\ast&\cdots&\ast\end{pmatrix}}_{J_g,\ m\times n}. $$

The inner dimension $m$ (the size of the intermediate space) is summed away, just as in any matrix product. For a scalar loss ($p=1$) the left factor is a $1\times m$ row, which is why backpropagation can carry a single row vector backward through the layers.

Intuition: shapes must chain like matrix multiplication; this is exactly backpropagation, one Jacobian per layer.

Example

With $g$ above and $h(y_1,y_2,y_3)=y_1+y_2+y_3$ (so $J_h=(1,1,1)$, a $1\times3$ row):

$$ J_k(1,1) = (1,1,1)\begin{pmatrix}1&1\\1&1\\2&0\end{pmatrix} = (4,\ 2). $$

Direct check: $k=x_1+x_2+x_1x_2+x_1^2$, $\nabla k = (1+x_2+2x_1,\ 1+x_1)$, at $(1,1)$: $(4,2)$. ✓