Dynamics in the eigenbasis

Ketan Suhaas Saichandran

linear-algebraeigenvalueseigenbasisdynamical-systems

Expanding in the eigenbasis

If $A$ is diagonalizable with eigenvectors $v_1,\dots,v_n$, write any $x_0$ as a combination of them:

$$ x_0 = c_1v_1 + \cdots + c_nv_n. $$

Shapes

Stacking the eigenvectors as columns of $V$ turns the sum into one matrix-vector product; solving for the coordinates uses $V^{-1}$:

$$ \underbrace{x_0}_{n\times1}=\underbrace{\begin{pmatrix}\vert&&\vert\\v_1&\cdots&v_n\\\vert&&\vert\end{pmatrix}}_{V,\ n\times n}\underbrace{\begin{pmatrix}c_1\\\vdots\\c_n\end{pmatrix}}_{c,\ n\times1} \quad\Longleftrightarrow\quad c=V^{-1}x_0. $$

Intuition: the eigenvectors are a new coordinate system; the $c_i$ are $x_0$'s coordinates in it.

Example

Reusing $A = \begin{pmatrix}2&1\\1&2\end{pmatrix}$, eigenvectors $v_1=(1,1), v_2=(1,-1)$ (from Diagonalization):

$$ x_0 = \begin{pmatrix}4\\0\end{pmatrix} = 2v_1 + 2v_2. $$

Repeated multiplication

Since $Av_i = \lambda_iv_i$, also $A^kv_i = \lambda_i^kv_i$, so by linearity:

$$ A^kx_0 = c_1\lambda_1^kv_1 + \cdots + c_n\lambda_n^kv_n. $$

Intuition: each eigen-direction grows or shrinks completely on its own, at its own geometric rate $\lambda_i$.

Shapes

In matrix form, $A^k=V\Lambda^kV^{-1}$: change to eigen-coordinates, scale each coordinate independently, change back. The middle matrix only needs its diagonal entries raised to the $k$-th power:

$$ \underbrace{A^kx_0}_{n\times1}=\underbrace{V}_{n\times n}\underbrace{\begin{pmatrix}\lambda_1^k&&\\&\ddots&\\&&\lambda_n^k\end{pmatrix}}_{\Lambda^k,\ n\times n}\underbrace{V^{-1}x_0}_{n\times1} =V\begin{pmatrix}\lambda_1^kc_1\\\vdots\\\lambda_n^kc_n\end{pmatrix}. $$

Example

With $\lambda_1=3, \lambda_2=1$: $A^kx_0 = 2\cdot3^k v_1 + 2\cdot1^k v_2$. Check against direct multiplication:

$$ Ax_0 = \begin{pmatrix}8\\4\end{pmatrix}, \quad A^2x_0 = \begin{pmatrix}20\\16\end{pmatrix}, \quad A^3x_0 = \begin{pmatrix}56\\52\end{pmatrix}. $$

Formula at $k=1$: $2\cdot3\cdot(1,1) + 2\cdot1\cdot(1,-1) = (8,4)$. ✓

The dominant eigenvalue wins

As $k\to\infty$, the largest $|\lambda_i|$ (with $c_i\neq0$) swamps every other term:

$$ A^kx_0 \approx c_1\lambda_1^kv_1 \quad (\lambda_1 \text{ the dominant eigenvalue}). $$

Intuition: repeatedly applying $A$ to almost any vector points it toward the dominant eigenvector, this is power iteration.

Example

Direction of $A^kx_0$ above: $(8,4)$, $(20,16)$, $(56,52)$, i.e. ratios $0.5, 0.8, 0.929,\dots \to 1$, converging to $v_1=(1,1)$.

Each multiplication by $A$ stretches the point by roughly $\lambda_1=3$ along its own direction, but since the $v_2$ part only grows at $\lambda_2=1$, the ray swings closer to $v_1=(1,1)$ every time.

Continuous time

The same expansion solves $\dot x = Ax$:

$$ x(t) = c_1e^{\lambda_1t}v_1 + \cdots + c_ne^{\lambda_nt}v_n. $$

Intuition: $\lambda_i>0$ grows, $\lambda_i<0$ decays, $\lambda_i$ complex rotates; same picture, continuous time instead of steps.

Shapes

Exactly the discrete formula with $\Lambda^k$ replaced by the matrix exponential $e^{\Lambda t}$, which for a diagonal matrix just exponentiates each entry:

$$ \underbrace{x(t)}_{n\times1}=\underbrace{V}_{n\times n}\underbrace{\begin{pmatrix}e^{\lambda_1t}&&\\&\ddots&\\&&e^{\lambda_nt}\end{pmatrix}}_{e^{\Lambda t},\ n\times n}\underbrace{V^{-1}x_0}_{n\times1}. $$

Intuition: same change-of-basis story as $A^k=V\Lambda^kV^{-1}$, with "raise to the $k$-th power" swapped for "exponentiate," since that is what solves $\dot y=\lambda y$ in each eigen-coordinate.

As $t\to\infty$, $x(t)$ aligns with $v_1$: the faster-growing term $e^{\lambda_1t}v_1$ eventually swamps $e^{\lambda_2t}v_2$.

Why this matters

Power iteration (and PageRank) repeatedly multiplies by $A$ to find the dominant eigenvector. A Markov chain is $x_{k+1}=Px_k$ with dominant eigenvalue $1$, so it converges to the stationary distribution. A deep network applies something like $A^k$ across its layers: eigenvalues (or singular values) far from $1$ are exactly why activations and gradients blow up or vanish with depth.