ModelRefs / Eigenvalues & Eigenvectors — Tutorial
Eigenvalues & Eigenvectors — Tutorial
The geometric heart of linear algebra — diagonalisation, PCA connections, and why they matter for ML. Covers What eigenvectors actually are.
Overview
The geometric heart of linear algebra — diagonalisation, PCA connections, and why they matter for ML
Level: Intermediate. Estimated reading time: 25 minutes.
What eigenvectors actually are
A matrix represents a linear transformation — it stretches, rotates, and shears space. For most vectors, multiplication by a matrix changes both their direction and magnitude.
An eigenvector is special: multiplying it by the matrix only scales it — it does not change direction. Formally:
A · v = λ · v
where v is the eigenvector and λ (lambda) is the eigenvalue — the scalar factor by which v is stretched or compressed. If λ > 1 the eigenvector grows; if 0 < λ < 1 it shrinks; if λ < 0 it flips direction; if λ = 0 the vector is in the null space.
A 2×2 matrix has at most 2 eigenvalue–eigenvector pairs. An n×n matrix has at most n.
Geometric intuition: imagine a transformation that squishes the y-axis and stretches the x-axis. Vectors pointing along x or y are eigenvectors — they only scale. All other vectors are deflected.
Computing eigenvalues and the characteristic equation
To find eigenvalues, start from A·v = λ·v, rearrange to (A − λI)·v = 0. For a non-zero solution v to exist, the matrix (A − λI) must be singular — its determinant must be zero:
det(A − λI) = 0
This is the characteristic equation. For a 2×2 matrix it produces a quadratic in λ; for n×n it produces a degree-n polynomial. Solving it gives the n eigenvalues (which may be complex, repeated, or zero).
Once you have an eigenvalue λ, substitute it back and solve (A − λI)·v = 0 to find the corresponding eigenvector.
In practice: NumPy's numpy.linalg.eig(A) returns eigenvalues and eigenvectors directly. For symmetric/Hermitian matrices (which arise naturally in ML — covariance matrices, Gram matrices, graph Laplacians), use numpy.linalg.eigh — it returns real eigenvalues sorted in ascending order and is numerically more stable.
Why eigenvalues matter for ML
Principal Component Analysis (PCA): the principal components are the eigenvectors of the data covariance matrix. The corresponding eigenvalues measure how much variance each component explains. Sorting eigenvectors by descending eigenvalue and projecting data onto the top-k gives the optimal k-dimensional subspace.
Spectral clustering: cluster data by the eigenvectors of the graph Laplacian matrix L = D − A (degree matrix minus adjacency matrix). The number of zero eigenvalues of L equals the number of connected components.
Stability analysis: in optimisation, the Hessian matrix H of a loss function encodes curvature. If all eigenvalues of H are positive (positive definite), you are at a local minimum. Negative eigenvalues = saddle point. The ratio of largest to smallest eigenvalue is the condition number — high condition number means ill-conditioned optimisation (slow convergence).
Singular Value Decomposition (SVD): SVD generalises eigendecomposition to non-square matrices. The singular values are the square roots of the eigenvalues of A^T·A. SVD underlies PCA, low-rank approximation, and collaborative filtering.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Eigenvalues & Eigenvectors — Tutorial.