ModelRefs / Support Vector Machines — Tutorial
Support Vector Machines — Tutorial
Maximum-margin classifiers, the kernel trick, and when SVMs beat neural networks. Covers Maximum margin classification, The kernel trick: non-linear boundaries.
Overview
Maximum-margin classifiers, the kernel trick, and when SVMs beat neural networks
Level: Advanced. Estimated reading time: 40 minutes.
Maximum margin classification
A Support Vector Machine finds the hyperplane that separates two classes with the largest possible margin. The margin is the distance between the hyperplane and the nearest point from each class — those nearest points are the support vectors.
Why maximise margin? The margin is a measure of confidence. Points far from the boundary are classified correctly even with some noise. A larger margin means better generalisation to unseen data.
The decision boundary is: w · x + b = 0, where w is the weight vector (perpendicular to the hyperplane) and b is the bias. The margin is 2/||w||. Maximising margin is equivalent to minimising ||w||.
Hard-margin SVM: every training point must be on the correct side. This fails if the data is not linearly separable. Soft-margin SVM: allow some misclassifications, controlled by hyperparameter C. High C = small margin, fewer errors on training data (risk: overfit). Low C = large margin, more errors allowed (risk: underfit).
The kernel trick: non-linear boundaries
Most real data is not linearly separable. The kernel trick maps data to a higher-dimensional space where linear separation is possible — without ever computing the high-dimensional coordinates.
Instead of computing φ(x) · φ(x') explicitly, a kernel K(x, x') computes the inner product in that space directly. This is the kernel trick.
Common kernels: - Linear: K(x, x') = x · x' (standard SVM, fast) - RBF (Gaussian): K(x, x') = exp(-γ||x-x'||²) — most popular, handles any shape boundary - Polynomial: K(x, x') = (x · x' + c)ᵈ
RBF SVM with tuned C and γ often matches or beats a neural network on tabular datasets with fewer than ~10,000 training examples. On larger datasets, the O(n²) kernel computation makes it slow.
When to choose SVM over a neural network
SVMs shine in specific scenarios:
Small datasets (< 10k samples): neural networks need large data to learn good representations. SVMs with RBF kernels can find complex boundaries from few examples.
Text classification: linear SVMs on TF-IDF features are very fast and often competitive with fine-tuned BERT on short-text tasks.
High-dimensional sparse data: SVMs handle thousands of features well (e.g., bag-of-words with 50k vocabulary).
Interpretability: for a linear SVM, the weight vector w directly shows which features matter — the feature with the highest weight is the most discriminative.
When NOT to use SVM: more than 100k training samples (training is O(n²–n³)), image/audio/video (CNNs dominate), sequence data (Transformers dominate).
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Support Vector Machines — Tutorial.