Support Vector Machines (SVM)

Draw the widest possible street between classes, not just any line that separates them.

Key Points

SVM maximizes the margin between classes, not just any line that happens to separate them.
Only the closest points to the boundary — the support vectors — determine its position; the rest of the data doesn't matter.
The kernel trick lets SVM draw non-linear boundaries by implicitly projecting data into higher dimensions.
The C parameter controls the trade-off between a wider margin and fewer misclassifications on training data.
Works well on small-to-medium datasets with a clear margin of separation, and in high-dimensional spaces.

What it is

A Support Vector Machine finds the decision boundary (hyperplane) that separates classes with the maximum possible margin — the widest gap between the closest points of each class. Those closest points are called support vectors because they alone determine where the boundary sits.

Why it exists

Many lines can separate two classes perfectly on training data, but not all of them generalize equally well to new data. SVM exists to pick the one boundary that is maximally 'safe' — as far as possible from both classes — which tends to generalize better than an arbitrary separating line.

Intuition

Imagine drawing a road between two neighborhoods that never touches a house on either side. You could draw a narrow, winding road that barely squeezes through, or the widest possible straight road with the most breathing room on both sides. SVM always chooses the widest road — new houses built later are less likely to end up on the wrong side of it.

Formulas

Margin Maximization Objective

\min_{w,b} \frac{1}{2}\|w\|^2 \quad \text{s.t. } y_i(w \cdot x_i + b) \geq 1

w, b — The weight vector and bias defining the separating hyperplane
\|w\|^2 — Minimizing this maximizes the margin width
y_i(w \cdot x_i + b) \geq 1 — Every point must be correctly classified and outside the margin

When to use

Use on small-to-medium datasets, especially with more features than samples (e.g. text classification, genomics).
Use when classes are separable with a clear margin, or nearly so.
Use the kernel trick (RBF kernel) when the boundary between classes is clearly non-linear.

When not to use

Do not use on very large datasets — training time scales poorly with number of samples.
Do not use without scaling your features first — SVM is sensitive to feature magnitude.
Do not expect easy interpretability compared to logistic regression or decision trees.

Common mistakes

Forgetting to scale/normalize features before training, which distorts the margin calculation.
Using a non-linear kernel by default without checking if a simple linear kernel would already work — adds unnecessary complexity.
Not tuning C and gamma — leaving them at defaults often gives poor results on real data.

In practice

Common in text classification (spam detection, sentiment analysis with TF-IDF features), bioinformatics, and image classification tasks with a moderate number of samples, though largely replaced by gradient boosting and neural networks for very large datasets.

Was this concept helpful?

Spot an error or gap? Suggest an edit →

Related concepts

kernel-tricklogistic-regressionfeature-scaling