k-nearest neighbours

The idea: you are like your neighbours

To guess whether a new customer will like a premium plan, look at the customers most similar to them. If most of those liked it, guess yes. That is the whole idea of k-nearest neighbours. It is one of the simplest ML methods and a good way to understand what "learning from examples" really means.

Measuring closeness

Each example is a point with one coordinate per feature. Euclidean distance is the straight-line distance: the square root of the sum of squared differences. Manhattan distance adds the absolute differences, like walking city blocks. For two customers described by (age, monthly spend in thousands), (25, 4) and (28, 8) have Euclidean distance √(3² + 4²) = 5.

import math

def euclidean(a, b):
    return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))

print(euclidean((25, 4), (28, 8)))

The algorithm

1. Store all training examples (there is no real training step; k-NN is a "lazy learner").
2. For a new point, compute its distance to every training example.
3. Take the k closest.
4. Classification: return the most common label among them. Regression: return their average value.

Choosing k

A small k (like 1) follows every noisy point and overfits. A large k smooths too much and underfits, eventually just predicting the majority class. Try several values with cross-validation. For two classes, an odd k avoids exact ties in the vote; you still need a tie rule for more classes, such as "the label of the nearest neighbour wins".

Scaling matters a lot

If one feature is income in rupees (values like 6,00,000) and another is age in years (values like 30), the distance is almost entirely about income. Age barely counts. Always scale features (for example, standardise them to mean 0 and standard deviation 1) before k-NN. Module 8 covers scaling.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
knn.fit(X_train, y_train)
print(knn.score(X_test, y_test))

Strengths and weaknesses

• Strengths: simple to explain ("these 5 similar cases"), no training time, works for odd-shaped class boundaries.
• Weaknesses: prediction needs a distance to every stored example, so it is slow with millions of rows (special indexes help); it struggles with many features, where all points start to look equally far apart (the "curse of dimensionality"); it needs careful scaling.

Common mistakes

• Forgetting to scale features. Fix: put a scaler in a pipeline.
• Using k = 1 and trusting a perfect training score; with k = 1, every training point is its own neighbour.
• Using k-NN on huge datasets that need fast answers without an index.