Linear regression and least squares

Why start with a line?

Many relationships are roughly straight over the range we care about: a bigger flat costs more, a longer distance takes longer. Linear regression is fast, easy to explain and a strong baseline. Even teams that use complex models usually try it first.

The model

With one feature x, the model is ŷ = w·x + b. ŷ (read "y-hat") is the prediction, w is the weight or slope (how much ŷ rises for each extra unit of x) and b is the bias or intercept (the value of ŷ when x is 0). Training means choosing w and b.

Residuals and least squares

A residual is the error for one example: actual minus predicted, y − ŷ. Some are positive, some negative. To get one number that measures how bad a line is, we square each residual (so negatives do not cancel positives, and big misses count more) and add them up. Least squares means choosing the w and b that make this sum of squared errors as small as possible.

The formula for one feature

1. Compute the means x̄ and ȳ.
2. w = Σ (x − x̄)(y − ȳ) ÷ Σ (x − x̄)². This is the covariance of x and y divided by the variance of x.
3. b = ȳ − w·x̄. The line always passes through the point (x̄, ȳ).

xs = [500, 750, 1000, 1250]        # flat area in sq ft
ys = [30, 42, 55, 66]               # price in ₹ lakh
mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
w = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
b = my - w * mx
print(round(w, 4), round(b, 2))
print(round(w * 1100 + b, 2))

More than one feature

Multiple linear regression uses many features: ŷ = w₁x₁ + w₂x₂ + … + b. Each weight is the effect of its feature when the others are held fixed. In scikit-learn it is the same LinearRegression().fit(X, y), and the learned weights are in model.coef_ and the bias in model.intercept_.

from sklearn.linear_model import LinearRegression

model = LinearRegression().fit(X_train, y_train)   # X: area, age, floor
print(model.coef_, model.intercept_)

When a line misleads

• Extrapolation: predicting far outside the data range. A line fitted to 500–1,250 sq ft flats says nothing reliable about a 10,000 sq ft bungalow.
• Curved relationships: if the true pattern bends, a straight line is systematically wrong. Plot the residuals; a curve in them is a warning.
• Outliers: one extreme point can pull the line a lot, because errors are squared.
• Correlated features: weights can become unstable and hard to interpret.

Common mistakes

• Reading the slope as cause. A positive slope between ad spend and sales does not prove ads caused the sales.
• Forgetting units. A slope of 0.0484 means ₹ lakh per sq ft, not rupees.
• Fitting when all x values are the same: the variance is 0 and the slope is undefined. Check for it.