Logistic regression and classification

Why not use a straight line for yes/no?

Suppose we label "will default on the loan" as 1 and "will repay" as 0, and fit a straight line. The line happily predicts 1.4 or −0.3, which make no sense as chances. We want an output between 0 and 1 that we can read as a probability. Logistic regression does exactly that, and despite its name it is a classification method.

The sigmoid function

First compute a score z = w·x + b, just like linear regression. Then pass it through the sigmoid, σ(z) = 1 ÷ (1 + e^(−z)). The sigmoid is an S-shaped curve: very negative scores give values near 0, very positive scores give values near 1, and z = 0 gives exactly 0.5.

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

for z in [-4, 0, 2]:
    print(z, round(sigmoid(z), 4))

From probability to decision: the threshold

The model outputs a probability, say 0.73 that an email is spam. To act, we need a yes/no. The usual rule is "predict 1 if the probability is at least 0.5". But 0.5 is not sacred. A bank worried about missing fraud may flag anything above 0.2, accepting more false alarms. A spam filter that must never hide important mail may only block above 0.9. Module 6 shows how to pick the threshold with metrics.

How it is trained: log loss

Logistic regression is trained by minimising log loss (also called binary cross-entropy). For one example with true label y (0 or 1) and predicted probability p, the loss is −(y·log p + (1 − y)·log(1 − p)). If y = 1 and the model says p = 0.9, the loss is small (about 0.105). If it says p = 0.01, the loss is large (about 4.6). Confident wrong answers are punished heavily. Training uses gradient descent on this loss.

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
proba = clf.predict_proba(X_test)[:, 1]   # probability of class 1
pred = (proba >= 0.5).astype(int)         # the same as clf.predict(X_test)

More than two classes

For three or more classes (for example, routing a support ticket to billing, delivery or technical), logistic regression is extended with the softmax function, which turns a score for each class into probabilities that add up to 1. You will meet softmax again in neural networks (module 9).

Reading the weights

A positive weight pushes the probability up as its feature grows; a negative weight pushes it down. Each weight is the change in log-odds per unit of the feature, which is why analysts in banks and insurance like logistic regression: it is accurate enough for many tasks and easy to explain to a regulator.

Common mistakes

• Treating the 0.5 threshold as fixed. Fix: choose it from the costs of each kind of mistake.
• Calling it a regression method because of the name. It is used for classification.
• Taking log(0) when computing log loss by hand. Fix: clip probabilities to a tiny range like [1e-15, 1 − 1e-15].
• Reading the probability as certain truth. Probabilities from a model can be badly calibrated; check them on held-out data.