Feature selection, leakage and pipelines

Why select features?

More features are not always better. Useless or duplicate features slow training, make overfitting easier, make models harder to explain, and cost money to collect in production. Some features are dangerous: they leak the answer or bring in unfair bias. Feature selection is the habit of keeping only what earns its place.

Three families of methods

• Filter methods: score each feature on its own, before modelling. Drop near-constant columns (very low variance), and drop one of two features that are almost perfectly correlated, such as "price incl. GST" and "price excl. GST".
• Wrapper methods: try subsets of features with the model and keep the best by cross-validation (for example, recursive feature elimination). Accurate but slow.
• Embedded methods: the model selects while training. Lasso (L1) pushes useless weights to zero; tree ensembles give importances.

Leakage features: the ones that must go

Ask of every feature: "Would I know this value at the moment I make the prediction?" Features filled in after the outcome, such as "loan_closed_date", "refund_amount" or "final_diagnosis_code", leak the answer. Also watch for ids that encode time or the outcome, and for aggregates computed over the future. When a single feature gives an amazing score on its own, investigate it before celebrating.

Pipelines: preprocessing done right

A Pipeline is a chain of steps, such as "fill missing values → scale → model", treated as one model. When you call fit, each step is fitted on the training data only; when you call predict, the same fitted steps are applied to new data. Inside cross-validation, the whole pipeline is refitted in each fold, so no information leaks across folds. And in production you save one object instead of five.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["age", "income", "existing_loans"]
categorical = ["city", "employment_type"]

prep = ColumnTransformer([
    ("num", Pipeline([("fill", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000))])
model.fit(X_train, y_train)
print(model.score(X_test, y_test))

Feature stores, briefly

In larger companies, a feature store is a shared system that computes and serves features (like "orders in the last 7 days") the same way for training and for live prediction, so the two never drift apart.

Common mistakes

• Selecting features using the whole dataset, then cross-validating. The selection itself leaked. Fix: put selection inside the pipeline.
• Dropping a feature only because it has low correlation with the target; it may matter in combination with others.
• Keeping an id column because it "improved" the score. It is usually leakage or chance.
• Writing preprocessing twice, once in a notebook and once in the app, and getting different results. Fix: one pipeline object.