Bias and fairness

Why fairness is a data problem

Models learn patterns from data, and data records the past, including its unfairness. If a company hired fewer women for engineering roles in the past, a model trained on those decisions may learn that being a woman lowers the chance of being hired. Nobody wrote that rule; the model found it. Because ML decisions now affect loans, jobs, insurance and access to services, checking for unfair outcomes is part of a data professional's job.

Where bias comes from

• Historical bias: the world itself was unequal, and the labels reflect it.
• Sampling (representation) bias: some groups are missing or rare in the data, such as rural users in an app that launched in cities, so the model works worse for them.
• Label bias: the "right answers" were produced by people or processes with their own biases.
• Measurement bias: a feature is measured worse for some groups (for example, cheaper phones taking poorer photos).
• Proxy features: even if you drop gender or caste, other columns such as pin code, first name, college or language can stand in for them.

Measuring fairness: simple metrics

Start by computing outcomes per group. The selection rate is the share of each group that gets the positive decision (approved, shortlisted). Demographic parity asks whether selection rates are similar. The disparate impact ratio divides the lowest group's selection rate by the highest; the "four-fifths rule" from US hiring guidance treats a ratio below 0.8 as a warning sign. It is a rule of thumb, not proof of discrimination or a legal standard everywhere.

decisions = [1, 1, 0, 1, 0, 1, 0, 0, 0, 1]
groups    = ["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"]
rate = {}
for g in set(groups):
    picked = [d for d, gg in zip(decisions, groups) if gg == g]
    rate[g] = sum(picked) / len(picked)
print(sorted(rate.items()), round(min(rate.values()) / max(rate.values()), 2))

Equal opportunity looks at errors instead: among people who truly qualify (true label 1), is the true positive rate similar across groups? A model can have equal selection rates and still miss qualified people in one group much more often. Different fairness definitions can conflict with each other, so teams must choose, with legal and domain advice, which matters most for the decision at hand.

Reducing unfairness

• Better data: collect more examples from under-represented groups; fix biased labels.
• Remove or test proxy features, and check results per group after every change.
• Reweight or resample training data so groups count fairly.
• Review thresholds per decision with domain experts and legal advice.
• Keep humans in the loop for high-stakes decisions, with a way for people to appeal.
• Document the model (a "model card"): what it is for, what data it used and how it performs for each group.

Common mistakes

• Believing that dropping a sensitive column makes a model fair. Proxies remain.
• Reporting only overall accuracy, which hides poor performance for small groups.
• Checking fairness once before launch and never again.