Exploratory data analysis (EDA)

Why explore first?

Jumping straight to a model is like a doctor prescribing medicine without examining the patient. EDA is the examination. It tells you whether the data can answer your question at all, which columns are useful, what needs cleaning, and what patterns a model should find. Many projects are saved, or quietly cancelled, during EDA.

An EDA checklist

1. Size and grain: how many rows and columns, and what is one row? (One order? One customer per day?)
2. Types: which columns are numbers, categories, dates, free text, ids?
3. Missing values: how many per column, and is the missingness random or meaningful?
4. Univariate look: for each important column, its distribution. Numbers: min, max, mean, median, histogram. Categories: counts.
5. Outliers and impossible values: negative quantities, ages of 200, dates in the future.
6. Bivariate look: how pairs of columns relate, such as delivery time vs distance (scatter), or cancellation rate by city (grouped bar).
7. The target: if you will predict something, how is it distributed? Is it rare (2% fraud)?
8. Write findings and questions down as you go.

Univariate and bivariate

Univariate means one variable at a time. Bivariate means two together. A histogram of delivery times is univariate. A scatter plot of delivery time against distance is bivariate. Correlation numbers (from module 2) summarise linear bivariate relationships, but always look at the chart too: a correlation near 0 can hide a strong curved pattern.

df.shape                          # (48210, 9)
df.isna().mean().round(3)         # share missing per column
df["delivery_min"].describe()
df["city"].value_counts().head(10)
df.groupby("city")["delivery_min"].median().sort_values()
df[["distance_km", "delivery_min"]].corr()

Ask questions, then check them

Good EDA is driven by questions. "Are weekend orders slower?" "Do new restaurants get more cancellations?" Write the question, compute the answer, then note it. Each answer usually raises a new question. Be careful: with enough slicing, something will always look interesting by chance, so treat surprising patterns as ideas to test, not facts.

Common mistakes

• Skipping EDA because "the model will figure it out". Fix: spend real time here; it pays back.
• Looking only at averages. Fix: always look at distributions; two very different datasets can share a mean.
• Not knowing what one row means, then counting customers when the rows are orders. Fix: state the grain first.
• Exploring the test data too and tuning decisions to it. Fix: once you split for modelling, explore only the training part.
• Forgetting to write findings down. Fix: keep a short "what I found" list at the top of the notebook.