pandas basics: Series and DataFrames

Why pandas?

Most business data comes as tables: sales exports, survey results, bank statements. Excel works for small tables, but it is hard to repeat the same cleaning every week, and it struggles with lakhs of rows. pandas lets you write the steps once in code, run them on any size of file and get the same result every time.

Series and DataFrame

• A Series is one column of values with labels (the index). Think of it as a smart list.
• A DataFrame is a whole table: several Series side by side, sharing the same index (row labels).
• The index is the set of row labels. By default it is 0, 1, 2 and so on.

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Ravi", "Priya", "Imran"],
    "city": ["Pune", "Delhi", "Pune", "Mumbai"],
    "marks": [78, 65, 91, 58],
})
print(df)
#     name    city  marks
# 0   Asha    Pune     78
# 1   Ravi   Delhi     65
# 2  Priya    Pune     91
# 3  Imran  Mumbai     58

First look at any dataset

Selecting columns and rows

df["marks"]                  # one column: a Series
df[["name", "marks"]]        # several columns: a DataFrame (note the double brackets)
df[df["marks"] > 70]         # rows where marks > 70 (a boolean mask, like NumPy)
df.loc[0, "name"]            # by LABEL: row label 0, column "name" -> 'Asha'
df.iloc[0, 0]                # by POSITION: first row, first column -> 'Asha'
df.loc[df["city"] == "Pune", ["name", "marks"]]   # filter rows and pick columns

Sorting and new columns

df.sort_values("marks", ascending=False)        # best marks first
df["passed"] = df["marks"] >= 60                # new column from a rule
df["percent"] = df["marks"] / 100 * 100         # maths on whole columns

Reading and saving files

pd.read_csv("sales.csv") reads a comma-separated file into a DataFrame. Useful options: sep=";" when the file uses semicolons, encoding="utf-8" for Hindi or other non-English text, parse_dates=["order_date"] to read dates as real dates, and usecols=[...] to load only the columns you need from a huge file. pd.read_excel reads .xlsx files (it needs the openpyxl package). When you are done, df.to_csv("clean.csv", index=False) saves the result; index=False stops pandas from writing the row numbers as an extra column.

Common mistakes

• df["name", "marks"] with single brackets raises a KeyError. Fix: use a list inside: df[["name", "marks"]].
• Combining filters with and: df[df.marks > 60 and df.city == "Pune"] fails. Fix: (df.marks > 60) & (df.city == "Pune").
• Chained assignment like df[df.marks > 60]["grade"] = "P" may not change df and shows a warning. Fix: df.loc[df.marks > 60, "grade"] = "P".
• Forgetting that most methods return a new DataFrame. df.sort_values("marks") does not change df unless you assign it back.