Pandas, DataFrames, Data Cleaning and Data Manipulation

Pandas Basics

pandas is the main Python library for working with tables. It has two building blocks:

• A Series: one column of data with labels.
• A DataFrame: a whole table of Series side by side, like a spreadsheet in code.

import pandas as pd
df = pd.read_csv("sales.csv")        # or pd.read_excel("sales.xlsx")

DataFrames: Inspecting and Selecting

Always look at your data before analyzing it:

Selecting: df["Sales"] gives one column; df[["Region","Sales"]] (double brackets) gives several. df.loc[row_label, column] selects by label and df.iloc[0:5] by position.

Filtering uses the same boolean idea as NumPy:

df[df["Sales"] > 500]
df[(df["Region"] == "North") & (df["Sales"] > 500)]   # and = &, or = |, wrap each test in ()

Data Cleaning in pandas

The same problems you fixed in Excel appear here, but now each fix is a line of code you can re-run on next month's file.

• Missing values: df.isna().sum() counts them per column. Then decide: df.dropna(subset=["Sales"]) removes rows missing Sales; df["Region"].fillna("Unknown") fills gaps.
• Duplicates: df.duplicated().sum() counts them; df = df.drop_duplicates() removes them.
• Wrong types: df["Date"] = pd.to_datetime(df["Date"]); df["Sales"] = pd.to_numeric(df["Sales"], errors="coerce") turns bad values into NaN instead of crashing.
• Messy text: df["Region"] = df["Region"].str.strip().str.title()
• Column names: df.columns = df.columns.str.lower().str.replace(" ", "_")

Note that most pandas methods return a new DataFrame; assign the result back (df = ...) to keep the change.

Data Manipulation

• New columns: df["Revenue"] = df["Units"] * df["Price"]
• Sorting: df.sort_values("Revenue", ascending=False)
• Grouping (like a PivotTable or SQL GROUP BY): df.groupby("Region")["Revenue"].sum(). Several statistics at once: df.groupby("Region")["Revenue"].agg(["sum","mean","count"])
• Pivot tables: pd.pivot_table(df, values="Revenue", index="Region", columns="Category", aggfunc="sum")
• Merging (like a lookup or SQL join): pd.merge(sales, products, on="product_id", how="left")
• Dates: df["Month"] = df["Date"].dt.month then group by Month.

Every one of these maps onto something you already know from Excel or SQL. pandas lets you chain them together and repeat the whole analysis with one click.