Pandas for Data Analysis: A Beginner Tutorial With Examples

pandas is the Python library most data analysts use to load, clean, transform and summarise tables of data. If you know Excel or SQL, pandas will feel familiar: a DataFrame is like a sheet or a table, and most operations have an Excel or SQL equivalent. This tutorial covers the parts you will use every day, with code you can copy.

What Is pandas?

pandas is an open-source Python library for data manipulation and analysis. The name comes from “panel data”. Its two main objects are the Series (one column) and the DataFrame (a table of columns). Install it with pip install pandas, or use Google Colab or Jupyter where it is already installed.

Load and Inspect Data

import pandas as pd

df = pd.read_csv("orders.csv")                 # or pd.read_excel("orders.xlsx", sheet_name="2026")
print(df.shape)          # (rows, columns)
print(df.head())         # first 5 rows
df.info()                # column types and non-null counts
print(df.describe())     # numeric summary

Detect and Fill Missing Values

print(df.isna().sum())                         # missing values per column

df["city"] = df["city"].fillna("Unknown")
df["age"] = df["age"].fillna(df.groupby("segment")["age"].transform("median"))
df = df.dropna(subset=["order_id"])            # a row without a key is unusable
df = df.drop_duplicates(subset="order_id", keep="last")

Fill values with a rule you can explain. A median per group is usually safer than one overall average, and a missing key should normally be investigated, not guessed.

Filter and Transform

df["amount"] = pd.to_numeric(df["amount"], errors="coerce")   # text to number
df["city"] = df["city"].str.strip().str.title()                 # " pune " -> "Pune"
big = df[(df["amount"] > 5000) & (df["status"] == "DELIVERED")]
df["amount_band"] = pd.cut(df["amount"], bins=[0, 1000, 5000, float("inf")],
                           labels=["Low", "Mid", "High"])

These are the same transformation rules an ETL job applies. Testers will recognise them from data validation in ETL.

Group and Summarise

summary = (df.groupby("city")
             .agg(orders=("order_id", "count"), revenue=("amount", "sum"), avg_order=("amount", "mean"))
             .sort_values("revenue", ascending=False))
print(summary.round(1))

pivot = df.pivot_table(index="city", columns="status", values="amount", aggfunc="sum", fill_value=0)

Merge Data Sources

orders = pd.read_csv("orders.csv")
customers = pd.read_csv("customers.csv")

merged = orders.merge(customers, on="customer_id", how="left", validate="many_to_one")
assert len(merged) == len(orders), "merge created duplicate rows"
print(merged["customer_name"].isna().sum(), "orders without a customer")

validate="many_to_one" makes pandas raise an error if the customer table has duplicate keys, a common cause of inflated totals.

Dates and Time Zones

df["order_time"] = pd.to_datetime(df["order_time"], utc=True)
df["order_time_ist"] = df["order_time"].dt.tz_convert("Asia/Kolkata")
df["order_date"] = df["order_time_ist"].dt.date
df["month"] = df["order_time_ist"].dt.to_period("M")

An order placed at 8 pm UTC on 31 March is 1:30 am IST on 1 April, so it falls in a different month. Mixed time zones are a frequent reason two reports disagree.

Save Data in Other Formats

df.to_csv("clean_orders.csv", index=False)
df.to_excel("clean_orders.xlsx", index=False)          # needs openpyxl
df.to_json("clean_orders.json", orient="records")
df.to_parquet("clean_orders.parquet")                  # needs pyarrow; small and fast
df.to_pickle("clean_orders.pkl")                       # Python-only, keeps dtypes

Compare Two DataFrames (Source vs Target)

“pandas compare two dataframes” is one of the most searched pandas tasks, and it is exactly what ETL testers do when checking a load. Use an outer merge with indicator=True to find missing rows, then compare values for matching keys:

src = pd.read_csv("source.csv")
tgt = pd.read_csv("target.csv")

m = src.merge(tgt, on="order_id", how="outer", suffixes=("_src", "_tgt"), indicator=True)
print(m["_merge"].value_counts())      # left_only = missing in target, right_only = extra in target

both = m[m["_merge"] == "both"]
diff = both[both["amount_src"].round(2) != both["amount_tgt"].round(2)]
print(len(diff), "rows with different amounts")
print(diff[["order_id", "amount_src", "amount_tgt"]].head())

For two frames with the same shape and index, df1.compare(df2) shows the differing cells side by side, and df1.equals(df2) returns True only if they are identical. More testing patterns are in the Python ETL testing framework guide.

Common Beginner Pitfalls

• Numbers stored as text. sum() on an object column joins strings. Convert with pd.to_numeric.
• Merges that add rows. Duplicate keys on the right side multiply rows. Use validate=.
• Chained assignment. df[df.a > 1]["b"] = 0 may not change df. Use df.loc[df.a > 1, "b"] = 0.
• NaN is not equal to NaN. Comparisons on missing values return False. Use isna().
• Forgetting the time zone. Convert to one zone before grouping by date.

The course Claude AI for Data Analysis & Business Intelligence teaches these pandas skills (transform, clean and merge data, serialise DataFrames to multiple formats, fill missing values, handle time zones and avoid beginner traps) alongside Claude AI, so you can use AI to write code and pandas to check it.

Data analysis with AI: Claude AI for data analysis · Claude prompts for data analysis · EDA on the Titanic dataset · data analyst interview questions · AI data analyst course in India · business intelligence course · data analyst roadmap 2026 · ETL testing vs data analyst · data analytics for working professionals · AI tools for data analysis

Frequently Asked Questions

What is pandas used for in data analysis?

pandas is used to load data from CSV, Excel, SQL and other sources, clean it, transform it, group and summarise it, merge tables and save the results.

Is pandas easy to learn for beginners?

Yes, especially if you know Excel or SQL, because most pandas operations map to a familiar filter, pivot or join.

How do I compare two DataFrames in pandas?

Merge them on the key with how="outer" and indicator=True to find missing rows, then compare columns for matching keys. For same-shaped frames, use df1.compare(df2).

What is the full form of pandas?

The name comes from “panel data”, an economics term for data observed over time.

Should I learn pandas or SQL first?

For testing and analyst roles, SQL first is common because most company data lives in databases. pandas is the next step for files, automation and analysis in Python.

AN

Written by

Asim Noaman Lodhi

Certified Google Partner · QA Consultant · 12+ Years IT

QA consultant specializing in ETL testing and data quality. Trained 913+ students to transition into data testing roles through hands-on, real-world instruction.

4.5 Rating 913+ Students Google Partner 79 Lectures