Exploratory Data Analysis (EDA)

Poking, prodding, and visualizing your data until it tells you its secrets.

Key Points

• EDA is not confirmatory. It's exploratory. You're not proving anything yet.
• Always start with the distribution plots. Histograms, box plots, density curves.
• Check for missing data and anomalies before you do anything else.
• Use aggregation (mean, sum, count) to see big picture, then drill down.
• The visualization pipeline: from raw data to visual form happens through mappings (marks + channels).

What it is

EDA is the detective work you do before any formal modeling. You plot, summarize, filter, and transform. You look for missing values, outliers, weird patterns, and surprising relationships. It's not about testing a hypothesis. It's about generating them. The goal is to know your data so well that when something looks wrong later, you notice immediately.

Why it exists

Because raw data is chaos. You can't just run a model and hope. EDA reveals data quality issues, suggests transformations, and points you toward the right analysis.

Intuition

How it works

1. Load & peek

   Look at first rows, column names, data types. Spot obvious errors like negative ages or 'NaN' placeholders.

2. Univariate analysis

   Plot each variable alone. Histograms for numbers, bar charts for categories. Check distribution shape, center, spread.

3. Missing & dirty data

   Count missing values per column. Look for duplicates, inconsistent formatting (e.g., 'NY' vs 'New York').

4. Bivariate & multivariate

   Scatter plots, box plots, heatmaps. Look for correlations, clusters, and separations.

5. Transform & simplify

   Filter irrelevant rows, aggregate, pivot, create new features, reduce dimensionality if needed.

When to use

• Every single time you get a new dataset. No exceptions.
• After each major transformation (after cleaning, after merging, after feature engineering) – re‑EDA to confirm.
• When a model gives surprising results – go back to EDA to find the cause.

Common mistakes

• Skipping EDA because you're eager to build models. That's like baking without tasting the batter.
• Only looking at summary statistics (mean, median) without plotting. Anscombe's quartet shows why this fails.
• Not checking for missing values or outliers before modeling. They will wreck your results.

Notes

• EDA is where most of real data science time goes. Up to 80 percent. Don't rush it.
• Tools like pandas profiling, Sweetviz, or D‑Tale can automate parts, but you still need human eyes.

In practice

EDA is the 'Explore' step in the data science workflow. It informs feature engineering, model selection, and even how you clean data. Interactive tools like Tableau, Plotly, and D3.js are modern EDA powerhouses.

Related concepts