Outlier Detection and Exploratory Analysis

Outlier Detection

An outlier is a value far away from the rest. Outliers can be errors (a price typed as 1000 instead of 10.00) or real and important (one customer who placed a huge order). Either way, you need to find them.

The IQR method, step by step:

1. Sort the data and find the quartiles. Q1 is the 25th percentile and Q3 the 75th. Excel: =QUARTILE.INC(range,1) and =QUARTILE.INC(range,3).
2. Interquartile range: IQR = Q3 − Q1. It covers the middle 50% of the data.
3. Lower fence = Q1 − 1.5 × IQR. Upper fence = Q3 + 1.5 × IQR.
4. Any value outside the fences is a potential outlier.

Example: Q1 = 30, Q3 = 45, so IQR = 15. Fences: 30 − 22.5 = 7.5 and 45 + 22.5 = 67.5. An order of 250 is an outlier.

Box plots show this visually: the box runs from Q1 to Q3 with a line at the median, the whiskers reach to the furthest non-outlier values, and outliers appear as dots. Excel: Insert > Charts > Box and Whisker.

The z-score method is an alternative for roughly normal data: z = (value − mean) / standard deviation, and |z| > 3 is often treated as an outlier.

What to do with an outlier: investigate before deleting. Fix it if it is an error, keep it if it is real (and maybe report it separately), and always write down what you decided and why.

Exploratory Analysis (EDA)

Exploratory data analysis is the first deep look at a new dataset, done before you build any report. Its goal is to understand the data, check its quality and find questions worth answering. A simple checklist:

1. Understand the structure: how many rows and columns? What does one row represent? What type is each column?
2. Check quality: missing values, duplicates, impossible values (negative ages, future dates).
3. Univariate analysis (one variable at a time): mean, median, standard deviation, min and max for numbers; histograms and box plots; value counts for categories.
4. Bivariate analysis (two variables): scatter plots and correlation for two numbers; grouped averages and bar charts for a category against a number.
5. Look at time: if there is a date, plot the main measure over time. Look for trends and seasonal patterns.
6. Write down findings and questions as you go: "Sales dip every February," "West has the highest average order but the fewest orders."

EDA ties together everything in this course. It is also the heart of the Project course later in the program.