Cleaning household health survey data on body weight and height Using Outlier Detection (IQR and Isolation Forest)

Project info

• Field of studyPublic Health and Epidemiology
• LevelDoctoral dissertation (S3)
• Method typeData Quality
• Estimated build time± 7 working days
• Toolspandas quantile; scikit-learn IsolationForest
• PriceUS$34 - US$222

Research goal

Apply Outlier Detection (IQR and Isolation Forest) to synthetic data to prepare household health survey data on body weight and height for further analysis.

Variables

Example result

• 279Observations
• 39IQR outliers
• 21Isolation Forest outliers
• 0.395Method agreement (Jaccard)
• 0.538IQR precision
• 0.619IF recall

Of 279 observations, the IQR rule flags 39 and Isolation Forest flags 21 outliers in Body weight; the two methods agree with a Jaccard index of 0.395.

Computed from illustrative synthetic data, not from a real study.

About Outlier Detection (IQR and Isolation Forest)

Outlier detection with the IQR rule flags values beyond the quartile fences, while Isolation Forest isolates anomalous points in multivariate data. It is used to find extreme data before further analysis and decide how to treat them.

Steps

1. 1Visualise distributions with boxplots and histograms
2. 2Compute Q1, Q3, IQR, and fences (Q1 - 1.5 IQR; Q3 + 1.5 IQR)
3. 3Flag univariate outliers
4. 4Run IsolationForest for multivariate outliers
5. 5Investigate the cause of each outlier (input error or real value)
6. 6Decide treatment: keep, correct, or remove

Assumptions to check

• The IQR rule suits distributions that are not strongly skewed
• The contamination parameter in Isolation Forest is an estimate, not a fact
• An outlier is not necessarily an error, so substantive review is needed

Reading and reporting the results

Report the count and percentage of outliers per method with the reason for the chosen treatment. Show results with and without outliers when they matter.

Example Python code

A short starting script with this project's variable names. The delivered version is adapted to your data and fully commented.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import IsolationForest

rng = np.random.default_rng(36)
n = 300
height = rng.normal(50, 5, n)
age = 0.8 * height + rng.normal(0, 3, n)
height[:5] += 30                      # inject univariate outliers
age[5:8] -= 25                     # inject multivariate outliers
body_weight = np.column_stack([height, age])

q1, q3 = np.percentile(height, [25, 75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
mask_iqr = (height < lo) | (height > hi)
print("IQR fences:", round(lo, 2), round(hi, 2), "outliers:", int(mask_iqr.sum()))

iso = IsolationForest(contamination=0.03, random_state=36).fit(body_weight)
mask_if = iso.predict(body_weight) == -1
print("Isolation Forest outliers:", int(mask_if.sum()))

fig, ax = plt.subplots(1, 2, figsize=(10, 4))
ax[0].boxplot(height, vert=False); ax[0].set_title("x1 boxplot")
ax[1].scatter(height[~mask_if], age[~mask_if], s=10, label="Normal")
ax[1].scatter(height[mask_if], age[mask_if], c="r", s=25, label="Outlier")
ax[1].legend()
plt.tight_layout()
plt.show()

Packages and prices

Choose how much help you need. Each higher package includes everything in the lower ones. Prices exclude 11% VAT.

• Payment: 50% down payment before work starts, balance after the deliverables are handed over
• Tax: Prices exclude 11% VAT
• Data and academic integrity: Results are computed from your own research data; you write and stand behind your manuscript, we help with the analysis and your understanding of it
• Revisions: Up to 2 revisions are included in the price (Package 3: 3 rounds)

Frequently asked questions