Applying Outlier Detection (IQR and Isolation Forest) to IoT device sensor data

Project info

• Field of studyInformatics and Information Systems
• LevelDoctoral dissertation (S3)
• Method typeData Quality
• Estimated build time± 7 working days
• Toolspandas quantile; scikit-learn IsolationForest
• PriceUS$34 - US$216

Research goal

Inspect and improve the quality of sensor temperature data with sensor humidity and battery voltage using Outlier Detection (IQR and Isolation Forest), then assess the impact on analysis.

Variables

Example result

• 230Observations
• 18IQR outliers
• 12Isolation Forest outliers
• 0.25Method agreement (Jaccard)
• 0.667IQR precision
• 0.5IF recall

Of 230 observations, the IQR rule flags 18 and Isolation Forest flags 12 outliers in Sensor temperature; the two methods agree with a Jaccard index of 0.250.

Computed from illustrative synthetic data, not from a real study.

About Outlier Detection (IQR and Isolation Forest)

Outlier detection with the IQR rule flags values beyond the quartile fences, while Isolation Forest isolates anomalous points in multivariate data. It is used to find extreme data before further analysis and decide how to treat them.

Steps

1. 1Visualise distributions with boxplots and histograms
2. 2Compute Q1, Q3, IQR, and fences (Q1 - 1.5 IQR; Q3 + 1.5 IQR)
3. 3Flag univariate outliers
4. 4Run IsolationForest for multivariate outliers
5. 5Investigate the cause of each outlier (input error or real value)
6. 6Decide treatment: keep, correct, or remove

Assumptions to check

• The IQR rule suits distributions that are not strongly skewed
• The contamination parameter in Isolation Forest is an estimate, not a fact
• An outlier is not necessarily an error, so substantive review is needed

Reading and reporting the results

Report the count and percentage of outliers per method with the reason for the chosen treatment. Show results with and without outliers when they matter.

Example Python code

A short starting script with this project's variable names. The delivered version is adapted to your data and fully commented.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import IsolationForest

rng = np.random.default_rng(36)
n = 300
sensor_humidity = rng.normal(50, 5, n)
battery_voltage = 0.8 * sensor_humidity + rng.normal(0, 3, n)
sensor_humidity[:5] += 30                      # inject univariate outliers
battery_voltage[5:8] -= 25                     # inject multivariate outliers
sensor_temperature = np.column_stack([sensor_humidity, battery_voltage])

q1, q3 = np.percentile(sensor_humidity, [25, 75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
mask_iqr = (sensor_humidity < lo) | (sensor_humidity > hi)
print("IQR fences:", round(lo, 2), round(hi, 2), "outliers:", int(mask_iqr.sum()))

iso = IsolationForest(contamination=0.03, random_state=36).fit(sensor_temperature)
mask_if = iso.predict(sensor_temperature) == -1
print("Isolation Forest outliers:", int(mask_if.sum()))

fig, ax = plt.subplots(1, 2, figsize=(10, 4))
ax[0].boxplot(sensor_humidity, vert=False); ax[0].set_title("x1 boxplot")
ax[1].scatter(sensor_humidity[~mask_if], battery_voltage[~mask_if], s=10, label="Normal")
ax[1].scatter(sensor_humidity[mask_if], battery_voltage[mask_if], c="r", s=25, label="Outlier")
ax[1].legend()
plt.tight_layout()
plt.show()

Packages and prices

Choose how much help you need. Each higher package includes everything in the lower ones. Prices exclude 11% VAT.

• Payment: 50% down payment before work starts, balance after the deliverables are handed over
• Tax: Prices exclude 11% VAT
• Data and academic integrity: Results are computed from your own research data; you write and stand behind your manuscript, we help with the analysis and your understanding of it
• Revisions: Up to 2 revisions are included in the price (Package 3: 3 rounds)

Frequently asked questions