Missing Data Imputation in Rubber Plantation Production Records and Daily Rainfall Data

Project info

• Field of studyAgriculture and Plantations
• LevelUndergraduate thesis (S1)
• Method typeData Quality
• Estimated build time± 3 working days
• Toolsscikit-learn SimpleImputer, KNNImputer, IterativeImputer
• PriceUS$16 - US$100

Research goal

Fill missing daily rubber production data by imputation using daily rainfall and air temperature and compare the methods.

Variables

Example result

• 23.662Missing %
• 10.768RMSE mean
• 10.871RMSE median
• 10.548RMSE kNN
• 9.923RMSE iterative
• 7.844Best-method gain % vs mean

With 23.7% of Daily rubber production missing, Iterative imputation gives the lowest RMSE (9.923) versus mean imputation (10.768).

Computed from illustrative synthetic data, not from a real study.

About Missing Data Imputation

Imputation fills missing data with estimated values so analyses do not lose many observations. Common methods are mean or median imputation, k-NN, and regression-based iterative imputation.

Steps

1. 1Compute the pattern and percentage of missing data
2. 2Consider the missingness mechanism (MCAR, MAR, MNAR)
3. 3Choose an imputation method (median, KNNImputer, IterativeImputer)
4. 4Fit the imputer on training data only, then apply to test data
5. 5Compare imputed values with true values (in simulation)
6. 6Check the effect on distributions and model results

Assumptions to check

• The missingness mechanism is at least MAR for imputation to be reasonable
• The imputer is fit on training data only to prevent leakage
• Single imputation shrinks variance so uncertainty should be discussed

Reading and reporting the results

Report the missing percentage, the imputation method with its rationale, and its effect on results (for example imputation RMSE or coefficient change). Add a sensitivity analysis.

Example Python code

A short starting script with this project's variable names. The delivered version is adapted to your data and fully commented.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import SimpleImputer, KNNImputer, IterativeImputer

rng = np.random.default_rng(37)
n = 300
daily_rainfall = rng.normal(0, 1, n)
air_temperature = 0.7 * daily_rainfall + rng.normal(0, 0.5, n)
daily_rubber_production = daily_rainfall + air_temperature + rng.normal(0, 0.5, n)
full = np.column_stack([daily_rainfall, air_temperature, daily_rubber_production])

X = full.copy()
miss = rng.random(n) < 0.2
X[miss, 1] = np.nan
print("Missing in x2 (%):", round(100 * miss.mean(), 1))

methods = {
    "Median": SimpleImputer(strategy="median"),
    "KNN": KNNImputer(n_neighbors=5),
    "Iterative": IterativeImputer(random_state=37),
}
rmse = {}
for name, imp in methods.items():
    out = imp.fit_transform(X)
    rmse[name] = np.sqrt(np.mean((out[miss, 1] - full[miss, 1]) ** 2))
    print(name, "imputation RMSE:", round(rmse[name], 3))

plt.bar(list(rmse.keys()), list(rmse.values()))
plt.ylabel("Imputation RMSE")
plt.show()

Packages and prices

Choose how much help you need. Each higher package includes everything in the lower ones. Prices exclude 11% VAT.

• Payment: 50% down payment before work starts, balance after the deliverables are handed over
• Tax: Prices exclude 11% VAT
• Data and academic integrity: Results are computed from your own research data; you write and stand behind your manuscript, we help with the analysis and your understanding of it
• Revisions: Up to 2 revisions are included in the price (Package 3: 3 rounds)

Frequently asked questions