Data Quality Assessment of web server access logs with Exploratory Data Analysis (EDA)

Project info

• Field of studyInformatics and Information Systems
• LevelUndergraduate thesis (S1)
• Method typeData Quality
• Estimated build time± 3 working days
• Toolspandas describe, isna, corr; matplotlib hist, boxplot, imshow
• PriceUS$19 - US$113

Research goal

Inspect and improve the quality of response time data with error status code and response size using Exploratory Data Analysis (EDA), then assess the impact on analysis.

Variables

Example result

• 331Observations
• 119.685Mean
• 18.475Std. deviation
• -0.19Skewness
• -0.081Excess kurtosis
• 0.34Max |correlation|

EDA of 331 observations: Response time averages 119.68 (SD 18.48) with skewness -0.190; the strongest pairwise correlation is 0.340.

Computed from illustrative synthetic data, not from a real study.

About Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) summarises and visualises data to understand distributions, relations, patterns, and quality issues before modelling. It is always an early step in the methodology or results chapter of a data-based thesis.

Steps

1. 1Load the data and check dimensions and variable types
2. 2Compute descriptive statistics and missing counts
3. 3Visualise each variable's distribution (histogram, boxplot)
4. 4Compute and visualise correlations
5. 5Check outliers and class imbalance
6. 6Note findings that affect modelling steps

Assumptions to check

• The data context is understood so units and variable types are clear
• Visualisations match variable types
• Exploratory findings are not used as formal hypothesis tests

Reading and reporting the results

Summarise descriptive statistics in a table and present two to four key plots supporting modelling decisions. Explain missing data, outliers, and strong correlations.

Example Python code

A short starting script with this project's variable names. The delivered version is adapted to your data and fully commented.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(35)
n = 250
error_status_code = rng.normal(40, 8, n)
response_size = rng.gamma(2.0, 3.0, n)
response_time = 0.5 * error_status_code + 2 * response_size + rng.normal(0, 5, n)
df = pd.DataFrame({"x1": error_status_code, "x2": response_size, "y": response_time})
df.loc[rng.choice(n, 12, replace=False), "x2"] = np.nan

print(df.shape)
print(df.dtypes)
print(df.describe().round(2))
print("Missing per column:\n", df.isna().sum())
corr = df.corr()
print(corr.round(2))

fig, ax = plt.subplots(2, 2, figsize=(10, 8))
ax[0, 0].hist(df["x1"], bins=25); ax[0, 0].set_title("x1")
ax[0, 1].hist(df["x2"].dropna(), bins=25); ax[0, 1].set_title("x2")
ax[1, 0].boxplot([df["x1"], df["x2"].dropna(), df["y"]])
ax[1, 0].set_xticklabels(["x1", "x2", "y"])
im = ax[1, 1].imshow(corr, vmin=-1, vmax=1, cmap="coolwarm")
ax[1, 1].set_xticks(range(3)); ax[1, 1].set_xticklabels(corr.columns)
ax[1, 1].set_yticks(range(3)); ax[1, 1].set_yticklabels(corr.columns)
plt.colorbar(im, ax=ax[1, 1])
plt.tight_layout()
plt.show()

Packages and prices

Choose how much help you need. Each higher package includes everything in the lower ones. Prices exclude 11% VAT.

• Payment: 50% down payment before work starts, balance after the deliverables are handed over
• Tax: Prices exclude 11% VAT
• Data and academic integrity: Results are computed from your own research data; you write and stand behind your manuscript, we help with the analysis and your understanding of it
• Revisions: Up to 2 revisions are included in the price (Package 3: 3 rounds)

Frequently asked questions