02 — EDA & Feature Engineering

Distribusi, skewness, zero inflation, transformasi, dan multikolinearitas

Notebook ini adalah POC portfolio yang dapat dijalankan ulang dari artefak lokal Pasuruan Lens. Kode produksi tetap berada di python/scripts/ dan python/src/pasuruan365/; notebook berfungsi sebagai narasi analitik yang ringkas, transparan, dan mudah dipresentasikan.

from pathlib import Path
import json
import sys

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns


def find_project_root(start: Path | None = None) -> Path:
    start = (start or Path.cwd()).resolve()
    for candidate in (start, *start.parents):
        if (candidate / "package.json").exists() and (candidate / "data").exists():
            return candidate
    raise RuntimeError("Root Pasuruan Lens tidak ditemukan. Jalankan notebook dari repository ini.")


ROOT = find_project_root()
sys.path.insert(0, str(ROOT / "python" / "src"))

pd.set_option("display.max_columns", 100)
pd.set_option("display.max_colwidth", 100)
sns.set_theme(style="whitegrid", context="notebook")
COLORS = {"A": "#2563eb", "B": "#f59e0b", "C": "#10b981"}

print(f"Project root: {ROOT}")
print(f"Python: {sys.version.split()[0]}")

Project root: PASURUAN365
Python: 3.12.6

Tujuan

EDA dipakai untuk memahami karakter data, bukan untuk mencari chart sebanyak mungkin. Fokusnya: distribusi tidak simetris, nilai nol yang dominan, outlier yang mungkin sah, perbedaan desa–kelurahan, dan representasi fitur sebelum modeling.

eda = pd.read_csv(ROOT / "data/analytics/village_eda_features.csv", dtype={"village_id": str})
numeric = [
    "population", "area_km2", "population_density", "sex_ratio",
    "polyclinic_count", "bts_count", "cellular_operator_count",
]

profile = pd.DataFrame({
    "coverage_pct": eda[numeric].notna().mean().mul(100),
    "median": eda[numeric].median(),
    "mean": eda[numeric].mean(),
    "skewness": eda[numeric].skew(),
    "zero_pct": eda[numeric].eq(0).mean().mul(100),
    "p95": eda[numeric].quantile(.95),
    "max": eda[numeric].max(),
}).round(2)
profile.sort_values("skewness", ascending=False)

	coverage_pct	median	mean	skewness	zero_pct	p95	max
bts_count	100.0	1.00	1.01	8.39	45.21	3.00	25.00
area_km2	100.0	2.79	3.66	5.14	0.00	9.29	38.18
population_density	100.0	1524.19	1862.49	4.84	0.00	4273.54	19675.51
polyclinic_count	100.0	0.00	0.19	3.42	86.85	1.00	4.00
population	100.0	3915.00	4564.17	1.76	0.00	9545.40	21782.00
sex_ratio	100.0	99.69	99.38	-0.19	0.00	104.70	109.14
cellular_operator_count	100.0	5.00	4.50	-0.36	0.27	7.00	7.00

POC 1 — Mengapa log1p digunakan secara selektif

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
sns.histplot(eda["population_density"], bins=30, ax=axes[0], color="#64748b")
axes[0].set(title="Population density — original", xlabel="penduduk / km²")
sns.histplot(eda["log1p_population_density"], bins=30, ax=axes[1], color="#2563eb")
axes[1].set(title="Population density — log1p", xlabel="log(1 + density)")
plt.tight_layout()

pd.Series({
    "skew_original": eda["population_density"].skew(),
    "skew_log1p": eda["log1p_population_density"].skew(),
}).round(3).to_frame("nilai")

	nilai
skew_original	4.843
skew_log1p	-0.420

POC 2 — Zero inflation adalah sinyal desain fitur

facility_columns = ["polyclinic_count", "pharmacy_count", "banking_facility_count", "bts_count"]
zero_profile = pd.DataFrame({
    "zero_count": eda[facility_columns].eq(0).sum(),
    "zero_pct": eda[facility_columns].eq(0).mean().mul(100),
    "positive_count": eda[facility_columns].gt(0).sum(),
}).sort_values("zero_pct", ascending=False)
zero_profile.round(1)

	zero_count	zero_pct	positive_count
polyclinic_count	317	86.8	48
banking_facility_count	310	84.9	55
pharmacy_count	301	82.5	64
bts_count	165	45.2	200

ax = zero_profile["zero_pct"].sort_values().plot.barh(figsize=(8, 3.8), color="#f59e0b")
ax.set(xlabel="Persentase nilai nol", ylabel="", xlim=(0, 100), title="Zero inflation pada fasilitas terpilih")
plt.tight_layout()

POC 3 — Pearson vs Spearman

corr_columns = [
    "population", "area_km2", "population_density", "sex_ratio",
    "cellular_operator_count", "bts_count", "banking_facility_count",
]
pearson = eda[corr_columns].corr(method="pearson")
spearman = eda[corr_columns].corr(method="spearman")

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
sns.heatmap(pearson, vmin=-1, vmax=1, cmap="vlag", square=True, ax=axes[0])
axes[0].set_title("Pearson — hubungan linear")
sns.heatmap(spearman, vmin=-1, vmax=1, cmap="vlag", square=True, ax=axes[1])
axes[1].set_title("Spearman — hubungan monotonik/rank")
plt.tight_layout()

Keputusan metodologis

Outlier tidak otomatis dihapus: nilai ekstrem dapat merepresentasikan pusat permukiman, wilayah sangat luas, atau konsentrasi fasilitas nyata. Transformasi dipilih per fitur, dan representasi count/rate/presence tidak dimasukkan bersamaan tanpa review karena dapat menduplikasi informasi.