{
"cells": [
{
"cell_type": "markdown",
"id": "898eff6f",
"metadata": {},
"source": [
"# 02 — EDA & Feature Engineering\n",
"\n",
"**Distribusi, skewness, zero inflation, transformasi, dan multikolinearitas**\n",
"\n",
"Notebook ini adalah POC portfolio yang dapat dijalankan ulang dari artefak lokal Pasuruan Lens. Kode produksi tetap berada di `python/scripts/` dan `python/src/pasuruan365/`; notebook berfungsi sebagai narasi analitik yang ringkas, transparan, dan mudah dipresentasikan."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "8aa9fe7e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-04T06:27:16.278163Z",
"iopub.status.busy": "2026-09-04T06:27:16.277981Z",
"iopub.status.idle": "2026-09-04T06:27:17.896543Z",
"shell.execute_reply": "2026-09-04T06:27:17.896118Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Project root: PASURUAN365\n",
"Python: 3.12.6\n"
]
}
],
"source": [
"from pathlib import Path\n",
"import json\n",
"import sys\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"import seaborn as sns\n",
"\n",
"\n",
"def find_project_root(start: Path | None = None) -> Path:\n",
" start = (start or Path.cwd()).resolve()\n",
" for candidate in (start, *start.parents):\n",
" if (candidate / \"package.json\").exists() and (candidate / \"data\").exists():\n",
" return candidate\n",
" raise RuntimeError(\"Root Pasuruan Lens tidak ditemukan. Jalankan notebook dari repository ini.\")\n",
"\n",
"\n",
"ROOT = find_project_root()\n",
"sys.path.insert(0, str(ROOT / \"python\" / \"src\"))\n",
"\n",
"pd.set_option(\"display.max_columns\", 100)\n",
"pd.set_option(\"display.max_colwidth\", 100)\n",
"sns.set_theme(style=\"whitegrid\", context=\"notebook\")\n",
"COLORS = {\"A\": \"#2563eb\", \"B\": \"#f59e0b\", \"C\": \"#10b981\"}\n",
"\n",
"print(f\"Project root: {ROOT}\")\n",
"print(f\"Python: {sys.version.split()[0]}\")"
]
},
{
"cell_type": "markdown",
"id": "96303a00",
"metadata": {},
"source": [
"## Tujuan\n",
"\n",
"EDA dipakai untuk memahami karakter data, bukan untuk mencari chart sebanyak mungkin. Fokusnya: distribusi tidak simetris, nilai nol yang dominan, outlier yang mungkin sah, perbedaan desa–kelurahan, dan representasi fitur sebelum modeling."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e8f47254",
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-04T06:27:17.900812Z",
"iopub.status.busy": "2026-09-04T06:27:17.900524Z",
"iopub.status.idle": "2026-09-04T06:27:17.933375Z",
"shell.execute_reply": "2026-09-04T06:27:17.932909Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" coverage_pct | \n",
" median | \n",
" mean | \n",
" skewness | \n",
" zero_pct | \n",
" p95 | \n",
" max | \n",
"
\n",
" \n",
" \n",
" \n",
" | bts_count | \n",
" 100.0 | \n",
" 1.00 | \n",
" 1.01 | \n",
" 8.39 | \n",
" 45.21 | \n",
" 3.00 | \n",
" 25.00 | \n",
"
\n",
" \n",
" | area_km2 | \n",
" 100.0 | \n",
" 2.79 | \n",
" 3.66 | \n",
" 5.14 | \n",
" 0.00 | \n",
" 9.29 | \n",
" 38.18 | \n",
"
\n",
" \n",
" | population_density | \n",
" 100.0 | \n",
" 1524.19 | \n",
" 1862.49 | \n",
" 4.84 | \n",
" 0.00 | \n",
" 4273.54 | \n",
" 19675.51 | \n",
"
\n",
" \n",
" | polyclinic_count | \n",
" 100.0 | \n",
" 0.00 | \n",
" 0.19 | \n",
" 3.42 | \n",
" 86.85 | \n",
" 1.00 | \n",
" 4.00 | \n",
"
\n",
" \n",
" | population | \n",
" 100.0 | \n",
" 3915.00 | \n",
" 4564.17 | \n",
" 1.76 | \n",
" 0.00 | \n",
" 9545.40 | \n",
" 21782.00 | \n",
"
\n",
" \n",
" | sex_ratio | \n",
" 100.0 | \n",
" 99.69 | \n",
" 99.38 | \n",
" -0.19 | \n",
" 0.00 | \n",
" 104.70 | \n",
" 109.14 | \n",
"
\n",
" \n",
" | cellular_operator_count | \n",
" 100.0 | \n",
" 5.00 | \n",
" 4.50 | \n",
" -0.36 | \n",
" 0.27 | \n",
" 7.00 | \n",
" 7.00 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" nilai | \n",
"
\n",
" \n",
" \n",
" \n",
" | skew_original | \n",
" 4.843 | \n",
"
\n",
" \n",
" | skew_log1p | \n",
" -0.420 | \n",
"
\n",
" \n",
"
\n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n",
"sns.histplot(eda[\"population_density\"], bins=30, ax=axes[0], color=\"#64748b\")\n",
"axes[0].set(title=\"Population density — original\", xlabel=\"penduduk / km²\")\n",
"sns.histplot(eda[\"log1p_population_density\"], bins=30, ax=axes[1], color=\"#2563eb\")\n",
"axes[1].set(title=\"Population density — log1p\", xlabel=\"log(1 + density)\")\n",
"plt.tight_layout()\n",
"\n",
"pd.Series({\n",
" \"skew_original\": eda[\"population_density\"].skew(),\n",
" \"skew_log1p\": eda[\"log1p_population_density\"].skew(),\n",
"}).round(3).to_frame(\"nilai\")"
]
},
{
"cell_type": "markdown",
"id": "cc461411",
"metadata": {},
"source": [
"## POC 2 — Zero inflation adalah sinyal desain fitur"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c7dbcc7b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-04T06:27:18.411198Z",
"iopub.status.busy": "2026-09-04T06:27:18.410965Z",
"iopub.status.idle": "2026-09-04T06:27:18.421002Z",
"shell.execute_reply": "2026-09-04T06:27:18.420465Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" zero_count | \n",
" zero_pct | \n",
" positive_count | \n",
"
\n",
" \n",
" \n",
" \n",
" | polyclinic_count | \n",
" 317 | \n",
" 86.8 | \n",
" 48 | \n",
"
\n",
" \n",
" | banking_facility_count | \n",
" 310 | \n",
" 84.9 | \n",
" 55 | \n",
"
\n",
" \n",
" | pharmacy_count | \n",
" 301 | \n",
" 82.5 | \n",
" 64 | \n",
"
\n",
" \n",
" | bts_count | \n",
" 165 | \n",
" 45.2 | \n",
" 200 | \n",
"
\n",
" \n",
"
\n",
"