01 — Data Foundation & Quality¶

Identitas kanonik, integritas join, missingness, dan lineage

Notebook ini adalah POC portfolio yang dapat dijalankan ulang dari artefak lokal Pasuruan Lens. Kode produksi tetap berada di python/scripts/ dan python/src/pasuruan365/; notebook berfungsi sebagai narasi analitik yang ringkas, transparan, dan mudah dipresentasikan.

from pathlib import Path
import json
import sys

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns


def find_project_root(start: Path | None = None) -> Path:
    start = (start or Path.cwd()).resolve()
    for candidate in (start, *start.parents):
        if (candidate / "package.json").exists() and (candidate / "data").exists():
            return candidate
    raise RuntimeError("Root Pasuruan Lens tidak ditemukan. Jalankan notebook dari repository ini.")


ROOT = find_project_root()
sys.path.insert(0, str(ROOT / "python" / "src"))

pd.set_option("display.max_columns", 100)
pd.set_option("display.max_colwidth", 100)
sns.set_theme(style="whitegrid", context="notebook")
COLORS = {"A": "#2563eb", "B": "#f59e0b", "C": "#10b981"}

print(f"Project root: {ROOT}")
print(f"Python: {sys.version.split()[0]}")

Project root: PASURUAN365
Python: 3.12.6

Mengapa tahap ini penting¶

Nama desa dapat berubah ejaan, kode antar-sumber belum tentu setara, dan nilai kosong tidak boleh otomatis dianggap nol. Karena itu, unit analisis dikunci lebih dahulu menggunakan master Kemendagri dan seluruh sumber dipetakan ke village_id yang sama.

master = pd.read_csv(ROOT / "data/reference/village_master.csv", dtype={"village_id": str})
features = pd.read_csv(ROOT / "data/processed/village_features.csv", dtype={"village_id": str})
lineage = pd.read_csv(ROOT / "data/processed/village_feature_lineage.csv", dtype={"village_id": str})

pd.DataFrame({
    "rows": [len(master), len(features), len(lineage)],
    "columns": [master.shape[1], features.shape[1], lineage.shape[1]],
    "unique_village_id": [master.village_id.nunique(), features.village_id.nunique(), lineage.village_id.nunique()],
}, index=["canonical_master", "processed_features", "value_lineage"])
rows columns unique_village_id
canonical_master 365 13 365
processed_features 365 98 365
value_lineage 17922 12 365

POC 1 — Validasi identitas dan grain¶

identity_checks = pd.Series({
    "365 unit aktif": len(master) == 365 and master["is_active"].astype(str).str.lower().eq("true").all(),
    "village_id unik": master.village_id.is_unique,
    "24 kecamatan": master.district_id.nunique() == 24,
    "341 desa": master.administrative_type.eq("DESA").sum() == 341,
    "24 kelurahan": master.administrative_type.eq("KELURAHAN").sum() == 24,
    "fitur tidak kehilangan unit": set(features.village_id) == set(master.village_id),
    "lineage hanya memakai ID kanonik": set(lineage.village_id).issubset(set(master.village_id)),
})
identity_checks.to_frame("lulus")
lulus
365 unit aktif True
village_id unik True
24 kecamatan True
341 desa True
24 kelurahan True
fitur tidak kehilangan unit True
lineage hanya memakai ID kanonik True

POC 2 — Missingness bukan zero¶

analysis_columns = [
    "population", "area_km2", "population_density", "pustu_count",
    "banking_facility_count", "flood_event_presence_2024",
    "temporary_waste_collection_presence",
]
quality_profile = pd.DataFrame({
    "dtype": features[analysis_columns].dtypes.astype(str),
    "non_null": features[analysis_columns].notna().sum(),
    "missing": features[analysis_columns].isna().sum(),
    "zero": features[analysis_columns].eq(0).sum(),
    "unique": features[analysis_columns].nunique(dropna=True),
})
quality_profile["coverage_pct"] = (quality_profile["non_null"] / len(features) * 100).round(1)
quality_profile.sort_values(["coverage_pct", "unique"])
dtype non_null missing zero unique coverage_pct
pustu_count float64 118 247 112 2 32.3
flood_event_presence_2024 int64 365 0 301 2 100.0
temporary_waste_collection_presence int64 365 0 191 2 100.0
banking_facility_count int64 365 0 310 9 100.0
area_km2 float64 365 0 0 270 100.0
population int64 365 0 0 359 100.0
population_density float64 365 0 0 365 100.0 
example = quality_profile.loc[["pustu_count", "flood_event_presence_2024"]]
assert example.loc["pustu_count", "missing"] > 0
assert example.loc["flood_event_presence_2024", "missing"] == 0
print("Pustu: sebagian null karena skema sumber tidak lengkap.")
print("Hazard biner: cakupan lengkap; 0 berarti tidak tercatat pada periode referensi, bukan 'aman'.")

Pustu: sebagian null karena skema sumber tidak lengkap.
Hazard biner: cakupan lengkap; 0 berarti tidak tercatat pada periode referensi, bukan 'aman'.

POC 3 — Audit lineage untuk satu desa dan fitur¶

sample_id = master.loc[0, "village_id"]
audit = lineage.loc[
    lineage.village_id.eq(sample_id)
    & lineage.feature_name.isin(["population", "area_km2", "population_density", "flood_event_presence_2024"]),
    ["village_id", "feature_name", "value", "source_id", "source_variable", "reference_year", "raw_table", "validation_status", "lineage_expression"],
].sort_values("feature_name")
audit
village_id feature_name value source_id source_variable reference_year raw_table validation_status lineage_expression
1 35.14.01.2001 area_km2 13.11 SRC-BPS-KDA-2025-TABLE-1.1 Luas Total Area (km2/sq.km) 2024 1.1 VALID NaN
10 35.14.01.2001 flood_event_presence_2024 0 SRC-BPS-KDA-2025-TABLE-4.4.2 Floods Ada/Tidak Ada 2024 4.4.2 VALID DIRECT_SOURCE_VALUE
21 35.14.01.2001 population 9387 SRC-BPS-KDA-2025-TABLE-3.1 Penduduk/Population - Jumlah/Total 2024 3.1 VALID NaN
22 35.14.01.2001 population_density 716.0183066361557 SRC-DERIVED-TASK03B population / area_km2 2024 DERIVED VALID population / area_km2

Keputusan metodologis¶

  • Join produksi menggunakan kode kanonik, bukan fuzzy matching nama.
  • Agregat kecamatan tidak disalin menjadi nilai desa.
  • Missing tetap missing kecuali sumber secara eksplisit mendefinisikan zero.
  • Setiap nilai yang dipromosikan memiliki sumber, periode, status validasi, dan—untuk fitur turunan—rumus lineage.