bluegreencity-ki

Validation and Current Operational ML Paths

Status (October 2026, v6): The retraining round 2026_10 is complete. The validation model (train 2018–2023, predict 2024, validate 2025) and the refit-final operational model (train 2018–2024, predict 2025) both live under data/models/autogluon/, and the 2026-10-09 operational-current run populated the canonical current table (irrigation_review.inputs.risk_path points at it). This document is kept in sync with the v6 operational panel (54 features, label future_drought_damage_severe_t1).

Purpose

BlueGreenCity-KI runs two separate prediction products from one versioned model:

Temporal validation path: train on 2018-2023 feature snapshots, predict 2024, and validate against 2025 damage observations.
Current operational path: load the refit-final operational model and apply it to the latest available observation per active tree in the v6 inventory for the current planning year.

The current path does not retrain and does not claim that every inventory row was observed in the planning year. operational-current selects each tree’s latest available observation up to the pinned run year, and each time-dependent criterion (age, DWSI, NDVI, seasonal weather) is additionally pinned to its own last usable year (plan F.3, decision D19), so every feature carries its own provenance-tracked reference year.

The inventory update also requires a fresh Water Balance state. The irrigation_planning safety gate (minimum_risk_join_fraction: 0.95) applies to any later irrigation-planning run; the 2026-10-09 operational run completed with 89,280 scored trees.

October 2026 End-to-End Results (current)

Validation round (data/models/autogluon/annual_validation/2026_10/train_2018_2023__predict_2024__validate_2025):

Temporal holdout 2024: ROC-AUC 0.9697 (OOF control 0.9861, gap 0.0164)
Grouped CV (5 folds, group objectid, 141,394 rows / 29,178 groups): pooled ROC-AUC 0.8881, PR-AUC 0.6654, Brier score 0.0898
Grouped CV (5 folds, group h3_res8, 261 groups): pooled ROC-AUC 0.8032
Results in the model directory; the successor report is pending
Previous round (2026_09): full numbers in reports/operational_panel_grouped_cv_report_2026-09-16.md

Operational model (data/models/autogluon/operational/2026_10/train_2018_2024__predict_2025): refit-final WeightedEnsemble_L2 trained on 365,059 panel rows (2018–2024) with the 54-feature allowlist, predicting 2025 (label future_drought_damage_severe_t1, Platt-calibrated, calibration.json: a=2.2831, b=0.6840, n=25,334). The 2026-10-09 operational-current run (data/output/2026_10_09_ml_operational_train_2018_2024_predict_2025) scored 89,280 trees and wrote the canonical current table.

Why the Paths Stay Separate

The validation path is immutable evidence for temporal generalization. Moving its 2024 holdout to the latest inventory would destroy comparability. The current path answers a different question: what does the validated model predict for the newest city inventory?

Outputs are isolated:

Path	Scenario	Main PostGIS table
Validation	`ml_validation_train_2018_2023_predict_2024_validate_2025`	`bgcki.analysis_tree_risk_scores_validation_2024`
Current	`ml_operational_train_2018_2024_predict_2025`	`bgcki.analysis_tree_risk_scores_operational_current`

ML prediction tables use the prefixes ml_predictions_validation_2024 and ml_predictions_operational_current respectively.

The model window is resolved only from ml.operational_panel. In 2027, advance its end year from 2024 to 2025 only after 2026 labels are available; the same code then resolves training 2018-2024, prediction 2025, and validation 2026.

Manual Runs

Validation and retraining:

conda run -n bgcki_wesemann python scripts/run_ml_path.py validation `
  --config config/config.yaml --train --shap high-risk --skip-approx

Current inference with the existing operational model:

conda run -n bgcki_wesemann python scripts/run_ml_path.py operational-current `
  --config config/config.yaml --current-year 2026 --shap off --skip-approx

--provisional writes to a separate scenario and analysis_tree_risk_scores_operational_current_provisional. It is suitable for technical integration tests and must not be presented as the current operational model result.

Overnight Refresh

The complete v6 refresh rebuilds panel snapshots for 2018-2024, regenerates the training dataset, retrains and validates (with SHAP for the high-risk subset in both the validation and operational runs), runs current inference only if all preceding steps succeed, refreshes the feature input used by the daily Water Balance module, updates the operational risk pointer, and finally synchronizes the Grafana analytics tables (validation metrics, feature importance, SHAP values, leaderboard):

pwsh -NoProfile -File scripts/run_ml_refresh_overnight.ps1 `
  -Config config/config.yaml -CurrentYear 2026 -StartDetached

The detached process survives VS Code closure and writes to logs/ml/ (notes the log file names still carry a cosmetic v4_ prefix).

Drift watch (issue #260). The metrics import keeps one row per monthly retrain in bgcki.ml_validation_history and compares the external holdout against the previous month: a ROC-AUC drop or Brier rise beyond ml.annual_validation.drift_watch.metric_delta (default 0.05) emits a WARNING in the log. The Validierung dashboard shows the monthly trend in two Drift-Watch panels.

Note: after a model feature-set change (e.g. new SWBM features), the SHAP stage silently skips recomputation when a SHAP cache exists. Pass --force-shap-recompute (via a direct python -m bluegreencity_ki.runners.cli --config <runtime.yaml> invocation after run_ml_path.py --write-config-only) to rebuild the SHAP artifacts before the Grafana sync.

Dashboard Direction

Grafana should retain the validation pages for model evidence and add a separate current-operations page sourced from bgcki.analysis_tree_risk_scores_operational_current. That page can later join the Phase 1 SWBM state and the irrigation_plan_decision table. Phase 3 Taipy remains responsible for changing resources, scenarios, overrides, and approvals; Grafana remains read-only monitoring and analysis.

Dashboard population numbers (2026-09 clarification)

The Lagebild counts trees of the current operational run (operational_tree_product, 2026-09-16: 89,280 = the full v6 inventory extract 09/2026, including trees first registered in 2026). The KPI “Bäume im aktuellen Datenstand” counts all scored trees, including vitality stages 4/5; map and analysis panels keep the 4/5 exclusion. The additional KPI “Bäume 2025 mit Prognose 2026” (years derived from ml.operational_panel.end_year) counts the subset with a 2025 inventory observation (~81,379) — the previous-year population that carries the 2026 forecast. The Vorjahresprognose page shows its 2024 holdout population (62,727) the same way.