bluegreencity-ki

Operational Panel Grouped CV Report (2026-09-16)

Successor to the 2026-06-08 grouped CV report, which is now archived. This report covers the 2026_09 retraining round on the v6 operational panel (35 features, label future_drought_damage_severe_t1, time limit 3600 s).

Purpose

This report documents the 5-fold grouped cross-validation of the September 2026 operational panel model. The goal is unchanged: provide a stricter internal robustness check for the panel dataset than AutoGluon’s row-wise out-of-fold validation, because the training data contain repeated yearly observations of the same tree. Grouped cross-validation assigns all rows of one tree to the same fold, so within-tree leakage across folds is excluded.

Run Setup

Parameter	Value
Label	`future_drought_damage_severe_t1`
Training data	`data/derived/ml/training_dataset_v6_operational_panel_2018_2024.csv` (v6, 35 features)
Group column	`objectid`
Splitter	`StratifiedGroupKFold`
Number of folds	`5`
AutoGluon preset	`best_quality` (documented portfolio, see `ml.training.hyperparameters`)
Time limit	`3600` seconds per fold
Threshold	`0.5`
Round	`2026_09` (run directory `annual_validation/2026_09/train_2018_2023__predict_2024__validate_2025`)

The grouped CV run is part of the validation round (scripts/run_ml_path.py validation --grouped-cv) and writes its artefacts to <run_dir>/grouped_cv/:

summary.json (pooled + per-fold + per-year metrics)
fold_metrics.csv
per_year_metrics.csv
decile_metrics.csv, top_k_metrics.csv
oof_predictions.csv (one row per panel observation, carrying fold + per-year OOF probabilities)

Companion artefacts of the same round live in the run directory: temporal_holdout/ (external 2024 holdout), leaderboards/, feature_importances/, and the refit-final operational model under data/models/autogluon/operational/2026_09/train_2018_2024__predict_2025.

Panel Changes vs. the June 2026 Run

Aspect	2026-06-08	2026-09-16
Dataset version	v4-era (`2026_06_07_` training CSV)	v6 operational panel
Rows / groups	127,416 / 23,870	141,394 / 29,178
Positive prevalence	0.1879	0.1675
Features	48	35 (Plan Block C + Issue #217 reductions)
Time limit per fold	1,800 s	3,600 s

Aggregated 5-Fold Results

From <run_dir>/grouped_cv/summary.json:

Metric	Value
ROC-AUC (pooled OOF)	`0.8563`
ROC-AUC (fold mean ± std)	`0.8655 ± 0.0113`
PR-AUC	`0.6121`
Precision	`0.6847` (mean `0.6893 ± 0.0334`)
Recall	`0.4211` (mean `0.4206 ± 0.0377`)
F1	`0.5214` (mean `0.5203 ± 0.0228`)
Accuracy	`0.8706`
Brier score	`0.099311` (mean `0.0993 ± 0.0073`)
Rows	`141,394`
Groups	`29,178`
Positive prevalence	`0.1675`

Fold-Level Results

Fold	Test rows	Test groups	ROC-AUC	Brier	Precision	Recall	F1
1	`28,054`	`5,802`	`0.8691`	`0.10074`	`0.6717`	`0.4713`	`0.5539`
2	`28,325`	`5,849`	`0.8703`	`0.09227`	`0.7327`	`0.3650`	`0.4873`
3	`28,243`	`5,833`	`0.8449`	`0.110677`	`0.6458`	`0.4288`	`0.5154`
4	`28,510`	`5,868`	`0.8643`	`0.102298`	`0.6723`	`0.4452`	`0.5357`
5	`28,262`	`5,826`	`0.8790`	`0.090577`	`0.7241`	`0.3926`	`0.5091`

Fold-wise ROC-AUC stays within 0.8449–0.8790; recall swings more (0.3650–0.4713), which is expected for a class-imbalanced panel.

Per-Year OOF Metrics

From <run_dir>/grouped_cv/per_year_metrics.csv (plan E.1):

Year	Rows	Prevalence	ROC-AUC	PR-AUC	Brier
2018	15,778	0.1675	`0.8800`	`0.6723`	`0.089714`
2019	17,361	0.1740	`0.8786`	`0.6768`	`0.09155`
2020	19,064	0.1675	`0.8685`	`0.6375`	`0.095624`
2021	21,048	0.1737	`0.8541`	`0.6140`	`0.103643`
2022	21,199	0.1647	`0.8541`	`0.6050`	`0.099428`
2023	21,610	0.1741	`0.8377`	`0.5749`	`0.110477`
2024	25,334	0.1544	`0.8337`	`0.5527`	`0.100158`

Discrimination declines gently toward the later (drier, higher-stress) years; there is no single weak year that the aggregate would hide. Prevalence drops to 0.1544 in 2024, consistent with the temporal holdout.

Ranking Quality (Top-k and Deciles)

From top_k_metrics.csv and decile_metrics.csv:

Fraction	Top-N	Prevalence	Capture rate	Lift	Score cutoff
5 %	7,070	0.7956	0.2376	4.75	0.8186
10 %	14,140	0.6902	0.4122	4.12	0.5155
25 %	35,349	0.4753	0.7096	2.84	0.1263

The top decile (D1) contains 41.2 % of all severe-damage positives at 4.12× lift; the extreme top-5 % captures 23.8 % of positives at 4.75× lift — comfortably above the random baseline (1.0×).

External Temporal Holdout (2024)

From layout.json / temporal_holdout/summary.json of the same round:

Metric	Value
ROC-AUC	`0.9647`
PR-AUC	`0.8968`
Precision	`0.9610`
Recall	`0.4162`
Accuracy	`0.9072`
Brier score	`0.062924`
Rows / prevalence	25,334 / 0.1544
Extreme group lift (top 5 %)	174.43

The best model (WeightedEnsemble_L2, refit-final) achieves the same ROC-AUC on the external 2024 holdout as in the leaderboard (score_test = 0.96469).

Interpretation

Robustness confirmed on v6. Pooled grouped-CV ROC-AUC 0.8563 (fold mean 0.8655 ± 0.0113) on 29,178 groups shows the 35-feature model still separates future severe drought damage well under group-aware splits.
Slightly below June 2026 (0.8817) for structural reasons. The panel is larger (+14k rows), prevalence is ~2 points lower (0.1675 vs 0.1879), and the feature set was reduced to 35 columns. Part of the gap is the harder, drier 2023–2024 tail (ROC-AUC 0.8337–0.8377).
Still materially below the internal OOF reference. The random-fold OOF value for this round is 0.9831 (layout.json::oof_metrics); the ~0.13 gap confirms that row-wise internal validation is optimistic for this panel structure. Grouped CV remains the preferred internal robustness estimate.
External generalization is strong. The out-of-time 2024 holdout (ROC-AUC 0.9647) stays far above the grouped-CV estimate, indicating the gap is mostly structural (repeated trees) rather than temporal generalization. The decisive forward test remains the 2025→2026 validation round.