bluegreencity-ki

Validation Framework

The validation framework assesses the predictive power of the drought risk model by validating pipeline scores (static and ML) against actual crown damage from historical tree inventory data.

Overview

The framework answers one central question: How well do our risk scores predict actual drought damage?

Four model variants are compared:

Model	Method	ROC-AUC
Static (April 2026 experiment, 20 criteria)	Weighted sum, blend 55/45	0.6148
Monotone Meta (April 2026 experiment)	OOF HistGradientBoosting on the 20 static risk features	0.7401
ML (OOF control)	AutoGluon out-of-fold validation on the 2026_10 panel model	0.9861
ML (Grouped CV by objectid)	StratifiedGroupKFold, 5 folds, one tree per fold only	0.8881

The ML rows reflect the current October 2026 reference run (2026_10, 54 features, label future_drought_damage_severe_t1); the grouped CV by h3_res8 scores 0.8032. The Static/Monotone rows are the April 2026 static weighting experiment (20 criteria; the active configuration now weights 25 criteria).

For the operational panel model, the standard AutoGluon OOF value is kept as an internal upper reference. The grouped CV value is the stricter internal robustness estimate because it prevents yearly observations of the same tree from appearing in different folds.

The monotone meta variant is documented here as an experimental offline benchmark only. It is not part of the productive pipeline recommendation and is not written as an operational risk column.

Validation runs automatically on every pipeline run (opt-out via --skip-validation).

The grouped CV workflow is deliberately separate from the default pipeline validation, because it retrains the model fold by fold with a custom splitter. When run as part of a training round (CLI stage 6 / run_ml_path.py validation --grouped-cv), its artefacts are written into the validation run directory (<run_dir>/grouped_cv/); the standalone script scripts/ml/evaluate_grouped_cv.py still writes to its own --output-dir (default data/models/autogluon/grouped_cv/).

For year-based external validation of the modelling approach, use scripts/ml/evaluate_temporal_holdout.py. This workflow trains on all panel rows with snapshot_year < holdout_year and validates on the exact holdout year snapshot_year == holdout_year. For example, holdout_year = 2024 answers the question: “If we had trained only on data available up to end-2024, how well would the approach have predicted the 2025 outcomes of the 2024 snapshot?”. The artefacts are written into one consistent run folder per model round, for example data/models/autogluon/annual_validation/2026_09/train_2018_2023__predict_2024__validate_2025/, with subfolders for model/, leaderboards/, feature_importances/, grouped_cv/, shap/, and temporal_holdout/. The <YYYY_MM> timestamp segment is created fresh for every training round, so each round’s metrics stay isolated. Label-specific file names and the short model alias remain inside that run folder.

In the operational pipeline, validation artefacts are now persisted in both places:

• as files in the report output directory,
• as tabular PostGIS tables for downstream BI tools such as Apache Superset.

Default table families are written with the prefixes validation_* and include, among others:

• run summary,
• static and ML overall metrics,
• extreme-group tables,
• top-k ranking tables,
• decile prevalence tables,
• calibration curves,
• species metrics,
• side-by-side comparison.

Architecture

CLI (--skip-validation)
  │
  ▼
risk_scoring.run_with_validation(ctx)         ← pure orchestration
  ├─ run(ctx)                                 ← 25 criteria + ML inference
  └─ plausibility checks

The legacy “pipeline run monitoring” validation (comparing the current operational inference against same-year ground truth) has been removed: for current-year inference the ground truth does not exist yet. Model quality is assessed by the temporal holdout (out-of-time) and the grouped CV instead.

Module Structure

bluegreencity_ki/
├── validation/
│   ├── __init__.py
│   ├── validator.py         # ModelValidator, metric helpers (reuse)
│   └── pipeline_validation.py  # Retrospective pipeline-run validation
├── pipelines/
│   └── risk_scoring.py      # run_with_validation() — pure orchestration
├── ml/
│   ├── labels/
│   │   └── label_extractor.py  # Ground-truth extraction
│   └── training/
│       ├── temporal_holdout.py # Temporal holdout / annual validation (Stage 2)
│       └── grouped_cv.py       # Grouped CV (separate)
└── runners/
    └── cli.py               # --skip-validation flag

Validation Categories

The validation framework covers four distinct categories, each with a clear purpose:

Category	Purpose	When to Use
Grouped CV	Internal robustness of the model on training panel	Training round (CLI stage 6) or separate script (scripts/ml/evaluate_grouped_cv.py)
Temporal Holdout	Out-of-time validation on the holdout year	Training round (CLI stage 2)
OOF Diagnostics	Internal training diagnosis (out-of-fold predictions)	Model development, not external validation
External 2025→2026 Validation	True future validation	Later: separate path for real temporal holdout

Important wording:

• Grouped CV remains separate because it retrains fold by fold.
• OOF is an internal training diagnostic, not the external validation of a prediction run.
• Rule-based score is a baseline diagnostic, not the main validation claim for the new operational ML model.

Ground Truth: Crown Damage

Labels are extracted from historical crown condition assessments in the tree inventory (LabelExtractor).

Current Target: future_drought_damage_severe_t1 (definition B+)

The operative panel label (LHH-aligned, 2026-08-30/31) is a next-year (t1) label per snapshot year:

score = 2*wipfelduerre + 1*belaubung + 1*vitality_change_t1
severe = score >= 2 OR felling_drought_t1

• Crown dieback (Wipfeldürre) alone is severe, or any two of the three signals.
• vitality_change_t1: last vitality observation of the label window year worse (0–5 scale) than the last observation of the snapshot year.
• Felling extension (issue #210): a tree felled inside the label window with reason „absterben” is a severe case even without a crown observation.
• Totholz is deliberately excluded: it is a cumulative record (91.8% prevalence in the label population), not a next-year signal.
• Panel prevalence: 16.75% (2026_10 validation panel, n=141,394).
• Extractor: LabelExtractor.extract_operational_drought_damage_labels().

Legacy Definition A (historical, pre-2026-08)

The initial target drought_damage_severe combined three indicators:

Indicator	Column in Tree Inventory	Weight	Binary
Crown dieback (Wipfeldürre)	Krone Wipfeldürre	×3	J=1, N=0
Dead wood (Totholzbildung)	Krone Totholzbildung	×2	J=1, N=0
Foliage anomalies (Belaubung)	Krone Belaubung (Auffälligkeiten)	×1	J=1, N=0

drought_damage_score = wipfelduerre × 3 + totholz × 2 + belaubung × 1

• Label: drought_damage_severe = 1 if score ≥ 3
• Prevalence at the time: 26.5% (7,119 of 26,856 trees)
• Aggregation: per tree, the MAX value across all observations up to cutoff_year (conservative: once damaged = damaged).

LabelExtractor Output

Column	Type	Description
ID	int	Tree ID (= objectid)
wipfelduerre_binary	Int64	0/1/NaN
totholz_binary	Int64	0/1/NaN
belaubung_auffaellig_binary	Int64	0/1/NaN
drought_damage_binary	int	1 if any damage observed
drought_damage_score	int	Weighted score (0–6)
damage_years_count	int	Number of years with damage
first_damage_year	int	Earliest damage year

CLI Integration

Default Behavior (Validation ON)

python -m bluegreencity_ki.runners.cli --config config/config.yaml

Disable Validation

python -m bluegreencity_ki.runners.cli --config config/config.yaml --skip-validation

Evaluate the Monotone Meta Variant Separately

python scripts/ml/archive/evaluate_monotone_static_meta_model.py --config config/config.yaml

The archived script reuses the current pipeline output, reconstructs the static score from the active config weights, trains a monotone 5-fold OOF comparison model on the *_risk features, and writes its artefacts to data/output/<study_area>/report/validation/monotone_meta/.

This evaluation is documented only for methodological comparison against the active static 55/45 configuration and the ML model. It is intentionally kept as a benchmark path and is not part of the productive output path.

Grouped CV for the Operational Panel Model

The grouped cross-validation is run separately because AutoGluon does not provide tree-grouped fold logic for panel data out of the box.

python -m scripts.ml.evaluate_grouped_cv \
  --config config/config.yaml \
  --labels future_drought_damage_severe_t1 \
  --group-column objectid \
  --n-splits 5

The evaluation uses StratifiedGroupKFold with objectid as the grouping column. All observations of a tree therefore stay within the same fold.

Outputs (standalone script; the training-round run writes to <run_dir>/grouped_cv/ instead):

• data/models/autogluon/grouped_cv/summary.csv
• data/models/autogluon/grouped_cv/future_drought_damage_severe_t1/summary.json
• data/models/autogluon/grouped_cv/future_drought_damage_severe_t1/fold_metrics.csv
• data/models/autogluon/grouped_cv/future_drought_damage_severe_t1/oof_predictions.csv

Current reference run (2026_10 round, group objectid):

• ROC-AUC: 0.8881
• PR-AUC: 0.6654
• Precision: 0.7035
• Recall: 0.4350
• F1: 0.5376
• Brier score: 0.0898

(Previous rounds: 2026_09 pooled AUC 0.8563; June 2026 0.8817 — archived report.)

For panel data, these grouped-CV metrics are the more relevant internal robustness estimate than the standard AutoGluon OOF output.

The current default parameter set of the monotone comparison variant is:

• max_depth=4
• learning_rate=0.05
• min_samples_leaf=30
• max_iter=300
• l2_regularization=0.0

config.yaml

validation:
  enabled: true                  # Default: true (opt-out instead of opt-in)
  mode: "pipeline_monitoring"    # pipeline_monitoring | expert_validation | external_2025_2026
  gisid_column: "objectid"       # ID column for pipeline ↔ labels join
  min_validation_matches: 50     # Minimum non-null labels required
  print_report: true             # Formatted report to console
  primary_score_column: "drought_risk_ml"
  threshold: 0.5                 # Binary classification threshold
  top_k_fractions: [0.05, 0.10, 0.20]  # Top-k ranking fractions
  baseline:
    enabled: true
    score_column: "drought_risk_rulebased"

output:
  postgis:
    enabled: true
    schema: "bgcki"
    artifacts:
      enabled: true
      validation_table_prefix: "validation"

ml:
  enabled: true
  drought_csv: "data/input/trees/Baum-2010-2025-v6_historisch_prod.csv"
  drought_start_year: 2018
  drought_cutoff_year: 2024
  training:
    model_path: "data/models/autogluon"
    num_bag_folds: 5       # Generates OOF predictions
  inference:
    label: "future_drought_damage_severe_t1"

API Reference

Data Classes

ValidationResult

@dataclass
class ValidationResult:
    overall_metrics: Dict[str, float]       # ROC-AUC, F1, Brier, etc.
    extreme_group: Dict[str, Any]           # Top vs bottom quantile
    top_k_metrics: Optional[pd.DataFrame]   # Top-k prevalence / capture / lift
    decile_analysis: Optional[pd.DataFrame] # Ranked decile prevalence table
    species_metrics: Dict[str, Dict]        # Per-species metrics
    calibration: Optional[pd.DataFrame]     # Reliability diagram
    baseline_comparison: Dict[str, Any]     # ML vs static comparison
    temporal_holdout: Dict[str, Any]        # Temporal holdout analysis

Method summary() returns a formatted summary string.

OofResult

@dataclass
class OofResult:
    ml_proba: pd.Series                     # Predicted probabilities
    y_true: pd.Series                       # Ground truth (aligned)
    source: str                             # "oof" or "inference"
    ml_proba_aligned: Optional[pd.Series]   # Full length, NaN for unmatched

ModelValidator

Central class for validating a single model.

Constructor

ModelValidator(y_true, y_proba, threshold=0.5)

Factory for Static Scores

ModelValidator.from_static_scores(y_true, scores, threshold_quantile=0.75)

Min-max normalizes the weighted-sum scores to [0,1] and derives the decision threshold from the specified quantile (default: top 25% = positive).

Methods

Method	Description
compute_overall_metrics()	ROC-AUC, Brier, F1, Precision, Recall, Accuracy
compute_extreme_group(quantile=0.05)	Top vs bottom quantile analysis
compute_top_k_metrics(fractions=[0.05, 0.10, 0.25])	Ranking quality for top-k percent cohorts
compute_decile_analysis(n_groups=10)	Decile prevalence, lift and cumulative capture
compute_species_metrics(species, min_samples=30)	Per-species evaluation
compute_calibration(n_bins=10)	Reliability diagram data
compare_with_baseline(baseline_scores)	Spearman correlation, disagreement rate
run_all(species, baseline_scores)	All analyses in a single call
export_report(result, output_dir)	CSV export of all artifacts

Metric Functions

Function	Description
roc_auc_score(y_true, y_proba)	ROC-AUC without sklearn (pure NumPy)
binary_metrics(y_true, y_pred)	TP, FP, FN, TN, Precision, Recall, F1
brier_score(y_true, y_proba)	Mean squared error of probabilities
spearman_rank_correlation(a, b)	Spearman rank correlation
calibration_curve(y_true, y_proba, n_bins)	Binned calibration data
extreme_group_comparison(scores, labels, quantile)	Lift analysis
top_k_capture_table(scores, labels, fractions)	Top-k prevalence, capture and lift
decile_prevalence_table(scores, labels, n_groups=10)	Prevalence by ranked deciles

Analysis Functions

load_oof_predictions(ml_cfg, merged, y_true, id_col)

Loads ML predictions with preference for OOF (out-of-fold) over pipeline inference:

1. Attempts to load AutoGluon predict_proba_oof() (honest cross-validation)
2. Maps OOF IDs to the merged pipeline data
3. Requires ≥ 50% match, otherwise falls back
4. Fallback: pipeline drought_risk_ml column (inference)

Returns OofResult with source="oof" or source="inference".

build_comparison(static_result, ml_result)

Builds a side-by-side comparison:

{  # example values from the April 2026 run
  "static_roc_auc": 0.6148,
  "ml_roc_auc": 0.9117,
  "auc_improvement": 0.2969,
  "static_f1": 0.3673,
  "ml_f1": 0.6587,
}

per_indicator_extreme_groups(labels_df, merged, id_col, static_scores, ml_proba)

Runs extreme-group analysis separately for each damage indicator:

Indicator	n	Prevalence	S-AUC	ML-AUC
Crown dieback (×3)	project dependent	project dependent	exported per run	exported per run
Dead wood (×2)	project dependent	project dependent	exported per run	exported per run
Foliage anomalies (×1)	project dependent	project dependent	exported per run	exported per run
Any damage (binary OR)	project dependent	project dependent	exported per run	exported per run
Severe damage (legacy def. A, score ≥ 3)	21,544 (Apr 2026 run)	20.47%	0.615	0.912

multi_quantile_extreme_groups(static_proba, y_true_static, ml_proba, y_true_ml)

Compares extreme groups at multiple quantile thresholds:

Quantile	Static Lift	ML Lift	Static Top%	ML Top%
5%	0.81×	1028×	31.5%	95.4%
10%	1.71×	470.75×	35.4%	87.4%
25%	2.58×	78.44×	33.4%	62.6%

Interpretation: In the active Hannover run, the static score is still only a coarse ranking signal, while the ML model strongly concentrates positives in the top-ranked cohorts. The new top-k and decile exports make this visible without relying only on ROC-AUC.

print_validation_report(results)

Prints a formatted report to stdout:

========================================================================
  VALIDATION REPORT - Static vs ML Drought Risk
========================================================================
  Trees: 21544 matched / 76616 total

  Static (weighted sum):
    ROC-AUC:   0.6148
    F1:        0.3673
    Top 10% prevalence: 35.4% (capture=17.3%, lift=1.73x)

  ML (AutoGluon, oof):
    ROC-AUC:   0.9117
    F1:        0.6587
    Top 10% prevalence: 87.4% (capture=42.7%, lift=4.27x)

  Improvement (ML vs Static):
    ROC-AUC: +0.2969
    F1:      +0.2914
========================================================================

Pipeline Flow in Detail

run_with_validation(ctx) — Orchestration

def run_with_validation(ctx) -> tuple[pd.DataFrame, dict, Profiler, dict, dict, dict]:

Steps:

1. Run pipeline — run(ctx) computes 25 criteria + ML inference → df (89,280 trees in the current v6 inventory)
2. Check config — validation.enabled → early return if false
3. Load labels — LabelExtractor (current target: future_drought_damage_severe_t1, definition B+)
4. Compute severity — severe = (2*wipfelduerre + 1*belaubung + 1*vitality_change_t1 >= 2) OR felling_drought_t1
5. Join — Pipeline output (objectid) with labels (ID) → inner join → merged
6. Static validation — ModelValidator.from_static_scores(y_true, drought_risk_rulebased)
7. ML validation — load_oof_predictions() → ModelValidator(oof.y_true, oof.ml_proba)
8. Comparison — build_comparison(static_result, ml_result)
9. Per-indicator — per_indicator_extreme_groups()
10. Multi-quantile — multi_quantile_extreme_groups()
11. Report — print_validation_report()

Error handling:

• Missing drought_csv → warning, empty dict returned
• < 50 matched trees → warning, {"error": "too few matches"}
• Analysis errors → caught per try/except, pipeline continues

OOF vs Inference

Property	OOF	Inference
Source	predict_proba_oof() from AutoGluon	predict_proba() on pipeline features
Honesty	Each prediction from a fold that did NOT see the tree	Model has seen all training data
ROC-AUC	~0.99 (OOF control, 2026_10)	run dependent fallback
Available for	Label-matched training subset (25,334 rows in the 2026_10 calibration panel)	All trees with valid features
Usage	Validation (preferred)	Fallback when OOF unavailable

OOF is preferred because it provides an honest estimate of generalization ability — no information leakage.

Tests

Validation logic is covered by tests/test_ml_validation.py:

Test Class	Tests	Description
TestBinaryMetrics	4	Precision, Recall, F1, Accuracy
TestROCAUC	3	ROC-AUC (perfect, random, edge cases)
TestBrierScore	3	Brier Score (perfect, worst, intermediate)
TestSpearman	3	Rank correlation
TestCalibration	2	Calibration curve
TestExtremeGroup	2	Extreme-group logic
TestRankingTables	2	Top-k capture and decile tables
TestModelValidator	12	Overall metrics, species, export, baseline

pytest tests/test_ml_validation.py -v

Changelog

Date	Change
2025-07-14	Refactoring: all validation logic consolidated in validator.py
2025-07-14	OofResult + load_oof_predictions() extracted from risk_scoring.py
2025-07-14	build_comparison(), per_indicator_extreme_groups(), multi_quantile_extreme_groups() moved to validator.py
2025-07-13	OOF predictions as primary validation source (ROC-AUC 0.91 instead of 0.52)
2025-07-12	Per-indicator binary flags added
2025-07-12	--skip-validation instead of --validate (validation default-on)
2025-07-11	Dual-score analysis (paper validation)
2025-07-10	Initial implementation: ModelValidator, metrics, CLI integration

Low Match Rate

If spatial join match rate is low (< 80%):

• Check CRS alignment (should be EPSG:25832)
• Increase max_distance parameter
• Verify geometry column exists

---