bluegreencity-ki

ML-Based Drought Risk Scoring – Complete Guide

BlueGreenCity-KI AutoGluon Framework

Table of Contents

  1. Overview & Motivation
  2. Current Operative Setup
  3. Development Journey
  4. Quick Start
  5. Labels
  6. Architecture & Complete Workflow
  7. Training
  8. Inference (Pipeline Integration)
  9. Validation Framework
  10. Decision Threshold Analysis
  11. Configuration
  12. Output Files
  13. Troubleshooting
  14. References

Overview & Motivation

The BlueGreenCity-KI project implements a machine learning pipeline for predicting drought-damage risk in urban trees. This complements the existing weighted-sum static scoring with data-driven predictions based on historical tree inventory records.

Why Machine Learning?

The first approach (master thesis) static scoring approach (weighted criteria) achieves low correlation with actual tree damage (Spearman ρ ≈ 0.007). An ML model trained on historical outcomes can learn which factors actually predict poor tree condition, providing:

System Architecture

┌─────────────────────────────────────────────────────────────┐
│         PIPELINE: Static Score vs. ML Score (Parallel)     │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Criterion Scores (25 criteria, 21 active)                 │
│  ↓                                                          │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ Static Path    │         ML Path                   │   │
│  │ Weighted Sum   │    (if enabled in config)         │   │
│  │ → drought_risk │                                   │   │
│  │   _rulebased   │    Tree Features + Training Data  │   │
│  │   (float 0–1)  │    ↓                              │   │
│  │                │    AutoGluon Model                │   │
│  │                │    ↓                              │   │
│  │                │    P(drought damage)              │   │
│  │                │    → drought_risk_ml (float)      │   │
│  │                │    → drought_risk_ml_class (cat)  │   │
│  │                │    → confidence / class margin    │   │
│  │                │    → local drivers for extremes   │   │
│  └─────────────────────────────────────────────────────┘   │
│                           ↓                                 │
│                 Output both columns in GPKG                │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Both scoring approaches run in parallel. ML does not replace the static score; they coexist for comparison.

Current Operative Setup (as of October 2026)

This section reflects the production state. Earlier sections of this guide document the development history and are kept for reference. When in doubt, this section takes precedence.

Parameter Value
Operative label future_drought_damage_severe_t1
Label strategy operational_panel_drought_damage — one label per tree×snapshot_year
Training approach Multi-year panel (2018–2024 snapshots)
Model AutoGluon WeightedEnsemble_L2 (refit_final; ensemble over GBM/CAT/XGB/NN_TORCH[GPU]/RF/XT candidates)
Eval metric ROC-AUC
Model path data/models/autogluon/operational/2026_10/train_2018_2024__predict_2025/model/drought_risk_t1/ (Stand: 2026-10-Re-Training)
Feature count 54 active features in the current training allowlist (2026-10 bundle: +7 spatial, +SWBM soil-water, +A/B re-entries; 4 removed on 2026-09-15, Issue #217)
Score calibration Platt sigmoid fitted on the external 2024 holdout each validation round (Issue #253); drought_risk_ml_calibrated is the operative decision score, Brier 0.041 → 0.029
Standard validation reference external 2024 holdout ROC-AUC 0.9697 (2026-10 validation run; OOF gap +0.016); the operational run (predict_2025 only) carries no test metrics
Grouped CV robustness check 5-fold StratifiedGroupKFold by objectid, ROC-AUC 0.888, Precision 0.704, Recall 0.435 (2026-10)
Thresholds on the calibrated score: critical ≥ 0.85 · high ≥ 0.30 · medium ≥ 0.10 · low < 0.10 (seit 2026‑10‑08, Issue #253; davor auf dem Rohscore 0.85/0.30/0.15)
Operative NDVI feature no active NDVI feature in the current allowlist
dataset_mode "operational_panel"

Label Definition: future_drought_damage_severe_t1

future_drought_damage_severe_t1 = 1  if  drought_damage_severe is observed
                                        in (snapshot_year + 1) to (snapshot_year + 1)
                                   0  otherwise

This is a next-year forecast: given tree features in year t, predict whether the tree shows severe drought damage in year t+1. The label is extracted by LabelExtractor.extract_operational_drought_damage_labels(snapshot_year=t).

Current definition B+ (LHH-aligned, decision 2026-08-30/31, see docs/plans/done/2026-08-30-kaskade-label-feature-reduktion.md, Block B):

score  = 2 * crown_dieback(Wipfeldürre) + 1 * foliage_anomaly(Belaubung)
         + 1 * vitality_change_t1(last observation t+1 > last observation t)
severe = score >= 2  OR  felled in t+1 with reason "absterbend"

Totholz is deliberately excluded from the score (cumulative record, not a next-year drought signal); it is still emitted as context (future_totholz_binary_t1).

Label-history note: Runs before 2026-09 used the older 3-point definition (A) which included Totholz in the score. The switch to B+ changes the labelled population (e.g. on the 2024 snapshot: −836 / +16 severe cases on the 23,390 common trees — fully explained by the definition change, not by the v6 inventory extract; see issue #228 comment 2026-10-06). Do not compare recall/precision numbers across the A/B+ boundary without this caveat.

2026-10-07 — B+ kept as-is after re-evaluation. Two follow-up ideas were analyzed and rejected: (1) a felling-age restriction (“absterbend” only for trees <10 years old at felling, via Hauptdaten Pflanzjahr) — felling contributes only ~1–2% of severe positives (168 in the 2019 window → 8 in the 2025 window), Pflanzjahr is known for only 33.6% of absterbend fellings (2019–2025), and the net label effect is 17.77% → 17.18% positive — negligible; (2) raising the positive rate to ~33% for class balance — the ~18% rate is the honest base rate of severe drought damage, is not a class-balance problem (the 0.96-AUC baseline was achieved at this rate), and forcing 33% would require including Totholz (label contamination). If imbalance ever becomes an issue, use ml.training.class_weight_positive (sample weighting) instead of diluting the label. Treelife-Behandlung is added as an independent validation signal (model top-risk vs. actual LHH interventions, issue #257); a second model with a Treelife label was discussed and rejected (endogenous treatment decisions, expected redundancy with B+).

Feature Set

The current operative model (“Modell Hannover”) uses a curated 54-feature allowlist defined in config/config.yaml (feature_columns). It combines:

The older 17-feature NDVI-heavy snapshot should be treated as a historical intermediate state. For the current panel-aware internal validation, the important distinction is not between individual feature ranks, but between the two internal validation levels:

Thresholds (operative)

Class Calibrated probability Action
critical ≥ 0.85 Immediate priority watering
high ≥ 0.30 Priority inspection / targeted watering
medium ≥ 0.10 Scheduled inspection
low < 0.10 Normal monitoring

Thresholds are operational policy, set in config/config.yaml under ml.thresholds. Since 2026‑10‑08 they apply to the Platt‑calibrated score (drought_risk_ml_calibrated), not the raw model output (Issue #253): the calibration makes the numbers honest probabilities, so the medium bound was re-anchored 0.15 → 0.10 (reference-stable ≈ 20% of trees on the 2024 holdout class) while high/critical become rarer but sharper. Before that, since 2026‑09‑19 the high/medium bounds were lowered on the raw score (0.60/0.40 → 0.30/0.15). The ML run stamps drought_risk_ml_class with the thresholds active at its run time, but the daily operational refresh re‑derives the class from the calibrated score (and the current config; reclassify_risk in bluegreencity_ki/operational_products.py), so a threshold change takes effect at the next refresh, not the next ML run. The raw score (drought_risk_ml) is a model artifact and is never modified. Irrigation stays hard‑gated on SWBM water demand (irrigation_demand_liters > 0); a class without an acute deficit means monitoring, not watering.

This section documents how we arrived at the current AutoGluon-based framework, including experiments, decisions, and lessons learned.

Timeline

Phase 1: First Try with ArcGIS AutoML

Motivation

We started with ArcGIS Pro AutoML because:

Setup & Workflow

  1. Software: ArcGIS Pro 3.6+
  2. Preparation: Export trees with features to feature class in File Geodatabase
  3. Training Tool: “Train Using AutoML” (GeoAI Tools)
    • Input: Feature class with ~27K trees + binary label column
    • Settings: binary classification, ROC-AUC metric, time limit = 2 hours
    • Output: .dlpk model package + HTML report
  4. Validation: Review leaderboard, SHAP plots, confusion matrix in report
  5. Inference: Apply trained model with “Predict Using AutoML” tool

Results

Limitations Encountered

Decision: Use ArcGIS AutoML as exploratory prototype only; migrate to open-source solution for production.

Phase 2: Label Selection Process

Problem: What to Predict?

Historical tree inventory contains multiple crown diagnosis columns:

Multiple observations per tree (2010–2025), encoded as J (Ja = Yes) or N (Nein = No).

Decision Criteria

  1. Label balance: Avoid extreme imbalance (99% positive = model predicts “always positive”)
  2. Sample size: Need ≥5,000 labeled trees for robust training
  3. Relevance: Should correlate with actual drought stress (not just general vitality)

Labels Evaluated

Label Positive Rate Sample Size Decision
wipfelduerre_binary 96.4% 5,357 trees ❌ Too imbalanced
totholz_binary 95.2% 4,821 trees ❌ Too imbalanced
belaubung_auffaellig_binary 87.3% 3,204 trees ❌ Too imbalanced
drought_damage_binary 99.7% 25,834 trees ❌ Extremely imbalanced
drought_damage_severe 25.8% 6,672 trees ✅ Chosen

Final Label Definition

drought_damage_severe = (score >= 3).astype(int)

where:
  score = Wipfeldürre_binary * 3 
          + Totholz_binary * 2 
          + Belaubung_binary * 1

Rationale:

Rejected alternative: Training separate models per symptom → too imbalanced for reliable predictions.

Phase 3: Framework Selection

Options Evaluated

Framework Pros Cons Decision
scikit-learn Simple, well-documented Manual hyperparameter tuning required ❌ Too much manual work
PyCaret Very beginner-friendly, unified API Maintenance mode since Apr 2024, no active development ❌ Long-term risk
H2O AutoML Mature, good documentation Java dependency, complex setup ❌ Infrastructure overhead
TPOT Genetic algorithm, creative Very slow, unstable ❌ Impractical runtime
MLjar-supervised AutoML with explanations Limited model stacking ❌ Not flexible enough
AutoGluon Stacking, fast, Apache 2.0, active development Learning curve ✅ Chosen

Why AutoGluon?

  1. Automatic stacking: Trains 10+ algorithms (RF, XGBoost, LightGBM, Neural Nets) and stacks them
  2. Fast: 30 min for best_quality (vs. 4 hours for TPOT)
  3. Bagging & cross-validation: Built-in k-fold CV → honest out-of-fold predictions for validation
  4. Apache 2.0 license: No vendor lock-in
  5. Active development: Regular updates (monthly releases), large community
  6. Easy integration: Pure Python, fits into existing pipeline

Key decision: Use presets="best_quality" with num_bag_folds=5 as the stable training shell, but control the benchmark portfolio explicitly via custom hyperparameters so feature ablations are compared under the same reproducible model families.

Phase 5: Benchmark Stabilization (Apr 2026)

After the first ablation runs, we found that benchmark fairness had become blurred by changes in the AutoGluon setup itself. In particular, the experimental extreme portfolio introduced additional foundation-model families, optional dependencies, GPU/backend variability, and very different inference costs. That is useful for method exploration, but not for the main scientific question of this project.

Decision: separate exploratory AutoML experimentation from the primary city-green benchmark.

The concrete comparison of the four stable reference and ablation runs is documented in reports/stable_ablation_report_2026-04-03.md.

The active reference benchmark is now intentionally conservative:

This change also reflects the project goal more accurately: the research focus is urban-tree drought risk, not AutoML portfolio research.

What About PyCaret?

PyCaret was initially considered due to its reputation for beginner-friendliness and unified API. However, critical research revealed:

Deal-breaker: Maintenance Mode (April 2024)

Comparison: PyCaret vs. AutoGluon (Feb 2026)

Aspect PyCaret AutoGluon FLAML
Development Status ⚠️ Maintenance only ✅ Active (Jan 2026 release) ✅ Active (Dec 2025)
Community Declining Growing (AWS-backed) Growing (Microsoft)
Long-term Support ❌ Uncertain ✅ Roadmap to 2030+ ✅ Enterprise support
Performance ⭐⭐⭐ Good ⭐⭐⭐⭐⭐ State-of-the-art ⭐⭐⭐⭐⭐ State-of-the-art
Ease of Use ⭐⭐⭐⭐⭐ Excellent ⭐⭐⭐⭐ Very good ⭐⭐⭐⭐ Very good
Model Stacking ⭐⭐⭐ Basic ⭐⭐⭐⭐⭐ Advanced (multi-level) ⭐⭐⭐⭐ Advanced
Sponsor Lock-in? ❌ No ❌ No (Apache 2.0) ❌ No (MIT)
Best For Quick prototyping (if maintained) Production, best performance Resource-constrained training

Verdict: While PyCaret’s simplicity is attractive, choosing a framework in maintenance mode would create technical debt. AutoGluon provides similar ease of use with active development and superior performance. FLAML (Microsoft) is also a good alternative but AutoGluon’s multi-level stacking and benchmark performance made it the preferred choice.

Phase 4: Key Architectural Decisions

1. Graceful Degradation

Decision: If ml.enabled=true but trained model doesn’t exist, log warning and continue (don’t crash pipeline).

Rationale: Allow gradual rollout; not all deployments have trained models yet.

try:
    predictor = DroughtRiskPredictor(model_path)
    trees["drought_risk_ml"] = predictor.predict_proba(trees)
except FileNotFoundError:
    logger.warning("ML model not found; skipping ML scoring")
    trees["drought_risk_ml"] = None

2. Parallel Scoring (Not Replacement)

Decision: Output both overall_drought_risk (weighted sum) and drought_risk_ml (AutoGluon).

Rationale:

3. Custom Validation Metrics (No sklearn)

Decision: Implement metrics in pure numpy/pandas (no sklearn dependency for validation).

Rationale:

Implemented metrics:

4. Out-of-Fold (OOF) Predictions for Threshold Analysis

Decision: Use OOF predictions from k-fold CV for threshold analysis, not direct predictions.

Why?

# ❌ Overfitted
proba = predictor.predict_proba(training_data)  # ROC-AUC = 1.0

# ✅ Honest
proba_oof = predictor.predict_proba_oof()       # ROC-AUC = 0.84

Lessons Learned

  1. Label balance matters: Imbalanced labels (>95% positive) produce models that just predict “always positive”
  2. ArcGIS AutoML is good for prototyping: Validates approach quickly, but not suitable for production
  3. AutoGluon stacking is powerful: Ensemble of 10+ models outperforms single best model by 3–5% ROC-AUC
  4. OOF predictions are essential: Avoid overfitting illusions when choosing thresholds
  5. Species matter: Model performance varies by species (Tilia: 0.81 AUC, Quercus: 0.75 AUC) → consider species-specific calibration in future
  6. Framework maintenance status is critical: PyCaret’s maintenance mode (Apr 2024) made it unsuitable despite good usability. Always check project activity before committing to a framework.

Quick Start

Prerequisites

Six-Step Workflow

Step 1: Generate Labels from Historical Data

Extract drought-damage labels from crown diagnosis CSV:

python -m scripts.ml.generate_ml_labels --drought-stats-only

Output: Statistics on label distribution.

Step 2: Create Training Dataset

Before creating the training dataset, materialize the yearly operational panel feature table in PostGIS for all configured snapshot years:

python scripts/ml/build_operational_panel_snapshots.py --config config/config.yaml

This step fills bgcki.analysis_tree_risk_scores_ml_operational_panel with one feature snapshot per tree and snapshot_year. The training dataset script does not compute these features itself; it only reads the already materialized panel rows from PostGIS and joins them with the extracted labels.

After the panel table is complete, join PostGIS features with CSV labels:

python -m bluegreencity_ki.ml.training.dataset_builder

Output: data/derived/ml/2026_07_23_training_dataset_ml_operational_panel_2018_2024.csv

The current production allowlist in config/config.yaml defines the operative ML feature set (54 features, see Feature Set in the “Current Operative Setup” section) plus the technical join column objectid. The table below is the September 2026 generation (35 features), kept for reference.

Feature Group Columns
Taxonomy and species profile species_genus, species_binomial
Individual tree attributes (inventory) st_umfang, kr_durchm
Street and site context dist_road_m, street_type_value, salt_value, land_use_value, planting_site_surface_type, planting_site_area_m2, planting_site_tree_count
Microclimate (thermal and shading) m_svf_value, m_sunlit_hours_value, m_air_temp_14h_value, m_air_temp_04h_value, m_cold_air_kvs_value
Groundwater groundwater_level_latest
Tree vitality (remote sensing) dwsi_summer_value
CiTree species profile citree_lichtanspruch, citree_bodenverdichtungstoleranz, citree_staunaessetoleranz, citree_salzvertraeglichkeit, citree_grundigkeit, citree_bodeneigenschaft, citree_bodenfeuchtetoleranz, citree_kronenform, citree_oberflaechennahe_wurzeln, citree_trockenheitstoleranz, citree_hitzevertraeglichkeit
Hydroclimatic drought context (DWD) precip_may_oct_mm, precip_nov_apr_mm, temp_mean_may_oct_c, et0_may_oct_mm, p_et0_ratio_may_oct, p_et0_ratio_anomaly_may_oct

Note: Several features formerly listed here (ga_lang, vitality_value, field_capacity_value, groundwater_trend, groundwater_level_median/max/std, construction_dist_m, planting_structure_neighbor_count, pit_size_value, tree_disc_value, m_uhi_intensity_value, m_cold_air_kpr_value, albedo_value, the seasonal m_sunlit_hours_* split, the NDVI columns, and citree_wuchsgeschwindigkeit / citree_kronendurchlaessigkeit) were removed during the correlation-cleaned selection (August 2026) or the 2026-09-15 reduction (Issue #217); see docs/risk_assessment_ml/automl_features.md for the per-feature rationale. They remain available as pipeline columns but are not ML features in the current allowlist.

The Citree matching metadata columns citree_art_id, citree_name, citree_match_rule, and citree_match_score are available in the pipeline output for QA, but they are intentionally not part of the ML allowlist.

Step 2a: Sanity-Check The Target Before Training

Before training a new model, the project now performs a dedicated sanity check on the final target drought_damage_severe after the drought-window filter and after the feature join.

This is important because two different failure modes are common in environmental ML workflows:

  1. The target becomes too imbalanced after narrowing the time window.
  2. The feature join silently removes a non-random subset of labelled trees and distorts the class distribution.

For the current production setup, the target is defined as:

drought_damage_severe = (drought_damage_score >= 3).astype(int)

with labels extracted only from the drought window 2018..2024.

Current Sanity-Check Result

The following numbers were computed from the April 2026 files drought_labels_2018_2024.csv / training_dataset.csv in data/derived/ml/ (superseded; the current panel is training_dataset_v6_operational_panel_2018_2024.csv):

For context, the earlier pre-window setup aggregated labels over the broader historical period and produced the following headline rates that were used in the first ML iteration:

Setup Target Positive rate Sample size
Earlier broad historical setup drought_damage_binary 99.7% 25,834 trees
Earlier broad historical setup drought_damage_severe 25.8% 6,672 trees
Current drought-window setup before join drought_damage_severe 22.22% 25,890 labelled trees
Current drought-window setup after join drought_damage_severe 22.14% 24,607 training rows

This comparison is intentionally kept in the documentation so the shift from the older broad label period to the current 2018..2024 drought window remains traceable.

Stage Rows Severe positives Severe rate
Windowed label dataset 25,890 5,752 22.22%
Final training dataset after feature join 24,607 5,448 22.14%

This means the PostGIS feature join retains 95.04% of the labelled trees. More importantly, the severe rate stays almost unchanged (22.22% -> 22.14%). That is a strong indication that the join is not introducing a major class bias.

Why This Is A Good Sign
Score Distribution

The final training dataset shows the following distribution for drought_damage_score:

Score Count
0 707
1 547
2 17,905
3 1,694
4 290
5 1,407
6 2,057

Interpretation:

Temporal Plausibility Check

For the severe class in the final training dataset:

For the non-severe class:

Interpretation:

Feature-Level Sanity Check

The following class-wise means were compared on the final training dataset:

Feature Severe mean Non-severe mean Delta
vitality_value 2.6017 1.9698 +0.6319
age_value 49.7430 56.2799 -6.5369
ndvi_value_2024 0.0586 0.0617 -0.0031
m_uhi_intensity_value 11.9204 11.1118 +0.8086
m_svf_value 0.5635 0.5645 -0.0010
groundwater_level_std 0.2376 0.1820 +0.0556
field_capacity_value 120.2580 122.4891 -2.2311
albedo_value 0.1364 0.1340 +0.0024
dist_road_m 8.8458 8.9593 -0.1135

Interpretation:

Missingness Check By Class

Some features are unequally available across classes and should be interpreted carefully during model diagnostics:

The strongest asymmetry is currently in m_svf_value. This should be kept in mind when interpreting future feature-importance rankings.

Species Concentration Check

To ensure the severe class is not dominated exclusively by one single species, the dataset was checked for species with at least 100 rows and the highest severe share:

Species Rows Severe rate
Ulmus hollandica 'Commelin', Stadt-Ulme 138 66.67%
Sorbus latifolia, Breitblättrige Mehlbeere 109 64.22%
Sorbus intermedia, Schwedische Mehlbeere 610 51.80%
Tilia cordata, Winter-Linde 289 35.64%
Acer pseudoplatanus, Bergahorn 553 35.08%

This does not prove causality, but it is another useful plausibility check: the severe target is not random noise. It clusters in recognisable species groups, which is consistent with the biological and site-related differences the model is supposed to learn.

Practical Conclusion

For the current 2018-2024 setup, drought_damage_severe is the right primary training target because:

This sanity check should be repeated whenever the drought window, the label logic, or the feature join changes.

Step 3: Train Models

Train AutoGluon models (1 hour per label with best_quality):

# All labels
python -m scripts.ml.train_autogluon --config config/config.yaml

# Quick test (2 min)
python -m scripts.ml.train_autogluon --config config/config.yaml \
    --presets medium_quality --time-limit 120

# Single label
python -m scripts.ml.train_autogluon --config config/config.yaml \
    --labels drought_damage_severe --time-limit 1800

The April 2026 ablation round used three focused experiment configs (removed again after the round; results in reports/stable_ablation_report_2026-04-03.md):

The pattern is still useful when the goal is not just maximum validation score, but also a better fachliche understanding of how much of the prediction comes from vitality, direct species identity, or species-profile proxies.

Output: data/models/autogluon/<label>/ (trained model + leaderboards + feature importances)

If ml.output.postgis.enabled=true is configured, the training artefacts are also written to PostGIS as tabular tables with the configured prefix (default: ml_training_*). This is intended for BI/Superset consumption of leaderboards and feature importances.

Step 4: Validate Model Quality

Assess model quality on two complementary levels:

  1. standard validation outputs for static and ML scores,
  2. grouped cross-validation for the operational panel model.

Standard validation:

python -m scripts.ml.validate_model --config config/config.yaml

Output: data/validation/autogluon/

Panel-aware grouped CV:

python -m scripts.ml.evaluate_grouped_cv --config config/config.yaml \
  --labels future_drought_damage_severe_t1 \
  --group-column objectid \
  --n-splits 5

Output: data/models/autogluon/grouped_cv/

Current reference run (October 2026 round, 2026_10):

The grouped CV numbers are the stricter internal estimates for the panel dataset because they keep all yearly observations of one tree (or one H3 cell) inside the same fold. Earlier rounds: 2026_09 grouped CV 0.8563 (see dual_validation_operational_workflow.md), June 2026 0.8817 (archived report).

Step 5: Analyze Decision Threshold

Find optimal probability threshold for your use case:

python scripts/ml/threshold_analysis.py

Output: Table showing precision, recall, F1, and % flagged trees at thresholds 0.10–0.90.

Examples:

Step 6: Run Pipeline with ML Scoring

Enable ML scoring in config:

# config/config.yaml
ml:
  enabled: true
  inference:
    label: "future_drought_damage_severe_t1"
  thresholds:
    critical: 0.85
    high: 0.30
      medium: 0.10
    low: 0.00

Run pipeline:

python -m bluegreencity_ki.runners.cli --config config/config.yaml

Output: Standard pipeline results plus:

Labels

The ML model learns from three base labels extracted from crown diagnosis columns in the historical tree inventory (2010–2025):

Base Labels (Extracted from CSV)

Label Source Column Meaning Positive Rate Sample Size
wipfelduerre_binary “Krone Wipfeldürre” Crown dieback observed (J=1, N=0) 96.4% 5,357 trees
totholz_binary “Krone Totholzbildung” Dead wood in crown observed 95.2% 4,821 trees
belaubung_auffaellig_binary “Krone Belaubung (Auffälligkeiten)” Foliage anomalies observed 87.3% 3,204 trees

These three labels are directly extracted from crown diagnosis assessments where foresters recorded visual symptoms (‘J’ = Yes, ‘N’ = No) during inspections from 2010–2025. For each tree, the label represents whether that symptom was EVER observed across all inspection years (MAX aggregation).

Derived Labels (Computed from Base Labels)

From the three base labels above, two additional labels are computed for different modeling purposes:

Derived Label Formula Meaning Positive Rate Sample Size
drought_damage_binary OR of three base labels Any crown damage observed 99.7% 25,834 trees
drought_damage_severe score ≥ 3 Multiple damage types or repeated years 25.8% 6,672 trees

Computation Logic:

# Step 1: Calculate hierarchical severity score
drought_damage_score = (
    wipfelduerre_binary * 3      # Crown dieback (most severe)
    + totholz_binary * 2          # Dead wood formation
    + belaubung_auffaellig_binary * 1  # Foliage anomalies (least severe)
)

# Step 2: Derive binary labels
drought_damage_binary = (wipfelduerre_binary | totholz_binary | belaubung_auffaellig_binary).astype(int)
drought_damage_severe = (drought_damage_score >= 3).astype(int)

Example Tree:

Tree 12345:
  2019: Wipfeldürre=J, Totholz=N, Belaubung=J  → score = 3+0+1 = 4
  2022: Wipfeldürre=N, Totholz=J, Belaubung=N  → score = 0+2+0 = 2
  
  Combined (MAX aggregation across years):
    wipfelduerre_binary = 1 (max of 1, 0)
    totholz_binary = 1 (max of 0, 1)
    belaubung_auffaellig_binary = 1 (max of 1, 0)
    
    → drought_damage_score = 3+2+1 = 6
    → drought_damage_binary = 1 (any damage present)
    → drought_damage_severe = 1 (score 6 ≥ 3)

Why Two Derived Labels?

Recommendation

Train primarily on drought_damage_severe (currently the best-balanced label with sufficient sample size, while still allowing comparison to the older 25.8% baseline).

Avoid training on drought_damage_binary (it was already unusable at 99.7% positive in the old setup and is still massively imbalanced at 97.1% positive in the current 2018-2024 training dataset).

Architecture & Complete Workflow

Five-Phase ML Pipeline

┌─────────────────────────────────────────────────────────────────────┐
│                      ML Pipeline Components                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  1. LABELS (bluegreencity_ki/ml/labels/)                            │
│     extract_drought_damage_labels()                                 │
│     → wipfelduerre_binary, totholz_binary, ...                      │
│     → drought_labels.csv                                            │
│                                                                     │
│  2. TRAINING DATA (python -m bluegreencity_ki.ml.training.dataset_builder) │
│     Join: PostGIS features + CSV labels                            │
│     → training_dataset.csv                                          │
│                                                                     │
│  3. TRAINING (bluegreencity_ki/ml/training/trainer.py)             │
│     AutoGluonTrainer.train_all_labels()                            │
│     → Model: TabularPredictor with 5-fold bagging                  │
│     → Artefacts: leaderboards, feature_importances                  │
│     → Output: data/models/autogluon/<label>/                       │
│                                                                     │
│  4. INFERENCE (bluegreencity_ki/ml/inference/predictor.py)         │
│     DroughtRiskPredictor.predict_proba()                           │
│     DroughtRiskPredictor.classify_risk()                           │
│     confidence(), class_margin(), explain_top_drivers()            │
│     → Integrated into risk_scoring pipeline                        │
│     → Output: score, class, confidence, margin, local drivers      │
│                                                                     │
│  5. VALIDATION (bluegreencity_ki/ml/validation/validator.py)       │
│     ModelValidator.run_all()                                       │
│     → Overall metrics (ROC-AUC, Brier, F1)                         │
│     → Species-specific performance                                 │
│     → Calibration & extreme-group analysis                         │
│     → Comparison with static weighted sum                          │
│     → Output: data/validation/autogluon/                           │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Phase 1: Label Extraction

Script: scripts/ml/generate_ml_labels.py

Input: data/input/trees/Baum-2010-2026-v4.csv

Code:

# bluegreencity_ki/ml/labels/label_extractor.py

@staticmethod
def extract_drought_damage_labels(
  csv_path: str,
  start_year: int | None = 2018,
  cutoff_year: int = 2024,
) -> pd.DataFrame:
    """
    Extract four binary labels + score from crown diagnosis data.
    
    Logic:
      1. Read CSV with crown diagnosis columns
    2. Filter to the configured drought window (2018-2024 for training)
      3. For each tree ID, aggregate MAX across years
      4. Encode: "J" → 1, "N" → 0, else NaN
      5. Compute score: Wip*3 + Tot*2 + Bel*1
      6. Derive severity: score ≥ 3 → 1
    
    Returns:
      DataFrame with columns:
        - objectid (tree ID)
        - wipfelduerre_binary
        - totholz_binary
        - belaubung_auffaellig_binary
        - drought_damage_binary (OR of above)
        - drought_damage_score (0-6)
        - drought_damage_severe (score ≥ 3)
        - damage_years_count
        - first_damage_year
    """

Output: data/derived/ml/drought_labels.csv

objectid,wipfelduerre_binary,totholz_binary,belaubung_auffaellig_binary,drought_damage_binary,drought_damage_score,drought_damage_severe,damage_years_count,first_damage_year
12345,1,1,1,1,6,1,2,2019
12346,0,0,1,1,1,0,1,2022
...

Phase 2: Create Training Dataset

Entry point: python -m bluegreencity_ki.ml.training.dataset_builder

What it does:

  1. Read the materialized panel table: Features from bgcki.analysis_tree_risk_scores_ml_operational_panel
  2. Filter by snapshot_year: One panel slice per configured year (2018–2024)
  3. Load CSV labels: Next-year drought labels for each snapshot year
  4. Left join labels onto features: On objectid == ID
  5. Feature selection: Keep configured raw attributes and criterion values, not risk scores
  6. Export: CSV ready for AutoGluon

Critical prerequisite:

python scripts/ml/build_operational_panel_snapshots.py --config config/config.yaml

Without this step, missing years or stale feature columns in bgcki.analysis_tree_risk_scores_ml_operational_panel will directly propagate into the training dataset.

Code:

# bluegreencity_ki/ml/training/dataset_builder.py

# Read one configured panel slice from PostGIS
query = """
  SELECT objectid, ga_lang, age_value, vitality_value,
       dwsi_summer_value, precip_may_oct_mm, temp_mean_may_oct_c,
       snapshot_year
  FROM bgcki.analysis_tree_risk_scores_ml_operational_panel
  WHERE snapshot_year = 2024
"""
features = pd.read_sql(query, engine)

# Load labels
labels = LabelExtractor.extract_operational_drought_damage_labels(
  csv_path="data/input/trees/Baum-2010-2026-v4.csv",
  snapshot_year=2024,
  label_horizon_years=1,
)

# Join
training_data = features.merge(labels, left_on="objectid", right_on="ID", how="left")

# Export
training_data.to_csv("data/derived/ml/2026_07_23_training_dataset_ml_operational_panel_2018_2024.csv", index=False)

Output: data/derived/ml/2026_07_23_training_dataset_ml_operational_panel_2018_2024.csv

24,607 rows × 57 columns
├── Tree attributes: objectid, ga_lang, st_umfang, st_durchm, kr_durchm
├── Remote sensing: obskh_volume, obskh_meanNDVI, obskh_meanGRI, ...
├── Criterion VALUES: age_value, vitality_value, street_type_value, ...
├── Labels: drought_damage_binary, drought_damage_score, damage_years_count, ...
└── No geometry column in the exported CSV

Important: Criterion VALUES (raw scores before normalization) are used, not risk scores!

Phase 3: Training

Script: scripts/ml/train_autogluon.py

Code:

# bluegreencity_ki/ml/training/trainer.py

class AutoGluonTrainer:
    def __init__(self, data: pd.DataFrame):
        self.data = data
        
    def prepare_features(self, label: str, excluded_columns=None) -> pd.DataFrame:
        """
        Drop leak prevention:
          - All OTHER labels (prevent model from cheating)
          - ID columns (objectid, BaumID)
          - Geometry column
        
        Keep:
          - Target label only
          - Raw features (tree attributes, criterion values, RS)
        """
        
    def train(self, config: TrainingConfig) -> Dict:
        """
        Train one TabularPredictor with:
          - problem_type: 'binary'
          - eval_metric: 'roc_auc'
          - presets: 'best_quality' (tries 10+ algos, stacks them)
          - time_limit: 3600 seconds per label
          - num_bag_folds: 5 (k-fold CV for bagging)
        
        Steps:
          1. Prepare features (drop other labels)
          2. Split X (features) and y (target label)
          3. Call TabularPredictor.fit(X, y, ...)
          4. Export leaderboard (all models + scores)
          5. Export feature importances (per model)
        
        Returns:
          {
            'label': 'drought_damage_severe',
            'model_path': 'data/models/autogluon/drought_damage_severe/',
            'best_model': 'WeightedEnsemble_L2',
            'leaderboard': DataFrame with all models + ROC-AUC
          }
        """

AutoGluon automatically:

  1. Tests 10+ algorithms: Linear, Random Forest, XGBoost, LightGBM, CatBoost, Neural Nets, tabular transformers
  2. Handles categorical variables (one-hot, target encoding, embeddings)
  3. Fills missing values (median for numeric, mode for categorical)
  4. Selects best models and stacks them into ensemble
  5. Returns single TabularPredictor wrapping final ensemble

CLI Usage:

# All labels (1 hour each)
python -m scripts.ml.train_autogluon --config config/config.yaml

# Custom time & presets
python -m scripts.ml.train_autogluon --config config/config.yaml \
    --time-limit 1800 --presets best_quality --labels drought_damage_severe

Output Directory Structure:

data/models/autogluon/
├── drought_damage_severe/
│   ├── model.pkl             # Saved TabularPredictor
│   ├── metadata.json         # Training info
│   ├── models/
│   │   ├── LightGBM_1/
│   │   ├── LightGBM_2/
│   │   ├── XGBoost_1/
│   │   ├── RandomForest_1/
│   │   ├── NeuralNet_1/
│   │   └── WeightedEnsemble_L2/  ← best model (usually)
│   └── preprocessor.pkl      # Feature transformations
├── leaderboards/
│   └── drought_damage_severe_leaderboard.csv
└── feature_importances/
    └── drought_damage_severe_importance.csv

Phase 4: Inference (Pipeline Integration)

Module: bluegreencity_ki/ml/inference/predictor.py

Code:

class DroughtRiskPredictor:
    def __init__(self, model_path: str):
        """Load trained TabularPredictor from disk."""
        self._predictor = TabularPredictor.load(model_path)
        
    def predict_proba(self, data: pd.DataFrame) -> pd.Series:
        """
        Return P(positive) for each tree.
        
        Steps:
          1. Align features (select expected columns only)
          2. Warn if missing features
          3. Call predictor.predict_proba(..., as_multiclass=False)
          4. Return Series of probabilities [0, 1]
        """
        
    def predict(self, data: pd.DataFrame) -> pd.Series:
        """Return binary predictions (0/1) at threshold=0.5."""
        
    def classify_risk(self, data: pd.DataFrame, thresholds=None) -> pd.Series:
        """
        Map probabilities to risk classes.
        
        Thresholds (configurable, descending; first match wins):
          - critical: 0.85
          - high: 0.60          # from config.yaml, not part of DEFAULT_THRESHOLDS
          - medium: 0.40
          - low: 0.00
        
        Logic:
          if P >= 0.85: return "critical"
          elif P >= 0.60: return "high"
          elif P >= 0.40: return "medium"
          else: return "low"
        """

Integration in Pipeline:

# Within bluegreencity_ki/pipelines/risk_scoring.py

if ctx.config["ml"]["enabled"]:
    try:
        model_path = ctx.config["ml"]["training"]["model_path"]
        label = ctx.config["ml"]["inference"]["label"]
        full_path = f"{model_path}/{label}"
        
        predictor = DroughtRiskPredictor(full_path)
        
        trees["drought_risk_ml"] = predictor.predict_proba(trees)
        trees["drought_risk_ml_class"] = predictor.classify_risk(
            trees, 
            thresholds=ctx.config["ml"]["thresholds"]
        )
        
        logger.info(f"ML scoring complete: {len(trees)} trees")
        
    except FileNotFoundError:
        logger.warning("ML model not found; skipping ML scoring")
        trees["drought_risk_ml"] = None
        trees["drought_risk_ml_class"] = None

Output Columns:

Phase 5: Validation

Script: scripts/ml/validate_model.py

Code:

# bluegreencity_ki/ml/validation/validator.py

class ModelValidator:
    def __init__(self, y_true: pd.Series, y_proba: pd.Series, threshold=0.5):
        """
        Initialize validator with ground truth labels and predictions.
        
        Args:
          y_true: Binary labels (0/1)
          y_proba: Predicted probabilities [0, 1]
          threshold: Decision threshold for binary metrics
        """
        
    def run_all(self, species=None, baseline_scores=None) -> ValidationResult:
        """
        Execute full validation suite:
        
        1. Overall metrics (ROC-AUC, Brier, precision, recall, F1)
        2. Extreme-group comparison (top 5% vs bottom 5% lift)
        3. Calibration curve (predicted vs observed frequency, 10 bins)
        4. Species-specific metrics (per-baumart performance)
        5. Baseline comparison (ML vs weighted-sum correlation)
        6. Temporal holdout (optional: train/test by year)
        
        Returns:
          ValidationResult dataclass with all metrics
        """

CLI Usage:

python -m scripts.ml.validate_model --config config/config.yaml

Output: data/validation/autogluon/

Training

AutoGluon Training Process

AutoGluon trains an ensemble of ensembles using stacking:

  1. Level 0 (Base Models): Trains the explicit benchmark portfolio on 5-fold bagged data (controlled via custom hyperparameters in config/config.yaml, so feature ablations are comparable under the same reproducible model families):
    • LightGBM/GBM (×2: extra-trees “XT” variant + standard)
    • XGBoost (×1)
    • CatBoost (×1)
    • Random Forest (×2: Gini, Entropy)
    • Extra Trees (×2: Gini, Entropy)
    • Neural Network NN_TORCH (GPU, min_time_limit: 300)
  2. Level 1 (Stacking): Trains meta-learner on base model predictions
    • Weighted ensemble of best Level 0 models
    • Uses out-of-fold predictions to avoid overfitting
  3. Level 2 (Final Ensemble): Weighted average of Level 1 outputs

Typical Result: WeightedEnsemble_L2 (final model) achieves 3–5% higher ROC-AUC than best single model.

Training Parameters

Configured in config/config.yaml:

ml:
  training:
    presets: "best_quality"     # best_quality, high_quality, medium_quality, good_quality
    time_limit: 3600            # seconds per label
    eval_metric: "roc_auc"      # for binary classification
    num_bag_folds: 5            # k-fold for bagging
Preset Time Models Tried Use Case
good_quality 5–10 min ~8 models Quick prototyping
medium_quality 10–20 min ~12 models Testing & CI/CD
high_quality 20–40 min ~20 models Production (balanced)
best_quality 30–60 min ~30 models Maximum performance

Portfolio Filtering and Audit

The productive “audit” logic now lives in bluegreencity_ki/ml/training/audit.py and is used directly by the trainer to keep AutoGluon portfolios runnable on the workstation. It is no longer a separate operational audit workflow.

select_hyperparameter_portfolio() performs three checks before training starts:

  1. It removes families that were explicitly excluded in the training config.
  2. It skips families whose optional Python package is missing, e.g. TABICL or TABDPT.
  3. It drops portfolio variants whose ag_args.max_rows limit is below the current training dataset size.

The resulting selection metadata is returned as PortfolioFamilySelection records and logged by the trainer. This makes it visible why a family was kept, excluded for configuration reasons, skipped because of missing dependencies, or removed because of row limits.

This filtering is enabled through the training configuration in config/config.yaml, especially:

That combination is the supported way to adapt the AutoGluon portfolio to the local workstation without maintaining a separate audit script.

Practical Training Controls

The training CLI now exposes the most important portfolio controls directly, so you can steer runs without rewriting the main config each time.

Common usage patterns are:

In practice this means the same training script can be used for both productive runs and controlled comparison experiments, while the fachliche interpretation still stays tied to the chosen feature set and label definition.

Feature Importance

After training, export feature importance:

trainer.export_feature_importances("data/models/autogluon/feature_importances/")

Output: drought_damage_severe_importance.csv

feature importance model
vitalitaet 0.2456 WeightedEnsemble_L2
pflanzjahr 0.1823 WeightedEnsemble_L2
obskh_hoehe 0.1234 WeightedEnsemble_L2
ndvi_value_2024 0.0987 WeightedEnsemble_L2
art 0.0876 WeightedEnsemble_L2

Interpretation: Vitality score and tree age are most predictive features.

SHAP Explainability for AutoGluon Outputs

AutoGluon’s native predictor.feature_importance() remains the default global permutation-ranking export. For BMUV KI-Leuchttürme transparency reporting, the trained TabularPredictor can additionally be explained post-hoc with model-agnostic Kernel SHAP via scripts/ml/explain_autogluon_shap.py. The script explains the positive-class drought-damage probability, so it works for stacked AutoGluon ensembles.

Example:

python -m scripts.ml.explain_autogluon_shap \
  --model-path data/models/autogluon/operational/2026_10/train_2018_2024__predict_2025/model/drought_risk_t1 \
  --background-sample-size 100 \
  --explain-sample-size 25 \
  --summary-output data/models/autogluon/operational/2026_10/train_2018_2024__predict_2025/shap/drought_risk_t1_summary.png \
  --waterfall-output data/models/autogluon/operational/2026_10/train_2018_2024__predict_2025/shap/drought_risk_t1_waterfall_row0.png

Without --data, the script derives the authoritative pipeline CSV from config/config.yaml via output.path and the active study area. Pass --data only when you intentionally want to explain a different export.

Outputs for reporting:

Kernel SHAP is computationally expensive. Start with the default 100 background rows and increase toward 200 only when runtime allows. Limit the explanation sample to representative records, for example the highest-risk trees or a balanced inspection subset.

XAI Method for Funding Progress Reporting

For BMUV progress reporting, reference SHAP as the XAI measure for ML transparency. The AutoGluon ensemble is treated as a black-box probability function and explained with Kernel SHAP on a subsampled background dataset. The exported global summary plot documents feature-level drivers across representative trees. The waterfall plot documents the feature contributions for an individual tree prediction.

Inference (Pipeline Integration)

How It Works

When ml.enabled=true, the pipeline:

  1. Loads trained model specified in ml.inference.label
  2. Prepares features (aligns columns with training data)
  3. Calls predict_proba() → returns probabilities
  4. Calls classify_risk() → maps probabilities to risk classes
  5. Computes confidence() and class_margin() for downstream prioritization
  6. Computes local driver strings only for configurable extreme groups
  7. Appends all ML output columns to the pipeline result

Local Explainability Strategy

The production default is a lightweight, model-agnostic perturbation approximation, not SHAP. For each eligible feature, it compares the original probability with the probability obtained after replacing only that feature with its fixed training reference value:

contribution_j = p(tree) - p(tree with feature_j set to training reference_j)

Numeric reference values are medians and categorical reference values are modes calculated from the exact rows used to train the deployed model. Training writes these values to explanation_reference_profile.json beside the AutoGluon model. All inference batches therefore use the same reference values. Models created before this artifact was introduced log a warning and fall back to batch median/mode references.

The current production approximation:

Default behaviour in the current pipeline configuration:

The contributions are individual counterfactual differences. They are not additive, do not sum to the predicted probability, and do not establish causal effects. Correlated features can also produce combinations that are uncommon in the training data. Candidate-selection methodology and runtime/quality tradeoffs are tracked separately in GitHub issue 198.

Runtime and Progress Reporting

Inference now reports progress in the terminal in the same spirit as the criterion loop:

These progress bars are especially useful for large city-wide runs where inference and local explanation operate on tens of thousands of rows.

Default Thresholds

# bluegreencity_ki/ml/inference/predictor.py
DEFAULT_THRESHOLDS = {
    "critical": 0.85,
    "medium": 0.40,
    "low": 0.00,
}

Mapping:

Override in config.yaml (the configured thresholds always win over the code default):

ml:
  thresholds:
    critical: 0.85
    high: 0.60
    medium: 0.40      # ← adjust via threshold analysis
    low: 0.00

Graceful Degradation

If model not found:

try:
    predictor = DroughtRiskPredictor(model_path)
except FileNotFoundError:
    logger.warning("ML model not found; skipping ML scoring")
    # Continue pipeline without ML columns

Pipeline does not crash if ML model missing.

Output Columns:

Validation Framework

Seven Categories of Metrics

1. Overall Classification Metrics

Computed at decision threshold (default: 0.5):

Metric Formula Good Value
Precision TP / (TP + FP) > 0.7
Recall TP / (TP + FN) > 0.7
F1 2 × (Prec × Rec) / (Prec + Rec) > 0.7
Accuracy (TP + TN) / Total > 0.75

2. ROC-AUC

Threshold-independent metric:

Implementation (custom numpy):

def roc_auc_score(y_true: pd.Series, y_proba: pd.Series) -> float:
    """Compute ROC-AUC using trapezoid rule."""
    # Sort by predicted probability
    order = np.argsort(y_proba)[::-1]
    y_sorted = y_true.iloc[order].values
    
    # Compute TPR and FPR at each threshold
    tps = np.cumsum(y_sorted)
    fps = np.cumsum(1 - y_sorted)
    tpr = tps / tps[-1]
    fpr = fps / fps[-1]
    
    # Integrate using trapezoid rule
    return np.trapz(tpr, fpr)

3. Brier Score

Mean squared error of probabilities:

\[\text{Brier} = \frac{1}{n} \sum_{i=1}^{n} (p_i - y_i)^2\]

4. Calibration Curve

Tests if model probabilities match reality:

def calibration_curve(y_true, y_proba, n_bins=10) -> pd.DataFrame:
    """
    Group predictions into bins and compare predicted vs observed frequency.
    
    Returns:
      DataFrame with columns: bin_mid, predicted_mean, observed_freq, count
    """

Example output:

bin_mid predicted_mean observed_freq count
0.05 0.048 0.10 342
0.15 0.155 0.18 521
0.50 0.502 0.48 634
0.95 0.946 0.92 287

Interpretation: If predicted_mean ≈ observed_freq for all bins, model is well-calibrated.

5. Extreme-Group Comparison

Compare damage prevalence in top 5% vs bottom 5% of predicted risk:

def extreme_group_comparison(scores, labels, quantile=0.05) -> Dict:
    """
    Compute lift: prevalence in top 5% / prevalence in bottom 5%.
    
    Returns:
      {
        'top_prevalence': 0.85,
        'bottom_prevalence': 0.15,
        'lift': 5.67,
        'top_n': 1292,
        'bottom_n': 1292
      }
    """

Example:

Top 5%:    85% have damage
Bottom 5%: 15% have damage
→ Lift = 85 / 15 = 5.67×

6. Species-Specific Metrics

Compute all metrics per tree species (min. 30 samples):

def compute_species_metrics(species: pd.Series, min_samples=30) -> Dict[str, Dict]:
    """
    Returns:
      {
        "Tilia cordata": {n: 4251, roc_auc: 0.81, f1: 0.71, ...},
        "Quercus robur": {n: 892, roc_auc: 0.75, f1: 0.64, ...},
        ...
      }
    """

Use case: Identify species where model struggles (e.g., less training data, different biology).

7. Baseline Comparison

Compare ML predictions with existing weighted-sum baseline:

def compare_with_baseline(baseline_scores: pd.Series) -> Dict:
    """
    Returns:
      {
        'spearman': 0.42,                  # Rank correlation ML ↔ baseline
        'disagreement_rate': 0.23,         # % trees with different risk class
        'ml_corr_with_truth': 0.38,        # ML ↔ actual labels
        'baseline_corr_with_truth': 0.007  # Baseline ↔ actual labels
      }
    """

Key insight: ML model correlates much better with actual damage than static weighted sum (0.38 vs 0.007).

Decision Threshold Analysis

Motivation

The trained model outputs probabilities (e.g., 0.72 = 72% chance of damage). To make binary decisions (“at risk” / “not at risk”), a threshold is needed.

Trade-off:

The right threshold depends on your use case, not on model quality.

Tool: threshold_analysis.py

Computes metrics at thresholds 0.10–0.90 using out-of-fold (OOF) predictions:

python scripts/ml/threshold_analysis.py

Example output:

Threshold Precision Recall F1 Trees Flagged % of Stock
0.20 57.4% 88.4% 0.696 10,972 40.9%
0.35 67.9% 78.3% 0.728 8,218 30.6%
0.50 77.3% 64.6% 0.703 5,954 22.2%
0.70 88.4% 45.0% 0.596 3,630 13.5%
0.90 95.2% 18.3% 0.307 1,364 5.1%

Interpretation (threshold = 0.35):

Choosing the Threshold

Recommendation process:

  1. Run threshold_analysis.py to generate table
  2. Select threshold based on priorities:
Use Case Threshold Reasoning
Prevention-focused (protect at-risk trees) 0.35 Recall ≥ 78% (catch most damaged trees)
Balanced (general use) 0.50 Precision ≈ Recall (77% vs 65%)
Resource-constrained (limited budget) 0.70 Precision ≥ 88% (only high-confidence trees)
  1. Update config/config.yaml:
ml:
  thresholds:
    critical: 0.85    # first match wins (descending)
    high: 0.30
    medium: 0.10      # ← your chosen threshold
    low: 0.00

Important: Use OOF Predictions

Threshold analysis must use out-of-fold (OOF) predictions from cross-validation, not direct predictions on training data:

Method ROC-AUC (June 2026 reference run) Reliable? Reason
predict_proba(training_data) 1.000 ❌ Overfitted (model seen this data)
predict_proba_oof() 0.9937 ✅ Honest row-wise CV predictions (unseen rows)
grouped CV by objectid 0.8817 ✅✅ Honest panel-aware CV predictions with same-tree rows kept together

OOF predictions are derived from k-fold cross-validation used during training:

For the operational panel dataset, this row-wise logic is still not the full story because the same tree can appear multiple times across years. That is why the project now also keeps a separate grouped-CV reference run by objectid.

How to get OOF predictions:

from autogluon.tabular import TabularPredictor

# point at the model directory the run created
# (operational runs: data/models/autogluon/operational/<YYYY_MM>_<run>/model/<label>/)
predictor = TabularPredictor.load("<run_model_dir>")

# ✅ Honest (for threshold analysis)
proba_oof = predictor.predict_proba_oof()

# ❌ Overfitted (DO NOT use for threshold analysis)
proba_direct = predictor.predict_proba(training_data)

Configuration

Enable and configure ML scoring in config/config.yaml.

The current main benchmark uses an explicit stable reference portfolio instead of the earlier experimental extreme track:

ml:
  # Enable ML scoring in pipeline (requires trained model)
  enabled: true

  # Path to training dataset CSV
  training_data: "data/derived/ml/training_dataset_v6_operational_panel_2018_2024.csv"

  # Crown diagnosis CSV for label extraction
  drought_csv: "data/input/trees/Baum-2010-2025-v6_historisch_prod.csv"
  labeling:
    strategy: "operational_panel_drought_damage"
    snapshot_year: 2025
    label_horizon_years: 1
    target_label: "future_drought_damage_severe_t1"

  # AutoGluon training parameters
  training:
    model_path: "data/models/autogluon"       # Base directory for saved models
    presets: "best_quality"                   # Training shell; portfolio is explicitly defined below
    time_limit: 3600                          # Seconds per label (2026-09-15: 2400 -> 3600 to
                                              # leave budget for NN_TORCH and the L2 stacking level)
    eval_metric: "roc_auc"                    # Evaluation metric (for binary classification)
    num_bag_folds: 5                          # Cross-validation folds for bagging
    num_gpus: 1
    fit_kwargs:
      dynamic_stacking: false                 # DyStack is unnecessary; num_stack_levels=1
    hyperparameters:
      GBM:
        - extra_trees: true                   # GBM (XT variant)
          ag_args:
            name_suffix: "XT"
          ag_args_fit:
            num_gpus: 0
        - ag_args_fit:
            num_gpus: 0
      CAT:
        ag_args_fit:
          num_gpus: 0
      XGB:
        ag_args_fit:
          num_gpus: 0
      NN_TORCH:
        ag_args_fit:
          num_gpus: 1
          min_time_limit: 300                 # Secured time share so that NN_TORCH can compete
      RF:
        - criterion: "gini"
          ag_args:
            name_suffix: "Gini"
          ag_args_fit:
            num_gpus: 0
        - criterion: "entropy"
          ag_args:
            name_suffix: "Entr"
          ag_args_fit:
            num_gpus: 0
      XT:
        - criterion: "gini"
          ag_args:
            name_suffix: "Gini"
          ag_args_fit:
            num_gpus: 0
        - criterion: "entropy"
          ag_args:
            name_suffix: "Entr"
          ag_args_fit:
            num_gpus: 0

  # Which model to use for pipeline scoring
  inference:
    label: "future_drought_damage_severe_t1"  # Model is auto-resolved: <training.model_path>/<label>/

  # Per-tree local explanation for all trees by default
  local_explainability:
    enabled: true
    scope: "all"                              # all | extreme_groups | none
    extreme_group_quantile: 0.05              # only used when scope=extreme_groups
    top_k: 3                                  # number of driver strings per side
    # feature_importance_path: optional override; defaults to the training-only
    # importance artifact stored beside the resolved model

  # Probability → risk class thresholds (descending; first match wins)
  thresholds:
    critical: 0.85
    high: 0.30
    medium: 0.10
    low: 0.00

Parameter Details

Parameter Options Description
enabled true/false Enable ML scoring in pipeline
presets best_quality, high_quality, medium_quality, good_quality Training time vs. quality trade-off
time_limit seconds Max training time per label
eval_metric roc_auc, f1, accuracy Metric for model selection (use roc_auc for imbalanced binary)
num_bag_folds 2-10 K-fold CV for bagging (5 is standard)
hyperparameters mapping Explicit benchmark portfolio; preferred for stable reference runs and fair ablations
hyperparameters_config AutoGluon portfolio name Base model family portfolio, e.g. zeroshot_2025_12_18_gpu
auto_filter_portfolio true/false Filter unusable families and variants before training
ignore_model_constraints_families family list Keep selected families even if AutoGluon marks them constrained
model_family_hyperparameter_overrides mapping Per-family overrides such as GPU settings or TABDPT.use_flash=false
num_gpus integer Number of GPUs exposed to trainer families that support GPU
label label name Which trained model to use for inference
local_explainability.enabled true/false Enable or disable local driver generation
local_explainability.scope all, extreme_groups, none Whether explanations are computed for all rows, only extremes, or disabled. Production default is all.
local_explainability.extreme_group_quantile 0.0-0.5 Tail share used for low/high extreme groups when scope=extreme_groups
local_explainability.top_k integer Number of positive/negative drivers returned per tree
local_explainability.feature_importance_path path CSV used to rank candidate features before local explanation
thresholds floats Probability cutoffs for risk classes

For full-city production runs, scope: all computes driver strings for every tree. If you need a shorter diagnostic run, switch to extreme_groups or none temporarily.

Output Files

Pipeline Output (when ml.enabled=true)

Standard pipeline output (GPKG, PostGIS) includes the static score plus the ML scoring bundle:

Column Type Description Example
drought_risk_ml float [0, 1] P(drought damage) from AutoGluon 0.7234
drought_risk_ml_class categorical Risk class based on thresholds “high”
drought_risk_ml_confidence float [0, 1] Confidence proxy based on distance from 0.5 0.4468
drought_risk_ml_class_margin float [0, 1] Distance to the nearest threshold boundary 0.1234
drought_risk_ml_top_positive_drivers_approx string Positive local drivers (approximation) for configured rows “vitality_value:+0.0522; m_svf_value:+0.0479”
drought_risk_ml_top_negative_drivers_approx string Negative local drivers (approximation) for configured rows “ndvi_change_2020to2024:-0.0003”
drought_risk_ml_top_positive_drivers_shap string Positive SHAP drivers (SHAP stage) “vitality_value:+0.0480”
drought_risk_ml_top_negative_drivers_shap string Negative SHAP drivers (SHAP stage) “m_svf_value:-0.0011”
overall_drought_risk float [0, 1] Weighted sum (unchanged) 0.6421

Example:

objectid,art,pflanzjahr,drought_risk_rulebased,drought_risk_ml,drought_risk_ml_class,drought_risk_ml_confidence,drought_risk_ml_class_margin,drought_risk_ml_top_positive_drivers_approx,drought_risk_ml_top_negative_drivers_approx
12345,Tilia cordata,1985,0.642,0.723,high,0.446,0.123,"vitality_value:+0.0522; m_svf_value:+0.0479","ndvi_change_2020to2024:-0.0003"
12346,Quercus robur,2005,0.321,0.145,low,0.710,0.145,"m_svf_value:+0.0310; kr_durchm:+0.0201","age_value:-0.0148"
12347,Acer platanoides,1995,0.558,0.812,critical,0.624,0.062,"dist_road_m:+0.0117; vitality_value:+0.0104",""

In the production default (scope: all), every tree receives driver strings in both columns. In extreme_groups mode the two columns stay empty for the middle bulk of the dataset and are only populated for the configured low/high score tails.

Validation Output

Run scripts/ml/validate_model.py to generate:

data/validation/autogluon/overall_metrics.csv

Single row with classification metrics:

Column Example Meaning
roc_auc 0.8421 ROC-AUC (threshold-independent); >0.7 is good
brier_score 0.1245 Mean squared error of probabilities; <0.25 is good
precision 0.6789 Of flagged trees, 68% truly have damage
recall 0.7830 Of damaged trees, 78% are detected
f1 0.7284 Harmonic mean (balanced metric)
accuracy 0.7124 Overall correctness
n 25834 Total trees evaluated
threshold 0.35 Decision threshold used

data/validation/autogluon/extreme_group.csv

Top vs bottom 5% comparison:

Column Example Meaning
quantile 0.05 Top/bottom 5%
top_prevalence 0.8524 85% of top 5% have damage
bottom_prevalence 0.1502 15% of bottom 5% have damage
lift 5.67 Top group is 5.67× more likely
top_n 1292 Trees in top 5%
bottom_n 1292 Trees in bottom 5%

data/validation/autogluon/calibration_curve.csv

Reliability diagram (10 bins):

bin_mid predicted_mean observed_freq count
0.05 0.048 0.10 342
0.15 0.155 0.18 521
0.50 0.502 0.48 634
0.95 0.946 0.92 287

Good calibration: predicted_mean ≈ observed_freq across all bins.

data/validation/autogluon/species_metrics.csv

Per-baumart metrics (min. 30 trees):

species n roc_auc f1 precision recall
Tilia cordata 4251 0.8124 0.7056 0.6842 0.7289
Quercus robur 892 0.7542 0.6385 0.6201 0.6589
Acer platanoides 567 0.7812 0.6814 0.6623 0.7021

data/validation/autogluon/baseline_comparison.csv

ML vs weighted-sum comparison:

Column Example Meaning
spearman 0.42 Rank correlation ML ↔ baseline
disagreement_rate 0.23 23% trees get different risk class
ml_corr_with_truth 0.38 ML’s correlation with actual labels
baseline_corr_with_truth 0.007 Baseline’s correlation (near zero!)

Model Training Output

After training with scripts/ml/train_autogluon.py:

data/models/autogluon/<label>/

Trained model directory:

drought_damage_severe/
├── model.pkl                # Saved TabularPredictor
├── metadata.json            # Training info (time, presets, etc.)
├── models/                  # Individual models
│   ├── LightGBM_1/
│   ├── LightGBM_2/
│   ├── XGBoost_1/
│   ├── RandomForest_1/
│   ├── NeuralNet_1/
│   └── WeightedEnsemble_L2/ ← Final ensemble (best)
└── preprocessor.pkl         # Feature transformations

data/models/autogluon/leaderboards/<label>_leaderboard.csv

All trained models with scores:

model_name score_val score_test fit_time pred_time
WeightedEnsemble_L2 0.8421 0.8387 1823.4 0.45
LightGBM_2 0.8312 0.8269 142.3 0.12
XGBoost_1 0.8289 0.8251 198.7 0.15
RandomForest_1 0.8156 0.8124 256.1 0.34

data/models/autogluon/shap/

Kernel SHAP explainability artefacts for reporting:

<label>_summary.png                 # global SHAP summary plot
<label>_summary.pdf                 # optional print/report version
<label>_waterfall_row0.png          # local explanation for one selected tree
<label>_waterfall_row0.pdf          # optional print/report version

data/models/autogluon/feature_importances/<label>_importance.csv

Feature importance (for best model):

feature importance model
vitalitaet 0.2456 WeightedEnsemble_L2
pflanzjahr 0.1823 WeightedEnsemble_L2
obskh_hoehe 0.1234 WeightedEnsemble_L2
ndvi_value_2024 0.0987 WeightedEnsemble_L2

Troubleshooting

Training Issues

Problem Solution
Training takes too long Reduce time_limit (e.g., 600s) or use presets="medium_quality"
Out of memory Reduce num_bag_folds (e.g., 3 instead of 5) or use smaller dataset
“No space left on device” AutoGluon saves many models; clear data/models/autogluon/ or increase disk space
Training crashes with Java error Ensure Java 8+ installed (required for CatBoost)
RealTabPFN fails on Windows during import/setup The trainer now applies a Windows-specific runtime workaround automatically when a TabPFN family is actually selected. If TabPFN still fails, treat it as an optional family and continue with the remaining portfolio.
GPU run falls back to CPU In the stable reference benchmark this is expected for GBM, CAT, XGB, RF, and XT; only NN_TORCH is assigned GPU. If GPU boosting is required, first install and verify matching GPU-enabled LightGBM/XGBoost/CatBoost builds on this workstation.

Inference Issues

Problem Solution
Model not found at inference Ensure training completed successfully; check data/models/autogluon/<label>/ exists
NaN values in predictions Ensure input data has all expected feature columns; check logs for “missing features” warning
“Missing 50 expected features” Training and inference data have different columns; re-create training dataset
Pipeline crashes with ml.enabled=true Check try/except block in risk_scoring.py; should log warning and continue

Validation Issues

Problem Solution
ROC-AUC < 0.55 Model not better than random; check data quality (labels correct? features complete?)
High Brier score (>0.3) Poor calibration; consider calibration layer or use predict_proba_oof()
Lift ≈ 1.0 Model doesn’t discriminate; check feature importance for uninformative features
High disagreement with baseline Expected! Models learn different feature importance; OK if ML is more predictive

Data Quality Issues

Problem Solution
Too few labeled trees (<1000) Widen the drought window (earlier drought_start_year or later drought_cutoff_year) or use the simpler label (drought_damage_binary)
Extreme imbalance (>99% positive) Use drought_damage_severe instead of drought_damage_binary
Many NaN values in features Check PostGIS query; ensure remote sensing data joined correctly
Duplicate tree IDs Check CSV for duplicate entries; use MAX aggregation in label extraction

Threshold Selection Issues

Problem Solution
Threshold analysis shows ROC-AUC = 1.0 Using direct predictions (overfitted); use predict_proba_oof() instead
All thresholds give same recall Model predicts same probability for all trees; check feature variance
Precision/recall don’t match config threshold Update config.yaml after running threshold_analysis.py

References

AutoGluon

Machine Learning Metrics

Changelog

2026-03-06

2026-03-05

2026-03 (March)

2026-02 (February)

2026-01 / 2025-12 (December/January)

Last updated: 2026-03-06
Framework: AutoGluon v1.5+
Python: 3.9+
License: Apache 2.0