bluegreencity-ki

Operational Panel ML Result Report (2026-04-08)

Purpose

This report answers three questions about the new operational panel model:

  1. Does the 2025 output differ meaningfully from the old baseline — or just superficially?
  2. How does this model compare to previous versions?
  3. What does the follow-up validation actually tell us?

Three models are compared:

  1. Baseline — the stable reference model
  2. Minimal — a lightweight model trained on the same target variable
  3. Operational Panel — the new model, trained to predict next-year severe drought damage (future_drought_damage_severe_t1)

Model Comparison

The AutoGluon leaderboard summarizes training accuracy (ROC-AUC: higher = better, max = 1.0):

Model Target variable Best algorithm ROC-AUC Prediction time Training time
Baseline drought_damage_severe WeightedEnsemble_L3 0.9090 7.09 s 1560 s
Minimal drought_damage_severe WeightedEnsemble_L3 0.9087 3.60 s 14 s
Operational panel future_drought_damage_severe_t1 WeightedEnsemble_L3 0.9872 22.38 s 145 s

What this means: The operational panel model scores noticeably higher. However, this jump is too large to simply call it a “better model.” The most likely explanation: predicting next year’s damage from this year’s data is an easier task, because damage patterns tend to repeat from year to year — a shortcut the baseline didn’t have.

Most Important Input Features

These are the top 5 features each model relies on most (higher importance = stronger influence on the prediction):

Baseline Top-5

Rank Feature Importance
1 vitality_value 0.01138
2 citree_krankheiten_schaedlinge 0.00361
3 m_svf_value 0.00285
4 groundwater_level_std 0.00164
5 ga_lang 0.00070

Minimal Top-5

Rank Feature Importance
1 ga_lang 0.10669
2 vitality_value 0.04434
3 groundwater_level_std 0.02077
4 m_svf_value 0.01866
5 ndvi_value_2020 0.00752

Operational Panel Top-5

Rank Feature Importance
1 ga_lang 0.12683
2 m_svf_value 0.01778
3 vitality_value 0.01616
4 ndvi_value_current 0.00150
5 age_value 0.00116

What this means: All three models are primarily driven by tree species (ga_lang) and site conditions. The operational panel model gives slightly more weight to current vegetation state (ndvi_value_current), but species and location still dominate.

Comparison of 2025 Outputs

Direct comparison on the same tree population (n = 61,319):

Metric Baseline Minimal Operational Panel
Mean risk score 0.1850 0.1694 0.1727
95th percentile 0.7063 0.7201 0.9072
Share with score ≥ 0.6 7.30 % 7.33 % 11.17 %
Share with score ≥ 0.8 3.22 % 3.68 % 7.45 %

Risk class distribution (number of trees):

Model low medium high critical
Baseline 51,504 5,338 1,944 2,533
Minimal 52,532 4,293 1,730 2,764
Operational panel 50,481 3,986 1,751 5,101

Pairwise agreement between models:

Comparison Score correlation Mean absolute difference Top-10 % overlap
Operational vs Baseline 0.7686 0.1125 70.5 %
Operational vs Minimal 0.7832 0.1030 73.1 %

What this means: The outputs are related but not identical. The biggest difference is in the high-risk tail: the operational panel model flags roughly twice as many trees as “critical” compared to the baseline. While general risk patterns on a map look similar, the rankings differ meaningfully at the individual tree level — especially for high-risk trees.

Validation: 2024 → 2025 Backtest

Because validating the 2025 production run requires 2026 damage data (not yet available), a historical backtest was run instead: the model was tested on 2024 features against known 2025 damage labels.

Updated Decision Thresholds

The original thresholds (medium=0.35, high=0.60, critical=0.75) were set before the final validation. Based on the backtest, the thresholds were updated:

Class New threshold Precision Recall Share in 2025 output
medium 0.40 0.8013 0.8120 16.02 %
high 0.60 0.8777 0.6986 11.17 %
critical 0.85 0.9589 0.4684 6.44 %

Precision = of all trees flagged as at-risk, how many actually suffered damage.
Recall = of all trees that actually suffered damage, how many were correctly flagged.

What this means: The critical threshold was raised from 0.75 to 0.85 because the new model tends to assign higher scores overall — keeping the old threshold would flag too many trees as critical. The result is a smaller but more reliable critical group.

Validation dataset:

Validation Metrics

Metric Rule-based model ML model
ROC-AUC 0.619 0.958
Brier Score (lower = better) 0.312 0.059
F1 Score 0.400 0.796
Precision 0.366 0.840
Recall 0.440 0.757
Accuracy 0.725 0.919

ROC-AUC measures how well the model separates at-risk trees from healthy ones (0.5 = random guess, 1.0 = perfect).
F1 Score balances precision and recall into a single number.

Ranking quality:

Metric Rule-based ML
Top-10 % prevalence 35.8 % 96.0 %
Top-10 % capture rate 17.2 % 46.1 %

Top-10 % prevalence: Of the 10 % of trees assigned the highest risk score, what share actually experienced severe damage?
Top-10 % capture rate: Of all trees that actually suffered severe damage, what share ended up in the top-10 % risk group?

What this means: The ML model far outperforms the rule-based approach. In the backtest, 96 % of trees flagged as highest risk actually went on to suffer severe damage — compared to only 36 % for the static model. The ML model also catches nearly three times as many truly damaged trees in its top-risk group.

Important caveat: These results come from a historical backtest using cross-validation, not a fully blind future test. A genuine external test on 2026 data is still pending.

Overall Assessment

Strengths

  1. The model is designed for the actual use case: it explicitly predicts next year’s severe drought damage.
  2. The 2024 → 2025 backtest shows strong performance, clearly beating the rule-based reference.
  3. The model meaningfully reshapes risk prioritization, especially at the high-risk end.

Limitations

  1. The ROC-AUC of 0.9872 is unusually high. Part of this likely reflects damage patterns repeating year to year — not purely better ecological understanding.
  2. Validation is based on cross-validation and a historical backtest, not a true future blind test.
  3. Only 20,826 of 54,563 backtest trees had matching damage labels; results apply to this subset only.
  4. The model is more aggressive in the high-risk range — without updated thresholds, it would over-flag trees.

Active Features (17 total)

ga_lang, age_value, vitality_value, dist_road_m, salt_value, street_type_value, ndvi_value_current, ndvi_age_years, ndvi_available, groundwater_level_latest, groundwater_level_median, groundwater_level_max, groundwater_level_std, m_svf_value, m_sunlit_hours_spring, m_sunlit_hours_summer, m_sunlit_hours_autumn

Bottom Line

The operational panel model is a methodological step forward: it is designed for the actual use case and performs well in backtesting. It should be used with the updated thresholds. A final verdict on production quality depends on the 2026 external validation.

Data Sources

  1. data/models/autogluon/leaderboards_2018to2024_baseline_stable_reference/leaderboard_drought_damage_severe.csv
  2. data/models/autogluon/feature_importances_2018to2024_baseline_stable_reference/feature_importance_drought_damage_severe.csv
  3. data/models/autogluon_ml_minimal_labels_2018_2024/leaderboards/leaderboard_drought_damage_severe.csv
  4. data/models/autogluon_ml_minimal_labels_2018_2024/feature_importances/feature_importance_drought_damage_severe.csv
  5. data/models/autogluon_ml_operational_panel/leaderboards/leaderboard_future_drought_damage_severe_t1.csv
  6. data/models/autogluon_ml_operational_panel/feature_importances/feature_importance_future_drought_damage_severe_t1.csv
  7. data/output/ml_operational_predict_2025/hannover_gesamt/report/trees_at_risk_ml_operational_predict_2025_hannover_gesamt_run_stats.json
  8. data/output/archive/cleanup_2026-04-09/ml_operational_backtest_2024/hannover_gesamt/report/trees_at_risk_ml_operational_backtest_2024_hannover_gesamt_run_stats.json