bluegreencity-ki

ML Drought Run Report — 2026-04-02

Scope

This report captures four connected results that should stay comparable across later runs:

1. the new AutoGluon training run on the drought-window labels 2018..2024
2. the comparison against the earlier broad historical 2010..2025 model run
3. the validation and interpretation of the new model
4. the 2025 pipeline run with ML inference and the verification of the inventory/status handling introduced under #122

Key Artefacts

• Current leaderboard: data/models/autogluon/leaderboards/leaderboard_drought_damage_severe.csv
• Current feature importance: data/models/autogluon/feature_importances/feature_importance_drought_damage_severe.csv
• Old leaderboard: data/models/autogluon/leaderboards_2010to2025/leaderboard_drought_damage_severe.csv
• Old feature importance: data/models/autogluon/feature_importances_2010to2025/feature_importance_drought_damage_severe.csv
• Current validation export: data/validation/autogluon_2018_2024/overall_metrics.csv
• Current pipeline log: logs/pipeline_ml_2025_20260402_115334.log
• Current 2025 output: data/output/hannover_gesamt/trees_at_risk_hannover_gesamt.gpkg

Training Summary — Drought Window 2018..2024

• Best model: WeightedEnsemble_L3
• Validation metric: roc_auc = 0.906279
• Models trained and inferable: 34
• Best direct base models under the ensemble:
  • LightGBMPrep_r41_BAG_L2 with roc_auc = 0.906018
  • LightGBMPrep_r31_BAG_L2 with roc_auc = 0.905555
  • CatBoost_c1_BAG_L2 with roc_auc = 0.905141
• OOF validation metrics:
  • ROC-AUC = 0.9063
  • Brier = 0.094289
  • F1 = 0.6602
  • Precision = 0.7578
  • Recall = 0.5848
  • Lift (top vs bottom 5%) = 579.5

Comparison To The Earlier 2010..2025 Model

Headline comparison

Run	Best model	ROC-AUC
2010..2025 historical setup	WeightedEnsemble_L3	0.913697
2018..2024 drought-window setup	WeightedEnsemble_L3	0.906279

Delta (new - old): -0.007418

Interpretation

The new model is slightly weaker in raw ROC-AUC, but the drop is small enough to be fully plausible given the tighter label definition:

• the new target is anchored in the explicit drought period instead of the broader archive
• the new label period is more specific and therefore harder to learn
• the model remains clearly strong enough for ranking and operational prioritisation

Feature-importance shift

Top signals in the old model:

Rank	Feature	Importance
1	ga_lang	0.020584
2	vitality_value	0.016544
3	m_svf_value	0.001105
4	groundwater_level_std	0.001006

Top signals in the new model:

Rank	Feature	Importance
1	vitality_value	0.039576
2	groundwater_level_std	0.015304
3	m_svf_value	0.013538
4	ga_lang	0.003692
5	citree_krankheiten_schaedlinge	0.002934

Interpretation:

• the old model leaned more heavily on species identity (ga_lang)
• the new model is still dominated by vitality_value, but groundwater variability and SVF now contribute materially
• this is directionally useful because the model appears less species-only than before, while still not escaping the influence of vitality

Validation Interpretation

The current validation results indicate a model that is operationally useful for ranking and class-based prioritisation:

• ROC-AUC = 0.9063 means strong separation overall
• Brier = 0.0943 indicates reasonable probability quality
• Precision = 0.7578 at threshold 0.5 means flagged positives are often true positives
• Recall = 0.5848 means a non-trivial part of the severe class is still missed at the default threshold
• the top 5% of the score distribution has severe-label prevalence 0.9415, while the bottom 5% is almost empty at 0.0016

This is a good model for prioritisation workflows. If operational use later needs more sensitivity, the threshold should be tuned rather than assuming the current 0.5 cut-off is final.

2025 Pipeline Run With ML Inference

Inventory and analysis rows

The latest inventory snapshot (2025-01-01) produced:

• raw rows: 89,286
• analysis rows: 61,319
• excluded raw rows: 27,967

Raw status counts:

• active_tree = 67,756
• replacement_marker = 12,227
• removed_tree = 7,224
• object_control_placeholder = 1,384
• erroneous_record = 685
• planned_not_planted = 10

Excluded active rows with invalid coordinates: 6,437

Interpretation:

• the current raw inventory is kept for QA via trees_raw
• the actual analysis layer contains only analysable active trees with valid geometry
• this confirms the intended #122 separation between raw inventory and filtered analysis rows

Output and inference

The 2025 pipeline run completed successfully and wrote:

• data/output/hannover_gesamt/trees_at_risk_hannover_gesamt.gpkg
• PostGIS table bgcki.analysis_tree_risk_scores with 61,319 rows

ML inference summary from the run:

• drought_risk_ml count: 61,319
• min: 0.006587
• mean: 0.170218
• max: 0.967135

ML class counts in the final output:

• low = 51,352
• medium = 5,369
• high = 1,841
• critical = 2,757

Run-time validation summary on the 2025 pipeline output:

• Static ROC-AUC: 0.501
• ML ROC-AUC: 0.9089
• Improvement: +0.4079

Plausibility summary:

• critical = 0
• warning = 3
• info = 0

Verification Of #122 In The Final Output

The finished 2025 output was checked both stichprobenartig and globally.

Confirmed behavior:

• active trees with valid geometry are present in the output
• active trees without valid geometry are not present in the output
• replacement_marker rows are not present in the output
• removed_tree rows are not present in the output
• object_control_placeholder rows are not present in the output
• all 61,319 rows in the final output have inventory_status = active_tree
• no output row has invalid coordinates

This means the #122 inventory/status handling is not only present in raw QA fields, but also works through to the final 2025 scored outputs.

Plausibility Spot-Check Of ML Classes

Class means show a clear monotonic increase in drought_risk_ml, while the static weighted score changes much less:

ML class	Mean ML proba	Mean static score
low	0.0843	0.4883
medium	0.4573	0.4961
high	0.6692	0.4955
critical	0.8787	0.4999

Interpretation:

• the ML model provides substantially more separation than the weighted-sum baseline
• the ranking appears stable and non-random
• the model is therefore useful as a prioritisation layer on top of the static score

Vitality dependence

Vitality remains important, but it does not fully explain the final classes.

Vitality share by class:

Class	Vitality 1	Vitality 2	Vitality 3	Vitality 4+
low	41.5%	41.8%	13.6%	3.0%
medium	37.3%	44.0%	15.4%	3.1%
high	35.7%	46.9%	14.0%	3.3%
critical	35.5%	45.5%	14.9%	4.0%

Interpretation:

• classes are not just a direct re-encoding of vitality
• but vitality remains the dominant single feature and should be ablated explicitly in the next training iteration

Prepared Next Training Variants

The corresponding explicit feature variants were prepared in:

• config/ml_feature_variants_2026-04-02.yaml

Recommended order:

1. no_vitality: remove only vitality_value
2. no_vitality_no_species_label: remove vitality_value and ga_lang
3. no_vitality_no_species_profile: remove vitality_value, ga_lang, and all citree_* profile features

Practical Conclusion

This run establishes a useful new reference point:

• the drought-window model remains strong enough for production-style ranking
• the 2025 pipeline run confirms that current active trees are now filtered and scored correctly after the inventory/status work from #122
• the model is not purely a copy of the static weighted sum
• the next scientifically useful experiment is now a controlled vitality ablation, not another generic retrain with the same features