bluegreencity-ki

Treelife Overlap Validation Report (2026-10-09)

Implements the validation step of issue #257 for the 2026_10 retraining round (label B+ future_drought_damage_severe_t1, 54 features, spatial_cv disabled). The check answers one question: does the model rank the trees that the city actually treats (Treelife root-zone therapy) higher than the average tree?

Purpose

Treelife is the root-zone treatment the Landeshauptstadt Hannover (LHH) prescribes for stressed trees. The decision is made by field staff from their own observation, independent of the ML model — which makes it an external, practice-based validation signal: if the model is useful operationally, the trees it flags as high-risk should be enriched in the set of trees that professionals actually intervene on.

Explicitly out of scope (per issue #257): using Treelife as a training label for a second model. The treatment decision is endogenous (budget, species, site, assessor judgement) and largely redundant with label B+.

Method

1. Treelife list extraction — from the historical LHH inventory CSV, all trees whose Maßnahmen festgelegt or Maßnahmen durchgeführt column contains the token “Treelife” (semicolon lists split, IDs deduplicated) in the reference year. Historical volumes: 2021: 1,383 / 2022: 1,636 / 2023: 1,255 / 2024: 1,862 / 2025: 1,061 trees.
2. Ranking overlap — model predictions are ranked by risk score; for the top 5 / 10 / 20 % of the ranked population the share of Treelife trees is compared against their population share, giving a lift (enrichment) factor.
3. Criterion (per issue #257): top-decile lift ≥ 2 → the model’s top ranking covers actual operational interventions.

Implementation: scripts/analysis/treelife_overlap_check.py (CLI: --holdout-csv, --trees-csv, --treelife-year, --top-fractions, --output-json). Result artefacts: data/derived/ml/treelife_overlap_2024.json and data/derived/ml/treelife_overlap_operational_2026.json (derived data, not committed).

Analysis 1 — 2024 Temporal Holdout (validation model)

2024 holdout predictions of the validation run (train_2018_2023__predict_2024__validate_2025, 25,334 trees, snapshot 2024 → damage 2025) against the Treelife list 2024 (1,862 trees; 1,318 of them present in the holdout, i.e. inspected in 2024).

• Treelife population share (ranked holdout): 5.2 %
• Mean risk score: Treelife trees 0.300 vs. population 0.190 (1.58×)

Ranking	Treelife in top group	Share of flagged	Share of Treelife caught	Lift
Top 5 %	90	7.1 %	6.8 %	1.37
Top 10 %	208	8.2 %	15.8 %	1.58
Top 20 %	557	11.0 %	42.3 %	2.11

Analysis 2 — Operational 2026 Scores vs. Treelife 2025

Operational 2026 scores of all 89,280 active v6 trees (new 54-feature model, run train_2018_2024__predict_2025) against the Treelife list 2025 (1,061 trees, last complete inventory year; 1,054 in the current inventory).

• Treelife population share: 1.18 %
• Mean risk score: Treelife trees 0.213 vs. population 0.154 (1.39×)

Ranking	Treelife in top group	Share of flagged	Share of Treelife caught	Lift
Top 5 %	105	2.4 %	10.0 %	1.99
Top 10 %	182	2.0 %	17.3 %	1.73
Top 20 %	314	1.8 %	29.8 %	1.49

Findings

• Criterion (top-decile lift ≥ 2) not met in either analysis (1.58 / 1.73). The model does not concentrate the top 10 % exclusively on professionally treated trees.
• Enrichment is real and substantial: treated trees score 1.4–1.6× the population mean, and the extreme top 5 % of the operational ranking reaches a lift of 1.99 — the ~4,500 highest-risk trees contain 10 % of all 2025-Treelife trees at only 1.2 % population share.
• The lift profiles differ in shape (2024 holdout: rising 1.37 → 2.11; operational 2026: falling 1.99 → 1.49). Plausible drivers: (a) the 2025 treatment list and the 2026 ranking are offset by one year (prediction horizon); (b) the 2025 cohort is smaller (1,061 vs. 1,862) → more noise; (c) on the holdout, the most extreme trees are often already felling candidates rather than treatment candidates, which dilutes the top decile.

Verdict

Treelife overlap is a useful, soft validation signal — not an acceptance gate. The models consistently rank actually-treated trees enriched (1.4–2.0× depending on top group and reference year) without being exclusive to them. Recommended ongoing use: rerun the check in future monthly cycles against the newest complete treatment year and track the top-5 % and top-10 % lifts over time (a decaying trend would be an early drift indicator complementing the #260 AUC drift watch).

Rerun

# 2024 holdout (validation model)
conda run -n bgcki_wesemann python scripts/analysis/treelife_overlap_check.py `
    --holdout-csv data/models/autogluon/annual_validation/2026_10/train_2018_2023__predict_2024__validate_2025/temporal_holdout/holdout_predictions.csv `
    --trees-csv <historical inventory CSV> --treelife-year 2024 `
    --output-json data/derived/ml/treelife_overlap_2024.json

# Operational 2026 (against latest operational score export)
conda run -n bgcki_wesemann python scripts/analysis/treelife_overlap_check.py `
    --holdout-csv <operational 2026 score CSV with objectid,y_proba> `
    --trees-csv <historical inventory CSV> --treelife-year 2025 `
    --output-json data/derived/ml/treelife_overlap_operational_2026.json