bluegreencity-ki

Pipeline Workflow – One Complete Run

This document describes one full execution of the BlueGreenCity-KI drought risk assessment pipeline, from raw input data to visualisation in Apache Superset. Each step is accompanied by a short explanation.

Table of Contents

Input Data

Category Dataset Key Fields / Notes
Vector – Trees Municipal tree inventory (CSV / PostGIS) objectid, pflanzjahr, vitalitaet, ga_lang, st_umfang, kr_durchm; Point geometry EPSG:25832
Vector – Roads OpenStreetMap highways or ALKIS cadastral roads highway class (OSM) or straßenname + winter-service class; fetched via Overpass API or WFS
Raster – Temperature FITNAH 3D climate model Daytime (14 h) and night-time (04 h) temperature grids, ~10 m
Raster – NDVI CIR aerial imagery (2020, 2024) ~0.5–2 m; multi-temporal vegetation health index
Raster – Albedo UrbanGreenEye surface reflectance ~1 m
Raster – Land Use Hannover 2022 land-use classification 10 m; codes 0–28
Raster – Field Capacity Soil map plant-available water capacity
Raster – Groundwater Hydrogeological maps ~250 m; groundwater table depth
Raster – Cold Air FITNAH 3D (KPR + KVS layers) ~10 m; cold air production rate and volume flow
Raster – Building Shading Urban climate model 5 m; Sky View Factor and Sunlit Hours
Tabular – Species Citree species catalogue (CSV) Drought/heat/salt tolerance, light demand; lookup by ga_lang
Tabular – Ground Truth Crown-diagnosis survey 2010–2025 (CSV) wipfelduerre, totholz, belaubung_auffällig; used for ML labels and validation only

Spatial Methods

Method Where It Is Used
Sample raster at points Air temperature, NDVI (point mode), albedo, land use, field capacity, groundwater, cold air, sky view factor, sunlit hours
Zonal statistics over crown buffer NDVI (optional), sunlit hours, sky view factor
Spatial join / buffer search Street type and salt criterion (nearest road within configurable radius)
Neighbor counting (radius) Planting structure (trees within 25 m radius via spatial index)
Nearest-neighbour join OBSkH crown-diameter enrichment (tolerance 3 m)
Attribute / species lookup Drought tolerance, heat tolerance (CSV join by ga_lang)
Categorical mapping Pit size, tree disc, land use code → risk score via config table (see config/config.yaml under each criterion’s params.mapping)
OSM Overpass / WFS fetch Road network retrieval bounded by study area bounding box

Enrichment Module

The enrichment module runs between tree-inventory loading and criteria evaluation. It attaches external attributes to each tree record so that all downstream criteria can use the enriched data without needing to know about the external sources.

For full technical detail see data_maintenance/enrichment.md.

Data Sources

Source Dataset Format Records Key Attributes Added
OBSkH Oberflächenbasierte Stadtbaumkartierung Hannover Shapefile (Point, EPSG:25832) ~143,748 Crown diameter (obskh_krone), tree height (obskh_hoehe), crown area/volume, mean NDVI, distance to buildings
Citree Citree species catalogue CSV 365 profiles Drought tolerance (citree_trockenheitstoleranz), heat tolerance (citree_hitzevertraeglichkeit), light demand, crown form, root pattern, growth rate

Methods

Method Source What it does
Coordinate standardisation Existing CSV WGS84 columns (Y (lat) / X (long)) plus tree geometry fallback Reuses source latitude/longitude where present and backfills missing values from geometry reprojected to WGS84 for downstream web-map consumers such as deck.gl
Nearest-neighbour spatial join (≤ 3 m) OBSkH Matches each inventory tree to the closest OBSkH point within the tolerance; adds obskh_* columns and QA metadata (obskh_match_found, obskh_spatial_distance_m)
Crown diameter merge with QA tracking OBSkH + inventory Combines kr_durchm from the inventory with obskh_krone; small deviations (< 10 %) are averaged, larger deviations keep the inventory value; source is recorded in kronendurchmesser_source
Species attribute lookup Citree Each unique ga_lang inventory label is normalised and matched to a Citree profile via botanical-name parsing, cultivar extraction, and a scoring rule set; matched attributes are added as citree_* columns

Enrichment in the Pipeline

flowchart TD
    E1["① Load Tree Inventory\nv.load_layer(trees_spec)"]

    subgraph ENRICH["② Enrichment Module  enrich_trees(trees, cfg)"]
        direction TB
        E2A["🌐 Coordinate Standardisation\nreuse source Y/X if present\notherwise geom -> WGS84"]
        E2B["🗺 OBSkH Spatial Join\nNearest-neighbour ≤ 3 m\nobskh_* columns added"]
        E2C["📐 Crown Diameter Merge\nkr_durchm updated\nkr_durchm_inventory_original preserved"]
        E2D["🌿 Citree Species Lookup\nga_lang → profile match\ncitree_* columns added"]
    end

    E3["③ ctx.vector[\"trees\"]\nEnriched GeoDataFrame available to all criteria"]

    E4["④ Criteria Loop\nAll 20 criteria read from enriched layer"]

    E1 --> E2A --> E2B --> E2C --> E2D --> E3 --> E4

Enrichment is configurable and optional via inputs.enrichment in config.yaml. Setting enabled: false for either sub-source skips that step entirely.

Main Pipeline Flowchart

flowchart TD
    %% ── INPUT DATA ──────────────────────────────────────────────
    subgraph INPUTS["Input Data"]
        I1["🗂 Tree Inventory\n(CSV / PostGIS)"]
        I2["🛣 Road Network\n(OSM / ALKIS)"]
        I3["🟩 Raster Layers\n(temperature · NDVI · albedo\nland use · field capacity\ngroundwater · cold air\nsky view factor · sunlit hours)"]
        I4["📋 Species Catalogue\n(Citree CSV)"]
    end

    %% ── PIPELINE ─────────────────────────────────────────────────
    A["1️⃣  Load Context\nParse YAML config · load vector & raster data\ninitialise schema and scenario provider"]
    B["2️⃣  Normalise Weights\nSum criterion weights · normalise to 1.0"]

    subgraph CRITERIA_LOOP["3️⃣  Criteria Loop  (one iteration per criterion)"]
        direction TB
        C1["🔢 age"]
        C2["💚 vitality"]
        C3["🛣 street_type\n(spatial join)"]
        C4["🌳 planting_structure\n(neighbor count)"]
        C5["🌡 air_temperature_day\n(sample raster)"]
        C6["🌙 air_temperature_night\n(sample raster)"]
        C7["🏙 uhi_intensity\n(sample raster)"]
        C8["🗺 land_use\n(sample raster)"]
        C9["☀️ albedo\n(sample raster)"]
        C10["🌿 ndvi\n(sample raster / zonal stats)"]
        C11["🌵 drought_tolerance\n(species lookup)"]
        C12["🔥 heat_tolerance\n(species lookup)"]
        C13["🪨 pit_size\n(categorical)"]
        C14["🪵 tree_disc\n(categorical)"]
        C15["🧂 salt\n(spatial join)"]
        C16["🌬 cold_air\n(sample 2 rasters)"]
        C17["💧 field_capacity\n(sample raster)"]
        C18["🚰 groundwater\n(sample raster)"]
        C19["🏢 sky_view_factor\n(sample raster)"]
        C20["🕐 sunlit_hours\n(sample raster / zonal stats)"]
    end

    D["4️⃣  Aggregate Static Risk\noverall_drought_risk = Σ score_i × weight_i"]

    ML_GATE{ml.enabled?}

    ML_INF["5️⃣  ML Inference\nDroughtRiskPredictor.predict_proba()\nclassify_risk() · confidence() · explain_top_drivers()"]

    PLAUS["6️⃣  Plausibility Check\nRange · monotonicity · correlation · distribution checks\nML vs static comparison (if ML enabled)"]

    VAL_GATE{validation.enabled?}
    VAL["7️⃣  Validation\nJoin ground-truth labels · ROC-AUC · Brier · F1\nspecies metrics · extreme-group analysis"]

    E["8️⃣  Write Outputs\nGeoPackage (GPKG) · CSV · JSON run report"]

    VIS["9️⃣  Apache Superset\nRisk maps · histograms · species comparison\ndashboards connected to PostGIS"]

    %% ── EDGES ───────────────────────────────────────────────────
    INPUTS --> A
    A --> B --> CRITERIA_LOOP --> D
    D --> ML_GATE
    ML_GATE -- yes --> ML_INF --> PLAUS
    ML_GATE -- no  --> PLAUS
    PLAUS --> VAL_GATE
    VAL_GATE -- yes --> VAL --> E
    VAL_GATE -- no  --> E
    E --> VIS

Step Descriptions

# Step What happens
1 Load Context The YAML configuration is parsed, all vector and raster datasets are loaded through the backend adapter (OSGeo or ArcPy), and the Context object (config, schema, scenario, vector dict, raster dict) is assembled.
2 Normalise Weights Each criterion weight from the config is divided by the total sum so that all weights together equal 1.0, regardless of the number of active criteria.
3 Criteria Loop For every enabled criterion the pipeline calls criterion.compute(ctx), which returns a CriterionResult containing a score series (index = tree ID, values in [0, 1]) and optional extra columns (raw values). The loop collects scores, extras, and per-criterion statistics (min / max / mean / histogram).
4 Aggregate Static Risk All criterion scores are combined into a single DataFrame and the weighted sum overall_drought_risk = Σ(score_i × weight_i) is calculated for every tree.
5 ML Inference (optional) When ml.enabled: true, the saved AutoGluon model is loaded and applied to the full feature set, producing drought_risk_ml (probability), drought_risk_ml_class, drought_risk_ml_confidence, drought_risk_ml_class_margin, and SHAP-based top-driver strings.
6 Plausibility Check Score distributions are validated for range [0, 1], monotonicity, inter-criterion correlation, and statistical outliers; results are flagged as green / yellow / orange / red without aborting the pipeline.
7 Validation (optional) Historical crown-damage labels are joined to the output and standard classification metrics (ROC-AUC, Brier score, F1) are computed for both the static and ML scores, enabling model quality monitoring.
8 Write Outputs Final results are written to GeoPackage (geometry + all score and extra columns), an optional CSV, and a JSON run report with pipeline metadata, per-criterion statistics, distribution histograms, and plausibility / validation summaries.
9 Apache Superset The GPKG results (or their PostGIS import) are queried by pre-built Superset dashboards to render interactive risk maps, per-species charts, and threshold-based alert views.

Criteria and Derived Features

All 20 criteria below feed into overall_drought_risk (static risk assessment). Columns marked ML are part of the default feature allowlist used by the AutoGluon model.

The current static weights are derived from a hybrid weighting route: first, stable-baseline AutoGluon permutation importances are aggregated to the matching static criterion; second, these ML-derived weights are blended with an explicit domain prior that keeps water availability, heat exposure, and site quality more visible in the final score; third, the blended weights are normalized to a total weight of 1.0. This active variant corresponds to 55% ML signal plus 45% expert prior and was selected as the best practical compromise between ROC-AUC, F1, extreme-group lift, and limited vitality dominance. Features without a clean 1:1 static criterion counterpart, such as ga_lang, OBSkH-only metrics, or broader Citree profile fields, are intentionally excluded from the ML aggregation step.

Criterion Static Weight Input Data Spatial Method Derived Feature(s) ML
vitality 0.412844 vitalitaet (stage 0–5) Categorical mapping vitality_risk, vitality_value ✅
sky_view_factor 0.109819 Urban climate model raster Sample raster at points or zonal stats sky_view_factor_risk, sky_view_factor_value ✅
groundwater 0.105861 Hydrogeological depth raster (~250 m) Sample raster at points (+ optional trend layer) groundwater_risk, groundwater_level_latest, groundwater_level_median ✅
ndvi 0.050970 CIR aerial imagery 2020 + 2024 Point sampling or zonal stats over crown buffer ndvi_risk, ndvi_value_2024, ndvi_value_2020, ndvi_change_2020to2024 ✅
age 0.038598 pflanzjahr / st_umfang Age calculation from planting year or circumference estimation age_risk, age_value, age_source ✅
field_capacity 0.045653 Soil map raster Sample raster at points field_capacity_risk, field_capacity_value ✅
uhi_intensity 0.032883 GEONET Waermeinsel rasters (siedlungsraum + verkehr, 10 m) Sample complementary rasters; outside coverage = green-space reference 0 K (#171) uhi_intensity_risk, m_uhi_intensity_value, m_uhi_source ✅
air_temperature_day 0.032755 FITNAH 3D raster (14 h) Sample raster at points air_temperature_day_risk, air_temperature_14h_value ✅
sunlit_hours 0.030051 Urban climate model raster (seasonal) Sample raster at points or zonal stats sunlit_hours_risk, sunlit_hours_value, sunlit_hours_spring/summer/autumn ✅
cold_air 0.028361 FITNAH 3D KPR + KVS rasters Sample two rasters, weighted combination cold_air_risk, cold_air_kpr_value, cold_air_kvs_value ✅
air_temperature_night 0.024931 FITNAH 3D raster (04 h) Sample raster at points air_temperature_night_risk, air_temperature_04h_value ✅
pit_size 0.019992 Tree attribute pit_size (category) Categorical mapping via config table pit_size_risk, pit_size_value ✅
albedo 0.015353 UrbanGreenEye reflectance raster (~1 m) Sample raster at points albedo_risk, albedo_value ✅
land_use 0.014763 Hannover 2022 land-use raster (10 m) + LHH planting-site polygons Sample raster at points, map codes; planting-site surface type overrides where covered (#194) land_use_risk, land_use_value, land_use_label, planting_site_surface_type, planting_site_area_m2, land_use_score_source ✅
street_type 0.010783 OSM highways / ALKIS roads Spatial join – nearest road within buffer street_type_risk, street_type_value ✅
planting_structure 0.006219 Tree point geometries + LHH planting-site polygons Neighbor counting within 25 m radius; shared planting-site membership where covered (#194) planting_structure_risk, planting_structure_neighbor_count, planting_site_tree_count, planting_structure_method ✅
salt 0.005871 OSM / ALKIS roads with winter-service class Spatial join to nearest salted road salt_risk, salt_value ✅
tree_disc 0.004945 Tree attribute tree_disc (category) Categorical mapping via config table tree_disc_risk, tree_disc_value ✅
drought_tolerance 0.004708 Citree species catalogue (CSV) Species attribute lookup by ga_lang drought_tolerance_risk, drought_tolerance_value ✅
heat_tolerance 0.004641 Citree species catalogue (CSV) Species attribute lookup by ga_lang heat_tolerance_risk, heat_tolerance_value ✅
gk25 0.0 GK25 geology polygons (LBEG WFS, NodeId=201) Spatial join to geology polygon; SCHICHTEN parsing → material class (#166) gk25_risk, gk25_material_class, gk25_schichten, gk25_klartext ✅
subway_tunnel 0.0 Overture Maps rail segments (class=subway, is_tunnel) Nearest-distance to tunnel segment, distance breaks (#139) subway_tunnel_risk, subway_tunnel_dist_m ✅
construction_sites 0.0 OSM highway=construction (fetched fresh, no cache) Nearest-distance to construction way, distance breaks (#139) construction_sites_risk, construction_dist_m ✅

See risk_assessment_static/static_weighting_comparison_2026-04-04.md for the full comparison of the tested weighting routes.

ML output columns (appended when ml.enabled: true): drought_risk_ml, drought_risk_ml_class, drought_risk_ml_confidence, drought_risk_ml_class_margin, drought_risk_ml_top_positive_drivers_approx, drought_risk_ml_top_negative_drivers_approx, drought_risk_ml_top_positive_drivers_shap, drought_risk_ml_top_negative_drivers_shap. The _approx driver columns hold the model-agnostic perturbation approximation (generated for every tree by default, reducible via ml.local_explainability.scope); the _shap columns hold KernelExplainer SHAP values written by the SHAP stage and are kept strictly separate.

ML Training Sub-Workflow

The training workflow is run once offline (not during a regular pipeline run) and produces a saved model artefact that the pipeline then loads at inference time.

flowchart TD
    T1["1️⃣  Extract Labels\nscripts/ml/generate_ml_labels.py\nRead crown-diagnosis CSV 2010–2025\nAggregate MAX damage per tree over 2018–2024\nEncode: score ≥ 3 → drought_damage_severe = 1"]

    T2["2️⃣  Build Panel Feature Table\nscripts/ml/build_operational_panel_snapshots.py\nRun yearly feature snapshots for 2018-2024\nWrite panel rows to bgcki.analysis_tree_risk_scores_ml_operational_panel"]

    T2b["2b️⃣  Create Training Dataset\npython -m bluegreencity_ki.ml.training.dataset_builder\nRead panel features from PostGIS by snapshot_year\nJoin next-year drought labels\nExport a date-prefixed operational panel CSV"]

    T3["2a️⃣  Sanity-Check Target\nVerify class balance and join quality\nCheck label distribution per species\nFlag suspicious class skew before training"]

    T4["3️⃣  Train Validation Model\nrun_ml_path.py validation --train\nAutoGluon TabularPredictor\n5-fold bagging · time-budget configurable\nArtefacts → data/models/autogluon/annual_validation/<YYYY_MM>/train_<a>_<b>__predict_<p>__validate_<v>/"]

    T5["4️⃣  Validate Model\nTemporal holdout (out-of-time)\nROC-AUC · PR-AUC · Brier · per-year metrics\nFeature-importance stability · SHAP\nComparison vs. static weighted-sum baseline"]

    T5b["4b️⃣  Grouped CV Robustness Run\nrun_ml_path.py validation --grouped-cv\nStratifiedGroupKFold grouped by objectid\nArtefacts → <run_dir>/grouped_cv/"]

    T6["5️⃣  Analyse Decision Threshold\nPlot TPR / FPR vs. threshold\nChoose operational threshold (Youden's index)\nStore threshold in config (ml.thresholds)"]

    T7["5b️⃣  Refit Operational Model\nrun_ml_path.py validation --refit-final\nTrain on full window incl. holdout year\nArtefacts → data/models/autogluon/operational/<YYYY_MM>/train_<a>_<b>__predict_<p>/"]

    T8["✅  Models Ready\nValidation model → holdout metrics (Grafana)\nOperational model → operational-current predictions\nLoaded by DroughtRiskPredictor at pipeline runtime"]

    T1 --> T2 --> T2b --> T3 --> T4 --> T5 --> T6 --> T7 --> T8
    T4 --> T5b

Training Step Descriptions

# Step What happens
T1 Extract Labels The historical crown-diagnosis CSV (2010–2025) is read; per-tree damage indicators are aggregated over the drought window (2018–2024) and encoded into a binary label drought_damage_severe (1 if damage score ≥ 3).
T2 Build Panel Feature Table scripts/ml/build_operational_panel_snapshots.py runs yearly feature snapshots for the configured panel years and writes them to bgcki.analysis_tree_risk_scores_ml_operational_panel in PostGIS.
T2b Create Training Dataset python -m bluegreencity_ki.ml.training.dataset_builder reads the materialized panel rows by snapshot_year, joins next-year labels by objectid, and exports a date-prefixed file such as 2026_07_23_training_dataset_ml_operational_panel_2018_2024.csv.
T2a Sanity-Check Target Class balance, label distribution per species, and join quality are inspected to catch biases or data-quality issues before committing to model training.
T3 Train Validation Model AutoGluon TabularPredictor is run with 5-fold bagging over a configurable time budget. It automatically selects and stacks multiple model families (gradient boosters, neural nets, linear). Every training round creates a fresh timestamped run directory data/models/autogluon/annual_validation/<YYYY_MM>/train_<a>_<b>__predict_<p>__validate_<v>/ holding model, leaderboards, feature importances, SHAP, and holdout metrics.
T4 Validate Model The trained model is evaluated out-of-time on the holdout year using ROC-AUC, PR-AUC, Brier score, per-year metrics, and feature-importance stability; results are compared with the static weighted-sum baseline.
T4b Grouped CV Robustness Run The grouped CV reruns the operational panel training with StratifiedGroupKFold grouped by objectid so that observations of the same tree never leak across folds. As part of a training round it writes its artefacts into the validation run directory (<run_dir>/grouped_cv/); the standalone script scripts/ml/evaluate_grouped_cv.py still writes to data/models/autogluon/grouped_cv/.
T5 Analyse Decision Threshold TPR / FPR curves are plotted across thresholds; the operational threshold (default: Youden’s index) is selected and stored in the pipeline config under ml.thresholds.
T5b Refit Operational Model After the validation round, --refit-final retrains on the full window including the former holdout year (e.g. train 2018–2024 → predict 2025). The operational model is stored under data/models/autogluon/operational/<YYYY_MM>/train_<a>_<b>__predict_<p>/ with the same timestamp as the validation round and is used for operational-current predictions.
T6 Models Ready The validation model provides the holdout metrics (synced to Grafana); the operational model is loaded at runtime by DroughtRiskPredictor for operational-current runs (falling back to the validation model until the first refit-final exists).

For the stable reference benchmark and the documented feature-ablation runs, see risk_assessment_ml/reports/stable_ablation_report_2026-04-03.md. For the current production-result interpretation of the highest ML-risk combinations, see risk_assessment_ml/reports/high_risk_feature_combinations_2026-04-07.md.

Plausibility Check and Validation

flowchart TD
    P_IN["Pipeline Output\n(all criterion scores + overall_drought_risk\n+ optional ML columns)"]

    subgraph PLAUS_BLOCK["Plausibility Check  (bluegreencity_ki/plausibility/checker.py)"]
        P1["Range Validation\nAll scores ∈ [0, 1]?"]
        P2["Monotonicity Check\ne.g. age ↑ → risk ↑\nfield_capacity ↑ → risk ↓"]
        P3["Correlation Analysis\nPearson / Spearman between criteria\nDetect redundancy or contradictions"]
        P4["Distribution Diagnostics\nmean · std · skewness · kurtosis per criterion"]
        P5["ML vs Static Comparison\n(if ML enabled)\nDifference histogram · correlation"]
    end

    subgraph VAL_BLOCK["Validation  (bluegreencity_ki/validation/validator.py)"]
        V1["Load Ground-Truth Labels\nHistorical crown-diagnosis CSV\nJoin to output by objectid"]
        V2["Static Metrics\nROC-AUC · Brier score · Precision · Recall · F1\n@ default threshold 0.5"]
        V3["ML Metrics\n(if ML enabled)\nSame metrics on drought_risk_ml\nAUC improvement vs. static baseline"]
        V4["Species-Level Analysis\nPer-genus ROC-AUC\nExtreme-group analysis (top / bottom 5 %)"]
        V5["Threshold Analysis\nYouden's index · multi-quantile TPR\nOptimal threshold recommendation"]
    end

    REPORT["Plausibility + Validation Summary\nIssues flagged 🟢🟡🟠🔴\nWritten into JSON run report"]

    P_IN --> P1 & P2 & P3 & P4
    P4 --> P5
    P1 & P2 & P3 & P5 --> REPORT
    P_IN --> V1 --> V2
    V2 --> V3 --> V4 --> V5 --> REPORT

Plausibility and Validation Step Descriptions

Step What happens
Range Validation Verifies that every score column contains only values in [0, 1]; out-of-range values are flagged as errors.
Monotonicity Check Tests expected directional relationships (e.g. older age → higher risk, greater field capacity → lower risk) and flags inversions.
Correlation Analysis Computes Pearson and Spearman correlations between all criterion pairs to surface unexpected redundancy or contradictions.
Distribution Diagnostics Reports mean, standard deviation, skewness, and kurtosis per criterion so analysts can spot degenerate distributions.
ML vs Static Comparison Plots a difference histogram and correlation between drought_risk_ml and overall_drought_risk to show where the two approaches diverge.
Load Ground-Truth Labels The historical crown-diagnosis CSV is read and joined to the pipeline output by objectid; trees without a label are excluded from metric calculation.
Static Metrics Standard binary-classification metrics (ROC-AUC, Brier score, precision, recall, F1) are computed for overall_drought_risk as a predictor of drought_damage_severe.
ML Metrics The same metrics are computed for drought_risk_ml; the AUC improvement over the static baseline is reported.
Species-Level Analysis Metrics are broken down by tree genus and an extreme-group check confirms that the top / bottom 5 % of scored trees contain a disproportionate share of damaged / healthy trees.
Threshold Analysis TPR and FPR are evaluated across the full probability range; Youden’s index identifies the optimal operating threshold for downstream alerting.