bluegreencity-ki

How to Run the Pipeline

Table of Contents

Overview

The BlueGreenCity-KI pipeline processes urban tree data, computes risk scores across multiple criteria, and writes results to files and PostGIS. It can be run:

All methods use the same CLI entry point and configuration file.

Prerequisites

  1. Conda environment activated:
conda activate <project-conda-env>
  1. Working directory must be the project root:
Set-Location <path-to-repository>

  1. Config file config/config.yaml must be present and valid (data sources, criteria weights, output paths configured).

  2. PostGIS must be reachable if PostGIS inputs or outputs are configured (see config/config.yaml → inputs / output.postgis).

Available Runners

Runner Entry Point Description
Risk Scoring (static) bluegreencity_ki.runners.cli Rule-based multi-criteria drought risk scoring

All runners read from config/config.yaml unless another path is passed via --config.

CLI Flags

Flag Description
--config <path> Path to the YAML config (required).
--output <path> Override the GeoPackage output path.
--train Force retraining even if a model for the configured window already exists.
--skip-holdout-validation Always skip stage-2 holdout validation.
--force-holdout-validation Force stage-2 holdout validation even when the model was only loaded (not retrained).
--skip-validation Skip run-monitoring validation against drought-damage ground truth.
--shap <scope> SHAP scope for operative predictions: high-risk (top-N trees), risk-classes (all trees in the configured risk classes), full (all trees), or off. Default from ml.shap.scope.

Stage-2 holdout validation runs only after training

Stage 2 (holdout validation) writes pure model-evaluation artefacts — leaderboards/, feature_importances/, temporal_holdout/, predictions_<year>.gpkg, layout.json. These change only when the model is retrained, so on an operational run that merely loads an existing model the stage is skipped automatically. It runs when the model was just (re)trained (--train) or when explicitly forced with --force-holdout-validation.

SHAP scope risk-classes (incremental)

--shap risk-classes computes SHAP drivers for the configured risk classes (ml.shap.risk_classes_include, default critical, high, medium). Set ml.shap.risk_classes_max_rows_per_class to a positive number for a deterministic, class-stratified sample; 0 explains every matching tree. Results are written incrementally per batch to two row-aligned files: risk_classes_shap_values.csv contains the contributions and risk_classes_feature_values.csv contains the original feature values, prediction score, and risk class required for a color-coded beeswarm. The *_drivers_shap columns are also UPDATEd in the database, so progress is visible during the run and partial results survive an interruption. A restarted run resumes by skipping trees that already carry SHAP drivers for the current run_date. Batch size is ml.shap.batch_size.

Execution Methods

Method 1: Direct Terminal Execution

Runs in the current terminal window. Output is printed directly to the console. Use this during development or when you want to observe the output in real time.

conda activate <project-conda-env>
python -m bluegreencity_ki.runners.cli --config config/config.yaml

Optional: Override the output path without changing the config:

python -m bluegreencity_ki.runners.cli --config config/config.yaml --output data/derived/my_run.gpkg

When to use:

Method 2: Detached Process via WMI (Survives VS Code Close)

Starts the pipeline as a fully detached Windows process via WMI (Win32_Process.Create). The process runs in session 0 under the Services context and is therefore not tied to the VS Code terminal session — it keeps running when VS Code or the terminal is closed. The full output (stdout + stderr) is written unbuffered to a timestamped log file in logs/.

Why not Start-Process from the VS Code terminal? Processes started with Start-Process from a VS Code terminal belong to the terminal’s job object and are killed when the VS Code window is closed. Start-Process -RedirectStandardOutput additionally buffers all output, so the log file stays empty during the run.

Why conda run instead of calling python.exe directly? Calling envs\<env>\python.exe directly leaves Library\bin of the environment off the PATH. psycopg2/libpq then loads incompatible DLLs from the system PATH, which crashes the process with an access violation (0xc0000005) at PostGIS writer initialisation. conda run -n <env> --no-capture-output sets up the environment correctly and streams output unbuffered.

$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$cmd = "cmd.exe /c set PYTHONUNBUFFERED=1&& C:\ProgramData\anaconda3\condabin\conda.bat run -n <project-conda-env> --no-capture-output python -u -m bluegreencity_ki.runners.cli --config config/config.yaml > logs\pipeline_run_$timestamp.log 2>&1"
$r = Invoke-CimMethod -ClassName Win32_Process -MethodName Create -Arguments @{
    CommandLine      = $cmd
    CurrentDirectory = '<path-to-repository>'
}
Write-Host "Pipeline gestartet - PID $($r.ProcessId) - Log: logs\pipeline_run_$timestamp.log"

When to use:

Monitor progress from a second terminal:

# Check if the process is still running (session-0 processes are not always
# visible to Get-Process filters - use tasklist)
tasklist /FI "IMAGENAME eq python.exe" /FO TABLE

# Tail the log in real time
Get-Content logs/pipeline_run_<timestamp>.log -Wait -Tail 30

Stop the detached process:

# Find the PID via tasklist (see above), then:
Stop-Process -Id <pid> -Force

Logging

Log Location

All pipeline logs are written to logs/ in the project root. Log files are not committed to Git (.gitignore).

File pattern Description
pipeline_run_YYYYMMDD_HHMMSS.log Full output of a Start-Process run (stdout + stderr combined)

Understanding Log Output

A successful run produces output in three phases:

  1. Data loading – tree inventory and context layers are read from PostGIS/files
  2. Criterion computation – each criterion prints stats (count, min, mean, max); spatial thresholds (e.g. temperature percentiles) are logged when computed from raster data
  3. Summary – profiling times per criterion, ASCII histograms of score distributions, output file paths

Warnings like the following are non-critical and can be ignored:

FutureWarning: Neither ogr.UseExceptions() nor ogr.DontUseExceptions() has been explicitly called.

Errors that indicate a failed run include:

Output Files

All outputs are written to data/derived/ (default) or the path configured under output.path in config/config.yaml.

File Description
trees_at_risk_<area>.gpkg GeoPackage with all risk scores (for QGIS)
trees_at_risk_<area>.csv Flat CSV with all risk scores
trees_at_risk_run_report.md Human-readable run summary (stats per criterion)
trees_at_risk_run_stats.json Machine-readable stats (for downstream processing)
report/validation/ Validation CSVs, charts, top-k and decile tables for static and ML
trees_at_risk_species_counts.csv Species counts for prioritizing tolerance factor research
trees_at_risk_dists/*.png Score distribution histograms per criterion

If output.postgis.enabled: true, results are additionally written to the PostGIS table bgcki.analysis_tree_risk_scores.

Troubleshooting

Symptom Likely cause Fix
No raster data available for day spatial threshold computation airtemp_raster_14h configured as PostGIS but table missing, or configured file path does not exist Check config/config.yaml → inputs.airtemp_raster_14h; switch to type: file if PostGIS table is not populated
No raster data available for land use landuse_raster PostGIS table missing or file not found Check inputs.landuse_raster in config
Log file contains only the GDAL FutureWarning, nothing else Python output is still buffered (process still running) Check with Get-Process -Name python; wait for completion
Output files not in data/output/ Default output path is data/derived/ Check config/config.yaml → output.path
PostGIS write fails DB not reachable or table schema mismatch Verify connection; check output.postgis config