bluegreencity-ki

Tree Inventory Import

Purpose

This document describes how the Hannover tree inventory CSV is loaded into the runtime pipeline, how raw source fields are mapped to pipeline columns, and how the inventory is interpreted from a domain perspective before criteria are computed.

The current source file (active study area hannover_gesamt) is configured as:

project:
  study_areas:
    active: "hannover_gesamt"
    definitions:
      hannover_gesamt:
        trees:
          type: "csv_inventory"
          path: "./data/input/trees/Baum-2010-2025-v6_historisch_prod.csv"
          delimiter: ";"
          timestamp_column: "Zeitpunkt"
          timestamp_filter: "latest"

A second study area definition points to the current operational snapshot (./data/input/trees/Baum-2026-09-11-v6_aktuell_prod_wie_WFS.csv); the ML trees_current spec in data/derived/ml/run_configs/ selects it for re-training.

Runtime Flow

The runtime import path is:

Context.from_yaml() loads config/config.yaml
bluegreencity_ki.io.readers.load_inputs() resolves the active study area
bluegreencity_ki.adapters.osgeo_backend.vector.load_layer() dispatches the source type
_load_csv_inventory() reads the tree CSV, applies timestamp filtering, maps columns, derives domain status flags, and builds geometry
load_inputs() stores the raw layer as ctx.vector["trees_raw"]
load_inputs() filters the raw layer down to analysable trees and stores them as ctx.vector["trees"]
bluegreencity_ki.io.enrichment.enrich_trees() enriches the analysable tree layer with OBSkH and Citree attributes
Criteria operate only on ctx.vector["trees"]

Raw Versus Analysis Layer

The project now distinguishes between two runtime representations of the same inventory snapshot:

`ctx.vector["trees_raw"]`

keeps the raw CSV snapshot after timestamp filtering,
can retain rows with invalid geometry,
includes derived status fields for QA,
is the correct layer for reporting and input-data diagnostics.

`ctx.vector["trees"]`

contains only rows that are valid for the current risk analysis,
excludes rows with empty geometry,
excludes planning placeholders, removed trees, administrative placeholders, and erroneous records,
is the layer used by all criteria and downstream scoring.

Timestamp Logic

The loader expects a snapshot-style CSV with a Zeitpunkt column.

timestamp_filter: latest keeps the newest available snapshot date when no project.feature_reference_year is configured. With a feature reference year, the runtime resolves latest to December 31 of that year so historical inference cannot load inventory attributes from a later validation year.
A fixed date such as 2025-01-01 pins the import to a specific snapshot.

The hannover_ml_training_2010 study area reuses the same tree specification through a YAML anchor and overrides only timestamp_filter: "2010-12-31". Current and historical feature extraction therefore use the same raw CSV and do not require a PostGIS input table.

This allows the project to use one consolidated historical CSV while still running the main pipeline against the current inventory state.

Column Mapping

The CSV contains very long source field names. The loader keeps the original columns internally and adds mapped aliases for pipeline use. Primary CSV, GeoPackage, PostGIS, and aggregated outputs omit a source column when its mapped alias is present, preventing semantically duplicate output columns. Explicit audit fields such as kr_durchm_inventory_original remain in the outputs.

Examples:

Source column	Pipeline alias	Purpose
`ID`	`objectid`	Technical pipeline key
`Baumnummer`	`baumnummer`	Candidate domain key for future identity handling
`Hauptdaten Pflanzjahr`	`pflanzjahr`	Age criterion and domain interpretation
`Hauptdaten Vitalität`	`vitalitaet`	Current vitality stage
`Hauptdaten Gattung/Art`	`ga_lang`	Species label
`Weitere Hauptdaten Baumnummer (Typ)`	`baumnummer_typ`	Domain interpretation of free sites and data artefacts
`Gefällt am`	`gefaellt_am`	Removal status
`Gelöscht am`	`geloescht_am`	Removal status
`Historisch seit`	`historisch_seit`	Historical state marker
`Pflanzplanung Pflanzstatus`	`pflanzplanung_status`	Planned planting workflow

For vitality-style text fields, the loader can also extract leading integers, for example 2 noch befriedigend vitaler Baum -> 2.

Geometry Handling

The Hannover inventory is interpreted primarily through projected coordinates:

Rechtswert = X / Easting
Hochwert = Y / Northing
source CRS = EPSG:25832
Temporal CRS override (issue #216): historical extracts 2009-2013 were exported in Gauss-Krueger zone 3 (EPSG:31467, Rechtswert ~3.55M); from 2014 the inventory is UTM 32N (Rechtswert ~0.55M) — verified empirically on the per-year Rechtswert medians of the v6 historical CSV. The override is configured via geometry.temporal_crs_overrides (year threshold 2014) and is applied per observation year before reprojection to the project CRS.

Rows with missing or zero coordinates are marked via has_valid_coordinates. If keep_invalid_geometry_rows: true is set, these rows remain in trees_raw with empty geometry. They are still excluded from the analysable layer later on.

Domain Interpretation

The loader derives a domain-level status model before the analysis filter is applied. The authoritative taxonomy is documented in Tree_Inventory_Status_Matrix.md.

In practice, the import distinguishes at least the following groups:

real current trees,
planned but not yet planted trees,
removed trees,
replacement markers,
object-control placeholders,
obvious administrative artefacts.

This is important because a missing coordinate does not always mean a data quality problem. In many cases it is a legitimate planning or placeholder state.

Why The Input Moved To `data/input/trees`

The historical tree inventory CSV is a source dataset delivered from outside the pipeline. It is therefore treated as input data rather than derived data.

The project now uses data/input/trees for:

the current inventory CSV,
ML crown-diagnosis label extraction,
future domain documentation around source snapshots.

The local data folders follow this lifecycle:

Folder	Contents
`data/input/trees/`	Current externally supplied tree inventory only
`data/derived/`	Reproducible processed datasets created from input or web sources
`data/cache/`	Refreshable downloads and intermediate caches (OSM, ALKIS, climate, STAC, raster sampling)
`data/output/`	Pipeline and ML result exports
`data/unused/`	Superseded local raw files and old derived artifacts retained for manual review

Additional raw criterion data remains in the external data sources workspace; the pipeline references it through project.raster_base_dir. It is not copied into data/input.

Implementation Notes

The current-tree pipeline still uses objectid as its technical join key.
The domain-level key transition based on Baumnummer is intentionally not forced yet.
The written DBG note is the leading rule for replacement identity: same site, new tree identity.