bluegreencity-ki

Plausibility Checks

Overview

The plausibility checks run automatically after the actual risk scoring. They are not intended to be a second modelling pass, but rather a technical and domain-level quality control layer.

In simple terms:

• The scoring calculates a risk.
• The plausibility check then verifies whether these results are internally consistent.
• Anomalies are reported, but the pipeline does not abort by default.

The implementation is located in bluegreencity_ki/plausibility/checker.py. Integration into the pipeline flow is done in bluegreencity_ki/pipelines/risk_scoring.py (repo-root paths — these links resolve in the repository, not on the docs site).

For operational use, plausibility artefacts are persisted both as files in the report directory and as PostGIS tables. This makes the diagnostic tables directly available to Apache Superset without parsing report files first.

Typical exported PostGIS table groups use the prefix plausibility_*, for example:

• summary and top findings,
• static criteria overview,
• spatial aggregation tables,
• static and ML correlation matrices,
• PCA embeddings and variance tables,
• cluster summaries,
• Ampelliste comparison,
• ML feature importance.

Why These Checks Were Introduced

The pipeline was already working correctly from a domain perspective, but with the new plausibility module it should become visible:

• whether individual criteria produce values outside the expected range,
• whether the direction of a criterion matches the domain understanding,
• whether spot-check recomputes yield the same values,
• whether spatial patterns are plausibly distributed,
• whether the ML input features and the ML model are internally consistent.

The plausibility module is therefore a combination of quality assurance, debugging support, and domain documentation.

Pipeline Flow

The order is now:

1. Calculate criteria
2. Aggregate total risk
3. Optionally calculate ML risk
4. Run plausibility checks
5. Run validation against ground truth
6. Write outputs and reports

Important:

• Plausibility runs after the calculations so that it evaluates the actual pipeline result.
• Errors in the plausibility checks are caught and logged, so that a pure reporting or diagnostics problem does not stop the entire pipeline.

Implemented Checks in the Static Branch

The static branch evaluates all activated criteria from config.yaml as well as the aggregated total risk.

1. Distribution Check

A simple statistical summary is calculated for each risk column:

• count of values
• missing values
• minimum and maximum
• mean and standard deviation
• median
• quantiles (q05, q95)
• count of exact 0, 1, and neutral values

Purpose:

• immediately detect gross imbalances,
• spot unusual concentrations at boundary values,
• quickly identify neutral or empty criteria.

2. Range Check

All risk values are checked for whether they fall within the expected range [0, 1].

Purpose in simple terms:

• A risk score greater than 1 or less than 0 is almost always a technical error.
• This check is the simplest and strictest safety check.

If values are outside the range, they are classified as critical.

3. Monotonicity Check

For selected criteria, it is verified whether the direction of risk development matches the domain expectation.

Examples:

• age: with increasing age, the risk should typically not decrease
• vitality: worse vitality should not lead to lower risk
• ndvi: better vegetation vitality should tend to lower risk
• drought_tolerance and heat_tolerance: higher tolerance should tend to lower risk

Technically, the following happens:

• the numeric value column and the risk column are merged,
• the value column is divided into quantile bins,
• the mean risk value is calculated per bin,
• the sequence of bin means is checked against the expected direction.

Important:

• The earlier Spearman correlation was removed from this part because that code path was unstable on native Windows.
• Instead, the trend direction is now checked directly via bin means.

Why this makes sense:

• The domain statement is preserved.
• The check is more robust.
• The plot/report remains understandable: you can directly see the mean risks per value range.

4. Spot Checks

A sample recompute is performed for a small allowlist of lightweight criteria.

The current default includes:

• salt
• street_type
• age
• vitality
• planting_structure
• land_use
• drought_tolerance
• pit_size
• tree_disc
• heat_tolerance

Procedure:

• the criterion is recalculated via its normal Criterion code,
• a few samples are compared with the value already in the result,
• deviations above the tolerance are reported.

Why not for all criteria?

• Some raster- or geometry-intensive criteria are expensive,
• the spot check should be reliable and fast,
• the pipeline should not be made unnecessarily slow or fragile by the plausibility checks.

5. Spatial Check

It is checked how risks are distributed spatially across administrative units.

By default, grouping is done by stadtteil. If this column is not present, a spatial join can be performed using polygon layers.

Default layers used:

• Districts: SKH20_Statistische_Gliederung/SKH20_Stadtteile/SKH20_Stadtteile.shp
• Boroughs: SKH20_Statistische_Gliederung/SKH20_Stadtbezirke/SKH20_Stadtbezirke.shp
• City boundary: SKH20_Statistische_Gliederung/SKH20_Stadtgrenze/SKH20_Stadtgrenze.shp

Calculated per area, among other things:

• number of trees
• mean risk
• risk minimum and maximum
• optionally mean criterion value, if a numeric value column is available

Why this is important:

• spatial outliers become visible,
• unusually large variation between districts is noticeable,
• mappable anomalies can be checked more quickly.

6. Separate Full Matrix of Static Value Columns

In addition, there is now a dedicated complete matrix for the static numeric value columns only.

This is explicitly different from the check on risk scores.

The question asked here is:

• Which raw criterion values are closely correlated with each other?
• Which domain input features might be measuring similar things?
• Where are there strong redundancies or opposing patterns already before conversion into risk scores?

Technically, the following happens:

• for all active static criteria, the available value columns are collected,
• duplicate columns are only considered once,
• the numeric value columns are transformed into a dedicated Pearson and Spearman matrix,
• strongly positive and strongly negative pairs are additionally exported as compact lists.

Why this is important:

• The domain raw data structure is separated from the later risk logic.
• It becomes visible whether two criteria already describe almost the same thing in their source values.
• It shows whether seemingly independent criteria are very closely coupled in the raw value space.
• Opposing domain patterns are also detectable, i.e., input features that systematically move in opposite directions.

Implemented Checks in the ML Branch

The ML branch only runs when drought_risk_ml is present.

1. Correlation Analysis

A sample is drawn from the numeric ML features. Pearson and Spearman correlation matrices are calculated on this sample.

Important for the current implementation:

• Correlation is no longer calculated via the problematic pandas/scipy shortcuts.
• Instead, a robust, manual correlation calculation is used.

Goal:

• make highly redundant feature pairs visible,
• detect potential leakage or duplicate effects,
• better interpret the model structure.

In the report, mainly the strongly correlated pairs are highlighted.

2. Explainability / Feature Importance

SHAP is currently not the default path.

Instead, the built-in AutoGluon feature importance is used. This is more stable and directly accessible via the loaded model.

In simple terms:

• The model reports which input features were most important for its decision.
• This is sufficient for global model interpretation in the report.

3. PCA

PCA projects the numeric ML input features into new axes that explain as much shared structure as possible.

In simple terms:

• Many individual input features are reduced to a few summarising directions.
• These new axes are called principal components.
• The first principal component explains the largest share of variance, the second the next largest, and so on.

Important:

• PCA is not a new model prediction.
• PCA does not directly explain risk, but the structure in the input space.
• If points in the PCA plot form clearly separated regions, this indicates systematic groups in the feature space.

When activated, the following are exported:

• the explained variance shares per component,
• an embedding of the sample in the PCA space,
• a scatter plot of the first two components.

Domain benefit:

• You can see whether the input features form a clearly structured feature space.
• You can tell whether groups are distributed more continuously or more cluster-like.
• You get a sense of how strongly the overall structure is shaped by a few main axes.

4. Species Clustering

The cluster analysis groups species by their position in the numeric feature space.

Important:

• Clusters are formed based on the aggregated input features.
• The risk scores themselves are not the primary basis for cluster formation.
• This is intentional: the clusters should show which species are similar in their feature profiles, not which species have already received the same final score.

Current procedure:

• numeric features are averaged per species,
• species with too few observations are excluded,
• features are imputed and scaled,
• then groups are formed in the feature space.
• afterwards, mean static and ML risks are additionally evaluated per cluster, without using these scores for the actual cluster formation.

Why risk scores are not directly included in cluster formation:

• Otherwise, the clusters would partially feed the pipeline result back directly.
• It would then be harder to distinguish whether groups arise from the input data or from the already calculated risk.
• From a domain perspective, it is more useful to see the structure of species independently of the final score first.

What this gives you:

• You can see which species have similar site and feature profiles.
• You can tell whether sensitive or robust species co-occur in the same structural spaces.
• You can later check whether species with similar profiles are also rated similarly, or whether contradictions exist.
• You can compare per cluster whether the static score and the ML score assess the same group similarly, or whether there are significant deviations.

5. Comparison with Ampelliste

This part is prepared but only active when an external “Ampelliste” list is configured later.

Then:

• species names are normalised,
• ML risks are aggregated at species level,
• external traffic light list ratings are translated into a scale,
• major deviations are reported.

Currently, this part is intentionally optional.

Configuration

Control is via config/config.yaml (section plausibility).

Important blocks:

• plausibility.enabled: master switch
• plausibility.spot_check_samples: number of samples
• plausibility.spot_check_tolerance: numeric tolerance
• plausibility.spot_check_criteria: allowlist for recomputes
• plausibility.visualizations.enabled: PNG/CSV artefacts for plausibility
• plausibility.monotonicity.*: direction and tolerance of the monotonicity check
• plausibility.spatial.*: spatial grouping and layers
• plausibility.ml.*: sample size, correlation, PCA, clustering, explainability
• plausibility.ampelliste.*: optional external comparison

Current production defaults are deliberately conservative:

• Spot checks only for lightweight criteria
• Correlation active
• Explainability active
• PCA and clustering enabled by default (plausibility.ml.pca_enabled: true, cluster_enabled: true)

Report and Artefacts

The run reports now include a dedicated plausibility section.

The following are written:

• a compact JSON summary of the plausibility results,
• a Markdown report with status, top findings, and ML summary,
• optionally visualisations and CSV artefacts under report/plausibility/.

Typical artefacts:

• summary figure by severity
• CSVs with spatial summaries
• CSVs and heatmap for the static value full matrix
• lists with the strongest positive and negative correlations of the static value columns
• correlation matrices
• CSV/PNG for feature importance
• PCA artefacts
• cluster artefacts

In addition, the report context contains two groups of images that are easy to confuse:

• report.save_hist_png: writes the histograms of result distributions to report/<runname>_dists/
• report.save_validation_png: writes the validation figures to report/validation/

In simple terms:

• If save_hist_png is set to false, exactly the histogram PNGs are missing from the *_dists folder.
• If save_validation_png is set to false, the validation images are missing, but not the rest of the report.

Stability Decisions During Implementation

When introducing the plausibility module, several things had to be made deliberately more robust:

Spearman Removed from Critical Paths

The original pandas/scipy correlation paths caused native crashes on Windows. They were therefore replaced with explicit, controlled calculations.

Recompute Spot Checks Limited

Some expensive raster criteria would effectively run the entire pipeline twice in a spot check. An allowlist exists for this reason.

Plausibility is Non-Blocking

If a plausibility part fails, the main result should not be lost. Therefore, errors are logged and included in the report rather than hard-stopping the pipeline.

Diagnostic Scripts

The diagnostic scripts were moved to scripts/debug/ to keep the regular project scripts clean.

There are now targeted helpers for:

• criteria ordering
• static plausibility
• ML plausibility
• age monotonicity
• validation without report

These scripts are explicitly debug tools and not part of the normal production run.

Domain Context of the ML Correlation Heatmap

The file report/plausibility/ml_correlation_heatmap.png shows the correlations between numeric ML input features, not between all criteria of the static model.

Important:

• The heatmap is based on the numeric ML feature frame.
• Non-numeric fields, IDs, and empty columns are removed beforehand.
• The visualisation deliberately shows only a subset of the matrix so that it remains readable.
• The complete values are available in the exported CSV files of the correlation matrices.

In simple terms:

• The heatmap is a diagnostic view for the ML model.
• It is not a complete overview of all static criteria in the pipeline.
• If a criterion is not present as a numeric ML feature, it will not appear there either.

Domain Context of the Static Value Full Matrix

The static value full matrix complements the ML heatmap with a second perspective.

It does not answer the question:

• “Which ML features are correlated with each other?”

but rather:

• “Which static criterion values are already closely correlated in their raw values?”

This makes it particularly important for the domain evaluation of the criteria logic.

Examples for interpretation:

• strong positive correlation: two criterion values often increase together,
• strong negative correlation: where one value increases, the other typically decreases,
• weak correlation: the criteria contribute rather independent information.

Particularly the negative correlations are of domain interest, as they can point to opposing site logics, for example more shading combined with lower heat load.

Are the Raw Criterion Values Also Analysed Against Each Other?

Yes. Since the current extension, there is a dedicated complete pairwise comparison of all available static numeric value columns.

The current state is:

• In the static branch, each criterion value is checked against its own risk score, primarily via monotonicity and spatial anomalies.
• In addition, there is now a dedicated separate matrix for the static value columns against each other.
• In the ML branch, numeric ML features continue to be correlated pairwise separately.

In simple terms:

• Yes, individual values are analysed.
• Yes, there is now also a dedicated complete comparison of the static raw values.
• And additionally, the comparison “value against its own score” and the separate ML feature structure analysis are retained.

Notes for Domain Users

For domain interpretation, three levels are important:

1. Static criterion: Checks whether an individual domain influence factor works sensibly on its own.

2. Spatial plausibility: Checks whether high or low risks cluster in comprehensible sub-areas.

3. ML plausibility: Checks whether the data-driven model operates with an internally comprehensible structure.

These levels answer different questions:

• “Is the individual criterion sensibly scaled from a domain perspective?”
• “Does the result show spatially unexpected jumps?”
• “Is the ML model learning redundant or surprising patterns?”

Particularly important for domain users:

• A warning in the plausibility checks does not automatically mean that the model is domain-incorrect.
• It means first of all: this is worth a closer examination.
• The plausibility module is therefore a diagnostic hint system, not an automatic judgement system.

Summary in Simple Terms

The plausibility module now answers four simple questions:

1. Are the values technically valid?
2. Do the criteria behave as one would expect from a domain perspective?
3. Can results be cleanly reproduced on a sample basis?
4. Does the ML model operate in an internally comprehensible and non-contradictory way?

That is exactly what this module was built for. It makes the pipeline more transparent, easier to debug, and more thoroughly verifiable from a domain perspective, without unnecessarily blocking the normal run.