Effluent monitoring / Forecast evaluation

See a nitrogen excursion
before it arrives.

Compare physics, learned process rates, and simple operational baselines on unseen wastewater scenarios. Inspect each forecast, its uncertainty, and the conditions where it fails.

Explore the recorded forecasts ↓
28Independent simulated scenarios
6 / 24 hForecast horizons
7Competing approaches
1 minuteExplicit BSM1 time step

01 / Inspect a rolling forecast

Each plotted prediction was issued 6 or 24 hours before its target timestamp. The current effluent and a past six-hour trend are available at issuance; future sensor readings are excluded. Lines join rolling forecasts, not a single multiday forecast.

Dark: BSM1 truth · Teal: forecast · Shaded: calibrated band · Dashed: illustrative alert threshold

02 / Require evidence against simple baselines

Mean ± standard deviation across five training-order/model seeds on one fixed simulator dataset. The same independent held-out scenarios are reused across seeds; they are not extra plants.

Held-out ammonia forecast errors
NH4_RMSE NOx_RMSE alert_recall false_positive_rate scenario_simultaneous_coverage interval_width_mgN_L
mean std mean std mean std mean std mean std mean std
split horizon_hours model
nominal 6 Black hybrid 1.217 0.013 0.406 0.002 0.911 0.005 0.108 0.008 1.00 0.0 10.493 0.308
Direct ML 1.134 0.036 0.445 0.013 0.905 0.008 0.041 0.005 1.00 0.0 23.132 0.282
Fixed physics 2.226 0.000 0.590 0.000 0.729 0.000 0.068 0.000 0.75 0.0 14.425 0.000
Grey hybrid 2.331 0.000 0.465 0.000 0.865 0.000 0.164 0.000 1.00 0.0 20.468 0.000
Local trend 1.847 0.000 0.837 0.000 0.829 0.000 0.164 0.000 1.00 0.0 9.271 0.000
Persistence 1.348 0.000 0.591 0.000 0.882 0.000 0.089 0.000 1.00 0.0 7.142 0.000
White hybrid 1.815 0.000 0.602 0.000 0.876 0.000 0.158 0.000 1.00 0.0 15.545 0.000
24 Black hybrid 5.088 0.106 1.037 0.026 0.800 0.006 0.275 0.016 0.75 0.0 30.080 0.491
Direct ML 1.087 0.032 0.328 0.012 0.915 0.003 0.031 0.011 1.00 0.0 22.578 0.247
Fixed physics 3.875 0.000 0.990 0.000 0.576 0.000 0.074 0.000 1.00 0.0 29.280 0.000
Grey hybrid 7.597 0.000 1.197 0.000 0.806 0.000 0.131 0.000 1.00 0.0 121.055 0.000
Local trend 4.784 0.000 1.896 0.000 0.718 0.000 0.254 0.000 1.00 0.0 25.923 0.000
Persistence 1.283 0.000 0.322 0.000 0.841 0.000 0.016 0.000 1.00 0.0 8.939 0.000
White hybrid 3.657 0.000 0.978 0.000 0.712 0.000 0.189 0.000 1.00 0.0 33.164 0.000
rain 6 Black hybrid 1.063 0.032 0.333 0.004 0.971 0.007 0.067 0.014 1.00 0.0 10.493 0.308
Direct ML 1.507 0.022 0.454 0.012 0.930 0.002 0.062 0.006 1.00 0.0 23.132 0.282
Fixed physics 3.333 0.000 0.738 0.000 0.836 0.000 0.178 0.000 0.50 0.0 14.425 0.000
Grey hybrid 3.078 0.000 0.384 0.000 0.814 0.000 0.200 0.000 0.75 0.0 20.468 0.000
Local trend 1.894 0.000 0.712 0.000 0.920 0.000 0.111 0.000 1.00 0.0 9.271 0.000
Persistence 1.413 0.000 0.515 0.000 0.925 0.000 0.089 0.000 1.00 0.0 7.142 0.000
White hybrid 2.942 0.000 0.706 0.000 0.903 0.000 0.267 0.000 1.00 0.0 15.545 0.000
24 Black hybrid 5.130 0.045 1.101 0.019 0.692 0.008 0.376 0.007 1.00 0.0 30.080 0.491
Direct ML 1.719 0.015 0.482 0.008 0.904 0.007 0.015 0.000 1.00 0.0 22.578 0.247
Fixed physics 5.417 0.000 1.326 0.000 0.615 0.000 0.242 0.000 1.00 0.0 29.280 0.000
Grey hybrid 5.990 0.000 1.009 0.000 0.699 0.000 0.212 0.000 1.00 0.0 121.055 0.000
Local trend 5.186 0.000 1.625 0.000 0.779 0.000 0.197 0.000 1.00 0.0 25.923 0.000
Persistence 1.951 0.000 0.549 0.000 0.850 0.000 0.015 0.000 0.75 0.0 8.939 0.000
White hybrid 5.013 0.000 1.247 0.000 0.726 0.000 0.303 0.000 1.00 0.0 33.164 0.000
storm 6 Black hybrid 1.512 0.021 0.298 0.004 0.991 0.000 0.093 0.006 1.00 0.0 10.493 0.308
Direct ML 2.514 0.035 0.406 0.004 0.956 0.005 0.013 0.005 1.00 0.0 23.132 0.282
Fixed physics 4.525 0.000 0.745 0.000 0.956 0.000 0.222 0.000 0.50 0.0 14.425 0.000
Grey hybrid 4.581 0.000 0.490 0.000 0.726 0.000 0.056 0.000 0.50 0.0 20.468 0.000
Local trend 1.975 0.000 0.642 0.000 0.951 0.000 0.111 0.000 0.75 0.0 9.271 0.000
Persistence 1.515 0.000 0.471 0.000 0.956 0.000 0.022 0.000 0.25 0.0 7.142 0.000
White hybrid 4.303 0.000 0.711 0.000 0.960 0.000 0.267 0.000 0.50 0.0 15.545 0.000
24 Black hybrid 5.140 0.045 0.924 0.022 0.869 0.005 0.382 0.007 1.00 0.0 30.080 0.491
Direct ML 2.787 0.019 0.570 0.002 0.896 0.004 0.000 0.000 1.00 0.0 22.578 0.247
Fixed physics 7.097 0.000 1.296 0.000 0.788 0.000 0.242 0.000 0.50 0.0 29.280 0.000
Grey hybrid 7.513 0.000 1.144 0.000 0.469 0.000 0.045 0.000 1.00 0.0 121.055 0.000
Local trend 5.934 0.000 1.526 0.000 0.814 0.000 0.106 0.000 0.00 0.0 25.923 0.000
Persistence 2.709 0.000 0.691 0.000 0.858 0.000 0.000 0.000 0.00 0.0 8.939 0.000
White hybrid 6.770 0.000 1.241 0.000 0.819 0.000 0.242 0.000 0.75 0.0 33.164 0.000
Alert metrics describe target-time NH4 exceedance at 4 mg N/L, an illustrative benchmark threshold, not a regulatory limit. Overlapping forecasts are correlated. Undefined false-positive rates or recall mean that the relevant class is absent.

Scored metrics · All predictions · Run manifest

03 / Test the data requirement

Learning curve for storm forecasts

Training sizes are 1, 2, 4, and 8 complete scenarios. Model settings are fixed before test evaluation. The hybrid's apparent reaction rates come from smoothed training sensors; they are not true simulator kinetics.

04 / Make uncertainty and failures visible

Eight calibration scenarios support finite nominal 80% simultaneous bands across both nitrogen outputs and all forecast origins at each horizon. Coverage assumptions do not extend to shifted rain/storm conditions. Wide bands can be operationally unhelpful even when they cover truth.

Worst cases by average ammonia error
NH4 MAE, mg/L
scenario split model horizon_hours
S16 nominal Grey hybrid 24 8.53
S25 storm Grey hybrid 24 8.37
Fixed physics 24 7.26
S27 storm Fixed physics 24 7.00
S25 storm White hybrid 24 6.93
S27 storm White hybrid 24 6.31
S20 rain Grey hybrid 24 6.10
S17 nominal Black hybrid 24 5.79
S21 rain Grey hybrid 24 5.68
S18 nominal Grey hybrid 24 5.63
Validity-domain evidence
flag_rate inside_RMSE outside_RMSE
mean std mean std mean std
split detector
nominal Convex hull 0.620 0.000 1.473 0.000 2.726 0.000
Hyperrectangle 0.133 0.000 2.306 0.000 2.487 0.000
K-means distance 0.304 0.000 2.345 0.000 2.296 0.000
One-class SVM 0.484 0.000 1.587 0.000 2.921 0.000
rain Convex hull 0.766 0.000 1.503 0.000 3.417 0.000
Hyperrectangle 0.285 0.000 2.530 0.000 4.146 0.000
K-means distance 0.345 0.000 2.670 0.000 3.731 0.000
One-class SVM 0.519 0.000 2.430 0.000 3.574 0.000
storm Convex hull 0.592 0.000 2.135 0.000 5.685 0.000
Hyperrectangle 0.332 0.000 2.192 0.000 7.315 0.000
K-means distance 0.347 0.002 2.381 0.001 7.060 0.016
One-class SVM 0.418 0.000 2.148 0.000 6.619 0.000
Sensor noise sensitivity
NH4_RMSE NOx_RMSE
mean std mean std
noise_fraction split horizon_hours model
0.01 nominal 6 Black hybrid 1.213 0.016 0.408 0.004
Direct ML 1.075 0.028 0.439 0.007
Fixed physics 2.203 0.007 0.590 0.001
Grey hybrid 2.393 0.016 0.472 0.002
Local trend 1.842 0.007 0.834 0.003
Persistence 1.347 0.004 0.592 0.001
White hybrid 1.788 0.010 0.602 0.001
24 Black hybrid 5.259 0.063 1.055 0.014
Direct ML 1.046 0.022 0.334 0.004
Fixed physics 3.845 0.013 0.983 0.001
Grey hybrid 7.794 0.213 1.230 0.005
Local trend 4.739 0.020 1.877 0.007
Persistence 1.275 0.005 0.320 0.003
White hybrid 3.631 0.020 0.972 0.003
rain 6 Black hybrid 1.027 0.037 0.326 0.008
Direct ML 1.465 0.028 0.451 0.004
Fixed physics 3.322 0.012 0.736 0.001
Grey hybrid 3.096 0.030 0.383 0.003
Local trend 1.845 0.010 0.705 0.004
Persistence 1.387 0.005 0.513 0.002
White hybrid 2.937 0.010 0.706 0.002
24 Black hybrid 5.115 0.021 1.101 0.018
Direct ML 1.683 0.037 0.487 0.007
Fixed physics 5.414 0.009 1.323 0.003
Grey hybrid 6.155 0.105 1.020 0.004
Local trend 5.107 0.012 1.604 0.009
Persistence 1.923 0.003 0.550 0.001
White hybrid 5.002 0.008 1.247 0.004
storm 6 Black hybrid 1.425 0.017 0.287 0.006
Direct ML 2.458 0.029 0.406 0.007
Fixed physics 4.525 0.005 0.744 0.001
Grey hybrid 4.535 0.030 0.498 0.004
Local trend 1.881 0.016 0.636 0.002
Persistence 1.481 0.007 0.468 0.001
White hybrid 4.314 0.014 0.711 0.003
24 Black hybrid 5.126 0.026 0.943 0.017
Direct ML 2.736 0.018 0.566 0.005
Fixed physics 7.075 0.012 1.291 0.002
Grey hybrid 7.427 0.041 1.173 0.005
Local trend 5.721 0.038 1.508 0.005
Persistence 2.703 0.003 0.693 0.001
White hybrid 6.756 0.018 1.235 0.005
0.05 nominal 6 Black hybrid 1.191 0.020 0.405 0.005
Direct ML 1.179 0.036 0.452 0.014
Fixed physics 2.264 0.052 0.591 0.003
Grey hybrid 2.081 0.036 0.461 0.008
Local trend 1.991 0.040 0.871 0.015
Persistence 1.389 0.021 0.603 0.006
White hybrid 1.918 0.052 0.607 0.005
24 Black hybrid 4.523 0.038 0.999 0.022
Direct ML 1.088 0.034 0.349 0.018
Fixed physics 3.933 0.073 0.983 0.008
Grey hybrid 4.929 0.624 1.122 0.024
Local trend 5.042 0.065 1.954 0.036
Persistence 1.303 0.026 0.338 0.013
White hybrid 3.786 0.103 0.984 0.013
rain 6 Black hybrid 1.258 0.075 0.346 0.012
Direct ML 1.564 0.033 0.457 0.007
Fixed physics 3.356 0.060 0.735 0.004
Grey hybrid 2.954 0.121 0.377 0.012
Local trend 2.061 0.057 0.727 0.021
Persistence 1.441 0.028 0.518 0.008
White hybrid 3.038 0.059 0.710 0.008
24 Black hybrid 4.836 0.130 1.067 0.037
Direct ML 1.702 0.032 0.493 0.010
Fixed physics 5.450 0.047 1.318 0.004
Grey hybrid 5.396 0.350 0.936 0.010
Local trend 5.593 0.154 1.649 0.052
Persistence 1.957 0.014 0.552 0.004
White hybrid 5.111 0.035 1.253 0.014
storm 6 Black hybrid 1.788 0.064 0.303 0.013
Direct ML 2.501 0.061 0.407 0.005
Fixed physics 4.550 0.020 0.742 0.004
Grey hybrid 4.268 0.177 0.421 0.014
Local trend 2.194 0.092 0.665 0.008
Persistence 1.560 0.043 0.476 0.004
White hybrid 4.372 0.044 0.710 0.012
24 Black hybrid 4.870 0.044 0.912 0.018
Direct ML 2.772 0.034 0.581 0.004
Fixed physics 7.110 0.056 1.286 0.010
Grey hybrid 7.410 0.217 1.009 0.019
Local trend 6.398 0.189 1.575 0.018
Persistence 2.747 0.019 0.698 0.003
White hybrid 6.824 0.064 1.230 0.026
0.10 nominal 6 Black hybrid 1.370 0.054 0.430 0.011
Direct ML 1.377 0.038 0.471 0.017
Fixed physics 2.459 0.099 0.599 0.005
Grey hybrid 2.027 0.032 0.486 0.010
Local trend 2.375 0.105 0.970 0.028
Persistence 1.508 0.048 0.636 0.011
White hybrid 2.250 0.096 0.625 0.009
24 Black hybrid 3.836 0.092 0.898 0.030
Direct ML 1.222 0.043 0.356 0.021
Fixed physics 4.168 0.133 0.983 0.015
Grey hybrid 3.716 0.317 0.999 0.044
Local trend 5.834 0.144 2.170 0.072
Persistence 1.402 0.055 0.391 0.022
White hybrid 4.151 0.199 1.010 0.020
rain 6 Black hybrid 1.757 0.145 0.402 0.015
Direct ML 1.758 0.067 0.467 0.002
Fixed physics 3.498 0.129 0.731 0.009
Grey hybrid 2.836 0.163 0.411 0.018
Local trend 2.647 0.124 0.795 0.043
Persistence 1.615 0.057 0.538 0.017
White hybrid 3.282 0.136 0.718 0.014
24 Black hybrid 4.574 0.162 1.044 0.055
Direct ML 1.781 0.028 0.511 0.008
Fixed physics 5.586 0.085 1.303 0.006
Grey hybrid 4.951 0.214 0.899 0.014
Local trend 6.791 0.420 1.815 0.110
Persistence 2.076 0.031 0.568 0.010
White hybrid 5.348 0.074 1.259 0.017
storm 6 Black hybrid 2.480 0.115 0.346 0.019
Direct ML 2.659 0.136 0.415 0.011
Fixed physics 4.675 0.036 0.736 0.009
Grey hybrid 3.911 0.305 0.395 0.013
Local trend 3.001 0.213 0.747 0.015
Persistence 1.806 0.099 0.500 0.008
White hybrid 4.540 0.070 0.708 0.018
24 Black hybrid 4.906 0.173 0.882 0.034
Direct ML 2.876 0.072 0.603 0.007
Fixed physics 7.248 0.109 1.272 0.017
Grey hybrid 7.357 0.539 0.887 0.030
Local trend 8.084 0.363 1.774 0.050
Persistence 2.882 0.050 0.716 0.008
White hybrid 6.993 0.108 1.219 0.039

05 / A concrete first engagement

Start with an alarm decision

Agree the pollutant, permitted operating modes, minimum useful lead time, review threshold, and false-alarm budget before fitting. Request 3–12 months of hourly influent flow and loading, aeration/DO, temperature, effluent NH4 and NOx, laboratory references, sensor maintenance, and operating-event logs.

Deliver a blind backtest

Audit timestamps and sensor delays, hold out contiguous weather/operating events, compare the current operating rule with persistence and local trend, and report missed excursions, false alarms, uncertainty width, and valid-data share. Keep process engineers in control of interventions.

Download the example hourly CSV schema · Source package includes a validate/infer command and exported JSON model.

06 / Reproduce and challenge the result

The benchmark runs custom influent scenarios for 14 days, discards the first four days, and evaluates full BSM1 outputs. This is not an official IWA benchmark certification or proof of steady-state initialization. NOx is combined oxidized nitrogen, not speciated nitrate. The plant, operating schedules, and disturbances are simulated.

Complete run configuration
{
  "project": "Wastewater Watch",
  "simulator": "bsm2-python BSM1 open loop, temperature effects enabled, explicit 1-minute step",
  "data": "28 custom simulated scenarios; not real plants or official IWA weather files",
  "scenario_counts": {
    "calibration": 8,
    "nominal": 4,
    "rain": 4,
    "storm": 4,
    "train": 8
  },
  "discarded_initial_hours": 96,
  "simulation_days": 14,
  "horizons_hours": [
    6,
    24
  ],
  "origin_spacing_hours": 3,
  "noise_fraction": 0.02,
  "seeds": 5,
  "seed_interpretation": "Training-subset/model seeds on one fixed simulator dataset, not independent plant replications",
  "alert_threshold_mgN_L": 4.0,
  "alert_note": "Illustrative NH4 threshold; not a regulatory limit. Labels concern target-time exceedance, not onset events or independent alarms.",
  "interval": "Nominal 80% simultaneous two-output prediction bands per scenario/horizon using maximum error over all origins; 8 independent calibration scenarios, rank 8 of 8. Exchangeability assumed; no guarantee under rain/storm shifts.",
  "forecast": "Current sensor readings plus past 6-hour trend only. Future measured controls are never given. The hybrid recursively updates predicted states with current controls held fixed. NOx is combined oxidized nitrogen, not speciated nitrate.",
  "limitations": [
    "Custom 14-day scenarios discard 4 days; no equilibrium/standard BSM benchmark certification",
    "Common initialized reactor states, constant simplified effective mixing volume",
    "Apparent net rates are not identified nitrifier maximum growth parameters",
    "Calibration uses noise-free simulator targets; field calibration needs validated reference sensors",
    "Synthetic cases share simulator architecture and source composition statistics; external plant validity is unknown"
  ],
  "runtime_seconds": 179.08,
  "versions": {
    "bsm2-python": "0.0.16",
    "numpy": "2.5.4",
    "scipy": "1.18.1",
    "pandas": "3.1.1",
    "scikit-learn": "1.9.1"
  },
  "python": "3.12.11",
  "data_sha256": "495cd917172ed76447159377d79127bf51ddaefd18045a4719c6846a2fb389f4",
  "source_sha256": "b38b41bbc08826341f04eb39680612eb52b2df3429d4e2c3afb5f17e81fc2800",
  "runs": [
    {
      "seed": 0,
      "training_ids": [
        "S00",
        "S01",
        "S02",
        "S03",
        "S04",
        "S05",
        "S06",
        "S07"
      ],
      "calibration_ids": [
        "S08",
        "S09",
        "S10",
        "S11",
        "S12",
        "S13",
        "S14",
        "S15"
      ]
    },
    {
      "seed": 1,
      "training_ids": [
        "S00",
        "S01",
        "S02",
        "S03",
        "S04",
        "S05",
        "S06",
        "S07"
      ],
      "calibration_ids": [
        "S08",
        "S09",
        "S10",
        "S11",
        "S12",
        "S13",
        "S14",
        "S15"
      ]
    },
    {
      "seed": 2,
      "training_ids": [
        "S00",
        "S01",
        "S02",
        "S03",
        "S04",
        "S05",
        "S06",
        "S07"
      ],
      "calibration_ids": [
        "S08",
        "S09",
        "S10",
        "S11",
        "S12",
        "S13",
        "S14",
        "S15"
      ]
    },
    {
      "seed": 3,
      "training_ids": [
        "S00",
        "S01",
        "S02",
        "S03",
        "S04",
        "S05",
        "S06",
        "S07"
      ],
      "calibration_ids": [
        "S08",
        "S09",
        "S10",
        "S11",
        "S12",
        "S13",
        "S14",
        "S15"
      ]
    },
    {
      "seed": 4,
      "training_ids": [
        "S00",
        "S01",
        "S02",
        "S03",
        "S04",
        "S05",
        "S06",
        "S07"
      ],
      "calibration_ids": [
        "S08",
        "S09",
        "S10",
        "S11",
        "S12",
        "S13",
        "S14",
        "S15"
      ]
    }
  ]
}
Exportable storm example