1.0.0: release review fixes (rollout leakage, convex CMNN, corrected paired t-test, merge of main)

CHANGELOG.md

@@ -5,7 +5,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
 [Semantic Versioning](https://semver.org/). The version is defined once in `pyproject.toml`
 and has to match the git tag of a release.
 
-## [1.0.0] – 2026-11-06
+## [1.0.0] – unreleased
 
 Major release, not backwards compatible except for the AgentLib-MPC integration
 (`agentlib_mpc.machine_learning_plugins.physXAI`). A migration guide is part of the documentation.
@@ -37,7 +37,7 @@ loads single step models only. The export is verified against that branch, but n
   regression on expressions whose prediction is monotonic or convex in raw data columns, guaranteed on a domain (the
   box of the data or `bounds`) by an exchange method for the semi-infinite program, with a certificate per
   constraint (`exact`, `verified`, `data`, `violated`), the cost of the constraints and a report table. Solved as QP
-  (HiGHS) in QR coordinates, about 1 s for 1e5 samples. `px.Linear(monotonicities=...)` bounds the sign of weights
+  (HiGHS) in QR coordinates. `px.Linear(monotonicities=...)` bounds the sign of weights
   with `solver='qp'`.
 - Warm start of `ModularANN` (`warm_start=True`): with a normalized `Linear` block as architecture, the gradient
   based training starts at the least squares solution of its data only terms, trainable terms (e.g. `px.MLP`) start
@@ -122,7 +122,8 @@ loads single step models only. The export is verified against that branch, but n
 - `PHYSXAI_REQUIRE_AGENTLIB=1` makes the AgentLib-MPC contract tests fail instead of being skipped if AgentLib-MPC
   is missing (set in the CI); the model saved with physXAI 0.x is part of the tests.
 - `ModelComparison.paired(baseline)`: paired comparison of the runs of every model with a baseline on the same
-  splits (mean difference with confidence interval, share of better splits, Wilcoxon p-value), also in
+  splits (mean difference with confidence interval and p-value of the corrected resampled t-test of Nadeau & Bengio
+  2003, which accounts for the overlap of repeated splits of the same data, and the share of better splits), also in
   `save(run, baseline=...)` and `--baseline` of the command line. The training data config stores a fingerprint
   of the splits (`summary.fingerprint`), which checks that paired runs used the same rows.
 - Sub-sampling to a longer `time_step` (point sampling) logs a warning: quantities over an interval (power, energy)
@@ -142,7 +143,7 @@ loads single step models only. The export is verified against that branch, but n
   modelled with lagged inputs, a prediction over several steps recursively (`eval_horizon`).
 - `random_seed` of the keras models defaults to the global seed (`physXAI.set_seed`), else a new seed is drawn and
   stored in the config (before: 42).
-- Python >= 3.11 is required (3.9 and 3.10 are end of life).
+- Python >= 3.11 is required (3.9 and 3.10 are end of life), as in 0.3.0.
 - CI uses the shared template of the institute (RWTH-EBC/ci_templates, like AgentLib and AgentLib-MPC): tests,
   coverage, package build, pylint/ruff reports, documentation and GitHub Pages; additionally the unit tests on
   Windows, a smoke run of the examples (one epoch, `tests/smoke_examples.py`) and strict ruff and Sphinx
@@ -154,6 +155,8 @@ loads single step models only. The export is verified against that branch, but n
 - `keras>=3.15.0`: older versions have known vulnerabilities in loading models (unsafe deserialization, file
   disclosure via HDF5), see the GitHub security advisories.
 - Default run folder names no longer contain `:` (invalid on Windows).
+- The embargo of multi step data no longer adds the warmup: the warmup lies inside the split of its sequence, so the
+  reach of a sample plus `shift` suffices (more training data at every boundary).
 - Single step data is split in blocks instead of random rows by default (random rows overestimate
   the test performance because of autocorrelation).
 - Data is loaded onto a regular time grid: missing time stamps are missing values instead of being
@@ -216,8 +219,17 @@ loads single step models only. The export is verified against that branch, but n
   normalization.
 - Recurrent models (`RNNModel`, `PCNNModel`) accepted lags, changes and derived columns of a predicted feature as
   inputs, which held its measured values over the horizon (data leakage); they are rejected now.
-- The recursive evaluation and training of single step models took inputs `Change(T)` (also lagged or inside a
-  derived column) of the recursive feature from the data; they are recomputed from the predicted trajectory.
+- The recursive evaluation and training of single step models (and the criterion of the feature selection) took
+  inputs `Change(T)` (also lagged or inside a derived column) of the recursive feature, and lags (`Q_lag1`) and
+  changes (`Change(Q)`) of derived columns depending on it, from the data; they are recomputed from the predicted
+  trajectory.
+- `shift=0` did not reject inputs holding the target through a change of a derived column (`Change(Q)` with Q of
+  the output) or nested derived columns.
+- `CMNNModel` and `PINNModel` without monotonicities had convex neurons only, which with the non negative weights of
+  the following layers made the prediction convex in all inputs. `activation_split` defaults to `[1, 1, 1]` now.
+- Settings of the split (`test_size`, `val_size`, `block_length`, `gap_steps`) next to a `split=BlockSplit(...)` were
+  ignored silently; different values raise a `ValueError`.
+- A constant training target gave an output rescaling with the scale 0 (a constant model); the scale is 1 then.
 - The default embargo of the split ignored the reach of `Change(X)` inputs (X(t-1)) and of lags inside derived
   columns, so a value of the data could be part of a training and a test sample. The documentation no longer claims
   that the embargo removes the autocorrelation (test blocks lie in the period of the training data).
@@ -254,4 +266,12 @@ loads single step models only. The export is verified against that branch, but n
 - `physXAI.utils.logging` (`Logger`, `get_full_path`, `create_full_path`): replaced by `physXAI.io.Run` and the
   `logging` module, see the migration guide. Training data is no longer pickled (`TrainingData.save`).
 - `GitPython` dependency; paths are no longer resolved relative to the git root.
-- `requirements.txt` with outdated pins (use `pip install -e .[dev]`).
+- `requirements.txt`: replaced by the bounds in `pyproject.toml` (`pip install -e .[dev]`) and the tested versions in
+  `constraints.txt`.
+
+## [0.3.0] – 2026-11-07
+
+### Changed
+- Python 3.11 - 3.13 (3.9 and 3.10 are end of life, 3.9 is no longer available on the GitHub runners).
+- `keras>=3.15.0` (older versions are vulnerable when loading models) and `GitPython>=3.2.1`.
+- `requirements.txt`: the first releases fixing the open security advisories (Dependabot).

README.md

@@ -84,7 +84,7 @@ Models in this repository are categorized into two main types:
 	- Constrained Monotonic Neural Network (CMNN): Enforces monotonicity in selected inputs (with convex, concave and saturated activations).
 	- Physics-Informed Neural Network (PINN): Allows to add physics-informed loss functions
 	- Modular ANN: physically motivated expressions and trainable blocks (linear, MLP, CMNN, RBF, ...) in one model
-	- Constrained Regression: linear regression on physical terms, monotonic or convex in raw inputs on the whole operating range (certified)
+	- Constrained Regression: linear regression on physical terms, monotonic or convex in raw inputs on the box of the training data or given bounds (with a certificate per constraint)
 - Multi-Step Models: Predicting a trajectory.
 	- Recurrent Neural Network: SimpleRNN, GRU and LSTM, warmed up on the measured past.
 	- Physically Consistent Neural Network (PCNN, Di Natale et al. 2022): the predicted temperature rises with heating and outside temperature by construction.

docs/branch_review.md

@@ -2,42 +2,21 @@
 
 Before the release of 1.0.0, all open feature branches were reviewed against the refactored code base
 (`refactoring_v1.0.0`). No code was taken over: useful ideas were re-implemented in the 1.0 architecture,
-methodological errors were not carried over. **All branches below can be deleted.**
+methodological errors were not carried over. **All branches below can be deleted.** Branches deleted since the
+review are no longer listed.
 
 Ancestor check: `git merge-base --is-ancestor origin/<branch> origin/refactoring_v1.0.0`.
 
 | Branch | In 1.0? | Result |
 |---|---|---|
 | `26-integrate-modular-anns` | yes (ancestor) | fully contained; the refactoring started from it |
-| `14-integrate-developments-from-gitlab` | no | covered; one bug fixed in 1.0 |
-| `58-preprocessingmultistep-avoid-data-leakage-with-overlapping-training-and-test-windows` | no | superseded by `BlockSplit` |
-| `simplify-executables-structure` | no | not adopted |
 | `45-add-interpolation-methods-for-input-output` | no | idea kept for later (see below) |
-| `18-integrate-physically-consistent-multi-step-models` | no | PCNN re-implemented after the paper |
 | `67-pinns` | no | ideas re-implemented (PINN loss, trainable residuals) |
 
 ## 26-integrate-modular-anns
 Its tip is an ancestor of `refactoring_v1.0.0`. The modular expressions, `closed_loop` and `init_layer` were replaced
 on purpose (see CHANGELOG, *Removed*); the constrained regression was kept and rebuilt as `ModularANN(solver='qp')`.
 
-## 14-integrate-developments-from-gitlab
-- `MultiStepModel.evaluate(..., clipped_warmup_length)`: in 1.0 the warmup is never part of loss or metrics.
-- `test_size=0`: the multi step part was covered. For single step data 1.0 still returned empty test arrays and
-  predicted on them; **fixed in 1.0** (missing splits are `None` and skipped everywhere, tested).
-- The branch cut `td.y_*` in place (twice on a second evaluation).
-
-## 58-…-avoid-data-leakage-with-overlapping-training-and-test-windows
-`_get_stratified_block_indices` shuffled block ids (no stratification despite the name, global random state), had no
-embargo between neighbouring blocks of different splits and concatenated the training blocks in time order. 1.0's
-`BlockSplit` (stratified blocks, embargo, seeded generator, windows never cross a split boundary) covers and
-exceeds it.
-
-## simplify-executables-structure
-A `TestCase` helper inside the library (`physXAI/utils`), run names from the caller's stack frame, shared mutable
-`Feature` objects between examples and `from configuration import *`, built on the removed `Logger` and constructed
-features. 1.0's examples are short (`Run`, preprocessing, model, `run.save`) and deliberately self-contained.
-Not adopted.
-
 ## 45-add-interpolation-methods-for-input-output
 **Idea worth keeping:** sample controls and disturbances as **interval means** over the time step (zero order hold
 of the control), states as point values. Not adopted as is: float modulo on the time index (fails for datetime),
@@ -50,20 +29,6 @@ AgentLib-MPC relies on. Design for a later version:
 - Error if a state or recursive feature is aggregated (averaging T over the next interval leaks the target).
 - Lags and `Change` derived after the aggregation; forecasts of disturbances in the MPC then also interval means.
 
-## 18-integrate-physically-consistent-multi-step-models
-Methodological errors found:
-- **Data leakage:** the example fed `T_amb − T` with the *measured* room temperature over the whole 48 step horizon,
-  so the multi step error was too optimistic and the model not usable in an MPC.
-- No feedback of the predicted temperature (pure integrator without temperature dependent losses).
-- Sign constraints on z-scored inputs with a free bias: "w ≥ 0" had no physical meaning; the nonlinear part was
-  unconstrained.
-- Double normalization in the sub networks; monotonicity constraints mapped by position, not by name.
-
-**Re-implemented:** `PCNNModel` after Di Natale et al. (2022), with the energy accumulator and the unforced
-dynamics as states, the predicted temperature fed back, provable consistency, warmup and export like an RNN.
-**Not adopted (idea for later):** a generic monotone RNN (sign patterns in one SimpleRNN; the guarantee only holds
-in open loop, the PCNN covers the closed loop case).
-
 ## 67-pinns
 12 commits of a student project (~10k lines), building specific (hard coded columns and RC values), largely
 duplicated, without tests, based on the `Logger`. Methodological findings:

docs/release_1.0.0.md

@@ -7,7 +7,12 @@ by a workflow, which needs a change of the repository settings.
 ## 1. Merge
 
 1. Merge all open PRs into `refactoring_v1.0.0`.
-2. Merge the PR `refactoring_v1.0.0` → `main`.
+2. If `main` got new commits since (e.g. security fixes of 0.x), merge `main` into `refactoring_v1.0.0` first, so
+   the PR to `main` has no conflicts. Files changed on `main` but removed in 1.0 (old workflows, `requirements.txt`,
+   `.github/copilot-instructions.md`) stay removed, `pyproject.toml` keeps the 1.0 version (static version, bounded
+   dependencies); check that the dependency floors of `main` are kept in `pyproject.toml` and `constraints.txt`.
+   `main` up to 0.3.0 is merged already.
+3. Merge the PR `refactoring_v1.0.0` → `main`.
    AgentLib-MPC installs `physXAI @ git+https://github.com/RWTH-EBC/physXAI.git` (`main`) in every CI run, so its
    tests run against 1.0.0 from now on (`tests/test_physXAI_plugin` passes with 1.0.0).
 
@@ -56,8 +61,8 @@ Every other branch gets its own preview after a push: `https://rwth-ebc.github.i
    v1.0.0 → Target: main*, release notes from `CHANGELOG.md`). Keep the **known limitation** at the top of the release
    notes: the export of `RNNModel` / `PCNNModel` needs AgentLib-MPC branch `143-feature-multistep-model-support`.
 2. The workflow **Version Check** runs on the tag: it compares the tag with `version = "1.0.0"` in `pyproject.toml`
-   and requires the entry `## [1.0.0]` in `CHANGELOG.md`. If the release date differs from the date in the
-   CHANGELOG, update it in a commit before tagging.
+   and requires the entry `## [1.0.0]` in `CHANGELOG.md`. Before tagging, replace `unreleased` in that entry by the
+   release date in a commit on `main`.
 3. PyPI is not used yet (`PYPI_RELEASE: false` in `ci.yml`).
 4. Later releases: update `constraints.txt` to the versions the release was tested with (the job `tested-versions`
    of `windows.yml` installs them on Linux and Windows).

docs/source/guide/data.md

@@ -30,7 +30,8 @@ training and test data and overestimates the test performance. physXAI splits bl
   every split,
 - an embargo of `gap_steps` separates blocks of different splits, so no value of the data is part of a training
   and a test sample. The default is the reach of a sample into the past (lags, `Change(X)` = X(t) - X(t-1) and the
-  variables of derived `columns`) plus `shift`, plus the warmup of multi step data. The rows are taken from the
+  variables of derived `columns`) plus `shift`. The warmup of multi step sequences needs no embargo, it lies inside
+  the split of its sequence. The rows are taken from the
   training side, so validation and test blocks stay complete,
 - windows of multi step data never cross a split boundary.
 

docs/source/guide/evaluation.md

@@ -57,12 +57,20 @@ the baseline run by run on the same split, so the spread caused by the split can
 |---|---|
 | `pairs` | number of runs on the same split (paired by the split seed, else by the run name, `by=`) |
 | `baseline`, `model_median` | medians of the score |
-| `mean_difference`, `ci_low`, `ci_high` | mean of model - baseline and its confidence interval (Student t, `confidence=0.95`) |
+| `mean_difference`, `ci_low`, `ci_high` | mean of model - baseline and its confidence interval (corrected resampled t-test, `confidence=0.95`) |
 | `median_difference` | median of model - baseline |
 | `better` | share of the splits on which the model is better (lower RMSE and absolute bias, higher R2) |
-| `p_value` | two-sided Wilcoxon signed rank test; it needs at least 6 pairs to reach 0.05 |
+| `p_value` | two-sided p-value of the corrected resampled t-test |
+| `resampling` | ratio n_test / n_train of the correction (0: all runs on the same split) |
 | `same_data` | share of the pairs evaluated on the same rows (fingerprint of the split, physXAI >= 1.0) |
 
+The runs are no independent samples: splits of the same data share most of their training data, and a plain
+paired t-test (or Wilcoxon test) is too optimistic. Confidence interval and p-value follow the corrected resampled
+t-test of Nadeau & Bengio (2003): the variance of the mean difference is `var(d) · (1/n + n_test/n_train)` instead
+of `var(d)/n`. With `vary=('init',)` all runs share one split and nothing is corrected; `corrected=False` gives the
+plain paired t-test. `paired` tests several scores and models at once: correct the p-values for multiple testing
+(e.g. Holm) before drawing conclusions from several of them.
+
 A `same_data` below 1 (also logged as a warning) means that the splits differ despite the same seed, e.g.
 because of another block length. Single step models and RNNs on the same split can still be evaluated on
 different windows (the warmup of the RNN has to lie inside the split).
@@ -142,3 +150,5 @@ result.save(run)              # feature_selection.json and .html
 - `controls={'u': 1}`: the prediction has to increase with `u` on (almost) all validation samples,
   otherwise a candidate is implausible and not selected. Controls (and with `criterion='horizon'` the recursive feature) are never removed.
 - `tolerance` selects the smallest set within (1 + tolerance) × the best criterion.
+- The criterion is computed on the validation data, which also stops the training early, so the score of the
+  selected set is optimistic. Judge the final model on the test data.

docs/source/guide/models.md

@@ -27,7 +27,7 @@ stored in the config. Unknown arguments raise a `TypeError`.
 ## Constrained regression
 
 A linear regression on physical terms whose prediction is monotonic or convex in raw data columns, guaranteed on a
-domain. The fit is a convex quadratic program: one global optimum, no seeds, about a second for 1e5 samples.
+domain. The fit is a convex quadratic program: one global optimum, no seeds.
 
 ```python
 from physXAI.models import ModularANN, ShapeConstraints

docs/source/guide/multi_step.md

@@ -17,8 +17,8 @@ With `eval_horizon`, every model predicting one of its inputs is applied recursi
 data, as in an MPC:
 
 - `Change(T)` or `T` updates `T`, and `T_lag1`, `T_lag2`, ... are shifted,
-- inputs `Change(T)` (= `T - T_lag1`, also lagged) and derived data columns depending on `T` are recomputed in
-  every step,
+- inputs `Change(T)` (= `T - T_lag1`, also lagged), derived data columns depending on `T`, their lags (`Q_lag1`) and
+  changes (`Change(Q)`) are recomputed in every step,
 - all other inputs are taken from the data.
 
 `td.horizon_metrics` holds RMSE, MAE, bias and the standard deviation of the error of the absolute value

examples/11_compare_models.py

@@ -70,7 +70,7 @@
 comparison = ModelComparison.load(run.path)
 print(comparison.summary(split='test').to_string(float_format='{:.4f}'.format))
 # Run by run on the same splits: mean difference to the linear regression with its confidence interval, share of the
-# splits on which a model is better, p-value (Wilcoxon, needs at least 6 seeds for 0.05), same_data = 1: same splits
+# splits on which a model is better, p-value (t-test corrected for the overlap of the splits), same_data = 1: same splits
 paired = comparison.paired('linreg', split='test')
 print(paired[['pairs', 'mean_difference', 'ci_low', 'ci_high', 'better', 'same_data']]
       .to_string(float_format='{:.4f}'.format))