Refactor PETSc preconditioner for CVODE by bendudson · Pull Request #3382 · boutproject/BOUT-dev

bendudson · 2026-05-29T22:47:37Z

Refactors the Jacobian coloring preconditioner and makes it available as a CVODE preconditioner.

test2 from scratch. 10 steps of size 1.

No preconditioning:

CVODE: nsteps 803, nfevals 967, nniters 966, npevals 0, nliters 4132
    -> Newton iterations per step: 1.202989e+00
    -> Linear iterations per Newton iteration: 4.277433e+00
    -> Preconditioner evaluations per Newton: 0.000000e+00
    -> Last step size: 2.046315e-02, order: 2
    -> Local error fails: 0, nonlinear convergence fails: 64
    -> Stability limit order reductions: 0
1.000e+01        292       1.22e+01     3.8    0.0   32.9    0.7   62.6
Run time : 3 m 38 s

User preconditioning:

CVODE: nsteps 237, nfevals 280, nniters 279, npevals 913, nliters 669
    -> Newton iterations per step: 1.177215e+00
    -> Linear iterations per Newton iteration: 2.397849e+00
    -> Preconditioner evaluations per Newton: 3.272401e+00
    -> Last step size: 1.171224e-01, order: 3
    -> Local error fails: 0, nonlinear convergence fails: 2
    -> Stability limit order reductions: 0
1.000e+01         69       5.32e+00   -10.9    0.0   31.0    2.4   77.5
Run time : 1 m 14 s

petsc preconditioning (ksp_type=preonly, pc_type=hypre):

CVODE: nsteps 282, nfevals 326, nniters 325, npevals 787, nliters 479
    -> Newton iterations per step: 1.152482e+00
    -> Linear iterations per Newton iteration: 1.473846e+00
    -> Preconditioner evaluations per Newton: 2.421538e+00
    -> Last step size: 1.976899e-01, order: 4
    -> Local error fails: 0, nonlinear convergence fails: 6
    -> Stability limit order reductions: 0
1.000e+01         16       1.36e+00     2.0    0.0   13.8    7.7   76.6
Run time : 2 m 55 s

With PETSc preconditioning the number of linear iterations per Newton step is small, and the timesteps seem to be bigger than other methods. When using STRUMPACK rather than hypre there is an error on the first iteration but it continues and accelerates after:

[9]PETSC ERROR: STRUMPACK error: factorization failed
[9]PETSC ERROR: #1 MatLUFactorNumeric_STRUMPACK() at /Users/dudson2/code/petsc/src/mat/impls/aij/mpi/strumpack/strumpack.c:986
[9]PETSC ERROR: #2 MatLUFactorNumeric() at /Users/dudson2/code/petsc/src/mat/interface/matrix.c:3312
[9]PETSC ERROR: #3 PCSetUp_LU() at /Users/dudson2/code/petsc/src/ksp/pc/impls/factor/lu/lu.c:121
[9]PETSC ERROR: #4 PCSetUp() at /Users/dudson2/code/petsc/src/ksp/pc/interface/precon.c:1086
[9]PETSC ERROR: #5 KSPSetUp() at /Users/dudson2/code/petsc/src/ksp/ksp/interface/itfunc.c:427

It runs fine with STRUMPACK after the first step, but this factorization failure makes me a bit suspicious.

Use the same coloring, preconditioning code in CVODE, PETSc and SNES time-integration solvers. Major refactoring by Codex. Much more testing needed.

Setting `cvode_precon_method` to `none` disables preconditioning. The default `auto` uses the user-supplied preconditioner if present, then PETSc if available, then SUNDIALS' BBD preconditioner.

cvode_lsetup_frequency and cvode_jac_eval_frequency modify how often the Jacobian and preconditioner are recalculated.

github-actions

clang-tidy made some suggestions

github-actions · 2026-05-29T22:52:37Z

+  PetscCall(VecGetArray(f, &fdata));
+
+  try {
+    s->rhs(s->petsc_t, const_cast<BoutReal*>(xdata), s->petsc_rhs_tmp.data(), true);


warning: do not use const_cast to remove const qualifier [cppcoreguidelines-pro-type-const-cast]

{ ^

`CVodeSetLSetupFrequency` and `CVodeSetJacEvalFrequency` are not available in Sundials version 4.1.0 on CI.

github-actions

clang-tidy made some suggestions

test-delp2 answer changes slightly (~1e-6) if preconditioning is enabled.

Hermes-3 braginskii_conduction creates a "type" subsection in every section of options. PetscLib throws an exception when the "petsc" section contains a subsection.

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

dschwoerer · 2026-06-03T07:19:43Z

+  if ((*options)["use_precon"].isSet()) {
+    output_warn << "WARNING: solver:use_precon is deprecated for CVODE and is now "
+                   "ignored. Use solver:cvode_precon_method=none to disable "
+                   "preconditioning.\n";
+  }


Should this be an error?
We throw also errors for unused options, so I think we should also force the users here to update.

dschwoerer · 2026-06-03T07:20:30Z

+  // around a bug in Hermes-3 braginskii_conduction that creates a
+  // "petsc:type" subsection.


How hard would it be to fix it in hermes?

It's fairly straightforward to fix (I've fixed it in a branch) so hopefully this is a short-lived patch.

dschwoerer · 2026-06-03T07:25:13Z

+      const int band_width_default = std::accumulate(
+          begin(f3d), end(f3d), 0, [](int a, const VarStr<Field3D>& fvar) {
+            const Mesh* localmesh = fvar.var->getMesh();
+            return a + localmesh->xend - localmesh->xstart + 3;


That means the ddx must only use one guard cell, and higher order stencils will perform poorly, do to wrong preconditioning?

The whole BBD preconditioner is of quite dubious benefit really. It doesn't know anything about spatial stencils so only preconditions coupling between variables that appear next to each other in the 1D state vector. In practice I think that means it's only good for preconditioning coupling between variables in the same cell. I've not yet seen a case where BBD preconditioning improved performance, so it's really here in case someone wants to try it as a last resort.

dschwoerer · 2026-06-03T07:28:00Z

+      output_progress.write("Creating Jacobian coloring\n");
+      updateColoring();
+
+#if 0


That seems to be the code that has been moved to petsc_preconditioner.cxx, thus I deleted it.

bendudson added 4 commits May 29, 2026 13:20

solvers:Refactor coloring-based preconditioner

d2790a4

Use the same coloring, preconditioning code in CVODE, PETSc and SNES time-integration solvers. Major refactoring by Codex. Much more testing needed.

cvode: Remove use_precon, use cvode_precon_method

1988ac4

Setting `cvode_precon_method` to `none` disables preconditioning. The default `auto` uses the user-supplied preconditioner if present, then PETSc if available, then SUNDIALS' BBD preconditioner.

cvode: Options to set preconditioner setup

623efc8

cvode_lsetup_frequency and cvode_jac_eval_frequency modify how often the Jacobian and preconditioner are recalculated.

cvode: Fix compile guard location

8aeab13

github-actions Bot reviewed May 29, 2026

View reviewed changes

cvode: Version guards around CVode calls

2ca5721

`CVodeSetLSetupFrequency` and `CVodeSetJacEvalFrequency` are not available in Sundials version 4.1.0 on CI.

github-actions Bot reviewed May 30, 2026

View reviewed changes

Comment thread src/solver/impls/cvode/cvode.cxx Outdated

bendudson added 2 commits May 30, 2026 09:49

cvode: Set default preconditioning to none

5667c05

test-delp2 answer changes slightly (~1e-6) if preconditioning is enabled.

cvode/snes: Initialise PetscLib early, tidying

8d734c2

Hermes-3 braginskii_conduction creates a "type" subsection in every section of options. PetscLib throws an exception when the "petsc" section contains a subsection.

github-actions Bot reviewed May 31, 2026

View reviewed changes

Comment thread src/solver/impls/cvode/cvode.cxx

Comment thread src/solver/impls/cvode/cvode.cxx Outdated

Comment thread src/solver/petsc_preconditioner.cxx

cvode/snes: Minor tidying, update manual

928f37e

github-actions Bot reviewed May 31, 2026

View reviewed changes

Comment thread src/solver/impls/cvode/cvode.cxx Outdated

dschwoerer added 7 commits June 3, 2026 08:34

Add missing headers

4d07c00

Prefer anonymous namespace

fe92b56

Add missing header

6491e33

Avoid nested conditional operator

59934f6

Update used headers

4ea2d47

Apply clang-tidy cleanups

a950cfa

Remove #if 0'ed code

0bcaf31

dschwoerer reviewed Jun 3, 2026

View reviewed changes

		// around a bug in Hermes-3 braginskii_conduction that creates a
		// "petsc:type" subsection.

Conversation

bendudson commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dschwoerer Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

dschwoerer Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

bendudson Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

dschwoerer Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

bendudson Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

dschwoerer Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bendudson commented May 29, 2026 •

edited

Loading