kms: support CephFS fscrypt encryption with a KMIP KMS by Greenpepper15 · Pull Request #6521 · ceph/ceph-csi

Greenpepper15 · 2026-09-01T07:16:12Z

Describe what this PR does

This PR makes a KMIP KMS usable for CephFS fscrypt encryption in the most
minimal way possible by implementing GetSecret on the existing kmip
provider.

ConfigureEncryption probes the KMS with GetSecret and rejects the
configuration when the KMS answers ErrGetSecretUnsupported, which the
kmip provider always did. The new implementation returns the key material
of the managed symmetric key, base64 encoded, and fscrypt uses it as a
custom_passphrase.

Fetching the key material requires the KMIP Get operation, so the change
is gated on USE_CRYPTO_RPC being "false".

The base64 encoding keeps the passphrase printable so that a volume remains
recoverable with the upstream fscrypt tool by running base64 on the key
exported from the KMS. I chose base64.StdEncoding but another ceph-csi
secret encoding uses base64.URLEncoding (see question section for the
motivation).

RBD with encryptionType: file would pass the same capability probe and use
the same fscrypt.Unlock, but that combination is untested, so this PR
explicitly rejects it in configureFileEncryption with an error. RBD
behavior is therefore unchanged.

Is there anything that requires special attention

Do you have any questions?

E2e placement: should the two e2e commits move to a separate PR, or is
keeping them here okay? Please check the KMIP e2e I added.
Passphrase encoding: should the passphrase use StdEncoding or the
URL-safe alphabet that generateNewEncryptionPassphrase uses? fscrypt
derives the protector key from the exact passphrase bytes, so the encoding
can never change afterwards. Either would work within the driver. What
breaks with URLEncoding is only the manual recovery path with the
base64(1) tool: base64(1) emits the standard alphabet, so a
by-the-book recovery would derive a different passphrase for any key whose
bytes encode to + or /, and the unlock will fail for some keys.
StdEncoding is therefore a recovery-contract choice, not a within-driver
correctness constraint.
Provider naming: should GetSecret own the bare kmip provider name,
or use a different name to keep the bare name free for other KMIP
integrations? This PR makes the bare kmip provider hand out key material
through GetSecret.
KMIP test server: should the KMIP e2e use PyKMIP? Almost nothing else
fits: the test server must speak exactly what the kmip provider speaks,
run in a plain Pod without a license, and let a script create a key.
PyKMIP is what Ceph itself tests the RGW KMIP encryption against
(qa/tasks/pykmip.py in the teuthology suite), and what OpenStack
Barbican's functional gate uses for its KMIP plugin. The one maintained
open-source alternative is Cosmian KMS (BUSL 1.1, KMIP 1.0-2.1, binary
socket server with TLS client authentication). If you prefer a maintained
implementation, I can switch the e2e to it.

PyKMIP 0.10.0 is the last release, so the pod pins the last dependency
versions it works with (Python 3.11, SQLAlchemy 1.x, cryptography
41.0.7). Newer cryptography removed the legacy ciphers PyKMIP imports
and the server crashes on startup.

Testing caveat: I could not run the new e2e spec on my setup, because it
needs a Ceph cluster together with a CephFS fscrypt capable kernel. The
spec itself has therefore not been executed and I am piggybacking the
ceph-csi CI to exercise it. Note that no CI job currently sets
--test-cephfs-fscrypt, so a run needs that flag. What I did verify
against a live PyKMIP server on a scratch cluster is the deployment and
provisioning flow.

Is the change backward compatible?

Existing kmip users are unaffected: RequiresDEKStore, EncryptDEK and
DecryptDEK are untouched, there is no configuration schema change, and
the only behavioral change is that a configuration that used to fail now
succeeds.
A CephFS volume encrypted with a KMIP KMS by this release cannot be staged
by an older release, which still answers the capability probe with
ErrGetSecretUnsupported.
Rotating or destroying the KMIP key, or pointing UNIQUE_IDENTIFIER at a
different key, makes every existing volume permanently unopenable. There
is no re-wrap path.

Related issues

Fixes: #6324

Future/now concerns

There are different ways to add KMIP support to CephFS fscrypt. I chose the
most minimal one, but the other directions remain possible and might make
more sense for different use cases.

Planned follow-ups:

validate RBD encryptionType: file with KMIP (an RBD e2e spec) and remove
the lock in configureFileEncryption.

Checklist:

Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
Reviewed the developer guide on Submitting a Pull
Request
Pending release
notes
updated with breaking and/or notable changes for the next major release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

Greenpepper15 · 2026-09-03T13:22:51Z

+// GetSecret gets the raw secret via KMIP.
+func (kms *kmipKMS) GetSecret(_ context.Context, _ string) (string, error) {
+	if kms.useCryptoRPC {
+		return "", fmt.Errorf("%w: fscrypt requires the key material, set %q to false",
+			ErrGetSecretUnsupported, kmipUseCryptoRPC)
+	}
+
+	key, err := kms.getKey(kms.uniqueIdentifier)
+	if err != nil {
+		return "", fmt.Errorf("failed to get key %q: %w", kms.uniqueIdentifier, err)
+	}
+
+	return base64.StdEncoding.EncodeToString(key), nil
+}


Dear ceph-csi maintainers,

This PR changes pretty big but the core change "just" involves this one function.

The rest are changes revolve around testing this change.

@Rakshith-R

black-dragon74 · 2026-09-08T06:19:30Z

/test ci/centos/mini-e2e/k8s-1.35

black-dragon74 · 2026-09-08T10:02:09Z

@Greenpepper15 could you please add e2e tests around fscrypt PVCs?

Greenpepper15 · 2026-09-09T08:43:01Z

 		if testCephFSFscrypt {
 			kmsToTest := map[string]kmsConfig{
 				"secrets-metadata-test": secretsMetadataKMS,
 				"vault-test":            vaultKMS,
 				"vault-tokens-test":     vaultTokensKMS,
 				"vault-tenant-sa-test":  vaultTenantSAKMS,
+				"kmip-fscrypt-test":     kmipKMS,
 			}

 			for kmsID, kmsConf := range kmsToTest {


@Greenpepper15 could you please add e2e tests around fscrypt PVCs?

@black-dragon74 I integrated this new KMIP fscrypt integration into the existing fscrypt KMS e2e testing suite.

I can add the kmip fscrypt testing to more KMS testing scenarios though like "encrypted PVC-PVC cloning" and "encrypted snapshot" testing. Is that fine?

I see you have enabled the said kms for fscrypt with the latest commit.

yep. Was that okay?

Greenpepper15 · 2026-09-09T09:12:05Z

Also it seems like the test run failed due to flake in the NVMe e2e testing suite.

• [FAILED] [187.180 seconds]
nvmeof Test NVMe CSI [It] test volumeGroupSnapshot
/go/src/github.com/ceph/ceph-csi/e2e/nvmeof.go:582
  [FAILED] failed to execute rados command 'rados listomapkeys csi.groups.default --pool=nvmeofpool | wc -l' : stdErr=2026-09-08T08:38:40.714+0000 7f3883fff640 -1 monclient: get_auth_request but no auth handler is set up
  In [It] at: /go/src/github.com/ceph/ceph-csi/e2e/utils.go:2150 @ 09/08/26 08:38:40.725
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

Summarizing 1 Failure:
  [FAIL] nvmeof Test NVMe CSI [It] test volumeGroupSnapshot
  /go/src/github.com/ceph/ceph-csi/e2e/utils.go:2150
Ran 121 of 195 Specs in 7088.765 seconds
FAIL! -- 120 Passed | 1 Failed | 0 Pending | 74 Skipped

Greenpepper15 · 2026-09-09T09:29:19Z

Also when looking at the logs I can see these messages

rbd.go:309] skipping RBD fscrypt file encryption test

I think the fscrypt tests are skipped for RBD:

�[0mRBD �[38;5;243mTest RBD CSI �[0m�[1mcreate a PVC and bind it to an app with encrypted RBD volume with VaultKMS (file)�[0m
�[38;5;243m/go/src/github.com/ceph/ceph-csi/e2e/rbd.go:307�[0m
  �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m@ 09/08/26 07:19:40.13�[0m
  I0908 07:19:40.130161   88594 util.go:414] >>> kubeConfig: /root/.kube/config
  �[1mSTEP:�[0m Building a namespace api object, basename rbd �[38;5;243m@ 09/08/26 07:19:40.13�[0m
  �[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m@ 09/08/26 07:19:40.135�[0m
  �[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m@ 09/08/26 07:19:40.235�[0m
  I0908 07:19:40.237726   88594 rbd.go:309] skipping RBD fscrypt file encryption test
  �[1mSTEP:�[0m Destroying namespace "rbd-3861" for this suite. �[38;5;243m@ 09/08/26 07:19:40.237�[0m
�[38;5;10m• [0.111 seconds]�

And I do not any other logs related to testing fscrypt in Cephfs. So I assume they are skipped too.

Can you pass the ceph-csi e2e test option "--test-cephfs-fscrypt" and "--test-rbd-fscrypt" to this command "/test ci/centos/mini-e2e/k8s-1.35" somehow. I think otherwise the fscrypt tests are not executed.

Or what do you think?

Run the encrypted PVC-PVC clone and the encrypted snapshot-backed volume specs against kmip-fscrypt-test. Both mount a volume that inherits the fscrypt metadata of its parent, a case the plain encrypted PVC spec does not cover. Requested: ceph#6521 (comment) Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

Greenpepper15 · 2026-09-10T06:32:37Z

Can you @black-dragon74 re-run the e2e-acceptance test? It failed at the build ceph-csi image stage.

The multi-arch-build / multi-arch-build also runs "build ceph-csi image" and succeeded on the same commit. So I am unsure what the problem can be except flake.

black-dragon74 · 2026-09-10T06:54:04Z

/test ci/centos/mini-e2e/k8s-1.35

mergify · 2026-09-11T06:29:51Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

Run the encrypted PVC-PVC clone and the encrypted snapshot-backed volume specs against kmip-fscrypt-test. Both mount a volume that inherits the fscrypt metadata of its parent, a case the plain encrypted PVC spec does not cover. Requested: ceph#6521 (comment) Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

black-dragon74 · 2026-09-11T07:26:01Z

/test ci/centos/mini-e2e/k8s-1.35

Greenpepper15 · 2026-09-14T12:04:23Z

Again like here #6521 (comment) when I go through the e2e tests run logs I do not find evidence that the any fscrypt e2e tests (including the new kmip e2e tests) were run.

I looked through the e2e log start and now it makes sense:

+ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@n27-27-105.pool.ci.centos.org 'cd /opt/build/go/src/github.com/ceph/ceph-csi && make run-e2e E2E_ARGS="--delete-namespace-on-failure=false "'
cephcsi image settings: quay.io/cephcsi/cephcsi version canary
cd e2e && \
../e2e.test -test.v -ginkgo.v -ginkgo.timeout="300m" --deploy-timeout="10" --cephcsi-namespace=cephcsi-e2e-857ca305 --delete-namespace-on-failure=false 
...

This command is run to start the e2e tests ../e2e.test -test.v -ginkgo.v -ginkgo.timeout="300m" --deploy-timeout="10" --cephcsi-namespace=cephcsi-e2e-857ca305 --delete-namespace-on-failure=false . However it misses the --test-cephfs-fscrypt or --test-rbd-fscrypt flags that enable the e2e fscrypt tests.

Should I open a PR on for the ci/centos branch to add these e2e test flags?

nixpanic · 2026-09-14T15:15:16Z

Ah, good catch! The tests run on CentOS Stream 9 (minikube with Podman driver). That means the kernel needs CephFS+fscrypt support. I do not know if that CentOS version is recent enough.

Ideally the --test-cephfs-fscrypt and --test-rbd-fscrypt flags default to true. You could set those in the e2e/ and see if the tests pass. Otherwise we need to move to CentOS Stream 10, and see if that kernel is recent enough.

Greenpepper15 · 2026-09-14T15:25:04Z

yup that sounds like a good idea.

Let's see which tests crash.

Greenpepper15 · 2026-09-14T16:52:34Z

/test ci/centos/mini-e2e/k8s-1.35

Greenpepper15 · 2026-09-14T20:11:48Z

@nixpanic I think this test may reflect a fscrypt test regression.

The current test failure is:

Summarizing 1 Failure:
  [FAIL] cephfs Test CephFS CSI [It] create a storageclass with pool and an encrypted PVC
        then bind it to an app with secrets-metadata-test
Ran 22 of 209 Specs in 1632.376 seconds
FAIL! -- 21 Passed | 1 Failed | 0 Pending | 187 Skipped
--- FAIL: TestE2E (1632.44s)

So the KMS test secrets-metadata-test fails not my KMIP tests (however I think they were not run yet).

As far as I see it this is the reason Reason:FailedMount,Message:MountVolume.MountDevice failed for volume "pvc-8d1b8796-3981-48ed-9b11-4a6dc35d8fce" : rpc error: code = Internal desc = not allowed to lock volume ID 7eb290a2-a1a0-404b-8f7a-07c1d7427046

This looks suspiciously like the new error type I introduced in #6539 specifically this commit 3d9af0e :(

I think the test dies at the exclusive lock function here:

ceph-csi/internal/cephfs/nodeserver.go

Line 190 in e056908

err = lock.LockExclusive(ctx)

(introduced 2 years ago. Lucky not me :D)

For this operation x caps are needed because the exclusive lock function is implement as a cls on the OSD-side.

BUT the e2e CephX user does not have x caps

ceph-csi/e2e/ceph_user.go

Line 84 in e056908

"osd", "'allow rw tag cephfs *=*'",

(this line is 5 years old)

(in normal deployment the CephFS user has x caps as per Ceph-csi recommendation https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md#cephfs. So this is why users do not run into this problem.)

I will add the x cap to this branch and see if this fixes this potential 2 year old regression. If my hypothesis is true we can figure out how to fix this.

fscrypt needs one deterministic secret from the KMS and builds its own key hierarchy on the volume, so the stub that always answered ErrGetSecretUnsupported was the only thing keeping a KMIP KMS from backing CephFS file encryption and RBD encryptionType file. Return the key material of the managed symmetric key, base64 encoded so a volume stays openable with the fscrypt tool, and keep rejecting the request when USE_CRYPTO_RPC keeps cryptographic operations on the server. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

Implementing GetSecret on the kmip provider makes it pass the fscrypt capability probe for RBD encryptionType file as well, but only the CephFS combination has been tested. Keep RBD failing at CreateVolume, as it does today, with an actionable error instead of the accidental probe rejection. The lock is removed once the combination is validated. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

Document that a KMIP KMS backs CephFS fscrypt encryption when USE_CRYPTO_RPC is disabled, and warn that rotating or replacing the managed key makes existing volumes permanently unopenable. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

No KMS speaking KMIP exists in the e2e environment, so the KMIP provider has never been covered there. Deploy PyKMIP, pinned together with the last dependency versions it works with, and provisions what the kmip KMS provider needs. PyKMIP identifies clients by the certificate CN and only lets the owner fetch a key, so the key-creating script uses the same client certificate as ceph-csi. PyKMIP 0.10.0 is the last release and predates current versions of its dependencies, so Python, SQLAlchemy and cryptography are pinned to the last versions it works with. Newer cryptography removed the legacy ciphers PyKMIP imports and the server crashes on startup. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

Add a kmip-fscrypt-test KMS configuration pointing at the PyKMIP Service, with USE_CRYPTO_RPC disabled so GetSecret may fetch the key material, and run the encrypted PVC and app binding spec against it. The KMIP server is only deployed for fscrypt test runs, keeping the PyPI dependency out of every other CI job. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

Run the encrypted PVC-PVC clone and the encrypted snapshot-backed volume specs against kmip-fscrypt-test. Both mount a volume that inherits the fscrypt metadata of its parent, a case the plain encrypted PVC spec does not cover. Requested: ceph#6521 (comment) Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: David Mohren <david.mohren@clyso.com>

Signed-off-by: David Mohren <david.mohren@clyso.com>

exclusive lock needs `x` caps which were not given to the e2e CephX user Signed-off-by: David Mohren <david.mohren@clyso.com>

Greenpepper15 · 2026-09-14T20:33:59Z

/test ci/centos/mini-e2e/k8s-1.35

Greenpepper15 · 2026-09-15T05:01:05Z

Okay after adding the x caps to the e2e user the test do longer fail to enable an exclusive lock.

Now its a different fscrypt on RBD test failure.

Reason:FailedMount,Message:MountVolume.MountDevice failed for volume "pvc-ad5978a0-38f0-432d-9ca0-
47d60a49c8f0" : rpc error: code = Internal desc = file system encryption unlock in
<long path>
failed: encryption not enabled on filesystem

Well I could look more into these fscrypt test failures in this PR or open a new PR that has the goal of making the fscrypt test suite work on CI. @nixpanic which option do you prefer?

Greenpepper15 requested review from a team as code owners September 1, 2026 07:16

Greenpepper15 force-pushed the wip-kmip-getsecret branch from d34803a to 1be881a Compare September 3, 2026 08:08

Greenpepper15 commented Sep 3, 2026

View reviewed changes

Greenpepper15 force-pushed the wip-kmip-getsecret branch from 1acd73d to b078ed6 Compare September 4, 2026 10:40

Greenpepper15 commented Sep 9, 2026

View reviewed changes

Greenpepper15 force-pushed the wip-kmip-getsecret branch from cad2b08 to 7fef5c3 Compare September 11, 2026 07:13

Greenpepper15 added 7 commits September 14, 2026 22:12

e2e: DO NOT MERGE, enable the fscrypt tests by default

2029426

Signed-off-by: David Mohren <david.mohren@clyso.com>

Greenpepper15 force-pushed the wip-kmip-getsecret branch from 6f1a9ea to 2029426 Compare September 14, 2026 20:18

e2e: DO NOT MERGE, align the CephFS user OSD caps with the docs

3cc7b09

exclusive lock needs `x` caps which were not given to the e2e CephX user Signed-off-by: David Mohren <david.mohren@clyso.com>

Conversation

Greenpepper15 commented Sep 1, 2026

Describe what this PR does

Is there anything that requires special attention

Related issues

Future/now concerns

Uh oh!

Greenpepper15 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Greenpepper15 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

black-dragon74 commented Sep 8, 2026

Uh oh!

black-dragon74 commented Sep 8, 2026

Uh oh!

Greenpepper15 Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

black-dragon74 Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Greenpepper15 Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Greenpepper15 commented Sep 9, 2026

Uh oh!

Greenpepper15 commented Sep 9, 2026

Uh oh!

Greenpepper15 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

black-dragon74 commented Sep 10, 2026

Uh oh!

mergify Bot commented Sep 11, 2026

Uh oh!

black-dragon74 commented Sep 11, 2026

Uh oh!

Greenpepper15 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixpanic commented Sep 14, 2026

Uh oh!

Greenpepper15 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Greenpepper15 commented Sep 14, 2026

Uh oh!

Greenpepper15 commented Sep 14, 2026

Uh oh!

Greenpepper15 commented Sep 14, 2026

Uh oh!

Greenpepper15 commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Greenpepper15 Sep 9, 2026 •

edited

Loading

Greenpepper15 commented Sep 10, 2026 •

edited

Loading

Greenpepper15 commented Sep 14, 2026 •

edited

Loading

Greenpepper15 commented Sep 14, 2026 •

edited

Loading