Expose tiered storage over FFI and add integration tests#871
Open
enigbe wants to merge 5 commits intolightningdevkit:mainfrom
Open
Expose tiered storage over FFI and add integration tests#871enigbe wants to merge 5 commits intolightningdevkit:mainfrom
enigbe wants to merge 5 commits intolightningdevkit:mainfrom
Conversation
This commit: Adds `TierStore`, a tiered `KVStore`/`KVStoreSync` implementation that routes node persistence across three storage roles: - a primary store for durable, authoritative data - an optional backup store for a second durable copy of primary-backed data - an optional ephemeral store for rebuildable cached data such as the network graph and scorer TierStore routes ephemeral cache data to the ephemeral store when configured, while durable data remains primary+backup. Reads and lists do not consult the backup store during normal operation. For primary+backup writes and removals, this implementation treats the backup store as part of the persistence success path rather than as a best-effort background mirror. Earlier designs used asynchronous backup queueing to avoid blocking the primary path, but that weakens the durability contract by allowing primary success to be reported before backup persistence has completed. TierStore now issues primary and backup operations together and only returns success once both complete. This gives callers a clearer persistence guarantee when a backup store is configured: acknowledged primary+backup mutations have been attempted against both durable stores. The tradeoff is that dual-store operations are not atomic across stores, so an error may still be returned after one store has already been updated. TierStore also implements `KVStoreSync` in terms of dedicated synchronous helpers that call the wrapped stores' sync interfaces directly. This preserves the inner stores' synchronous semantics instead of routing sync operations through a previously held async runtime. Additionally, adds unit coverage for the current contract, including: - basic read/write/remove/list persistence - routing of ephemeral data away from the primary store - backup participation in the foreground success path for writes and removals
Add native builder support for tiered storage by introducing `TierStoreConfig`, backup and ephemeral store configuration methods, and a `build_with_dynstore` path that wraps the provided store as the primary tier and applies any configured secondary tiers. This makes tiered storage a builder concern while keeping the Rust `build_with_store` API ergonomic for native callers. Note: The temporary `dead_code` allowance will be removed in a follow-up commit once the new tier-store builder APIs are exercised.
Add UniFFI-facing store abstractions and builder APIs so foreign-language callers can configure backup and ephemeral stores when constructing nodes with custom storage. This introduces `FfiDynStoreTrait` as an FFI-safe equivalent of `DynStoreTrait`, along with a Rust-side adapter that bridges foreign store implementations into the internal dynamic store abstraction used by the builder. As part of this change: - add UniFFI bindings for custom primary, backup, and ephemeral stores - expose `Builder::set_backup_store`, `Builder::set_ephemeral_store`, and `Builder::build_with_store` on the FFI surface - route FFI-backed builder construction through the native dyn-store path - move FFI IO-related types into a dedicated module - preserve per-key write ordering across the FFI boundary - route Rust-side sync access through the async mutation path to avoid runtime-sensitive locking behavior
Extend the Rust test harness to support tiered store configurations and add integration coverage for routing data across primary, backup, and ephemeral stores. This introduces tier-store-aware test helpers, including support for configuring separate stores per node and inspecting persisted data across native and UniFFI-enabled test builds. Add an integration test covering the tiered-storage channel lifecycle and verifying that: - durable node data is persisted to both the primary and backup stores - ephemeral-routed data is stored in the ephemeral tier - ephemeral data is not mirrored back into the durable tiers Also update existing Rust test call sites and benchmark helpers to match the new tier-store-aware setup APIs.
Add a Python in-memory KV store implementation for the UniFFI-backed builder APIs and extend the Python test suite with tiered-storage coverage. This adds a tier-store test that builds nodes with custom primary, backup, and ephemeral stores through the FFI surface and exercises a full channel lifecycle against them. The test verifies that: - primary-backed data is persisted to both the primary and backup stores - ephemeral-routed data is stored in the ephemeral store - backup contents converge to the primary store contents
|
I've assigned @tnull as a reviewer! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR is the second half of the tiered-storage work and is intended to be reviewed on top of #692. This split is intended to keep review manageable:
While Support tiered data storage #692 is limited to the Rust-side
TierStoreimplementation and nativeNodeBuilderintegration, this PR adds the FFI surface and end-to-end test coverage on top of that foundation.The changes introduced include:
Expose tiered storage over FFI
This introduces
FfiDynStoreTraitas an FFI-safe analogue toDynStoreTraitand wires theArcedNodeBuilderAPIs so foreign-language callers can configure:It also routes FFI-backed builder construction through the same internal
dyn-storepath used by the native builder.To support FFI-backed custom stores safely, this PR also:
Add Rust integration coverage
This extends the Rust test harness with tier-store-aware setup helpers and adds integration coverage verifying that:
Add Python FFI tests
This adds a Python in-memory key-value store implementation and a Python FFI test that exercises a full channel lifecycle using custom primary, backup, and ephemeral stores configured through the exposed builder APIs.
The Python test offer the same verification as the native Rust test.