[rkistner]

For MongoDB Storage v4 compacting, we have two types of compact process, picked per bucket:

1. Chunk-merge compact: This only merges recently-added small chunks into larger ones. This is incremental and fast.
2. Full bucket compact: This scans a bucket for multiple operations on the same source rows, and replaces older ones with tombstones. This scans through the entire bucket.

Both of them have an optimization to not re-write any chunk if it did not change. It could change either when merging multiple chunks into one larger one, and/or replacing operations with tombstones.

This now adds an exception to that rule: If the chunk exceeds the object storage threshold (16KiB by default), but is still stored inline, it is now moved to object storage. Since the logic is already there to re-write chunks, this is a simple change to exclude this case from skipping the chunk.

This specifically helps for the case of an instance that is already using storage version 4, but did not have object storage enabled yet. This will effectively move the data over to object storage as buckets are compacted, without requiring full reprocessing.

I expect this to only have a minor effect on chunk-merge compact: Since it primarily scans new chunks, those should already be on object storage if object storage is enabled. This nonetheless enables the check for chunk-merge compact. But for a full compact, the entire bucket would be scanned and moved.

This would still have no effect if a bucket doesn't receive any changes. But even a single new operation on a bucket would eventually trigger a full compact.

This can also move existing chunks when the inline threshold is decreased, if they change to now being above the threshold.

AI Usage

Implemented using gpt-6.1-sol; manually reviewed.

[changeset-bot]

🦋 Changeset detected

Latest commit: 4049616

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-mongodb-storage	Patch
@powersync/service-schema	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-core	Patch
@powersync/service-module-core	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR