Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions architecture/powersync-service.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ When a change occurs in the source database that affects a certain bucket (based

The PowerSync Service persists the bucket state in durable storage: there is a pluggable storage layer for bucket data, and MongoDB and Postgres are currently supported as _bucket storage_ databases. The _bucket storage_ database is separate from the connection to your _source database_ (Postgres, MongoDB, MySQL, SQL Server or Convex). Our cloud-hosting offering (PowerSync Cloud) uses MongoDB Atlas as the _bucket storage_ database.

With [S3 object storage](/sync/advanced/storage-version-4#s3-object-storage) (Beta, requires storage version 4), larger blocks of prepared sync data are stored in S3 and synced to clients directly from there, while MongoDB keeps the metadata that locates them. Clients continue to connect only to the PowerSync Service.
With [S3 object storage](/sync/advanced/storage-version-4#s3-object-storage) (requires storage version 4), larger blocks of prepared sync data are stored in S3 and synced to clients directly from there, while MongoDB keeps the metadata that locates them. Clients continue to connect only to the PowerSync Service.

Persisting the bucket state in a database is also part of how PowerSync achieves high scalability: it means that the PowerSync Service can have a low memory footprint even as you scale to very large volumes of synced data and users/clients.

Expand All @@ -84,7 +84,7 @@ When the PowerSync Service replicates data from the source database, it:

### Initial Replication vs. Incremental Replication

Whenever a new Sync Config is deployed, initial replication takes a snapshot of all tables or collections it references. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), a Sync Streams deployment only snapshots the streams that were added or changed. Clients continue using the current copy of the data until the new one is ready.
Whenever a new Sync Config is deployed, initial replication takes a snapshot of all tables or collections it references. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing), a Sync Streams deployment only snapshots the streams that were added or changed. Clients continue using the current copy of the data until the new one is ready.

After that, data is incrementally replicated using a change data capture stream. The specific mechanism depends on the source database type: Postgres logical replication, MongoDB change streams, the MySQL binlog, SQL Server Change Data Capture, or Convex document deltas.

Expand Down
6 changes: 3 additions & 3 deletions configuration/powersync-service/self-hosted-instances.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
# Note only 1 connection is currently supported
# Multiple connection support is on the roadmap
connections:
- type: postgresql

Check warning on line 27 in configuration/powersync-service/self-hosted-instances.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/powersync-service/self-hosted-instances.mdx#L27

Use 'Postgres' instead of 'postgresql'.
# The PowerSync server container can access the Postgres DB via the DB's service name.
# In this case the hostname is pg-db

Expand Down Expand Up @@ -475,8 +475,8 @@
Maximum number of connections to the storage database, per Service process.
</ResponseField>

<ResponseField name="default_storage_version" type="number" default="2">
Storage version to use when deploying a Sync Config that does not specify a storage version. To opt in to [storage version 4](/sync/advanced/storage-version-4) (Beta), set `config.storage_version: 4` in your Sync Config. An explicit `config.storage_version` takes precedence over this default. Available since Service version 1.26.0.
<ResponseField name="default_storage_version" type="number">
Storage version to use when deploying a Sync Config that does not specify a storage version. When unset, MongoDB bucket storage uses [storage version 4](/sync/advanced/storage-version-4) and Postgres bucket storage uses version 2. Service versions 1.26.x default to version 2. An explicit `config.storage_version` takes precedence over this default. Available since Service version 1.26.0.

<Note>
Postgres bucket storage does not currently support version 4.
Expand Down Expand Up @@ -522,7 +522,7 @@
</ResponseField>

<ResponseField name="object_storage" type="object">
Stores large blocks of bucket data in Amazon S3 or an S3-compatible object store instead of MongoDB. Requires Service version 1.26.0 or later with [storage version 4](/sync/advanced/storage-version-4) (Beta), which is not currently the default. Sync Configs on version 2 keep all data in MongoDB even when this is configured.
Stores large blocks of bucket data in Amazon S3 or an S3-compatible object store instead of MongoDB. Requires Service version 1.26.0 or later with [storage version 4](/sync/advanced/storage-version-4), the default since Service version 1.27.0. Sync Configs on version 2 keep all data in MongoDB even when this is configured.

Follow [Self-Hosted S3 Setup](/sync/advanced/storage-version-4#self-hosted-s3-setup) to create the bucket, configure permissions and lifecycle settings, and enable object storage in the Service. The fields below describe the available configuration options.

Expand Down
2 changes: 1 addition & 1 deletion maintenance-ops/compacting-buckets.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ Key considerations:

## Sync Streams Deployments

Without incremental reprocessing, deploying [Sync Streams](/sync/streams/overview) recreates all buckets from scratch. This has the same effect as defragmenting and compacting every bucket, and clients download all their data again. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), unchanged streams keep their buckets and operation history, so a deployment no longer clears their history.
Without incremental reprocessing, deploying [Sync Streams](/sync/streams/overview) recreates all buckets from scratch. This has the same effect as defragmenting and compacting every bucket, and clients download all their data again. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing), unchanged streams keep their buckets and operation history, so a deployment no longer clears their history.

Do not redeploy a Sync Config for routine history cleanup. Use compacting, targeted source-row updates, or the Dashboard's **Defragment** action instead.

Expand Down
2 changes: 1 addition & 1 deletion resources/feature-status.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Below is a summary of the current main PowerSync features and their release stat
| Postgres | GA |
| | |
| **PowerSync Service** | |
| Storage Version 4 (incremental reprocessing, S3 object storage) | Beta |
| Storage Version 4 (incremental reprocessing, S3 object storage) | GA |
| Open Edition | GA |
| Enterprise Self-Hosted | GA |
| Sync Streams | GA |
Expand Down
2 changes: 1 addition & 1 deletion resources/usage-and-billing/reducing-usage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ PowerSync processes source changes at the row level. Rewriting a row selected by

### Deployments

Deploying Sync Streams recreates the buckets, and existing clients download all their data again even if the underlying rows did not change. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), clients download only the data for streams you added or changed. In both cases, group related changes when practical instead of deploying many small updates.
Deploying Sync Streams recreates the buckets, and existing clients download all their data again even if the underlying rows did not change. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing), clients download only the data for streams you added or changed. In both cases, group related changes when practical instead of deploying many small updates.

## Reducing Data Hosted

Expand Down
2 changes: 1 addition & 1 deletion resources/usage-and-billing/usage-and-billing-faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ Each connected `PowerSyncDatabase` counts as a concurrent client. If one user se
Some events increase usage temporarily and don't indicate a problem:

- New app installations download all their subscribed data, including the operations history, during the initial sync. Expect higher data synced when onboarding new groups of users. A large operations history makes these initial syncs bigger; see [More Operations Than Rows](#more-operations-than-rows).
- Deploying changes to [Sync Streams](/sync/streams/overview) recreates buckets, and existing clients download the new buckets. Data hosted can also increase temporarily while the previous and new bucket data exist concurrently. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), only the streams you added or changed are affected. See [Deployments](/resources/usage-and-billing/reducing-usage#deployments).
- Deploying changes to [Sync Streams](/sync/streams/overview) recreates buckets, and existing clients download the new buckets. Data hosted can also increase temporarily while the previous and new bucket data exist concurrently. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing), only the streams you added or changed are affected. See [Deployments](/resources/usage-and-billing/reducing-usage#deployments).
- Defragmentation reduces what new installations download, but existing clients re-sync each touched row.

## Accident Forgiveness
Expand Down
7 changes: 3 additions & 4 deletions sync/advanced/compatibility.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,10 @@ streams:
query: SELECT * FROM todos WHERE owner_id = auth.user_id()
```

When you omit `storage_version`, the PowerSync Service uses its default, which is version 2 in v1.26.0. On PowerSync Cloud, PowerSync manages the default. For self-hosted deployments, set `config.storage_version` explicitly to select a different version.
When you omit `storage_version`, the PowerSync Service uses its default. Since v1.27.0, the default is version 4 with MongoDB bucket storage, which all PowerSync Cloud instances use, and version 2 with Postgres bucket storage. Service versions 1.26.x default to version 2.

Set `storage_version` when you need to:

- Use [storage version 4](/sync/advanced/storage-version-4), which is in Beta and enables incremental reprocessing and S3 object storage.
- Delay a storage upgrade. When the default moves to a newer version, pin `storage_version` to the version your data already uses. This keeps later Sync Config deployments on that format. Remove the pin when you are ready for the new format.
- Prepare for a Service downgrade. Select a version supported by the older Service, deploy the Sync Config, and wait for the new copy to finish before downgrading.

Expand All @@ -90,9 +89,9 @@ All PowerSync Cloud instances use MongoDB bucket storage, so they are compatible
| Version | Bucket storage | Status |
| --- | --- | --- |
| `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. |
| `2` | MongoDB or Postgres | Stable. The default in v1.26.0. |
| `2` | MongoDB or Postgres | Stable. The default with Postgres bucket storage. Also the default with MongoDB bucket storage in Service versions 1.26.x. |
| `3` | MongoDB | Experimental. The unstable predecessor of version 4, with the same format. Do not use it in production. Deploy with version 4 instead. |
| `4` | MongoDB | Stable. Enables [incremental reprocessing and S3 object storage](/sync/advanced/storage-version-4) (Beta). |
| `4` | MongoDB | Stable. The default with MongoDB bucket storage since Service version 1.27.0. Enables [incremental reprocessing and S3 object storage](/sync/advanced/storage-version-4). |

Version numbers follow a pattern. Even numbers are stable formats: they stay backwards compatible and later Service versions continue to support them. Stable makes no guarantee that a format is bug-free. Odd numbers are experimental formats: their layout can change without notice and support can be removed in a later release, so use them only for testing, never in production.

Expand Down
50 changes: 18 additions & 32 deletions sync/advanced/storage-version-4.mdx
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: "Storage Version 4"
description: "Opt in to storage version 4 for faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage."
description: "Storage version 4 provides faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage."
---

{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/storage-version-4.mdx. It is a short summary that links here for details, so keep the anchors opt-in, incremental-reprocessing, s3-object-storage, and self-hosted-s3-setup stable. Mention Sync Rules here only to state that incremental reprocessing requires Sync Streams. */}
{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/storage-version-4.mdx. It is a short summary that links here for details, so keep the anchors opt-in, switching-storage-versions, incremental-reprocessing, s3-object-storage, and self-hosted-s3-setup stable. Mention Sync Rules here only to state that incremental reprocessing requires Sync Streams. */}

Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0.
Storage version 4 is the current format the PowerSync Service uses to store the data it syncs to clients. It is the default for MongoDB bucket storage since PowerSync Service v1.27.0.

Compared to version 2, it provides:

Expand All @@ -15,56 +15,42 @@ Compared to version 2, it provides:

## Availability

Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported.
Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported, so instances that use it stay on version 2.

The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup).

| | Source database | PowerSync Cloud (Beta) | Self-hosted (Beta) |
| | Source database | PowerSync Cloud | Self-hosted |
| --- | --- | --- | --- |
| Storage version 4 | Any | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) |
| Storage version 4 | Any | [Default](#switching-storage-versions) | [Default](#switching-storage-versions) with MongoDB bucket storage |
| Incremental reprocessing | MongoDB | Included with version 4 | Included with version 4 |
| S3 object storage | Any | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) |

Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams).

## Opt In
<span id="opt-in" />

Version 4 is not the default in PowerSync Service v1.26.0. Moving a Sync Config to version 4 runs like any other deployment:
## Switching Storage Versions

1. PowerSync reprocesses all data selected by your Sync Config in the background. The current version keeps serving clients, so there is no downtime.
2. When the new copy is ready, PowerSync switches to it. On PowerSync Cloud, this appears as a new deploy event in the PowerSync Dashboard.
3. Clients download their data again once, as after any deployment before version 4. On self-hosted deployments with many clients, scale out the API before the switch to absorb the re-sync.
Since PowerSync Service v1.27.0, MongoDB bucket storage defaults to version 4. If `config.storage_version` is unset, the Service uses its default the next time you deploy the Sync Config. Upgrading the Service alone does not change the storage version of an existing Sync Config.

After this first deployment, later Sync Streams deployments use incremental reprocessing automatically when your instance meets its requirements. There is no separate setting.
A switch runs like any other deployment:

### PowerSync Cloud
1. PowerSync reprocesses all data selected by your Sync Config in the background. The current version keeps serving clients, so there is no downtime.
2. When the new copy is ready, PowerSync switches to it. On PowerSync Cloud, this appears as a new deploy event in the PowerSync Dashboard.
3. Clients download their data again once. On self-hosted deployments with many clients, scale out the API before the switch to absorb the re-sync.

Free plan instances are upgraded automatically during the Beta. No action is needed.
After the switch to version 4, later Sync Streams deployments use incremental reprocessing automatically if your instance meets its requirements.

On other plans, add `storage_version: 4` to the `config` block of your Sync Config and deploy it:
To use a different version, set `storage_version` in the `config` block of your Sync Config and deploy it. For example, to stay on version 2:

```yaml
config:
edition: 3
storage_version: 4
storage_version: 2

streams:
todos:
query: SELECT * FROM todos WHERE owner_id = auth.user_id()
```

### Self-Hosted

<Note>
Postgres bucket storage is not supported with version 4.
</Note>

Add `storage_version: 4` to the `config` block of each Sync Config as shown above, then deploy or redeploy it to use version 4.

To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess.

To also enable S3 object storage, follow the [self-hosted S3 setup instructions](#self-hosted-s3-setup) to prepare a bucket and configure the Service.

## Incremental Reprocessing

Incremental reprocessing is active when you use a MongoDB source database, Sync Streams, and storage version 4.
Expand Down Expand Up @@ -129,7 +115,7 @@ S3 object storage requires storage version 4. It is compatible with all PowerSyn

### PowerSync Cloud

During the Beta, PowerSync enables S3 object storage per instance. [Contact us](/resources/contact-us) if you want it on your instance before we enable it for all instances.
PowerSync enables S3 object storage per instance on request. We plan to enable it on all PowerSync Cloud instances automatically in the future. [Contact us](/resources/contact-us) if you want it on your instance sooner.

### Self-Hosted S3 Setup

Expand Down Expand Up @@ -162,7 +148,7 @@ During the Beta, PowerSync enables S3 object storage per instance. [Contact us](
Restart or redeploy the PowerSync Service to load the updated `service.yaml`. If you run replication, API, and compacting in separate containers or jobs, apply the same object storage configuration to each.
</Step>
<Step title="Deploy on Storage Version 4">
Deploy your Sync Configs on storage version 4 as described in [Opt In](#opt-in). Sync Configs on version 2 keep all data in MongoDB, even when `object_storage` is configured.
Make sure your Sync Configs use storage version 4, as described in [Switching Storage Versions](#switching-storage-versions). Sync Configs on version 2 keep all data in MongoDB, even when `object_storage` is configured.

Once replication reaches a healthy checkpoint, confirm that objects appear under the prefix, run a test initial sync, and run `compact` once to surface permission errors early.
</Step>
Expand Down
Loading
Loading