Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/vale/config/vocabularies/PowerSync/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,8 @@ ECDSA
E2EE
ECS
EdDSA
GTID
GTIDs
HA
HS256
IDE
Expand Down Expand Up @@ -240,6 +242,7 @@ varchar

# Products/Companies
Auth0
Aurora
Avalonia
Bavard
Blazor
Expand Down
77 changes: 45 additions & 32 deletions configuration/source-db/connection.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,30 @@ Select your Postgres hosting provider for steps to connect your newly-created Po
If you get an error such as "IPs in this range are not supported", the instance is likely not configured to be publicly accessible. A DNS lookup on the host should give a public IP, and not for example `10.x.x.x` or `172.31.x.x`.
</Accordion>

<Accordion title="AWS Aurora">
1. In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to the **Database Connection** view.
2. Select the **Postgres** tab.
3. Use the cluster (writer) endpoint from the cluster's details page in the RDS console as the "**Host**". Aurora supports logical replication only from the writer. The cluster endpoint follows the writer through a failover. Do not use the reader endpoint or an instance endpoint.
4. Complete the remaining fields:
- "**Name**" can be any name for the connection.
- "**Port**" is 5432 for Postgres databases.
- "**Database name**" is the database in your cluster to replicate.
- "**Username**" and "**Password**" maps to the `powersync_role` created in [Source Database Setup](/configuration/source-db/setup#postgres).
- PowerSync has the AWS RDS CA certificates pre-configured, so the default `verify-full` SSL mode works without additional configuration.
5. Click **Test Connection** and fix any errors.
6. Click **Save Connection**.

PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete.

### Troubleshooting

If you get an error such as "IPs in this range are not supported", the host resolves to a private IP. Make the cluster publicly accessible, or connect through a [Private Endpoint](/configuration/source-db/private-endpoints).

If Sync Config validation fails with `postgres query failed`, check the instance logs for the underlying error.

If the logs show `Hostname/IP does not match certificate's altnames`, the server certificate does not cover the hostname in "**Host**". Check that "**Host**" is your cluster endpoint. If you use a Private Endpoint, check that the target group behind it forwards to the current writer in the same cluster. Correct the hostname or update the target group as needed.
</Accordion>

<Accordion title="Azure Postgres">
1. In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to the **Database Connection** view.
2. Select the **Postgres** tab.
Expand Down Expand Up @@ -217,37 +241,26 @@ To connect to **Azure DocumentDB** (formerly Azure Cosmos DB for MongoDB vCore),
Select your MySQL hosting provider for steps to connect your newly-created PowerSync instance to your MySQL database:

<Accordion title="AWS Aurora">
To enable binary logging and GTID replication in AWS Aurora, you need to create a [DB Parameter Group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Reference.ParameterGroups.html) and configure it with the necessary parameters. Follow these steps:

1. Navigate to [Amazon RDS console](https://console.aws.amazon.com/rds/) In the navigation pane, select **Parameter groups** and click **Create Parameter Group**:
<Frame>
<img
src="/images/installation/aurora-create-parameter-group.png"
/>
</Frame>
2. Add all the required [binlog configuration](/configuration/source-db/setup#binlog-configuration) parameters. For example:
<Frame>
<img
src="/images/installation/aurora-set-gtid-flags.png"
/>
</Frame>
3. Associate your newly created parameter group with your Aurora cluster:
1. In the navigation pane, select **Databases**.
2. Select your Aurora cluster.
3. Select **Modify**.
4. In the **DB Parameter Group** section, select the parameter group you created.
5. Click **Continue** and then **Apply** immediately.
4. Whitelist PowerSync's IPs in your Aurora cluster's security group to allow access. See [Security and IP Filtering](/configuration/source-db/security-and-ip-filtering) for more details.
5. In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to the **Database Connection** view.
6. Select the **MySQL** tab.
7. Fill in your MySQL connection details from AWS Aurora:
1. "**Name**", "**Host**", "**Port**", "**Database name**", "**Username**", "**Password**" are required.
2. "**Name**" can be any name for the connection.
3. "**Host**" is the endpoint for your Aurora cluster.
4. "**Database name**" is the default database to replicate.
5. "**Username**" and "**Password**" maps to your database user.
8. Click **Test Connection** and fix any errors.
9. Click **Save Connection**.
Before connecting, complete [Aurora MySQL Setup](/configuration/source-db/setup#aws-aurora-mysql).

<Steps>
Comment thread
joshuabrink marked this conversation as resolved.
Resolved
<Step title="Open the Connection Settings">
In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to **Database Connection**. Select the **MySQL** tab.
</Step>
<Step title="Enter the Connection Details">
Use the cluster (writer) endpoint as **Host**. Do not use the reader endpoint: Aurora MySQL binary logs are available only on the writer. The cluster endpoint follows the writer through a failover, while an instance endpoint does not. If the new writer has a different server UUID, PowerSync takes a new snapshot before resuming replication.
Comment thread
joshuabrink marked this conversation as resolved.
Resolved

Complete the remaining required fields:

- **Name** can be any name for the connection.
- **Port** is your cluster's database port (`3306` by default).
- **Database name** is the default database to replicate.
- **Username** and **Password** are the credentials for the replication user created in [Database User Configuration](/configuration/source-db/setup#database-user-configuration).
</Step>
<Step title="Test and Save the Connection">
Click **Test Connection** and fix any errors. Click **Save Connection**.
</Step>
</Steps>

PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete.
</Accordion>
Expand All @@ -266,7 +279,7 @@ For other providers and self-hosted databases:
4. Click **Test Connection** and fix any errors.
5. Click **Save Connection**.

PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete.
PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete.
</Accordion>

<Info>
Expand Down
10 changes: 7 additions & 3 deletions configuration/source-db/private-endpoints.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,11 @@ The flow is the same for both supported source databases: configure an Endpoint
To expose a Postgres database via PrivateLink, you need a Network Load Balancer that forwards traffic to the database. This works for Postgres running on EC2 or RDS.

<Warning>
For AWS RDS, the steps below do not handle dynamic IPs if the RDS instance's IP changes. This is specifically relevant when using an RDS cluster with failover support. See this [AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for handling IP changes automatically.
The target group holds a fixed IP address. It does not follow the database when the IP changes, which happens on an Aurora failover, an instance replacement, or a migration to a new cluster. PowerSync then cannot connect, or reaches a different instance and rejects its certificate because the names in the certificate do not include the configured host. Update the target group when the writer changes, or automate it as described in this [AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/).
</Warning>

1. **Create a Target Group**:
1. Obtain the RDS instance's private IP address. Make sure this points to a writable instance.
1. Obtain the private IP address of the instance that PowerSync must replicate from. For Aurora, resolve the cluster endpoint to get the current writer's IP. For RDS, use the instance's private IP.
2. Create a **Target Group** with IP addresses as the target type, using the IP from above. Use TCP protocol and the database port (typically `5432` for Postgres).
2. **Create a Network Load Balancer (NLB)**:
1. Select the same VPC as your RDS instance.
Expand Down Expand Up @@ -135,12 +135,16 @@ The flow is the same for both supported source databases: configure an Endpoint
2. Select the **Postgres** or **MongoDB** tab.
3. In the **Private Endpoint** dropdown, select your endpoint. Only endpoints in the same region as the instance with status `Available` are selectable.
4. Fill in the rest of the connection details:
- **For Postgres**: enter your database connection details as usual. PowerSync routes traffic through the Private Endpoint to your load balancer.
- **For Postgres**: enter your database connection details as usual, with the database's own hostname as the "**Host**": the cluster endpoint for Aurora, or the instance endpoint for RDS. PowerSync uses this hostname to verify the server certificate and sends the traffic through the Private Endpoint to your load balancer. Do not enter the load balancer or Endpoint Service DNS name as the host, because it is not in the database's certificate.
- **For MongoDB**: on the Atlas cluster, click **Connect**, choose **Private Endpoint** as the connection type, select the provisioned endpoint, choose **Drivers** as the connection method, and copy the resulting connection string. It should look something like `mongodb+srv://<db_username>:<db_password>@your-cluster-pl-0.abcde.mongodb.net/`. Paste it into the **URI** field in the Dashboard.
5. Click **Test Connection** and resolve any errors.
6. Click **Save Connection**.

PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes. Monitor the logs to confirm the instance connects.

<Note>
**Test Connection** checks that the configuration is complete but does verify reachability. After deployment, confirm the connection in the instance's **Health** view and logs.
</Note>
</Step>
</Steps>

Expand Down
116 changes: 102 additions & 14 deletions configuration/source-db/setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

import PostgresPowerSyncUser from '/snippets/postgres-powersync-user.mdx';
import PostgresPowerSyncPublication from '/snippets/postgres-powersync-publication.mdx';
import PostgresRdsPowerSyncUser from '/snippets/postgres-rds-powersync-user.mdx';
import NeonDatabaseSetup from '/snippets/neon-database-setup.mdx';
import SupabaseGuidedSetup from '/snippets/supabase-guided-setup.mdx';

Expand Down Expand Up @@ -66,24 +67,47 @@

### 2. Create a PowerSync database user

Create a PowerSync user on Postgres:
<PostgresRdsPowerSyncUser />

### 3. Create `powersync` publication

<PostgresPowerSyncPublication />
</Accordion>

<Accordion title="AWS Aurora">
### Supported Scaling Options

PowerSync replicates only from the writer. Clusters can also include reader instances.

| Scaling Option | Scaling Model | PowerSync Support |
| --- | --- | --- |
| Provisioned | Configured instance capacity | Supported |
| Aurora Serverless v2 | Automatically adjusts instance capacity | Supported |
| Aurora Postgres Limitless | Horizontally scales writes and reads across multiple nodes | [Not supported](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/limitless-reqs-limits.html#limitless-limitations) |
Comment thread
joshuabrink marked this conversation as resolved.
Resolved

### Prerequisites

PowerSync must be able to reach the writer. Either make the writer publicly accessible over IPv4 and restrict access to PowerSync's IPs (see [IP Filtering](/configuration/source-db/security-and-ip-filtering)), or connect through a [Private Endpoint](/configuration/source-db/private-endpoints) using AWS PrivateLink. If you use public access, apply the same network settings to reader instances that can become the writer.

### 1. Ensure logical replication is enabled

In Aurora, `rds.logical_replication` is a setting of the **DB cluster parameter group**, not of a DB instance parameter group.

1. In the [Amazon RDS console](https://console.aws.amazon.com/rds/), open your cluster's **Configuration** tab and follow the **Parameter group** link of type **DB cluster parameter group**. If the cluster still uses a default parameter group, [create a custom DB cluster parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithDBClusterParamGroups.html) and assign it to the cluster first. Default parameter groups cannot be edited.
2. Set `rds.logical_replication` to `1` and save.
3. Reboot the writer. The change only takes effect after the reboot.
4. Connect to the writer through the cluster endpoint and confirm the settings:

```sql
-- SQL to create powersync user
CREATE ROLE powersync_role WITH BYPASSRLS LOGIN PASSWORD 'myhighlyrandompassword';

-- Allow the role to perform replication tasks
GRANT rds_replication TO powersync_role;

-- Set up permissions for the newly created role
-- Read-only (SELECT) access is required
GRANT SELECT ON ALL TABLES IN SCHEMA public TO powersync_role;

-- Optionally, grant SELECT on all future tables (to cater for schema additions)
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO powersync_role;
SHOW rds.logical_replication; -- on
SHOW wal_level; -- logical
```

To restrict read access to specific tables, explicitly list allowed tables for both the `SELECT` privilege, and for the publication (as well as for any other publications that may exist).
For the related parameters `max_replication_slots`, `max_wal_senders` and `max_worker_processes`, see [Setting up logical replication for your Aurora PostgreSQL DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.Configure.html). PowerSync uses one replication slot per instance, plus a second slot while it processes a new Sync Config version. See [Postgres Maintenance](/configuration/source-db/postgres-maintenance).

### 2. Create a PowerSync database user

<PostgresRdsPowerSyncUser />

### 3. Create `powersync` publication

Expand Down Expand Up @@ -322,7 +346,7 @@
- `find`: This privilege must be granted either at the database level or on specific collections.
- `changeStream`: This privilege must be granted at the database level (not on individual collections). In MongoDB Atlas, set `collection: ""` or check `Apply to any collection` in MongoDB Atlas if you want to apply this privilege on any collection.
- If replicating from multiple databases, this must apply to the entire cluster. Specify `db: ""` or check `Apply to any database` in MongoDB Atlas.
- For the `_powersync_checkpoints` collection add the following privileges: `createCollection`, `dropCollection`, `find`, `changeStream`, `insert`, `update`, and `remove`

Check warning on line 349 in configuration/source-db/setup.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/setup.mdx#L349

Did you really mean 'createCollection'?

Check warning on line 349 in configuration/source-db/setup.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/setup.mdx#L349

Did you really mean 'dropCollection'?

Check warning on line 349 in configuration/source-db/setup.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/setup.mdx#L349

Did you really mean 'changeStream'?
- To allow PowerSync to automatically enable [`changeStreamPreAndPostImages`](#post-images) on replicated collections (i.e. the [**Post Images**](#post-images) setting for the MongoDB connection on your PowerSync instance is set to **Auto-Configure**, which is the default for new PowerSync instances), additionally add the `collMod` permission on the database and all collections being replicated.

### Post Images
Expand Down Expand Up @@ -416,6 +440,70 @@

PowerSync reads from the MySQL [binary log](https://dev.mysql.com/doc/refman/8.4/en/binary-log.html) (binlog) to replicate changes. We use a modified version of the [Zongji MySQL](https://github.com/powersync-ja/powersync-mysql-zongji) binlog listener to achieve this.

For Aurora MySQL, follow the provider-specific steps below instead of the generic [Binlog Configuration](#binlog-configuration).

<Accordion title="AWS Aurora MySQL">
PowerSync supports provisioned Aurora MySQL clusters and Aurora Serverless v2 clusters with a single writer. The cluster can include reader instances, but PowerSync replicates only from the writer.

### Prerequisites

For PowerSync Cloud, make the writer publicly accessible over IPv4 and allow inbound traffic from [PowerSync's IP addresses](/configuration/source-db/security-and-ip-filtering) on the database port (`3306` by default). Apply the same network settings to reader instances that can become the writer. [Private Endpoints](/configuration/source-db/private-endpoints#current-limitations) are not currently supported for MySQL.

<Steps>
<Step title="Configure a DB Cluster Parameter Group">
In the [Amazon RDS console](https://console.aws.amazon.com/rds/), open the **DB cluster parameter group** linked from your cluster's **Configuration** tab. If the cluster uses a default group, [create a custom DB cluster parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithDBClusterParamGroups.html) with the family that matches your Aurora MySQL version. Default parameter groups cannot be edited.

<Frame caption="Aurora MySQL DB cluster parameter group creation">
![Amazon RDS creation form with Aurora MySQL, the aurora-mysql8.0 family, and DB Cluster Parameter Group selected](/images/installation/aurora-create-parameter-group.png)
</Frame>

Set these parameters to enable binary logging and global transaction identifiers (GTIDs):

| Parameter in the AWS Console | Required Value |
| --- | --- |
| `binlog_format` | `ROW` |
| `binlog_row_image` | `FULL` |
| `enforce_gtid_consistency` | `ON` |
| `gtid-mode` | `ON` |

<Frame caption="GTID settings in the cluster parameter group">
![Amazon RDS parameter editor with enforce_gtid_consistency and gtid-mode set to ON](/images/installation/aurora-set-gtid-flags.png)
</Frame>

Aurora [enables binary logging](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_LogAccess.MySQL.BinaryFormat.html) through `binlog_format`. You do not need to configure `log_bin` or `server_id` separately. The AWS console calls the GTID parameter `gtid-mode`; its SQL variable is `gtid_mode`.
</Step>
<Step title="Apply the Parameter Group">
If you created a new parameter group, assign it to the cluster through **Modify** in the RDS console. Save your changes and reboot the writer for the settings to take effect. If reader instances do not restart with the writer, reboot them manually to apply the same settings.
</Step>
<Step title="Verify the Settings">
Connect to the writer through the cluster endpoint and run this query:

```sql
SELECT
@@GLOBAL.log_bin AS log_bin, -- 1
@@GLOBAL.gtid_mode AS gtid_mode, -- ON
@@GLOBAL.enforce_gtid_consistency AS enforce_gtid_consistency, -- ON
@@GLOBAL.binlog_format AS binlog_format, -- ROW
@@GLOBAL.binlog_row_image AS binlog_row_image; -- FULL
```
</Step>
<Step title="Set Binlog Retention">
Set [binlog retention](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/mysql-stored-proc-configuring.html#mysql_rds_set_configuration-usage-notes.binlog-retention-hours) long enough to cover the initial snapshot and expected downtime or replication lag. If required binlogs are purged before PowerSync reads them, PowerSync must take a new snapshot.

Replace `24` with the number of hours your workload requires:

```sql
CALL mysql.rds_set_configuration('binlog retention hours', 24);
```

Monitor storage usage after increasing retention.
</Step>
<Step title="Create a PowerSync Database User">
Follow [Database User Configuration](#database-user-configuration) to create a user with replication privileges and read access to the tables you want to sync.
</Step>
</Steps>
</Accordion>

### Binlog Configuration

To ensure that PowerSync can read the binary log, you need to configure your MySQL server to enable binary logging and configure it with the following server command options:
Expand Down Expand Up @@ -709,7 +797,7 @@

PowerSync reads Convex data using the Convex Streaming Export API. Initial replication pins a single Convex snapshot cursor and snapshots each selected table at that cursor. Streaming replication then reads the global `document_deltas` stream and filters rows according to your Sync Streams.

Convex does not support user-defined database schemas or namespaces in the same way as SQL databases. You do not need to qualify Convex table names in Sync Streams. If you do choose to qualify them, use the default `convex` schema.

Check warning on line 800 in configuration/source-db/setup.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/setup.mdx#L800

Did you really mean 'namespaces'?

### Connection Requirements

Expand Down
Loading
Loading