You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4e7900c
Browse filesBrowse the repository at this point in the historyBrowse files
description: "Store AI chats as synced rows, and stream replies to clients while the model generates them."
4
+
sidebarTitle: "AI Chat"
5
+
keywords:
6
+
- "ai"
7
+
- "llm"
8
+
- "chat"
9
+
- "chatbot"
10
+
- "streaming"
11
+
- "server-sent events"
12
+
- "sse"
13
+
---
14
+
15
+
With PowerSync, an AI chat can be stored as ordinary synced rows. Your backend calls the model and writes each reply to your source database, and PowerSync syncs it to the client. Chat history then works offline, and the same chat shows on every device.
16
+
17
+
Showing a reply while the model is still generating it builds on this, and is covered in the [second half](#streaming-replies-while-the-model-generates-them) of this guide.
18
+
19
+
## Storing Chats as Synced Data
20
+
21
+
Your backend calls the model, never the client, so model API keys stay on your server. The code that calls the model can live in your backend API, a separate process, or a background job, as long as it can write to your source database.
22
+
23
+
Because the client reads and writes only to its local database:
24
+
25
+
- A chat opens without a network request.
26
+
- A reply in progress survives a page refresh, and shows in every tab and device where the chat is open.
27
+
- A message sent offline waits in the [upload queue](/client-sdks/writing-data#write-operations-and-upload-queue) until the client reconnects.
28
+
- If several users can access a chat, for example through a chat members table in your Sync Streams, they all see new messages and replies as they arrive.
29
+
30
+
### Tables
31
+
32
+
Create a `chats` table and a `messages` table. Give AI-generated messages a `status` column so that the client can show whether a reply is pending, being generated, done, stopped, or failed. If generation fails, store the error message on the row as well.
33
+
34
+
### Sync Streams
35
+
36
+
Define two [Sync Streams](/sync/streams/overview). The `chats` stream syncs the user's chats, and the client subscribes to `chat_messages` for the chat that is open:
37
+
38
+
```yaml
39
+
config:
40
+
edition: 3
41
+
42
+
streams:
43
+
chats:
44
+
auto_subscribe: true
45
+
query: SELECT * FROM chats WHERE user_id = auth.user_id()
46
+
47
+
chat_messages:
48
+
query: |
49
+
SELECT * FROM messages
50
+
WHERE chat_id = subscription.parameter('chat_id')
51
+
AND user_id = auth.user_id()
52
+
```
53
+
54
+
See [Client-Side Usage](/sync/streams/client-usage) for how to subscribe in each SDK.
55
+
56
+
### Generating Replies
57
+
58
+
1. When the user sends a message, the client writes two rows in one local transaction: the user's message, and an empty message for the AI reply, with a pending status. The UI shows the pending reply right away, even offline.
59
+
2. The client SDK uploads both rows to your backend through `uploadData()` in your [backend connector](/configuration/app-backend/client-side-integration).
60
+
3. The backend claims the pending reply with a conditional update, so that only one backend process works on it.
61
+
4. The backend sends the chat history to the AI model.
62
+
5. When the model finishes, the backend saves the full text and marks the reply as done.
63
+
64
+
The client receives each reply when the write from step 5 syncs.
65
+
66
+
## Streaming Replies While the Model Generates Them
67
+
68
+
To show reply text before the reply is complete, the backend has to send partial text to the client during step 4. We have tested two ways to do this, and we have a [demo](#example-implementation) available that implements both.
69
+
70
+
### 1. Streaming Through the Database
71
+
72
+
As new text arrives, the backend writes the text it has so far into the reply's row, at most once per interval, for example every 100 ms. PowerSync syncs each update, and a [watch query](/client-sdks/watch-queries) on the client re-renders the message. The interval caps the write rate no matter how fast the model produces tokens.
73
+
74
+
```mermaid
75
+
flowchart LR
76
+
model["Model"] -->|tokens| backend["Backend"]
77
+
backend -->|"text so far,<br/>at most every 100 ms"| db[("Source<br/>database")]
78
+
db --> service["PowerSync<br/>Service"]
79
+
service -->|"syncs each write"| client["Client"]
80
+
```
81
+
82
+
This is the simpler of the two, because PowerSync is the only channel to the client. A client that opens the chat mid-reply catches up through normal sync.
83
+
84
+
The downside is volume. Each write is an update in your source database, and it syncs the whole row, so a long reply sends its earlier text again with every write, to every client subscribed to the chat.
85
+
86
+
### 2. Streaming Over a Separate Connection
87
+
88
+
The backend keeps the reply in memory and sends the text to clients over a separate connection, for example [Server-Sent Events (SSE)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events). It writes the row once, when the reply is finished. In the diagram, dashed arrows carry the finished reply.
89
+
90
+
```mermaid
91
+
flowchart LR
92
+
model["Model"] -->|tokens| backend["Backend"]
93
+
backend -->|"SSE: text so far,<br/>then new text"| client["Client"]
service -.->|"syncs the<br/>finished reply"| client
97
+
```
98
+
99
+
When a reply is being generated, the client opens a connection to the backend and shows the text as it arrives. If the connection drops, it opens it again. When the finished reply syncs through PowerSync, the client shows the row and closes the connection.
100
+
101
+
Each reply is written to your database once, however long it is. On the other hand, you have to build and secure the streaming endpoint yourself. If you run more than one backend process, each client also has to connect to the process that is generating its reply. [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/) and similar services solve that.
102
+
103
+
### Smoothing the Text
104
+
105
+
With either option, text arrives in steps, and some models send it in large chunks. If the UI renders each step as it arrives, the reply jumps and pauses. Keep the received text separate from the displayed text, and reveal the difference a few characters per animation frame, at a speed based on how often text arrives. Show existing messages without animation.
106
+
107
+
## Example Implementation
108
+
109
+
The [AI chat demo](https://github.com/powersync-ja/powersync-js/tree/main/demos/react-ai-chat) implements both options in a React app with Postgres. Its Node.js backend applies uploads and calls the model with the [Vercel AI SDK](https://ai-sdk.dev).
`requestCheckpoint()` requires that the database is connected or connecting. The device must be online for the request to reach the PowerSync Service. If it is offline or the sync client is reconnecting, the call waits and continues once the Service is reachable.
189
+
`requestCheckpoint()` requires that the database is connected or connecting. The device must be online for the request to reach the PowerSync Service. If it is offline or the sync client is reconnecting, the call waits without a timeout and continues once the Service is reachable. Cancel the calling task if you need to stop waiting.
190
190
191
-
Creating the request has no timeout of its own, so it can stay suspended while the sync client retries its connection. Cancel the calling task if you need to stop waiting.
191
+
After the request reaches the Service, the request may time out depending on the SDK. If the request times out, `requestCheckpoint()` fails with an error. Call it again to retry.
192
192
193
193
Aborting a `waitForSync` call or configuring a timeout does not remove the checkpoint. It only limits how long you wait for the checkpoint to sync and apply locally.
194
194
@@ -428,7 +428,8 @@ To make the PowerSync client SDK post checkpoint requests to your backend instea
428
428
- For Kotlin, implement `CustomCheckpointRequestConnector`.
429
429
- For Swift, make your connector conform to `CustomCheckpointRequestConnector`.
430
430
- For .NET, implement `ICustomCheckpointRequestConnector`.
431
-
- For Rust and Tauri, implement `BackendConnector::post_checkpoint_request`.
431
+
- For Rust, implement `Authenticator::post_checkpoint_request` on your `Authenticator`.
432
+
- For Tauri, implement `BackendConnector::post_checkpoint_request`.
432
433
433
434
Forward the request to your application backend:
434
435
@@ -454,6 +455,7 @@ final class MyBackendConnector extends PowerSyncBackendConnector
454
455
Future<String> postCheckpointRequest(
455
456
String clientId,
456
457
String requestId,
458
+
Future<void> abortSignal,
457
459
) async {
458
460
final response = await myBackend.createCheckpointRequest(
459
461
clientId,
@@ -510,8 +512,8 @@ class BackendConnector : ICustomCheckpointRequestConnector
510
512
511
513
```rust Rust
512
514
#[async_trait]
513
-
implBackendConnectorforTodoDatabase {
514
-
// ... also implement fetch_credentials and upload_data
### Android: Flipper Network Plugin for HTTP Streams
443
-
444
-
If you connect to PowerSync with HTTP streams (the default), Android requires additional configuration. React Native does not support streams natively, so the SDK uses the [polyfills](/client-sdks/reference/react-native-and-expo#installation) from the installation steps. An open Flipper [issue](https://github.com/facebook/flipper/issues/2495) prevents the Flipper network plugin from firing stream events. [Disable the plugin](https://stackoverflow.com/questions/69235694/react-native-cant-connect-to-sse-in-android/69235695#69235695) so that HTTP streams work.
445
-
446
-
**If you are using Java (Expo \< 50):**
447
-
448
-
Uncomment the following in `android/app/src/debug/java/com/<ProjectName>/ReactNativeFlipper.java`:
449
-
450
-
```java
451
-
// NetworkFlipperPlugin networkFlipperPlugin = new NetworkFlipperPlugin();
452
-
// NetworkingModule.setCustomClientBuilder(
453
-
// new NetworkingModule.CustomClientBuilder() {
454
-
// @Override
455
-
// public void apply(OkHttpClient.Builder builder) {
If you test offline mode on an iOS simulator by disabling the host machine's internet connection, the simulator stays offline even after you restore the connection. This affects all network requests in the app.
Copy file name to clipboardExpand all lines: client-sdks/reference/rust.mdx
+67-33Lines changed: 67 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -214,55 +214,74 @@ Next, create a database:
214
214
</Tab>
215
215
</Tabs>
216
216
217
-
Finally, instruct PowerSync to sync data from your backend:
217
+
<LocalOnly />
218
+
219
+
### 3. Stream Changes from PowerSync
220
+
221
+
The PowerSync client SDK connects to the PowerSync Service to download changes. To connect, it needs the Service endpoint and an `Authenticator` that provides an [authentication token](/configuration/auth/overview).
222
+
223
+
The authenticator must implement `resolve_credentials`, which returns the token. The SDK does not cache the returned token and can call `resolve_credentials` several times while connected, so a typical implementation returns a token cached in your app's state. Optionally, implement `invalidate_credentials` to clear that cache. The SDK calls it when the Service rejects the token with a 401 or when the token has expired or is about to expire. See [Authentication Setup](/configuration/auth/overview) for how to generate tokens.
224
+
225
+
**Example:**
218
226
219
227
```Rust
220
-
// MyBackendConnector is defined in the next step...
// Use a development token (see Authentication Setup https://docs.powersync.com/configuration/auth/development-tokens) to get up and running quickly.
239
+
// For an example implementation, see https://github.com/powersync-ja/powersync-native/blob/d2841961303c4478bece620a0c12f980a74188c5/examples/egui_todolist/src/database.rs#L129-L151
1. Get an auth token to connect to the PowerSync instance.
233
-
2. Upload client-side writes to your backend API. The SDK places every write to the SQLite database in an upload queue and uploads the queue to your backend when the user is connected. Your backend then applies the changes to the source database.
261
+
### 4. Upload Local Mutations
234
262
235
-
The connector must implement two methods:
263
+
The SDK places every write to the SQLite database in an upload queue and calls the `MutationUploader` when the queue has writes to upload. Your backend then applies the changes to the source database. See [When `uploadData()` is Called](/configuration/app-backend/client-side-integration#when-uploaddata-is-called) for triggers, throttling, and retry behavior, and [Writing Client Changes](/handling-writes/writing-client-changes) for the app backend implementation.
236
264
237
-
1.`fetch_credentials` - The SDK calls this method to get authentication credentials. It caches the credentials and calls the method again only when needed, for example on the first connection or when the token is near expiry. See [When `fetchCredentials()` is Called](/configuration/app-backend/client-side-integration#when-fetchcredentials-is-called) for details and [Authentication Setup](/configuration/auth/overview) for how to generate credentials.
238
-
2.`upload_data` - The SDK calls this method whenever it has client-side writes to upload to your backend API. Implement how those writes are processed and uploaded. See [When `uploadData()` is Called](/configuration/app-backend/client-side-integration#when-uploaddata-is-called) for triggers, throttling, and retry behavior, and [Writing Client Changes](/handling-writes/writing-client-changes) for the app backend implementation.
265
+
`SyncOptions::download_only` does not upload local writes. If the app writes to the local database while connected in download-only mode, the sync status reports a download error because the SDK cannot apply downloaded data over pending local writes. Skip this step only if the app never writes to the local database.
// implement fetchCredentials to obtain the necessary credentials to connect to your backend
251
-
// See an example implementation in https://github.com/powersync-ja/powersync-native/blob/508193b0822b8dad1a534a16462e2fcd36a9ac68/examples/egui_todolist/src/database.rs#L119-L133
252
-
253
-
Ok(PowerSyncCredentials {
254
-
endpoint:"[Your PowerSync instance URL or self-hosted endpoint]".to_string(),
255
-
// Use a development token (see Authentication Setup https://docs.powersync.com/configuration/auth/development-tokens) to get up and running quickly
0 commit comments