The commit log causes infinite expansion of data dir.

[kdletters]

My current project is increasing commit logs at a rate of 1G per day, causing issues with disk usage, backup, and restart times.
Due to the lack of a hot backup at the moment, I must perform a cold backup. The backup time plus restart time has been almost half an hour.

[gefjon]

We periodically capture a snapshot of the current database state, at which point the commitlog prior to that snapshot is no longer necessary to restart the database. We don't automatically delete those outdated commitlogs because they contain potentially interesting historical data.

SpacetimeDB-cloud (the proprietary system which runs Maincloud) automatically archives unneeded commitlog segments and snapshots to cold storage, to reduce usage of relatively expensive local disks. SpacetimeDB-standalone (the source-available system which you're using to self-host) does not contain any automatic archival or deletion logic.

If that historical data is not useful to you, you can delete any commitlog segment whose entire contents are strictly before the most recent snapshot. Note that commitlog segments are named for the earliest transaction they contain, and should be treated as containing the range of transactions from their own file name to the next segment's file name. For example, if you have segments 100.stdb.log, 200.stdb.log, 300.stdb.log and a snapshot 250.snapshot_dir, it would be safe to delete the segment 100.stdb.log, but you still need 200.stdb.log and 300.stdb.log. Similarly, you can delete all but the most recent snapshot.

Cold backups should not be necessary. So long as you don't disturb the most recent snapshot and the commitlog suffix which are used when the database restarts, it is safe to move or delete the historical commitlog and snapshots during SpacetimeDB's execution.

How long does the restart, specifically, take? What hardware are you operating on, and particularly, what sort of disks or local storage? We target sub-10-second restarts when running on modern middle-to-high-end hardware with medium-to-fast local disks. If your database is taking significantly longer to restart, it may be due to poor performance of network-attached storage, but it may also be a SpacetimeDB bug which we'd like to investigate.

[kdletters]

Here are my server info.

Here are my data-dir looks like. It currently 19G.

The latest reboot only took 33.5 seconds on STDB, and the rest of the time was spent on cold backup, compression, and other tasks.