Repository navigation
Upstream sync through 6c159ad53: self-describing saves, Linux fixes, ten of the team's later fixes taken early - #39
Merged
Conversation
48 commit(s) from the team since dkfans/keeperfx@fb0ab7f4d: - Multiplayer: lobby buttons respond instantly (#5496) (6c159ad53) - Disable hand light in possession (#5479) (a0efb54e6) - Multiplayer: fixed spectators being unable to sync (#5495) (38f0783cc) - Computer Player avoids reusing creatures to defend it just dropped (#5480) (e2df7360d) - Chickens can no longer breathe fire (#5481) (94691033f) - Parchment map zoom now unaffected by MP input lag (#5491) (50c55bb4d) - Spectators switch view with space or clicking the name (#5489) (aa3d4633c) - Multiplayer: smoothen sidebar toggling (TAB) for spectator (#5492) (a95930972) - Fix camera speed for arrow keys + edge panning (#5490) (3d1eed14d) - Multiplayer: better disconnect messages (#5488) (29e414d21) - Fixed 'Off-by-one'-error related to gold veins (#5476) (470dd6799) - Potential fix on SET_SACRIFICE_RECIPE (#5475) (c78faa424) - Health flower: same height independent of mood icons (#5483) (cb3334719) - Fix possession intro sliding and turning the creature north (#5473) (5d94e335f) - Merge adjacent rooms of same type on claim (#5467) (7e5a38ae5) - Multiplayer: Reduce stutter when spectator joins, and request resync (#5484) (3bbcdc200) - Break all saves, so saves will never break again in the future (#5424) (f03798d71) - Multiplayer: Assume 2 players if there is no player count in lof (#5474) (43dcdfda5) - Fix build: free Lua serialized data in replay rewind (#5470) (bb5aca64f) - Hide chat messages from log (#5469) (a09f55ca2) - Multiplayer: improve lua serialization (#5430) (e9f47ebfe) - Multiplayer: fix dig tagging misclicks and roomspace box lag (#5466) (814d3ce2e) - Multiplayer: Rework input lag calculation (#5462) (abb1582f7) - Show an on-screen message and timer while recording a movie (#5461) (b91db1159) - Clear on-screen message when a level loads (#5460) (48adf1b0d) - Allow escape characters in .po files (#5453) (facca7c44) - Hold backspace to rewind replays (#5450) (54f9ee23d) - Extra sprite space (#5387) (3bffc8fe3) - Added 4 player MP map: Purple Frontier (58f5bf87e) - Spectator instance bugfixes (#5452) (a885de72b) - Read orientation from slabset.toml (#5458) (521cb1c52) - VIEW_TYPE is 0 again when no view type is found (#5457) (6af711db1) - Multiplayer: more assorted log fixes (#5454) (6d4f458b2) - Reveal map after heart explode, not before (#5445) (9144deb4d) - Fix tooltips hidden behind the keeper hand (#5451) (a5120455a) - Assorted Spectator and Replay improvements (#5449) (d77a62cf7) - Equally share spectators among players (#5448) (4b7c77655) - Replays: disable lua IO, move some state to LocalState (#5441) (3e44716a2) - Hide vision of preplaced computers on replays (#5447) (7b88c8e7d) - Multiplayer: set your player colour in the lobby (#5439) (d073f1983) - Allow Alternate player names in locations (#5444) (b755369a9) - Packetload skips intro cinematic (#5446) (c759d28d6) - Assorted small bug fixes that resulted in log warnings/errors (#5431) (b284fd1e1) - Add CHAT_FONT_SIZE keeperfx.cfg setting (#5440) (cb146b949) - Faster software-rendered parchment fade (#5429) (7d46f39c3) - UserState refactor (4/9) (#5428) (75256d22f) - Russian text tweaks to spectator strings (#5426) (6f1d0f706) - Add USE_SPECIAL_STEAL_HERO script command (#5422) (d4e34a163) Filtered with packaging/linux-only/filter-tree.sh --upstream: removed 290 Windows-only file(s) dropped #include of removed headers from 1 file(s) stripped Windows conditionals from 32 source file(s) Linux-Snapshot-Of: 6c159ad534c11414bf5c83b4ea1231db3eb524b2 Linux-Snapshot-Upstream-Count: 5539
…-10-19) Upstream brought, among others: a new save format (#5424) -- every file describes its own layout and fields are read by name, so saves stop breaking whenever a struct changes -- the UserState refactor (#5428), replay rewind (#5450) and Lua stubs for replays (#5441), the remaining #5431 fixes, a reworked dig-tag and room-build input path (#5466), claimed rooms merging with adjacent rooms of the same kind (#5467), log fixes (#5454), the faster software parchment fade (#5429, #5491), the heart-explode reveal and tooltip fixes (#5445, #5451), slabset orientation (#5458), VIEW_TYPE (#5457), gameplay fixes (#5422, #5473, #5475, #5476, #5479, #5480, #5481, #5483, #5490) and multiplayer and spectator work (#5439, #5447-#5449, #5452, #5462, #5474, #5484, #5488, #5489, #5492, #5495, #5496). Conflicts, 21 files: The save code -- game_saves.c/.h, frontmenu_saves.c/.h, highscores.c, game_loop.c, console_cmd.c: #5424 moves saving and loading into src/kfx/save/ and leaves game_saves.c with the campaign helpers. Taken from upstream, then this branch's own save features put back on its SaveManager API: - deleting a save into the system trash or save/trash/ moves out of game_saves.c into its own save_trash.c, so the next upstream edit to the save files does not touch it; a save this build cannot read is now trashed before a save is written into its slot, rather than lost; - the save/load list stays packed with the free slots last, and the delete buttons and their confirmation are unchanged; - a refused load from the in-game menu gives back the pause state the menu found (upstream toggles it); - high scores: the 80-byte entries 64-bit Linux builds write are read as well as KFXS and the 72-byte raw layout, and this edition keeps writing the 80-byte layout while stable 1.4.0.5660, which reads only that, shares the game folder. Dropped as upstream now does the same: the fork's whole-file check before loading and its refusal messages (upstream checks and decodes into scratch memory first, and greys and explains an unloadable save), the re-entry to the load list after a failed load (upstream's frontend_load_game_failed), the dual chunk-header reader for listing old saves. The 64-bit campaign-progress import and the CI save-format check follow in their own commits on this branch. config.c, packets_input.c, net_exchange_common.c, roomspace.c/.h, room_garden.c, room_treasure.c, thing_objects.c -- upstream's, which now carries the fixes this branch had ported early (#5431's roomspace, CTA unset state, treasury recount, the desynced spectator's resync) and the UserState fields. room_treasure.c equals upstream again: its recount starts from zero, and the fork's read-only seeding pass goes (measured on 10-16: identical figures, no guard errors on either build). creature_states_spdig.c -- ours: imp_toking keeps the continue state unless it is unset. light_data.c -- ours with upstream's kfx_rewindable: light_bitmask stays uint32_t, as upstream's 256-byte memcpy into the 128-byte saved field is still there on 64-bit. lua_base.c -- upstream's replay stubs, then lua_make_files_read_only(), so replays read files and refuse writes as the game they recorded did. config_campaigns.h, lvl_script.h, lvl_script_commands.c -- upstream now uses int32_t and PRIu32 as this branch did; kept the comments. Also in this merge: - linux.mk: the 31 new sources (src/kfx/save/**, FileSystemLb.cpp, kernel_memtracker.c, replay_rewind.c, roomspace_input.c) and save_trash.c, roomspace_prediction.c dropped, by sync-linux-mk-sources.sh. - console_cmd.c cmd_game_check printed a long with %d; packets.c's target_idx is initialised (always set before use, but gcc can't see it). Checked so far: check-tree, linux.mk sources, a clean build with no warnings but the farm's usual fall-through false alarm.
…he new save code Upstream's new save code (#5424) imports the old fx1contn.sav only in the 32-bit layout (67241 bytes). Every Linux build wrote it with 8-byte campaign flags (67401 bytes), stable 1.4.0.5660 included, so for a 5660 player the file counted as absent: no Continue, no progress percent, and the campaign, its bonus levels, carried creatures and flags started over. 9447a49 fixed the same thing in the old save code; this puts it back. - The decoder moves out of the save code into legacy_progress_lp64.c/.h, so upstream's files carry only the calls. Its static asserts still pin the 67401-byte layout. - SaveContinue.cpp reads either size, and narrows the 64-bit one. - fx1contn.sav is kept after its import, as 2888dad did: stable 5660 shares the game folder and reads nothing else. - Continue offers fx1contn.sav only while its campaign has no file of its own yet (3f1a162), and a campaign whose own file is there but can't be read no longer falls back to the older stable-era position. Upstream deletes the old file at import, so it never meets either case. - tests/run.sh compiles the real decoder file instead of cutting it out of game_saves.c, and links FileSystemLb.cpp, which PlatformManager now needs for the file-enumerator and Lua read-only tests. Unit tests pass, normally and with ASan/UBSan.
The new save format (upstream #5424) stores only the fields its tables name. trap.rearm_route_failed_turn, this edition's (527bc71, PR #31: imps skip a trap for 1000 turns after failing to reach it), was not in Thing_trap, so a load would have survived it only as leftover raw union bytes, if at all. It is now a field of its own, read back by name. Found by the new table check (packaging/ci/check-save-tables.sh, next commit), which now reports no missing member.
Since upstream #5424 a saved game describes its own layout and is read field by field, so sizeof(struct Game) no longer decides whether a save loads, and the probe that measured it no longer compiles (the chunk header and catalogue entry moved to kfx/save/SaveTypes.h). check-save-format.sh now: - builds upstream's tools/save-tool and compares what the field tables say (schema, migrations, format version) with the committed copy in packaging/ci/save-format/. A difference fails, printing upstream's own policy verdict (tools/save-tool/policy.py) on whether older saves still load; --update accepts it, in the commit that makes the change. - runs check-save-tables.sh, upstream's check that the tables list every member of every saved struct, done with the native gcc and gdb instead of 32-bit MinGW and gdb-multiarch (it compares names, not sizes). A member the tables miss is simply never saved, and nothing else fails. - --release-note <tag> warns when the tag's saves will not load: always for a tag from before the new format (the one-time break, with a note that campaign progress and high scores carry over), otherwise when policy.py finds a break. A tag with no baseline at all still exits 3. save-format-baseline becomes the history: what the check compares now, how to react, the 2026-10-19 entry, and the old entries kept as record. Both workflows install gdb; the sync bot's warning and comments describe the new check. save_format_probe.c and its linux.mk rule are gone. Checked: the check passes on this tree (format 1.1, 1319 fields, no missing member); --release-note v1.4.0.5660 prints the one-time warning, v1.4.0.5000 exits 3; policy.py flags a version bump as a break, an added field as compatible and a rename without an alias as missing_alias.
Saves written before the new format are listed greyed, by the title in their info block. Upstream reads that block after a 12-byte chunk header, but every save 64-bit Linux builds wrote before 1.4.0.5751 -- stable 1.4.0.5660 included -- has 24-byte headers (three unsigned longs, until upstream #5353), so those saves showed as "fx1g0003.sav" instead of the name the player gave them. ReadLegacyTitle() now reads either layout, told apart as 5d5d2d1 did (PR #27): the wide layout's second 32-bit word is the high half of a length, always zero, and a chunk id never is. Checked against a copy of a save stable 5660 wrote: a 24-byte header, INFO, length 412, then its title. A static_assert pins the 412-byte info block.
Upstream #5467 merges a claimed room into adjacent rooms of the same kind, through merge_room_into(), which building a connecting slab uses too. It moves the room's workers to the surviving room, but two more room ids a creature keeps still named the absorbed room after its index was freed for the next room built: - target_room_id, for a creature on its way there: move_check_on_head_for_room() never sees it arrive in the merged room (the index differs), and if the index is reused, it "arrives" in a stranger's new room instead; - last_work_room_id, which the room-deletion sweep then cleared. Both now point at the surviving room. Reasoned from the code with the upstream review; not reproduced.
ReadReplayHeader() reads a replay's chunks straight off the file and parsed their headers by hand, skipping the checks the in-memory reader makes (save_chunk_head). A damaged or hostile replay -- they are shared files -- could: - size the body buffer from any stored_len, up to 4 GiB: out of memory, or std::bad_alloc thrown through extern "C" and the game terminated; - give an uncompressed chunk a raw_len longer than its body, which save_chunk_unpack() then memcpy'd past the end of the buffer. The header now goes through save_chunk_head() against what is left of the file, so both are refused as damaged, as they are in a saved game.
SaveManager::Load() copied the save's full path into a DISKPATH_SIZE (144-byte) buffer. A save folder deep enough -- a long home directory, a Flatpak data directory -- gave a cut name, the file read as missing, and the slot was disabled. The copy is FULLPATH_SIZE now, as long as the path it copies.
…ease The payload's fxdata/gtext_*.dat and the campaign and level texts come from upstream's last stable release (1.4.0, July), while the engine is upstream's master, so every string added since is missing: the engine shows nothing where it asks for one. Upstream's new save code (#5424) explains a refused save with three such strings (guitext 1155-1157), so a player clicking a save from an earlier version would get an empty error box. The same gap blanks the newer spectator and lobby messages. Both release workflows now run upstream's own pkg-languages after pkg-gfx (as PR #22 did for the artwork): it fetches the prebuilt Linux po2ngdat and builds fxdata, campaign and level texts from lang/*.po, and they go over upstream's release data with the artwork. The Flatpak step is guarded like pkg-gfx, for a stable tag that predates pkg_lang.mk. Checked here: make -f Makefile pkg-languages builds 95 files in 6 s; the new gtext_eng.dat carries "This save was made by an incompatible version and can't be loaded.", the shipped one does not.
…les the last one REMOVE_SACRIFICE_RECIPE (or SET_SACRIFICE_RECIPE with no action) shifts the recipes after the removed one down a slot, which leaves the last slot as it was. With the table full -- all 99 usable slots taken -- the last recipe then sat in both slot 98 and slot 99. The last slot is cleared after the shift. Upstream's #5475 fixed the matching and the full-table check in the same function; this was the part it left, found in the review of that sync. Reasoned from the code.
Upstream's own codec and file tests for the new save format (#5424, tests/save) now build and run in tests/run.sh, normally and under ASan/UBSan with SANITIZE=1, as CI runs it. They round-trip a real struct Game through the field tables and compare digests, so a table that reads differently on this 64-bit target than on upstream's 32-bit Windows build fails here rather than in a player's save. Here: 14571 + 36 checks, 0 failed, normally and with the sanitizers.
When the tables differ, check-save-format.sh showed the first lines of the schema diff and then stopped: diff exits 1 on a difference, and under pipefail that ended the script before it printed upstream's policy verdict and what to do. It still failed, so CI went red, but the one line that says whether older saves load was never shown. Found by running it on a trial merge of the next upstream batch, which now prints "Older saves keep loading" with its one finding.
(cherry picked from commit 1b8545219ed08014194a7956c664c26774019f28)
Fixes #5499 Fixes #5498 (cherry picked from commit f5bcdf10e46822d2653f54c0d50935abbf266365)
(cherry picked from commit ddd6fccc82a88ff97ff790558263e3753d61cd61)
…517) (cherry picked from commit b5f85b15a57cb318b6f90b4c93ca01face0e1fa9)
Implements #3131 (cherry picked from commit 9f192b60e39576f6caded840635c6554a75784d6)
…523) Fixes crash report 787 (cherry picked from commit b520ee28cd83a5e9deff098a0b4d71627d108e6d)
(cherry picked from commit 52fc911d4fb79b3cf1dbfabd5dc0848f2c61b43b)
Fixes #1555 - You will still get notified when your portal suggests you need one - If you can research one but cannot build it, you still do not get notified. (cherry picked from commit 08fdc40901e4b58f62408cbcf1cbd60ce48ee856)
Fixes #1423 (cherry picked from commit 3e7490839c6a38cd09911117886562f206cc83e3)
Fixes #2517 (cherry picked from commit 5c85b64ffc3201d179618ac3ca43d0c5289fee66)
power_lost_side_effects() cleared the chosen power whenever the power lost matched it, whoever lost it. The chosen power is this machine's spell-panel selection, so an enemy losing a book cleared the player's own selection of that power: claiming an enemy library, stealing a book from one, or (since upstream #5516) a neutral library shrinking. It is now cleared only when the local player is the one who lost the power. game.chosen_spell_type is saved but not synced, and resync keeps it, so this changes nothing another machine sees.
The section still described saves as a raw dump of struct Game checked by its size. Since upstream #5424 saves are read field by field and check-save-format.sh compares the field tables; say what can still break a save and how a change is accepted.
…atures
Upstream #5424 (in this sync) narrowed the range check for long and
unsigned long config fields to 32 bits, to reject what Windows rejects.
On 64-bit Linux int64_t and uint64_t are long and unsigned long, so
var_type() files every 64-bit field under them. The Castability flags
of POWER_HAND, POWER_SLAP and POWER_PICKUP_OBJECT go above bit 31, were
refused ("magic.cfg(line 1720): Value out of range for unsigned long:
34900738054"), and stayed empty.
Measured with qa-harness/handpick_1022.py on keeporig level 1: four
imps added beside the heart, then a click on the creature tab's idle
count, which picks one up with the hand. Alpha 02dd676: an imp in the
hand. This sync without the fix: none, and the three warnings at every
level start. With the fix: an imp in the hand, no warnings.
The checks are back to the platform's own long, as stable 1.4.0.5660
and alpha have them. Elsewhere these types are still treated as 32-bit
(the datatype limits and the field reader), as before this sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The team's master through 6c159ad53 (48 commits, 13–19 October), merged by hand through the Linux-only snapshot because the bot hit conflicts (#25). On top: Linux fixes the batch needed, ten of the team's fixes from the following week taken early, and two fixes of ours.
The batch
The big change is upstream #5424, the self-describing save format: saved games, the continue link and campaign progress carry their own layout and are read field by field, so later engine changes stop breaking saves. Saves from 1.4.0.5762-alpha and stable 1.4.0.5660 can't be loaded by it, one last time; the release notes say so automatically.
Each commit was read and judged: none makes things worse for us. The riskiest is 814d3ce2e, the team's rewrite of the dig and build input; its one regression (pickup priority, #5500) is among the fixes taken early below. The hand-resolved conflicts are in the merge commit's message.
Linux fixes the batch needed
longconfig fields at 32 bits, to reject what Windows rejects. On 64-bit Linuxuint64_tisunsigned long, so the Castability flags of POWER_HAND, POWER_SLAP and POWER_PICKUP_OBJECT (bits above 31) were refused at every level start. The checks are back to the platform's ownlong, as stable and alpha have them.fx1contn.savonly in the 32-bit layout; every Linux build, stable 5660 included, wrote the 64-bit one.sizeof(struct Game)(fc75f7e), and prints that policy verdict when the format changes (b9b2813). The team's save tests run with ours (1934bea).docs/releasing.mddescribes the new check (7d7200f).Ten of the team's fixes from 20–22 October, taken early
Cherry-picked unchanged (
git cherry-pick -x; each patch-id equals upstream's), so Monday's sync merges them cleanly:#5500 pickup priority, #5501 possession camera, #5503 replay-rewind minimap and event boxes, #5517 previous work room, #5510 alliances stop fights, #5523 MP crash after a failed join, #5478 slab flags, #5482 lair requests, #5516 books turning powers off, #5512 navigation around redrawn slabs. The two multiplayer refactors and the rest of that week wait for the bot.
Next to #5516, ours (8b43450): another keeper losing a power no longer clears your own selection of it (claiming or robbing an enemy library, a script taking it from the computer, a neutral library shrinking).
Checks
Run on a clean build of this branch merged with
fix/work-room-step-off(the 2026-10-23 alpha candidate), against alpha 02dd676 and stable 5660:tests/run.shpasses (and withSANITIZE=1on the build before 34e50b3; 14,571 save-format checks, 0 failed);check-save-format.shOK; Linux-only guard clean.qa-harness/handpick_1022.py(keeporig level 1, the creature tab's idle count picks an imp up with the hand): alpha 02dd676 holds an imp; this sync without 34e50b3 holds none and logs three Castability warnings at every level start; with it, an imp and no warnings.qa_sync_check12/12,qa_newall pass, 35/35 level starts inlevels_sweep(the only new warning kind was the Castability one, now gone; the rest differ by #5454's corrected file names), 300 s soaks on levels 10 and 20 alive with no new warning kinds. On the build before 34e50b3 also:qa_commonclean, OpenGL smoke 4/4, three resolution switches without a black screen.Closes #25.