borg2 check: implement repository repair (index and pack rebuild)

Refs #9696, #8466, #8476, #9901, #9825. Tracking: #9998.

Problem

The read-only repository check is done: it hashes index/ and packs/ objects via Store.hash,
checks the index first and the packs only if the index is intact, and never rebuilds the index on
that path. But when it finds damage, there is no repair. Repository.check(repair=True) just logs
and returns:

# repository.py, check()
elif repair:
    logger.error("Finished ... repository check, errors found (repository repair not implemented).")

and the corrupt-index branch has the placeholder for the missing part:

# TODO: --repair will rebuild the index from the packs here instead of stopping (refs #8572).
logger.error("Repository index is corrupted and must be repaired; skipping the pack check.")

A repo with a corrupt index/ fragment or a corrupt pack therefore cannot be repaired at the
repository level. This ticket splits that work out of #9696 ("... to be continued").

Scope

Repository-level repair only: rebuilding the chunks index and salvaging packs. Archives repair
(dropping archives with corrupt/lost metadata, recreating lost entries) already exists, and missing
file-content chunks heal on the next borg create. Both are out of scope.

Work items, in order

Command surface. Decide before writing repair code: keep it under borg check --repair, or
split off a read-only borg check and a separate borg repair with explicit targets
(--index, --packs PACKID...) (borg2: improve borg check #9696). The items below assume the existing check paths; only
the CLI shape depends on this.
Index repair from headers. Rebuild the chunks index by range-reading pack object headers and
skipping the payloads, recording chunk id -> (pack id, offset, size). Reading full packs is too
slow to be the routine repair path (borg2: avoid doing an unnecessary rebuild of the index #8476, to pack or not to pack ... #8572). Replaces the TODO in the corrupt-index branch.
Do not trust the headers blindly (borg2: index rebuild trusts pack headers #9901). Headers are unauthenticated, so a tampered header
(wrong id or sizes) would poison the rebuilt index. Authenticate on first real read (AEAD binds
the id to the content) and/or verify ids for mismatching entries. Part of item 2, not a follow-up.
Pack repair. For a pack that fails Store.hash, keep every object that still
AEAD-authenticates in a new pack, drop the corrupt pack, and update the index. Objects that no
longer decrypt are already gone; drop them from the index so the affected archives report missing
chunks and heal on the next create (borg2: avoid doing an unnecessary rebuild of the index #8476, to pack or not to pack ... #8572). Needs item 2.
Consume the persisted corrupt-pack list. A read-only check records corrupt pack ids in
cache/checked-packs (borg2 check: move from marker to set-of-checked-packs #9897; PR check: keep pack check results, add --max-age to reuse them #9925 makes them survive a completed check). Repair reads that
list and touches only the flagged packs, then clears their records once fixed (borg2: improve borg check #9696). Needs
PR check: keep pack check results, add --max-age to reuse them #9925 plus items 2 and 4.
Keep the rebuilt index (borg2: check [--repair] improvements #8466). ArchiveChecker.finish() deletes the cached chunks index
after repair, so the next command pays for a slow rebuild. Write the index rebuilt in item 2 back
instead. Needs item 2.
Repair progress (Add progress indicator to repository check loop #9443, Progress of Remote: checking segment file /whatever/data/17/17078... #8009). The read-only check shows per-loop percentages; the repair
paths (header scan, pack salvage, verify-data delete pass) need the same. Needs items 2 and 4.

Out of scope (tracked elsewhere)

Vanished-pack detection, cheap cloud check mode (borg2 check: vanished packs? #9898, PR check: keep pack check results, add --max-age to reuse them #9925).
Reporting/UX: findings+repairs summary (check: summarize findings, --repair: summarize repairs #2397, PR check: summarize findings and repairs #9891), grouped missing-chunk reports (Improve borg2 check reporting of missing chunks #9218,
PR check: report missing chunks grouped as chunk -> files -> archives, #9218 #9965), Ctrl-C at safe boundaries (Ctrl-C does not interrupt borg check #7893, PR check: handle Ctrl-C at safe boundaries (#7893) #9966).
Post-corruption docs and acknowledging known-lost chunks so check/compact stop returning exit 2
(borg2: improve dealing with repo corruption #9825).
Repair from a redundant related repo (borg check: doing a repair using redundant related repos #6584, milestone 2.1).

Data-safety constraints

Never delete an object after a single failed read; a second read must also fail first. Transient
network or RAM errors must not destroy data.
A rebuilt index must never claim objects that do not exist. A lying index breaks borg create
dedup (borg2: avoid doing an unnecessary rebuild of the index #8476).
An interrupted repair must leave the repository consistent (Ctrl-C does not interrupt borg check #7893).

check --repair: rebuild a corrupt repository index from the packs, bo…

Merge pull request #10048 from mr-raj12/check-repair-index-rebuild-10026

On point 1, where repair should live: I would suggest keeping it under borg check --repair instead of adding a separate borg repair command.

Reasoning - no new command surface to document and maintain, users already reach for borg check when something is wrong, and the repair work needs the same repository walk that check is doing anyway. So it will be additional flags under check rather than a new top level command.

Will proceed on this basis unless you would prefer it the other way.

OK, we can keep it as check --repair for now.

Status on master (0dd454d), a reproduction, and a proposal for the rest. Points marked (decide) need your call.

Done

Command surface: check --repair (Sep 12).
- 1. Index rebuild from pack headers, every object validated with the key: check --repair: rebuild a corrupt repository index from the packs, #10026 #10048, validate object headers when walking a pack (PackReader.iter_headers) #10083, check --repair: resync past corrupt object headers when rebuilding the chunks index #10094, index rebuild: abort cleanly on a corrupt object header, #10122 #10168, check --repair: validate the repository index rebuild with the key, #9901 #10369, remove the drop_corrupt_tail mechanism, refs #10369 #10378, check: make the chunk index rebuild interruptible, fixes #10042 #10382; check: always require the key #10385 makes every check load the key.
ArchiveChecker.finish() stores the rebuilt index; check --repair: re-read only the packs the repair wrote, refs #8466 #10379 narrows it to the packs the repair wrote.

Left

Pack salvage.
Use the checked-packs records during repair.
Progress output for the repair loops (Add progress indicator to repository check loop #9443).

Also: --repository-only --repair leaves index entries that point at missing packs.

Why 4 is a data-safety issue, not only a failing check

The rebuild's validator (object_validator) checks the header and the metadata slot, not the data slot. Reproduced with one flipped byte in a chunk's data slot:

borg check --repair            # rc 0, "corrupt pack(s) found" as a warning; the chunk stays indexed
borg check                     # rc 1, Corrupt pack: 9248e7c0...  (every run from now on)
borg create a2 src             # deduplicated size 566 B: the new archive reuses the corrupt chunk
borg extract a2                # rc 113, "replaced by all-zero data"

--verify-data --repair removes such a chunk. Damage outside any object (e.g. bytes appended to a pack) cannot be repaired at all: the pack keeps failing its store hash.

Proposal

Where. In Repository.check(repair=True), after the pack loop and before the rebuild gate. Every pack that failed its store hash in this run gets salvaged. Afterwards the gate is "every pack intact or salvaged", so a corrupt index gets rebuilt in the same run.

Authentication (decide). A second callback next to validate: authenticate(chunk_id, obj) = RepoObj.parse(chunk_id, obj, decompress=False, want_compressed=True, ro_type=ROBJ_DONTCARE). It decrypts and authenticates the data slot with the chunk id in the AAD, without decompressing or hashing the plaintext. Salvage copies the ciphertext unchanged and never gives content a new id, so the id check of the repair place is not needed.
Salvage refuses to run under BORG_WORKAROUNDS=authenticated_no_key, where decrypt does not verify the tag.

What is kept (decide). Walk the pack with iter_headers(validate) and keep only the objects authenticate accepts, including objects the index does not list. Everything else is dropped: objects that fail, bytes no index entry covers, and ranges the walk skips. This differs on purpose from compact_pack, which copies uncovered bytes unchanged.

Safety rules

Re-hash the pack just before salvaging it. If it is intact now, record it as ok and leave it alone.
Drop an object only if a second read straight from the store (bypassing _pack_cache) fails authenticate too.
A read error (OSError, StoreReadError from check: survive I/O errors while reading repository objects #10194) leaves that pack untouched.
If no object in the pack authenticates, leave the pack and report it. This matches the rebuild's refusal when not one object validates.

Order per pack

Store the new pack.
Set the index invalid marker.
Update the in-memory index: point the kept objects at the new pack, remove the dropped ones. Skip this if the index is corrupt; the rebuild follows.
Delete the old pack.
Remove its checked-packs record.

After all packs: store the index, delete the marker, save the records. An interruption leaves either the old pack indexed, or the marker set so the next use rebuilds. This resolves the TODO in the check() docstring.

Result. Dropped objects are data loss.

--repository-only then fails, like skipped byte ranges today.
A full check passes the drops to the archives phase (chunks_modified, or written_packs from check --repair: re-read only the packs the repair wrote, refs #8466 #10379), which reports the affected files; they heal on the next create.

Missing packs. With --repair, remove the index entries of missing packs and store the index. Then test_empty_repository can be un-skipped.

Recorded results (decide). Allow --repair --max-age. Corrupt records are always re-hashed already (entry.result and max_age). The rebuild gate (pack_files == len(pack_infos)) has to count reused intact records as intact.

Progress/Ctrl-C. Show a progress bar for the salvage loop, and stop only between packs.

Cleanup. The #8572 references in repository.py, check_cmd.py (code and help text) and the tests point to a closed issue; they will point here.

PRs, in order

Missing-pack index repair (small), after authenticate (and encrypt) index and cache store objects, fixes #9819, fixes #10235 #10406.
The salvage function plus unit tests.
Wiring into check --repair, passing drops to the archives phase, help text and docs; needs check --repair: re-read only the packs the repair wrote, refs #8466 #10379.
--repair --max-age.
Progress output; then check --repair: share one object header parser, stop re-validating unchanged packs #10318 point 3 (double rebuild); then salvage per pack for --verify-data defect chunks, replacing the per-chunk delete().

Separately: compact makes a corrupt pack pass check again, see #10410.

#10406, #10379 and #10414 are merged. #10411 (index entries of missing packs, PR 1 of the list above) and #10412 (compact / repo-compress skip packs recorded corrupt, #10410) are open.

Before I start the salvage PR, the three (decide) points from above:

Authentication: keep an object only if RepoObj.parse(id, obj, decompress=False, want_compressed=True, ro_type=ROBJ_DONTCARE) accepts it. OK?
Keep only authenticated objects, also ones the index does not list. Drop failing objects, uncovered bytes and skipped ranges (compact_pack copies uncovered bytes). OK?
Allow --repair --max-age, or keep --repair re-hashing every pack?

Unless you object, I'll do 1 and 2 as described and leave 3 out for now.

repository: salvage_pack calls before_old_pack_delete only if the old…

repository: salvage_pack review fixes, refs borgbackup#10026

check: --repair stores the index after a salvage also if it is empty,…

check: --repair reuses the intact pack check results within --max-age…

check: --repair salvages the packs recorded corrupt, refs borgbackup#…

check: --repair salvages only packs still present and keeps the pack …

check: --repair stores the index after a salvage also if it is empty,…

check: raise a dedicated error if objects can not be authenticated, r…

check: --repair stores the chunk index right after the pack salvage, …

check: --repair fails if a pack reads corrupt and then intact, refs b…

docs: the pack salvage removes packs and invalidates the stored chunk…

check: --repair reuses the intact pack check results within --max-age…

Merge pull request #10455 from mr-raj12/check-repair-wire-salvage-10026

check: --repository-only --repair ignores --max-age for any index err…

Merge pull request #10464 from mr-raj12/check-repair-max-age-10026

All seven work items are done on master (4e99898).

Command surface: check --repair, decided in this thread on 09-12.
- 1. Index rebuild from pack headers, every object authenticated with the key: check --repair: rebuild a corrupt repository index from the packs, #10026 #10048, validate object headers when walking a pack (PackReader.iter_headers) #10083, check --repair: resync past corrupt object headers when rebuilding the chunks index #10094, index rebuild: abort cleanly on a corrupt object header, #10122 #10168, check --repair: validate the repository index rebuild with the key, #9901 #10369, remove the drop_corrupt_tail mechanism, refs #10369 #10378, check: make the chunk index rebuild interruptible, fixes #10042 #10382, check: always require the key #10385.
Pack salvage: repository: add salvage_pack, keeping only the authenticated objects of a corrupt pack, refs #10026 #10419 adds Repository.salvage_pack, check: --repair salvages the packs recorded corrupt, refs #10026 #10455 wires it into check --repair.
The checked-packs records drive the repair: check: --repair salvages the packs recorded corrupt, refs #10026 #10455 salvages the packs recorded corrupt and drops their records, check: --repair reuses the intact pack check results within --max-age, refs #10026 #10464 lets --repair reuse intact results within --max-age and ignores --max-age when --repository-only finds an index error.
The rebuilt index is kept: check --repair: re-read only the packs the repair wrote, refs #8466 #10379, check: a full --repair rebuilds a corrupt index once, in the archives phase, fixes #10434 #10445.
Repair progress: check: show progress while --repair removes defect chunks and re-reads the written packs, fixes #9443 #10458.

The out of scope items stay with their own issues: #8476, #9696, #8466, #10410.

Navigation Menu

Repository navigation

borg2 check: implement repository repair (index and pack rebuild) #10026

borg2 check: implement repository repair (index and pack rebuild)

Problem

Scope

Work items, in order

Out of scope (tracked elsewhere)

Data-safety constraints

Done

Left

Why 4 is a data-safety issue, not only a failing check

Proposal

PRs, in order

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

borg2 check: implement repository repair (index and pack rebuild) #10026

Description

borg2 check: implement repository repair (index and pack rebuild)

Problem

Scope

Work items, in order

Out of scope (tracked elsewhere)

Data-safety constraints

Activity

mr-raj12 commented on Sep 12, 2026

ThomasWaldmann commented on Sep 12, 2026

mr-raj12 commented on Sep 23, 2026

Done

Left

Why 4 is a data-safety issue, not only a failing check

Proposal

PRs, in order

mr-raj12 commented on Sep 23, 2026

12 remaining items

mr-raj12 commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions