An --update pass overwrites the previous WARC and leaves a page-less WACZ

[xroche]

A second crawl into the same output with --warc-file replaces the previous archive instead of adding to it, so the bodies the first crawl captured are gone, and the --wacz package built from it comes out empty.

Measured on master + #754, against tests/server-root/mini304:

httrack -O out URL/mini304/index.html --warc-file warc-out --wacz
  warc-out.warc.gz  3 response + 3 request + warcinfo
  warc-out.wacz     pages.jsonl has 2 page rows

httrack -O out URL/mini304/index.html --warc-file warc-out --wacz --update
  warc-out.warc.gz  1 response (robots.txt) + 2 revisit + warcinfo, file replaced
  warc-out.wacz     pages.jsonl has 0 page rows

After the update pass the revisit records point at payloads that exist nowhere, and a replay tool opening the WACZ finds no pages at all. warc_wacz_package() already keeps the existing package when the build fails ("kept existing %s untouched"), but nothing notices that the package it is about to move into place carries no pages.

This is why 74_local-warc-wacz.test drives its second pass with --rerun-args '-C0' instead of --rerun: a cacheless pass re-downloads and produces a real package, while the update pass produces one that fails wacz-validate with "pages.jsonl has no page rows after the header". The workaround is in the test because the test needed a second package to clobber, not because the update behaviour is fine.

tests/local-crawl.sh already snapshots the first-pass .warc.gz before a rerun for the same reason, so the overwrite has been known at the test level for a while without being decided at the engine level.

What --update should mean for WARC output is the actual question: append to the existing archive, write a new segment per run the way --warc-max-size already does, or refuse to overwrite an archive holding records the new run does not have.

[xroche]

PR #777 stops the destruction but does not answer the question in your last paragraph. Here is what I found on the three options. The decision is yours; I have a preference but did not act on it.

First, a measurement that reframes all three. Neither pywb nor wabac.js reads WARC-Profile. Both dispatch revisit resolution on WARC-Refers-To-Target-URI alone (pywb resolvingloader.py, wabac.js warcloader.ts), and neither looks at WARC-Refers-To. Our 304 revisits carry none of those fields, and our CDXJ line for them says "status": "200" with no digest. So even with the original responses preserved in the same package, today's revisit records resolve to "Archived Page Not Found". Filed as #778. Append and segment-per-run are both worth little until that lands; refuse is the only option that does not depend on it.

Append. Spec-legal and better supported than I expected. WARC 1.1 Annex D blesses record-at-time gzip members, and 6.2 says warcinfo records may appear anywhere "to allow the concatenation of WARC files into a larger valid WARC file"; RFC 1952 2.2 makes the gzip side append-safe. Existing record offsets stay valid, so the CDXJ only needs the new members indexed and merged rather than rebuilt. The catch is that CDXJ 0.1.0 makes sorted order a MUST, so a plain >> onto the index silently breaks binary search and returns false negatives instead of errors. In the ecosystem it is the road less travelled: of the writers I read, only wpull offers it, opt-in via --warc-append. For WACZ it saves nothing, since datapackage.json carries a sha256 and byte count per file and any change forces a full repackage.

Segment per run. The ecosystem consensus and the smallest change here. Heritrix, warcprox and browsertrix all mint a new file per run rather than touching the old one; browsertrix's default output is already a multi-WARC WACZ. The WACZ spec requires archive/ to hold "one or more files", CDXJ carries filename per line, and cross-file revisit resolution is the routine path in both replay engines (pywb re-queries the whole index; wabac.js lookupUrl has no file scoping). We already have most of it: %s-%05u segment naming, per-line CDXJ filename, and a WACZ packer that already loops over segments. The missing piece is scanning for the first free serial at startup, which is exactly what wpull does. Cost: the CDXJ still needs a merge and re-sort across runs, and segments accumulate until someone prunes them.

Refuse. Has real precedent for the default (grab-site raises FileExistsError, Heritrix throws on an existing launch dir, browsertrix needs an explicit --overwrite), and it is the only option with no index work at all. But both of those tools pair refusal with a unique default target, so their users never hit it. Refusing without a rotation fallback breaks the plain httrack --update workflow, which is what #777 does today, and the cost is measurable: I ran a pass where index.html changed, a.html 304'd and a new b.html appeared. All three landed in the mirror, and the guard dropped the whole archive, so the two genuinely new captures reached no WARC at all.

A fourth option I considered and rejected: write a full response record from the cached body instead of a revisit. Every run then stands alone and pages reappear, but it re-stores every unchanged body on every run, defeats revisit dedup entirely, and records a response the server did not send.

My recommendation is segment per run, with #778 first. It is what the tools we interoperate with do, it is what WACZ was designed for, it needs the least new code here, and unlike refuse it keeps the update workflow working. Append's only advantage over it is one fewer file, and it buys that with a sorted-index invariant that fails silently when broken.

Worth noting for whichever way this goes: wget has the same bug. warc_start_new_file() opens the WARC "wb+" with no existence check, while opening the CDX "a+", so a second run truncates the archive and appends to the index.