No single-file HTML output with inlined assets#720
Conversation
Add --single-file (-%Z): after the mirror completes, rewrite every saved page in place with its stylesheets, scripts, images and fonts embedded as data: URIs, while links between pages stay relative. The mirror remains a browsable tree and each page also stands alone. This cannot reuse the -%M path: MHT streams a MIME part per file as it is saved, but a data: URI needs the asset's bytes when the page is written, and pages are normally saved before their assets are fetched. The new pass runs over the finished tree at the tail of httpmirror(), after the update purge. Audio, video, page-to-page links and anything over --single-file-max-size (10 MB default) keep an ordinary link. References carrying a scheme are skipped, which covers data: and makes a second --update run a no-op. Resolution is clamped to the mirror root: the HTML is hostile input. htsopt.h gains two tail-appended fields; VERSION_INFO is untouched. Closes #713 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
Master's #709 took test number 82, so the single-file crawl test moves to 86. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Master took 82 through 85 while this branch was open. The engine self-test also moves to the end of the script: it and the crawl assertions cover different ground, and failing first hid the crawl half. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
sf_relative_from() indexed one byte past from_dir's terminator whenever the page's own directory was a prefix of the asset path, which is the ordinary mirror layout: an out-of-bounds read that also miscounted the ../ prefix and emitted a broken link. Two review agents hit the same ASan trace; the branch had no test because every nested asset in the fixture was under the cap. Also from review: an escaped quote no longer ends a CSS string early (which exposed its contents to the url()/@import scanner), @import url(...) now inlines like the quoted form, a raw-text element ends only on a real end tag rather than any prefix of one, an over-wide tag is copied through by the same quote-aware scan instead of a second quote-blind one, <!--> is an empty comment, whitespace before a tag's > survives, and a page cannot inline more than SINGLEFILE_MAX_PAGE_SIZE, which bounds the multiplicative @import fan-out. A failed encode no longer leaves a payload-less data: prefix behind, and base_dir is matched against the root on a component boundary. sf_readfile now wraps a new readfile2_utf8() rather than being a fifth copy of the readfile family. Docs place --single-file beside -%M instead of implying it supersedes it: MHT stays the better container, this wins on opening anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
copy_htsopt's two new fields, including the >0 guard that must not let an unset source clear the target's default; -%Z0; and a rejected --single-file-max-size argument falling back to the built-in cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
The Windows job builds httrack.exe only, and reaches this file through its *_local-*.test glob, so requiring htsserver failed the whole test there even though every crawl assertion had passed. Skip that half instead, and run the engine self-test before it so the skip cannot swallow it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
Master took test number 86 for the proxytrack cache-longfields test; both that and 87_local-single-file.test now sit in TESTS. The only conflict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
#722 gave slprintfbuff the attribute, so the over-wide-tag fixture's call became the one warning this branch adds over master's baseline. Handle it the way that PR's own self-test code does; the buffer cannot truncate here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
|
Two notes for anyone reading the commit list rather than the diff. The branch's first commit still says Separately, #726 tracks something this PR noticed but deliberately did not fix: |
…les (#739) The feature PR added the four LANG_SINGLEFILE* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Test 87 was taken by master (proxytrack-nodate) and 90 by the web GUI checkbox test, so the single-file test moves to 91. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
…overage The rewrite spools to a .sfnew temp and renames it over the page, which bypasses filecreate() and with it the HTS_ACCESS_FILE chmod the engine puts on every other mirrored file. Under a restrictive umask the pages came out 0600 while their assets stayed 0644, so a mirror served by a webserver or shared with a group lost read access on exactly the pages. chmod the spool before the rename. Two guards had no coverage: deleting the scheme/data: check in sf_resolve left both the self-test and the crawl test passing, because the fixtures resolved to paths that were absent either way, and the per-page inline budget was never exercised. The fixtures now plant a file where each guard's removal would land the walk, and a self-importing stylesheet measures the budget against a large-budget control. Charging that budget after the nested rewrite instead of before let an @import chain spend what its ancestors had already claimed and drove it negative; charge it up front and refund on failure. The attribute table missed the lazy-loading attributes hts_detect[] already downloads, so a modern page inlined almost nothing: add data-src, data-srcset, lowsrc, object@data and embed@src, and record why the rest stay links. The new web GUI checkbox had no hidden companion input, so it could be ticked but never cleared, which is what #725 fixed for every other box. Master's test 90 catches it once the branch merges. Adds a libFuzzer harness over the rewriter, since it re-serializes hostile HTML, and renames the crawl test to 91 now that master holds 87 and 90. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Master took test 91, so the single-file test moves to 92. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Recorded in htssinglefile.h already; the user-facing guide is where someone raising --single-file-max-size will look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Conflict in tests/Makefile.am only: master took 89, so the change-report test moves to 93 (PR #720 holds 92). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Its GUI half needs htsserver, which the Windows job does not build, so the test now exits 77 there instead of reporting a pass for assertions it never ran. The skip list is pinned, so it has to be declared. Both Windows jobs reported fail=0; only the list check was red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Master took test 92 and added an expected Windows skip; the single-file test moves to 93 and both skip lists are combined. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
HTTrack's only one-file output was
-%M, amultipart/relatedMHT archive. Firefox and Safari never read those, so you cannot hand someone an.mhtand count on it opening.--single-file(-%Z) produces ordinary HTML instead: once the mirror completes, every saved page is rewritten in place with its stylesheets, scripts, images and fonts embedded asdata:URIs. Links between pages stay relative, so the mirror is still a browsable tree and any single page can also be handed to someone on its own. The assets do end up on disk twice.-%Mstays, and the two are good at different things: MIME carries text parts without the base64 tax, keeps each resource's original URL and stores a shared asset once, while single-file HTML wins on opening by double-click anywhere. The help text and the guide say which to reach for; neither presents one as the successor.It cannot reuse the MHT path.
postprocess_file()streams a MIME part per file as it is saved, but adata:URI needs the asset's bytes at the moment the page is written, and pages are normally saved before their assets are fetched. So this is a pass over the finished tree, hooked at the tail ofhttpmirror()after the update purge. By then the engine has made every reference relative, so resolving one is path arithmetic against the page's own directory, clamped to the mirror root: the HTML is hostile input and a crafted../../..must not reach a file outside the project. A reference carrying a scheme is skipped, which coversdata:and makes a second--updaterun a no-op rather than a double-encode. Audio and video keep their links, as do page-to-page links and anything over--single-file-max-size(10 MB by default, reported in the log).htsopt.hgains two tail-appended fields; I have not touchedVERSION_INFO.An inlined stylesheet becomes a
data:URL, and per the URL Standard adata:URL has an opaque path. So a relative reference left inside one does not resolve: the parser returns failure, whatever path we write there. That only bites an asset the stylesheet referenced and the cap kept as a link, and raising--single-file-max-sizepast it inlines the asset too.htssinglefile.hand the docs record the limitation.-#test=singlefilebuilds a fixture mirror and asserts what is inlined, what keeps its link, the cap, the per-page inline budget, the mirror-root clamp and idempotence.tests/93drives a real crawl, decodes the emitted payloads with Python and compares them byte for byte against the served files, checks the rewritten page keeps the mode the engine gave it, and checks the webhttrack form still emits the right command line. The negative-case fixtures plant a real file wherever deleting a guard would send the resolver, so no assertion can pass just because the target was missing; I deleted each guard in turn and watched the suite go red. A libFuzzer harness covers the rewriter, since it re-serializes attacker-controlled HTML.The option is wired into the webhttrack Build tab, all 30 locales, the help text, a regenerated man page and the command-line guide.
Closes #713