Storing warc files?

[Shrinks99]

Hey there! I noticed in this reply you (Mikkel) mention that you're crawling to WARC files. Are you keeping these around after indexing? I imagine the storage requirements for that would get quite large... but there are some cool web archiving possibilities / serving cached pages if you are!

[mikkeldenker]

Hi!
Yes, the warc files for all the pages in the production index are stored in a self-hosted minio bucket.
The idea is indeed to use S3 range requests to provide a link to the cached page. This is a bit further down the road though, as it would also require us to store all the assets from the page (images, css, js) which we don't currently do and that would increase the storage requirements quite substantially.

[Shrinks99]

Coooooool!

I'll link you to WACZ then if you haven't heard about it. The TL;DR is that it's WARC files with an index in a ZIP. Not really a requirement if your WARC file only has one record, but if you start storing other assets from the page while crawling it can take a while for our replay viewer (ReplayWebpage) to parse everything in the file and build the index.

Aside from capturing all the data, if you did implement ReplayWebpage it would be pretty close to having Google's (now removed) cached pages feature with hopefully little work? With WACZs content is loaded with range requests from inside the archive file so that saves a lot on bandwidth!

I will flag as somebody that does a lot of high-fidelity web archiving, your crawls would get quite large. 🙃 Storing just the HTML and CSS could be a good middle ground that would get you reasonably far? JS wouldn't be a bad idea but you'd probably want some form of deduplication — you'd be storing a lot of JQuery files haha.

Also full disclosure, I work on all the tools mentioned above :P

[mikkeldenker]

This seems very interesting indeed, thank you for pointing me in this direction!

Do you know if there are any benchmarks of the read performance as a function of the archive file size? I don't know exactly how the cdx files are read, but I imagine the lookup performance will be quite a bit slower if I have, say, 5000 warc files in a single wacz archive compared to 10.

[edsu]

The issue is probably more the number of URLs in the index rather than the size or quantity of the WARC files? I wonder if @ikreymer has done benchmarking like this before?

[Shrinks99]

I know in ReplayWebpage we don't load them all client side for search — which also results in a bunch of UX issues — presumably for this reason.