scraper: update VirtualRealPorn for the new site - #2244
Open
DarXV05 wants to merge 1 commit into
Open
Conversation
The site moved to Laravel Livewire and every selector the scraper used is gone, so it matched nothing, left SceneID empty and returned without emitting a scene or an error. Verified against a live scene page: the old scene-ID, tag, gallery and download-link selectors all return zero elements. Builds on xbapps#2233 by ShadowOps, which covers the selector migration, with three further fixes found while verifying it against the live site: - Performer names carry a " VR" suffix on the new site. Stored verbatim they create a second actor record beside the existing one, so strip the suffix. - The JSON-LD VideoObject no longer has a "trailer" object, so the trailer config pointed at a field that no longer exists and resolved zero sources. The stream sources are on the dl8 player element instead, which is the shape badoink.go already handles: scrape_html over `dl8-video source`. - image.Decode is used to validate covers, but this package registers no image format decoder. It worked only because pkg/server happens to import image/jpeg; nothing in pkg/scrape guaranteed it, and a test binary that omits pkg/server rejected every cover. Register gif, jpeg and png explicitly. Co-authored-by: ShadowOps <1622411+ShadowOps@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #2233 by @ShadowOps, which covers the selector migration. Credited via
Co-authored-by.Problem
The site moved to Laravel Livewire and every selector the scraper used is gone. Checked against a live scene page:
script#virtualreal_video-streaming-js-extra(scene ID)script#virtualreal_download-links-js-extra(filenames)div.metaSingleData a span(tags)figure[itemprop="associatedMedia"] a(gallery)Scene ID comes from the first, and the scraper ends with
if sc.SceneID != "" { out <- sc }— so it emits nothing and returns no error. Silent failure.Additional fixes beyond #2233
Found while verifying that PR against the live site:
Performer names carry a
" VR"suffix on the new site. Stored verbatim, they create a second actor record alongside the existing one. Stripped.Trailers resolve zero sources. The PR keeps
TrailerType: "scrape_json"pointed attrailer.contentUrlin the JSON-LD, but theVideoObjectno longer has atrailerkey at all — its keys are@context, @type, actor, description, duration, genre, interactionStatistic, name, thumbnailUrl, uploadDate, url. The stream sources are on the player element instead, which is the shapebadoink.goalready handles:scrape_htmloverdl8-video source. Verified to return 5 sources with quality labels.Covers silently rejected in some builds.
image.Decodevalidates covers, butpkg/scraperegisters no image format decoder. It works in the shipped binary only becausepkg/server/heatmapproxy.gohappens to importimage/jpeg— nothing inpkg/scrapeguarantees it, and a build excludingpkg/serverrejects every cover. Registers gif/jpeg/png explicitly.Verification
Single-scene scrape against the live site:
Listing pagination also verified:
/videos/?page=1yields 20 scene links viaa.data-title.