fix(crawler): recycle the PKC client after every crawl pass - #3
Conversation
5archive-server-1 restarted 143 times in the 23 hours since it moved to v0.4.0: every ~8 minutes Node aborted with "Reached heap limit ... JavaScript heap out of memory" at the 1 GiB cap (exit 134), and seeditarchive hit the same wall at its 512 MiB cap. No requests, no new comments, no errors: the crawl loop alone leaked about 2.4 MB/s. pkc-js keeps what a crawl loads (community instances, their pages, the verification caches behind them) for the life of the PKC instance, and none of it is reachable from anything the crawler holds. Reproduced inside the container with the crawler's exact page walk over a 721-comment board: the heap grew ~2 MB per crawl with no growth in any enumerable pkc-js map, and destroying and recreating the PKC instance between crawls kept it flat. 68 boards refreshed every few seconds is ~20 MB per pass, hence the cap. So a pass that crawled anything now retires the client on the way out and the next pass reconnects (a local WebSocket, a few hundred ms). The recycle is bounded to 10 s so a daemon that will not close the socket cannot stall the loop, and it is quiet: the "[pkc] connected" line is kept for connects that follow a timeout or a daemon restart instead of printing once per pass. An idle pass leaves the cache alone. Cut as 0.4.1.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (4)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe crawler now recycles the PKC client after passes that process communities. Quiet reconnect handling suppresses routine connection logs. Tests verify client destruction after active work and no destruction during idle passes. The package version increases to ChangesPKC client recycling
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to Active crawler passes now recycle the PKC client to limit retained memory, while idle passes retain the existing client and shutdown is bounded. No merge-blocking production or user-impact risk is currently identified. Sequence Diagram(s)sequenceDiagram
participant CrawlerTick
participant Database
participant PKCClient
participant PKCConnection
CrawlerTick->>Database: process queued community
Database-->>CrawlerTick: crawled row
CrawlerTick->>PKCClient: resetPkcClient({ quiet: true })
PKCClient->>PKCConnection: reconnect with quiet flag
PKCConnection-->>PKCClient: connection result
PKCClient-->>CrawlerTick: complete recycling
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 3 files. (1 skipped: 1 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
5archive-server-1 restarted 143 times in the 23 hours since it moved to v0.4.0: every ~8 minutes Node aborted with
Reached heap limit … JavaScript heap out of memoryat the 1 GiB cap (exit 134), and seeditarchive hit the same wall at its 512 MiB cap. No requests, no new comments, no errors: the crawl loop alone leaked about 2.4 MB/s.pkc-js keeps what a crawl loads (community instances, their pages, the verification caches behind them) for the life of the PKC instance, and none of it is reachable from anything the crawler holds. Reproduced inside the container with the crawler's exact page walk over a 721-comment board: the heap grew ~2 MB per crawl with no growth in any enumerable pkc-js map, and destroying and recreating the PKC instance between crawls kept it flat. 68 boards refreshed every few seconds is ~20 MB per pass, hence the cap.
So a pass that crawled anything now retires the client on the way out and the next pass reconnects (a local WebSocket, a few hundred ms). The recycle is bounded to 10 s so a daemon that will not close the socket cannot stall the loop, and it is quiet: the
[pkc] connectedline is kept for connects that follow a timeout or a daemon restart instead of printing once per pass. An idle pass leaves the cache alone. Cut as 0.4.1.Verified: typecheck clean, 119/119 server tests (new regression test covers the recycle and the idle-pass no-op).
Note
Medium Risk
Changes end-of-pass PKC client lifecycle and reconnect behavior on the crawler’s critical path; mitigated by timeout bounds, generation guards, and a regression test.
Overview
Fixes a heap leak where
pkc-jsretained crawl-loaded data (~2 MB per board pass) until process OOM. After any crawl pass that actually indexed communities, the crawler now destroys and recycles the PKC client viaresetPkcClient({ quiet: true }), bounded by a 10s timeout so a stuckdestroy()cannot block the loop.Idle passes (nothing due) skip recycle to avoid pointless WebSocket reconnect churn. Routine recycles are quiet: they suppress the
[pkc] connectedlog so only timeouts or daemon restarts stay visible.tick()is exported for tests; a new regression test asserts destroy runs once after a crawling pass and not again on an idle second pass. Server version 0.4.1.Reviewed by Cursor Bugbot for commit 35c494a. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
Bug Fixes
Tests
Chores