Beacon includes a spider crawler that follows links across your site using Chrome to render each page (so JS-rendered links are captured). It uses PHP Fibers for concurrent page loading — multiple Chrome sessions load pages in parallel, bounded by maxConcurrent.
It tracks every URL it finds, whether internal or external, where each was linked from, and the HTTP status code.
$results = beacon()->visit('https://example.com')->crawl();
// How many URLs found
echo $results->count();
// Iterate over results
foreach ($results as $url => $result) {
echo "{$url} — {$result->statusCode}\n";
}Pass a CrawlConfig to control the crawl behaviour:
use Myerscode\Beacon\Crawler\CrawlConfig;
$config = (new CrawlConfig())
->maxDepth(4) // Only go 4 levels deep (default: 5)
->maxConcurrent(3) // 3 Chrome instances in parallel (default: 5)
->timeout(15) // Page load timeout in seconds (default: 30)
->maxRetries(2) // Retry failed pages twice (default: 0)
->requestDelay(500) // 500ms delay between requests (default: 0)
->exclude(['/admin', '/logout', '.pdf'])
->shouldCrawl(fn (string $url) => !str_contains($url, 'private'));
$results = beacon()->visit('https://example.com')->crawl($config);Patterns are matched with str_contains against the full URL:
$config = (new CrawlConfig())->exclude([
'/admin', // Skip admin pages
'/api/', // Skip API endpoints
'.pdf', // Skip PDF files
'logout', // Skip logout URLs
]);The shouldCrawl closure receives the URL and returns a bool. It runs after exclude patterns, so excludes take precedence:
$config = (new CrawlConfig())->shouldCrawl(function (string $url): bool {
// Only crawl blog pages
return str_contains($url, '/blog');
});Failed page loads can be retried automatically:
$config = (new CrawlConfig())->maxRetries(2);Each retry yields to other Fibers so the crawl continues progressing while waiting.
You can configure exponential backoff for retries:
$config = (new CrawlConfig())
->maxRetries(3)
->retryDelay(1000) // 1 second base delay (default: 1000ms)
->retryBackoff(2.0); // Double the delay each attempt (default: 2.0)With these settings, retries wait 1s, 2s, then 4s before each attempt.
Add a delay between requests to avoid overwhelming the target server:
$config = (new CrawlConfig())->requestDelay(500); // 500ms between requestsJitter adds randomness to delays, preventing multiple crawlers or retries from hitting the target in lockstep (the "thundering herd" problem). Inspired by AWS Step Functions error handling, the FULL strategy randomizes the wait between 0 and the calculated interval.
Apply jitter to request delays:
use Myerscode\Beacon\Crawler\JitterStrategy;
$config = (new CrawlConfig())
->requestDelay(500)
->requestWithJitter(); // Randomize between 0–500ms per requestApply jitter to retry delays:
$config = (new CrawlConfig())
->maxRetries(3)
->retryDelay(1000)
->retryBackoff(2.0)
->retryWithJitter(); // Randomize: 0–1s, 0–2s, 0–4sCombine both for maximum spread:
$config = (new CrawlConfig())
->maxRetries(3)
->retryDelay(1000)
->retryBackoff(2.0)
->retryWithJitter()
->requestDelay(500)
->requestWithJitter();You can also explicitly pass the strategy enum:
$config = (new CrawlConfig())
->requestWithJitter(JitterStrategy::FULL) // Randomized (default)
->retryWithJitter(JitterStrategy::NONE); // Deterministic (no jitter)Use onCrawled to get notified as each URL is processed:
use Myerscode\Beacon\Crawler\CrawlResult;
$config = (new CrawlConfig())
->onCrawled(function (string $url, CrawlResult $result): void {
$type = $result->internal ? 'internal' : 'external';
$status = $result->statusCode ?? '---';
echo "[{$status}] ({$type}) {$url}\n";
});
beacon()->visit('https://example.com')->crawl($config);The CrawlResultCollection provides several ways to filter and query results:
$results = beacon()->visit('https://example.com')->crawl();
// All results
$results->all();
// Only internal URLs
$results->internal();
// Only external URLs
$results->external();
// URLs with a specific status code
$results->withStatus(404);
// All broken links (4xx and 5xx)
$results->broken();
// Check if a specific URL was found
$results->has('https://example.com/about');
// Get a specific result
$result = $results->get('https://example.com/about');Export results for storage or reporting:
$results = beacon()->visit('https://example.com')->crawl();
// As a plain array
$array = $results->toArray();
// As JSON
$json = $results->toJson();
// Save to file
file_put_contents('/tmp/crawl-results.json', $results->toJson());Each CrawlResult contains:
$result = $results->get('https://example.com/about');
$result->url; // 'https://example.com/about'
$result->internal; // true (same domain as start URL)
$result->statusCode; // 200 (null for external URLs that weren't visited)
$result->linkedFrom; // ['https://example.com', 'https://example.com/contact']
$result->depth; // 1 (how many clicks from the start URL)$results = beacon()->visit('https://example.com')->crawl();
foreach ($results->broken() as $url => $result) {
echo "Broken: {$url} (HTTP {$result->statusCode})\n";
echo " Linked from:\n";
foreach ($result->linkedFrom as $source) {
echo " - {$source}\n";
}
}- Uses Panther (Chrome) to render each page, so JS-generated links are captured
- PHP Fibers run page loads concurrently across multiple Chrome sessions
- Resolves relative URLs, protocol-relative URLs, and root-relative URLs
- Strips fragments (#) and normalises trailing slashes to avoid duplicates
- Skips
mailto:,tel:,javascript:, and empty hrefs - Records external links but doesn't follow them
- Shares a single ChromeDriver process across all sessions
| Method | Description |
|---|---|
maxDepth(int $depth): self |
Max link depth to follow (default: 5) |
maxConcurrent(int $n): self |
Chrome instances in pool (default: 5) |
timeout(int $seconds): self |
Page load timeout (default: 30) |
maxRetries(int $n): self |
Retry failed pages (default: 0) |
retryDelay(int $ms): self |
Base delay between retries in ms (default: 1000) |
retryBackoff(float $rate): self |
Backoff multiplier for retries (default: 2.0) |
retryWithJitter(JitterStrategy): self |
Randomize retry delays (default: NONE) |
requestDelay(int $ms): self |
Delay between requests in ms (default: 0) |
requestWithJitter(JitterStrategy): self |
Randomize request delays (default: NONE) |
exclude(array $patterns): self |
URL patterns to skip |
shouldCrawl(Closure $fn): self |
Custom filter closure |
onCrawled(Closure $fn): self |
Callback after each URL is processed |
| Method | Description |
|---|---|
all(): array |
All results |
internal(): array |
Internal URLs only |
external(): array |
External URLs only |
withStatus(int $code): array |
Filter by status code |
broken(): array |
4xx and 5xx results |
has(string $url): bool |
Check if URL exists |
get(string $url): ?CrawlResult |
Get specific result |
count(): int |
Total URL count |
toArray(): array |
All results as plain array |
toJson(int $flags): string |
All results as JSON string |
| Property | Type | Description |
|---|---|---|
url |
string |
The URL |
internal |
bool |
Same domain as start URL |
statusCode |
?int |
HTTP status (null if not fetched) |
linkedFrom |
string[] |
Pages that link to this URL |
depth |
int |
Depth from start URL |