Skip to content

Latest commit

 

History

History
280 lines (205 loc) · 8.15 KB

crawler.md

File metadata and controls

280 lines (205 loc) · 8.15 KB

Crawler

Beacon includes a spider crawler that follows links across your site using Chrome to render each page (so JS-rendered links are captured). It uses PHP Fibers for concurrent page loading — multiple Chrome sessions load pages in parallel, bounded by maxConcurrent.

It tracks every URL it finds, whether internal or external, where each was linked from, and the HTTP status code.

Basic Usage

$results = beacon()->visit('https://example.com')->crawl();

// How many URLs found
echo $results->count();

// Iterate over results
foreach ($results as $url => $result) {
    echo "{$url} — {$result->statusCode}\n";
}

Configuration

Pass a CrawlConfig to control the crawl behaviour:

use Myerscode\Beacon\Crawler\CrawlConfig;

$config = (new CrawlConfig())
    ->maxDepth(4)           // Only go 4 levels deep (default: 5)
    ->maxConcurrent(3)      // 3 Chrome instances in parallel (default: 5)
    ->timeout(15)           // Page load timeout in seconds (default: 30)
    ->maxRetries(2)         // Retry failed pages twice (default: 0)
    ->requestDelay(500)     // 500ms delay between requests (default: 0)
    ->exclude(['/admin', '/logout', '.pdf'])
    ->shouldCrawl(fn (string $url) => !str_contains($url, 'private'));

$results = beacon()->visit('https://example.com')->crawl($config);

Exclude Patterns

Patterns are matched with str_contains against the full URL:

$config = (new CrawlConfig())->exclude([
    '/admin',       // Skip admin pages
    '/api/',        // Skip API endpoints
    '.pdf',         // Skip PDF files
    'logout',       // Skip logout URLs
]);

Custom Crawl Filter

The shouldCrawl closure receives the URL and returns a bool. It runs after exclude patterns, so excludes take precedence:

$config = (new CrawlConfig())->shouldCrawl(function (string $url): bool {
    // Only crawl blog pages
    return str_contains($url, '/blog');
});

Retries

Failed page loads can be retried automatically:

$config = (new CrawlConfig())->maxRetries(2);

Each retry yields to other Fibers so the crawl continues progressing while waiting.

You can configure exponential backoff for retries:

$config = (new CrawlConfig())
    ->maxRetries(3)
    ->retryDelay(1000)    // 1 second base delay (default: 1000ms)
    ->retryBackoff(2.0);  // Double the delay each attempt (default: 2.0)

With these settings, retries wait 1s, 2s, then 4s before each attempt.

Throttling

Add a delay between requests to avoid overwhelming the target server:

$config = (new CrawlConfig())->requestDelay(500); // 500ms between requests

Jitter

Jitter adds randomness to delays, preventing multiple crawlers or retries from hitting the target in lockstep (the "thundering herd" problem). Inspired by AWS Step Functions error handling, the FULL strategy randomizes the wait between 0 and the calculated interval.

Apply jitter to request delays:

use Myerscode\Beacon\Crawler\JitterStrategy;

$config = (new CrawlConfig())
    ->requestDelay(500)
    ->requestWithJitter();  // Randomize between 0–500ms per request

Apply jitter to retry delays:

$config = (new CrawlConfig())
    ->maxRetries(3)
    ->retryDelay(1000)
    ->retryBackoff(2.0)
    ->retryWithJitter();  // Randomize: 0–1s, 0–2s, 0–4s

Combine both for maximum spread:

$config = (new CrawlConfig())
    ->maxRetries(3)
    ->retryDelay(1000)
    ->retryBackoff(2.0)
    ->retryWithJitter()
    ->requestDelay(500)
    ->requestWithJitter();

You can also explicitly pass the strategy enum:

$config = (new CrawlConfig())
    ->requestWithJitter(JitterStrategy::FULL)   // Randomized (default)
    ->retryWithJitter(JitterStrategy::NONE);    // Deterministic (no jitter)

Live Callbacks

Use onCrawled to get notified as each URL is processed:

use Myerscode\Beacon\Crawler\CrawlResult;

$config = (new CrawlConfig())
    ->onCrawled(function (string $url, CrawlResult $result): void {
        $type   = $result->internal ? 'internal' : 'external';
        $status = $result->statusCode ?? '---';
        echo "[{$status}] ({$type}) {$url}\n";
    });

beacon()->visit('https://example.com')->crawl($config);

Working with Results

The CrawlResultCollection provides several ways to filter and query results:

$results = beacon()->visit('https://example.com')->crawl();

// All results
$results->all();

// Only internal URLs
$results->internal();

// Only external URLs
$results->external();

// URLs with a specific status code
$results->withStatus(404);

// All broken links (4xx and 5xx)
$results->broken();

// Check if a specific URL was found
$results->has('https://example.com/about');

// Get a specific result
$result = $results->get('https://example.com/about');

Serialisation

Export results for storage or reporting:

$results = beacon()->visit('https://example.com')->crawl();

// As a plain array
$array = $results->toArray();

// As JSON
$json = $results->toJson();

// Save to file
file_put_contents('/tmp/crawl-results.json', $results->toJson());

CrawlResult Properties

Each CrawlResult contains:

$result = $results->get('https://example.com/about');

$result->url;        // 'https://example.com/about'
$result->internal;   // true (same domain as start URL)
$result->statusCode; // 200 (null for external URLs that weren't visited)
$result->linkedFrom; // ['https://example.com', 'https://example.com/contact']
$result->depth;      // 1 (how many clicks from the start URL)

Finding Broken Links

$results = beacon()->visit('https://example.com')->crawl();

foreach ($results->broken() as $url => $result) {
    echo "Broken: {$url} (HTTP {$result->statusCode})\n";
    echo "  Linked from:\n";
    foreach ($result->linkedFrom as $source) {
        echo "    - {$source}\n";
    }
}

How It Works

  • Uses Panther (Chrome) to render each page, so JS-generated links are captured
  • PHP Fibers run page loads concurrently across multiple Chrome sessions
  • Resolves relative URLs, protocol-relative URLs, and root-relative URLs
  • Strips fragments (#) and normalises trailing slashes to avoid duplicates
  • Skips mailto:, tel:, javascript:, and empty hrefs
  • Records external links but doesn't follow them
  • Shares a single ChromeDriver process across all sessions

API Reference

CrawlConfig

Method Description
maxDepth(int $depth): self Max link depth to follow (default: 5)
maxConcurrent(int $n): self Chrome instances in pool (default: 5)
timeout(int $seconds): self Page load timeout (default: 30)
maxRetries(int $n): self Retry failed pages (default: 0)
retryDelay(int $ms): self Base delay between retries in ms (default: 1000)
retryBackoff(float $rate): self Backoff multiplier for retries (default: 2.0)
retryWithJitter(JitterStrategy): self Randomize retry delays (default: NONE)
requestDelay(int $ms): self Delay between requests in ms (default: 0)
requestWithJitter(JitterStrategy): self Randomize request delays (default: NONE)
exclude(array $patterns): self URL patterns to skip
shouldCrawl(Closure $fn): self Custom filter closure
onCrawled(Closure $fn): self Callback after each URL is processed

CrawlResultCollection

Method Description
all(): array All results
internal(): array Internal URLs only
external(): array External URLs only
withStatus(int $code): array Filter by status code
broken(): array 4xx and 5xx results
has(string $url): bool Check if URL exists
get(string $url): ?CrawlResult Get specific result
count(): int Total URL count
toArray(): array All results as plain array
toJson(int $flags): string All results as JSON string

CrawlResult

Property Type Description
url string The URL
internal bool Same domain as start URL
statusCode ?int HTTP status (null if not fetched)
linkedFrom string[] Pages that link to this URL
depth int Depth from start URL