Bot verification

Identification and signals

The feed helpers use fixed official URLs, five-second deadlines, a one-MiB response ceiling and at most 10,000 validated prefixes. Refresh only runs in background tasks. Refresh before snapshot expiry; refresh failure does not extend expired evidence. Bing lookups are deduplicated with bounded pending work/cache size. Positive cache age is at most 30 seconds or a shorter observed forward-record TTL; Node's reverse lookup does not expose the PTR TTL.

Supported provider and class IDs

Seven provider integrations are available; Meta requires the separate trusted WAF configuration described below. Verification identifies a supported network source. Operators still choose class/route policy and measured budgets. Adding a feed does not activate an allowance or bypass rate/work caps. Unsupported or ambiguous claims retain conservative treatment.

Googlebot covers its desktop/mobile Googlebot/ identity. AdsBot desktop and mobile-web identities share adsbot-google and the special-crawler feed; retired mobile-app identifiers are not recognized. Google-Extended and Applebot-Extended are not treated as HTTP crawler identities. Other Google products and bots without an implemented proof mechanism are not implicitly verified.

Perplexity’s documentation currently links .com feed URLs that redirect to the .ai endpoints above. The integration uses the inspected canonical .ai destinations and rejects redirects during refresh. A future URL/schema change must be reviewed before changing the built-in catalog.

import { CRAWLER_FEEDS, refreshCrawlerFeed, refreshCrawlerFeeds } from '@valiron/crawler-protection';

// Background startup/schedule, never within a website request:
const status = await refreshCrawlerFeeds(registry, { ttlMs: 60 * 60 * 1000, concurrency: 3 });
// Returns each built-in class as 'updated' or 'unavailable'. Monitor failures.
// Individual refresh remains available:
await refreshCrawlerFeed(registry, 'googlebot', { ttlMs: 60 * 60 * 1000 });

CRAWLER_FEEDS is an immutable catalog of nine class-to-provider/URL entries. refreshCrawlerFeed accepts a catalog class ID, not a URL. refreshCrawlerFeeds bounds concurrency (default three, maximum eight), isolates failures and preserves an older validated snapshot only until its original expiry. OPENAI_FEEDS and refreshOpenAiFeed remain compatible OpenAI-only exports. The standalone gateway refreshes all nine feeds every five minutes with a one-hour TTL; feed network I/O is outside the admission path. In-process applications schedule the helper themselves.

To configure budgets, put the provider ID and its class IDs under policy.providers. For Google, configure separate googlebot and adsbot-google class rates under the shared google provider rate. For Perplexity, configure perplexitybot and perplexity-user separately under perplexity. Optional crawlerAccess entries can deny a class on specific groups. Verified classes absent from the configured provider budgets still use the conservative unknown-traffic budget. Dashboard filters and aggregate reporting use these configured IDs and show the actual verification/action; identifying a provider does not grant it unrestricted access.

Only verified providers/classes with an explicit allowance receive their crawler budgets. Other requests use the conservative unknown-traffic budget and retain global/IP/route caps. A mismatch is not automatically a 403. Verification establishes network provenance, not human identity, software behavior, intent or unlimited permission.

Meta classes can use verified AWS WAF evidence over an authenticated origin path. This requires WAF to reject viewer-supplied protected headers before it inserts a class only on matching verified-bot and bot-name labels. A Meta user-agent or broad ASN is insufficient. Real Meta fetch/ad diagnostics remain unverified. Cryptographic crawler signatures, learned behavioral reputation and a general bot/human detector are not implemented. Do not substitute a broad ASN/cloud range for crawler-specific evidence.

Broader crawler directory

The standalone gateway also refreshes the Verified Bots Directory in the background every five minutes. It imports a broader discovery catalog without treating directory membership as permission. Existing official feed, Bing DNS and trusted Meta WAF verifiers remain authoritative. Webhooks, overlapping official identities and unsupported UA patterns are excluded. Only literal UA names with optional version suffixes are recognized; remote regular expressions are never executed.

Directory data attribution: Verified Bots Directory by Mayank Gupta (https://verifiedbots.dev), CC-BY-4.0. This is an independent third-party catalog. Its resolved ranges are directory evidence, not a direct official-provider verification. No directory guarantees complete bot coverage.

To enable identity verification for additional crawlers:

1. Run node gateway/directory.cjs > directory-review.json from the supplied package directory, or run the equivalent file in the supplied container. This downloads the current catalog and emits candidates, operator/docs links, stable provider/class keys and approval hashes. It does not activate anything.
2. Review each selected bot's operator documentation and network scope. Create a JSON object mapping selected bot IDs to their exact 64-character approval hashes. Never approve the whole catalog automatically.
3. Set CRAWLER_DIRECTORY_APPROVALS_FILE to that file's mounted, read-only path and restart the gateway. With Compose, explicitly mount it into the container and set the environment variable; it is not mounted by the default template.
4. Add the returned provider and crawlerClass to the site's explicit rate/class policy and optional route access rules. For example, directory ID ahrefsbot uses provider directory-ahrefsbot and class ahrefsbot. Follow the existing policy-validation and explicit recovery/activation procedure after changing policy.

Approval binds the bot ID, operator URL, documentation, category, UA patterns, verification recipes and tier. Changed identity metadata requires a new review/hash. CIDR ranges from an unchanged approved recipe can refresh automatically. Only reviewed entries with CIDR evidence qualify; DNS-only, ASN-only, signature-only and listed-only entries receive no new identity privilege. directory_cidr evidence still obeys customer route/rate/work limits. Even a verified directory crawler keeps unknown limits until its provider/class budget is configured. Directory keys identify individual bots, not an aggregate organization quota.

Downloads use one fixed HTTPS URL, reject redirects, time out after ten seconds and cap the body at 4 MiB, with bounded entries/ranges. URLs and DNS/signature recipes from the directory are never fetched or executed. Approval validation rejects invalid or overly broad CIDRs. Replacements install atomically; failed refreshes retain the previous snapshot only to its original expiry. Evidence expires after one hour at most, and a directory generation older than 48 hours or older than the installed generation is rejected. A failed refresh logs crawler_directory_unavailable; unknown/expired evidence remains subject to conservative limits. Downloads run outside request admission.

For in-process integrations, construct new CrawlerDirectory(reviewedApprovals), attach it with registry.setDirectory(directory), and schedule refreshCrawlerDirectory(directory) independently of refreshCrawlerFeeds(registry, ...). These exports are available from @valiron/crawler-protection; applications supply their own refresh scheduler and error reporting. directory.review() reports recognized candidates and approval status. SDK integrations do not fetch a directory merely by constructing a registry.