# Valiron Crawler Protection

Website guide: [Valiron Crawler Protection documentation](https://www.valiron.co/docs/crawler-protection). Browse the [quickstart](https://www.valiron.co/docs/crawler-protection/quickstart), [integration](https://www.valiron.co/docs/crawler-protection/integration), [bot verification](https://www.valiron.co/docs/crawler-protection/bot-verification), [policies](https://www.valiron.co/docs/crawler-protection/policies), [analytics](https://www.valiron.co/docs/crawler-protection/analytics), and [recovery](https://www.valiron.co/docs/crawler-protection/operations) guides.

**Plan: Pro (also available to Enterprise accounts).** Starting with SDK 0.2.0, supported runtime use requires your account’s server-side operator API key. The earlier 0.1.0 package does not contain this activation check.

Install `@valiron/crawler-protection` from npm with Node.js 22 or later:

```sh
npm install @valiron/crawler-protection
```

The package contains the standalone gateway, Node/Nuxt adapters, private work API contract and Python helper. It is separate from `@valiron/sdk`; installing the SDK does not install crawler verification or shared workload admission. Follow the deployment guide after installation.

The guard combines source verification, weighted route budgets and concurrency controls before protected server work. Crawlers do not need a Valiron account, agent key, wallet or blockchain identity.

## Website setup

Open **Dashboard → Valiron Crawler Protection** (`/dashboard/crawler-protection`). Free accounts see the Pro upgrade path. Pro and Enterprise accounts can copy integration instructions, validate a schema-v1 site configuration with an optional measured baseline, review rollout blockers and download the validated configuration scoped to their account. The server checks the current authenticated plan; a client-supplied plan does not unlock validation.

Validation and export do not save or deploy policy, connect to your Redis/database, or turn protection on. Install the integration package with `npm install @valiron/crawler-protection`. Your team installs the runtime and follows the deployment/recovery procedure in its own infrastructure. Optional runtime telemetry powers hosted crawler analytics. Automatic policy activation remains unavailable. A passing configuration check does not prove production capacity or genuine Meta acceptance.

## Start here

1. Sign up, upgrade to Pro (or use Enterprise), and create an operator API key in **Dashboard → API Keys**. Open **Dashboard → Valiron Crawler Protection** and install `@valiron/crawler-protection`. Keep the key in the server environment as `VALIRON_API_KEY`; never put it in browser code or the site configuration JSON.
2. Choose the standalone gateway for an HTTP website, or an in-process adapter for a Node application. Inventory the pages, APIs and expensive operations you need to protect.
3. Validate and export your configuration, then deploy the gateway/runtime with shared Redis in your infrastructure. Route protected traffic through it and prevent direct-origin bypass.
4. For query-level protection, wrap every protected database operation with the private work API or in-process admission. Start work only after admission succeeds; release capacity only after the operation actually stops.
5. Test observation mode, hard workload caps, application authorization and recovery on your own environment before enabling broader crawler access.

### Pro API-key activation (SDK 0.2.0+)

The standalone gateway requires `VALIRON_API_KEY`. In-process integrations call `await activateCrawlerProtection({ apiKey, tenant })`, then pass the returned `activation` to `createCrawlerRuntime()` or `CrawlerGuard`. Use the `operator_<account ID>` tenant from the dashboard export. Missing credentials, an invalid/revoked key, a Free account or a foreign tenant cannot activate the supported runtime.

Startup checks the current account through `POST /operator/crawlers/activation`. A valid lease lasts at most five minutes and refreshes every minute in the background. Admission checks use local memory, without an HTTP request per visitor or database query. A 401/403 refresh response invalidates the lease immediately; temporary network/5xx/429 failures preserve only its remaining lifetime. After expiry, new protected requests and operations fail closed with 503/errors. Existing operations may settle and release capacity; private release/recovery tools and explicitly bypassed static routes remain usable. Call `activation.stop()` on shutdown. `--check` validates configuration/credential syntax offline; it does not prove the account is paid.

The account API must be deployed before upgrading to the requiring SDK. Startup needs network connectivity; keys and leases are not written to disk. Activation is independent of optional analytics reporting. Public SDK checks can be bypassed in modified code, and 0.1.0 cannot be retroactively gated; hosted Pro endpoints enforce the account plan server-side.


## Website compatibility

Use the **standalone HTTP gateway** in front of an HTTP(S) origin, regardless of its frontend framework or backend language. The gateway service runs on Node.js 22+, while the website application can run on PHP, Python, Ruby, Java, Go or another HTTP-capable stack. In-process Node middleware and a Nuxt/Nitro adapter are also available.

| Layer | Integration | Verification scope |
| --- | --- | --- |
| Frontend | Gateway placement does not depend on React, Vue, server templates or another frontend framework. | A real Python origin and a Nuxt/Vue SSR fixture are covered. Every frontend/platform has not been tested individually. |
| Backend | Any origin that accepts HTTP/1.1 over HTTP or HTTPS can sit behind the gateway when you control routing and origin access. | Python behind the gateway and in-process Node/Nuxt are tested. PHP, Ruby, Java and Go can use the same HTTP boundary; separate native clients for them are not included. |
| Database | Admission wraps an operation through the private HTTP protocol or a Node callback, independently of the database client. | SQLite and MariaDB are tested. Other SQL/NoSQL systems require integration and driver-specific settlement/cancellation validation; Valiron does not automatically instrument their drivers or pools. |
| Runtime | The standalone service requires Node.js 22+ or its supplied container; the origin application does not need Node.js. | Node/Nuxt in-process adapters and a standard-library Python client are included. |
| Hosting/transport | Configure the CDN/reverse proxy to reach the gateway and restrict the origin. Terminate public TLS/HTTP2/HTTP3 at the ingress as needed. | This service's incoming listener is HTTP/1.1; its fixed upstream supports HTTP/1.1 with optional TLS. Hosted routing restrictions and other protocols need an integration. |

The npm package includes `gateway/service.cjs`, a Dockerfile, a Compose template and deployment instructions in `gateway/README.md`. Route origin-bound website traffic through the gateway, configure allowed public hosts and a fixed origin, authenticate gateway-to-origin requests, and restrict direct-origin access. Protect every relevant origin and backing API; traffic sent to an unprotected origin remains outside this deployment. A hosted platform must permit those routing/origin controls; there is no universal bypass of a platform's hosting restrictions. HTTP WebSocket upgrades, CONNECT tunnels and Expect/Continue are rejected by this gateway.

The gateway applies source checks, route/rate budgets and shared concurrent origin-request limits **before forwarding**. It streams HTTP responses and preserves application cookies and authorization. Viewer forwarding/internal headers are replaced; internal response headers are stripped. It does not follow origin redirects. Incomplete responses, origin timeouts and uncertain failures retain nonexpiring holds for confirmed-drain recovery. Browser disconnects do not release still-running origin work.

**Database work is a separate boundary.** Other backend languages call the private work API before starting each protected operation, then release only after confirmed settlement. The gateway forwards a signed request capability scoped to the original method/path/query, tenant/policy and expiry; the API authenticates the application and applies shared parent fanout and work limits. Its acquire/release protocol and included standard-library Python client are documented in `gateway/README.md`. No Node SDK is required inside the non-Node website. Gateway-only installation bounds HTTP requests, not every query or background job they may launch.

The work API has no public administrative or automatic hold-reset endpoint. Keep it private and authenticated. Releases work across broker replicas and remain possible after request expiry; missing/ambiguous release retains capacity. Existing authorization, connection-pool and query safeguards remain necessary.

## Placement

```text
Crawler/browser -> CDN/WAF
                       |-> eligible public cache hit
                       +-> gateway or middleware -> protected handler
                                                       |
                                              operation admission -> database
```

Place server-side admission before protected rendering, data fetching and API handlers. Client-side navigation checks run too late to protect server work. Wrap each expensive database operation, including parallel fanout, in work admission. For the included adapters, use Nuxt/Nitro server middleware before rendering and Node middleware before API handlers.

CDN cache hits never reach this guard and are not charged as origin work. Keep your CDN/WAF, existing authorization, bounded connection pools, query timeouts and cancellation controls. Unwrapped queries and routes that bypass middleware remain outside its protection. Bound SSR work independently.

## Reusable site configuration

The integration package now provides `compileCrawlerConfiguration()` and `createCrawlerRuntime()`. The same engine supports site-specific configuration for ecommerce and content routes; company names, URL discovery and database credentials are not part of the engine.

Schema version 1 contains `tenant`, `policy`, explicit `inventory`, `rollout`, optional `crawlerAccess` and optional public `cache` settings. Inventory entries declare IDs, HTTP methods, paths, protected groups or explicit bypasses. Path patterns support literal segments, `:parameter` and terminal `**`. GET entries include HEAD. Unknown fields, overlaps, missing groups, expired policy and ambiguous/unmapped targets reject. Every route group must have an inventory entry.

```ts
import { activateCrawlerProtection, createCrawlerRuntime } from '@valiron/crawler-protection';

const activation = await activateCrawlerProtection({
  apiKey: process.env.VALIRON_API_KEY!,
  tenant: reviewedSiteConfiguration.tenant,
});
const runtime = createCrawlerRuntime({
  activation,
  configuration: reviewedSiteConfiguration,
  redis: sharedAdmissionRedis,
  clientIp: trustedClientIp,
  customer: validateServerSideShoppingGrant,
});
app.use(runtime.nodeMiddleware);
// In protected handlers:
const product = await runtime.withDatabaseAdmission(
  req.valironCrawler!, () => queryProduct(req.params.id),
);
```

The named configuration, Redis, trusted ingress, shopping-grant validation and query functions are application dependencies. Runtime construction does not initialize or activate shared policy. Keep application authorization on checkout/account/write routes; matching an inventory entry never grants those permissions. Nuxt uses `defineEventHandler(runtime.nuxtMiddleware)`; wrap actual SSR with `runtime.withRenderingAdmission()`.

`rollout.mode = 'observe'` evaluates candidate crawler rates/access in separate buckets and records proposed decisions. It **still enforces** configured safety rates, customer rates, actual SSR/DB caps, fanout, expiry, trusted ingress and dependency failures. Existing CDN/WAF restrictions stay active. In enforce mode, only explicit `enforceGroups` enforce candidate policy; remaining catalog groups stay in observation. Work caps cannot be disabled by observation.

Optional crawler rules select `provider`, `crawlerClass`, route `groups` and `action: 'budget' | 'deny'`. Only verified catalog classifications qualify; validated customer grants keep their separate policy. These are origin-admission decisions: eligible public cache hits remain outside classification. Apply edge restrictions too if a class must be denied cached public content.

`runtime.publicResponse(request, admittedLoader, background?)` authenticates ingress, resolves inventory and applies configured public cache groups. The loader must admit before SSR and wrap every expensive operation. `runtime.report.snapshot()` returns bounded **process-local origin** aggregates for actual/proposed request decisions, wrapped work outcomes and cache events. It excludes raw IPs, URLs, UA, sessions and request IDs. Proposed throttles are not blocked requests; rejected callbacks are not a measured count of SQL queries saved. Reports cover guard-evaluated requests and wrapped work/cache events; early adapter rejections and edge traffic require existing ingress logs. Optional authenticated reporting connects these aggregates to the dashboard, as described below. Hosted policy deployment controls remain unavailable; Pro API-key activation authorizes SDK use, not Redis policy activation.

`planCrawlerRollout()` validates supplied pool/headroom/checkout measurements and returns rollout blockers; missing measurements do not generate guessed production limits. `createEcommerceConfiguration()` builds the same schema from operator-supplied routes and policy. Budgets, verified ingress and actual grant integration remain deployment responsibilities. Your team installs and operates the runtime in its own infrastructure.

## Identification and signals

| Signal | Implemented check | Boundary |
| --- | --- | --- |
| Client IP | Strict IPv4/IPv6 parsing and mapped IPv4 normalization; CloudFront helper authenticates the proxy path and configured hops | A caller-supplied forwarding header is not trusted source evidence. Restrict direct origin access. |
| OpenAI crawler claim | Match GPTBot, OAI-SearchBot or ChatGPT-User against its matching installed official CIDR snapshot | UA alone is spoofable. Feed refresh is a separate background operation. |
| Google crawler claim | Match Googlebot or AdsBot-Google against the matching common-crawler or special-crawler official feed | Search and advertising classes have separate feed snapshots and policy budgets. Generic Google Cloud ranges are not crawler evidence. |
| Applebot claim | Match Applebot against Apple’s official crawler CIDRs | Applebot-Extended is a robots.txt usage-control token, not a separate crawler identity. |
| DuckDuckBot claim | Match DuckDuckBot against DuckDuckGo’s official crawler CIDRs | A matching name without the matching source IP grants no privilege. |
| Perplexity claim | Match PerplexityBot or Perplexity-User against its own official feed | Search crawling and user-triggered fetching have separate classes. User-triggered does not establish shopper identity or authorization. |
| Bingbot claim | Bounded reverse DNS, approved hostname suffix and forward resolution back to the original IP | PTR alone is insufficient; cold requests initially receive conservative treatment. |
| Route cost and volume | Weighted global, route, IP, provider and crawler-class token buckets | Cost is application configuration, not a measurement inferred from the caller. |
| Concurrent work | Shared work slots, local concurrency ceiling and per-request operation bound | Every protected operation must use the wrapper. |

The feed helpers use fixed official URLs, five-second deadlines, a one-MiB response ceiling and at most 10,000 validated prefixes. Refresh only runs in background tasks. Refresh before snapshot expiry; refresh failure does not extend expired evidence. Bing lookups are deduplicated with bounded pending work/cache size. Positive cache age is at most 30 seconds or a shorter observed forward-record TTL; Node's reverse lookup does not expose the PTR TTL.

### Supported provider and class IDs

Seven provider integrations are available; Meta requires the separate trusted WAF configuration described below. Verification identifies a supported network source. Operators still choose class/route policy and measured budgets. Adding a feed does not activate an allowance or bypass rate/work caps. Unsupported or ambiguous claims retain conservative treatment.

| Provider ID | Class IDs | Official verification source |
| --- | --- | --- |
| `openai` | `gptbot`, `oai-searchbot`, `chatgpt-user` | [GPTBot](https://openai.com/gptbot.json), [SearchBot](https://openai.com/searchbot.json), [ChatGPT-User](https://openai.com/chatgpt-user.json) |
| `google` | `googlebot`, `adsbot-google` | [Common crawler CIDRs](https://developers.google.com/static/crawling/ipranges/common-crawlers.json), [special crawler CIDRs](https://developers.google.com/static/crawling/ipranges/special-crawlers.json); [verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) |
| `apple` | `applebot` | [Applebot CIDRs](https://search.developer.apple.com/applebot.json); [Apple’s guide](https://support.apple.com/en-us/119829) |
| `duckduckgo` | `duckduckbot` | [DuckDuckBot CIDRs](https://duckduckgo.com/duckduckbot.json); [DuckDuckGo’s guide](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot) |
| `perplexity` | `perplexitybot`, `perplexity-user` | [Bot CIDRs](https://www.perplexity.ai/perplexitybot.json), [user-fetch CIDRs](https://www.perplexity.ai/perplexity-user.json); [Perplexity’s guide](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| `bing` | `bingbot` | [Bing reverse/forward DNS verification](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) |
| `meta` | `social-preview`, `advertising`, `search`, `training`, `fetcher` | Authenticated AWS WAF verified-bot evidence and matching supported UA, as described below |

Googlebot covers its desktop/mobile `Googlebot/` identity. AdsBot desktop and mobile-web identities share `adsbot-google` and the special-crawler feed; retired mobile-app identifiers are not recognized. Google-Extended and Applebot-Extended are not treated as HTTP crawler identities. Other Google products and bots without an implemented proof mechanism are not implicitly verified.

Perplexity’s documentation currently links `.com` feed URLs that redirect to the `.ai` endpoints above. The integration uses the inspected canonical `.ai` destinations and rejects redirects during refresh. A future URL/schema change must be reviewed before changing the built-in catalog.

```typescript
import { CRAWLER_FEEDS, refreshCrawlerFeed, refreshCrawlerFeeds } from '@valiron/crawler-protection';

// Background startup/schedule, never within a website request:
const status = await refreshCrawlerFeeds(registry, { ttlMs: 60 * 60 * 1000, concurrency: 3 });
// Returns each built-in class as 'updated' or 'unavailable'. Monitor failures.
// Individual refresh remains available:
await refreshCrawlerFeed(registry, 'googlebot', { ttlMs: 60 * 60 * 1000 });
```

`CRAWLER_FEEDS` is an immutable catalog of nine class-to-provider/URL entries. `refreshCrawlerFeed` accepts a catalog class ID, not a URL. `refreshCrawlerFeeds` bounds concurrency (default three, maximum eight), isolates failures and preserves an older validated snapshot only until its original expiry. `OPENAI_FEEDS` and `refreshOpenAiFeed` remain compatible OpenAI-only exports. The standalone gateway refreshes all nine feeds every five minutes with a one-hour TTL; feed network I/O is outside the admission path. In-process applications schedule the helper themselves.

To configure budgets, put the provider ID and its class IDs under `policy.providers`. For Google, configure separate `googlebot` and `adsbot-google` class rates under the shared `google` provider rate. For Perplexity, configure `perplexitybot` and `perplexity-user` separately under `perplexity`. Optional `crawlerAccess` entries can deny a class on specific groups. Verified classes absent from the configured provider budgets still use the conservative unknown-traffic budget. Dashboard filters and aggregate reporting use these configured IDs and show the actual verification/action; identifying a provider does not grant it unrestricted access.

| Classification | Meaning |
| --- | --- |
| `verified` | The claimed crawler passed its configured source check. |
| `unverified` | No supported, unambiguous claim was identified. This does not mean human. |
| `claim_mismatch` | Available source evidence does not match the claim. |
| `verification_unavailable` | Required evidence is absent, expired or being resolved. |

Only verified providers/classes with an explicit allowance receive their crawler budgets. Other requests use the conservative unknown-traffic budget and retain global/IP/route caps. A mismatch is not automatically a 403. Verification establishes network provenance, not human identity, software behavior, intent or unlimited permission.

Meta classes can use verified AWS WAF evidence over an authenticated origin path. This requires WAF to reject viewer-supplied protected headers before it inserts a class only on matching verified-bot and bot-name labels. A Meta user-agent or broad ASN is insufficient. Real Meta fetch/ad diagnostics remain unverified. Cryptographic crawler signatures, learned behavioral reputation and a general bot/human detector are not implemented. Do not substitute a broad ASN/cloud range for crawler-specific evidence.

## Broader crawler directory

The standalone gateway also refreshes the [Verified Bots Directory](https://verifiedbots.dev/data-docs/) in the background every five minutes. It imports a broader discovery catalog without treating directory membership as permission. Existing official feed, Bing DNS and trusted Meta WAF verifiers remain authoritative. Webhooks, overlapping official identities and unsupported UA patterns are excluded. Only literal UA names with optional version suffixes are recognized; remote regular expressions are never executed.

Directory data attribution: **Verified Bots Directory by Mayank Gupta (https://verifiedbots.dev), CC-BY-4.0**. This is an independent third-party catalog. Its resolved ranges are directory evidence, not a direct official-provider verification. No directory guarantees complete bot coverage.

To enable identity verification for additional crawlers:

1. Run `node gateway/directory.cjs > directory-review.json` from the supplied package directory, or run the equivalent file in the supplied container. This downloads the current catalog and emits candidates, operator/docs links, stable provider/class keys and approval hashes. It does not activate anything.
2. Review each selected bot's operator documentation and network scope. Create a JSON object mapping selected bot IDs to their exact 64-character `approval` hashes. Never approve the whole catalog automatically.
3. Set `CRAWLER_DIRECTORY_APPROVALS_FILE` to that file's mounted, read-only path and restart the gateway. With Compose, explicitly mount it into the container and set the environment variable; it is not mounted by the default template.
4. Add the returned `provider` and `crawlerClass` to the site's explicit rate/class policy and optional route access rules. For example, directory ID `ahrefsbot` uses provider `directory-ahrefsbot` and class `ahrefsbot`. Follow the existing policy-validation and explicit recovery/activation procedure after changing policy.

Approval binds the bot ID, operator URL, documentation, category, UA patterns, verification recipes and tier. Changed identity metadata requires a new review/hash. CIDR ranges from an unchanged approved recipe can refresh automatically. Only reviewed entries with CIDR evidence qualify; DNS-only, ASN-only, signature-only and listed-only entries receive no new identity privilege. `directory_cidr` evidence still obeys customer route/rate/work limits. Even a verified directory crawler keeps unknown limits until its provider/class budget is configured. Directory keys identify individual bots, not an aggregate organization quota.

Downloads use one fixed HTTPS URL, reject redirects, time out after ten seconds and cap the body at 4 MiB, with bounded entries/ranges. URLs and DNS/signature recipes from the directory are never fetched or executed. Approval validation rejects invalid or overly broad CIDRs. Replacements install atomically; failed refreshes retain the previous snapshot only to its original expiry. Evidence expires after one hour at most, and a directory generation older than 48 hours or older than the installed generation is rejected. A failed refresh logs `crawler_directory_unavailable`; unknown/expired evidence remains subject to conservative limits. Downloads run outside request admission.

For in-process integrations, construct `new CrawlerDirectory(reviewedApprovals)`, attach it with `registry.setDirectory(directory)`, and schedule `refreshCrawlerDirectory(directory)` independently of `refreshCrawlerFeeds(registry, ...)`. These exports are available from `@valiron/crawler-protection`; applications supply their own refresh scheduler and error reporting. `directory.review()` reports recognized candidates and approval status. SDK integrations do not fetch a directory merely by constructing a registry.


## Shared policy

`RedisAdmission` atomically checks/debits all related rate buckets using Redis server time. A failed compound check does not partially debit the other buckets. Verified IPs share provider/class budgets, and crawler classes share a parent provider ceiling. Unknown traffic has both aggregate and per-IP budgets.

`GuardPolicy` requires:

| Field | Purpose |
| --- | --- |
| `version`, `expiresAt` | Policy identifier and absolute epoch-millisecond expiry. |
| `globalRate`, `unknownRate`, `perIpRate` | `{ capacity, refillPerSecond }` token buckets. |
| `providers` | Parent provider rate and explicitly allowed `classes`, each with its own rate. |
| `routes` | Named groups with a positive `cost` and rate; middleware cost must match. |
| `globalWorkLimit`, `unknownWorkLimit`, `providerWorkLimit` | Shared work-slot ceilings. |
| `localWorkLimit` | Concurrent operations per guard instance. |
| `maxOperationsPerRequest` | Maximum operations a context may attempt to start. |
| `maxContextAgeMs` | Context lifetime from 1 to 60,000 milliseconds. |

Unknown budgets cannot exceed configured provider/class budgets; route costs must fit burst capacities. Choose limits from measured peak workload and database headroom, not daily averages. Preserve parameters that change content, such as filters, pagination, currency and locale.

Use bounded Redis command timeouts, no offline queue and no automatic command retries. Eviction or lost state after failover can invalidate shared limits. Stop admission during uncertain recovery until prior work is drained/reconciled; retain independent pool/process limits. Automatic failover reconciliation is not implemented.

## Node integration

These exports belong to the integration package, not `@valiron/sdk`. Construct one `CrawlerGuard` with the started `activation` handle, a policy, `CrawlerRegistry`, `BingVerifier` and shared admission store. The integration uses `ManagedRedisAdmission` with an explicitly activated immutable policy. The remaining dependencies below belong to your application.

```ts
import {
  AdmissionError, createNodeCrawlerMiddleware, sendDenial,
} from '@valiron/crawler-protection';

app.use(createNodeCrawlerMiddleware({
  guard,
  clientIp: trustedClientIp,
  route: mapProtectedRoute, // { group, cost }, or undefined for unprotected paths
}));

app.get('/products/:id', async (req, res, next) => {
  try {
    const product = await guard.withDatabaseAdmission(
      req.valironCrawler!,
      () => queryProduct(req.params.id),
    );
    res.json(product);
  } catch (error) {
    if (error instanceof AdmissionError) return sendDenial(res, error.decision);
    next(error);
  }
});
```

Use the exported `GuardedRequest` type or application request augmentation in TypeScript. Register this handler only on routes covered by the mapping. The query promise must settle after the operation stops and releases its connection. An HTTP timeout does not by itself terminate database work.

`createCloudFrontClientIpResolver()` supports an authenticated distribution-to-origin path with configured trusted CIDRs and additional proxy-hop count. Other platforms need their own trusted-ingress adapter; never infer a trusted source from arbitrary client headers.

## Nuxt integration

```ts
// server/middleware/crawler.ts
import { createNuxtCrawlerMiddleware } from '@valiron/crawler-protection';
import { guard, trustedClientIp, mapProtectedRoute } from '../utils/crawlerGuard';

export default defineEventHandler(createNuxtCrawlerMiddleware({
  guard, clientIp: trustedClientIp, route: mapProtectedRoute,
}));
```

Admitted context is attached to `event.context.valironCrawler`; a denied response is ended before the adapter returns. The structural event adapter is tested, while actual Nitro cache ordering/render termination requires validation against the application's installed version.

Contexts belong to the guard instance that created them. Managed mode supports `InternalAdmission`: Nuxt issues a signed, short-lived child through `guard.forward()`, and Node uses `guard.acceptInternal()` on a protected internal route. Tokens bind issuer/audience, policy, method, exact path/query and raw body. The page quota is charged once; every DB operation still consumes shared fanout and work admission. A replay cannot execute another DB operation. Keep signing credentials and tokens on servers; a forwarded `alreadyAdmitted` flag never grants access.

Use `withRenderingAdmission(context, renderAndFetch)` to bound SSR separately from database operations, holding capacity until all actual work settles. Keep direct public API routes under ordinary request admission.

## Shopper isolation and catalog cache

HEAD requests can reuse an existing cached GET representation, with body handling left to the HTTP adapter. A HEAD cache miss runs the admitted loader without storing its result; a stale HEAD never starts regeneration. This prevents an empty HEAD response from replacing the HTML served to GET requests.

An optional customer policy reserves shopper and checkout work capacity and separates their rate budgets. Priority requires a trusted application callback that validates a server-side shopping/session/cart grant; arbitrary cookies, caller flags and user-agents do not qualify. Checkout priority is limited to explicitly configured checkout route groups. Use separate bounded DB pools as well. Anonymous shoppers without a validated grant still share conservative traffic limits; measure false throttles.

`CatalogCache` stores only eligible public HTML. Explicit meaningful query keys form the cache key; personalized/authenticated requests and unknown query parameters bypass it. Denials, private responses, Set-Cookie and Vary responses are never stored. The application must ensure public representations depend only on configured keys. Shared regeneration holds deduplicate a key and bound concurrent cold misses across distinct keys. Stale refreshes require an explicit background-lifetime hook and the same admitted loader. A busy cold miss returns an uncached 503.

## Decisions and recovery

`guard.admit()` returns `{ decision, context? }`. A decision contains `action`, `reason`, `requestId`, `policyVersion` and applicable `retryAfterSeconds`. The guard emits `allow`, `limit`, `unavailable` or `block`. An enforced deny rule for a verified crawler can emit `block`; applications can also pass a block decision to `sendDenial()`.

The Node adapter and `sendDenial()` use compact `{ code, requestId }` bodies:

| HTTP / code | Cause | Recovery |
| --- | --- | --- |
| 429 / `CRAWLER_LIMIT` | Rate budget exhausted | Honor `Retry-After`; do not rotate IP/identity to evade it. |
| 503 / `CRAWLER_UNAVAILABLE` | Admission cannot complete safely, ingress is untrusted or work capacity is exhausted | Honor retry guidance and investigate the decision reason. Protected work must not run. |
| 403 / `CRAWLER_BLOCK` | An enforced crawler deny rule or an application-defined block | Resolve policy; repetition does not grant access. |

Reasons include `rate_budget_exhausted`, `admission_unavailable`, `local_work_unavailable`, `work_store_unavailable` and `work_capacity_exhausted`. Work rejection throws `AdmissionError`, not the main SDK's `ValironError`. Rejections include `Cache-Control: no-store`; ensure the CDN does not cache caller-specific denials.

### Gateway and private work API errors

The gateway also returns `{ code }` for proxy-specific failures: 400 `INVALID_GATEWAY_REQUEST`, 408 `REQUEST_BODY_TIMEOUT`, 413 `REQUEST_TOO_LARGE`, 503 `GATEWAY_BODY_BUSY` or `ORIGIN_CAPACITY_EXHAUSTED`, 502 for incomplete/failed origin connections, and 504 `ORIGIN_TIMEOUT`. WebSocket upgrades return 426 and Expect/Continue returns 417. Failed admission does not forward the request. A timeout after forwarding can leave live origin work and retains its hold; repeating requests must not reclaim it.

The private work API requires service authentication and a valid signed request capability for acquisition. Invalid authentication returns 401; invalid capability scope/expiry returns 403; invalid JSON/fields return 400; oversized bodies return 413; non-JSON bodies return 415. Capacity, fanout and dependency failures return 503. Acquire only starts the operation on HTTP 200. Do not retry an ambiguous acquisition automatically. Release requires the returned permit and confirmed settlement; cross-worker release is idempotent. The exact contract is shipped in `gateway/work-api.openapi.json` inside the npm package; these endpoints are not part of Valiron's hosted public REST API.

Work holds do not expire automatically. A crash, client disconnect or HTTP timeout must not free capacity while a query might remain alive. Release follows actual operation settlement. For Node callback wrappers, a release error preserves the operation result, leaves the hold and emits `work_release_failed` through the decision callback. The Python helper surfaces release failures; its context manager is suitable only when returning/raising means the query stopped. For asynchronous drivers, acquire explicitly and release after confirmed completion or cancellation. Managed mode exposes hold inventory, pause, exact-ID reconciliation with confirmed-drain evidence, drained policy activation and resume. Reclaim orphaned holds only after admission is stopped and affected work is confirmed stopped. Missing control state or a mismatched policy fails closed. These controls do not prove that an old query stopped; that confirmation remains operational. Measured DB outcomes can open a shared catalog circuit; controlled probes and explicit recovery preserve shopper/checkout reservations.

## Validation boundary

A real Python HTTP origin executing SQLite queries is tested behind gateway/broker replicas, including pre-query rejection, shared limits, scoped capabilities, incomplete uploads, timeouts, truncated responses and disconnects. The protocol is database-independent, but these results do not prove every language, framework, driver, hosted platform or database has been tested.

Local tests cover spoof rejection, source parsing, middleware, shared Redis contention, operation bounds, dependency failure and retention of capacity after disconnect. An isolated Nuxt 4.1.3/Nitro Vue SSR → Node → MariaDB 10.11.8 fixture additionally covers physical checkout pools, actual SQL timeout/disconnect/process-death recovery, cache storms, replay/fanout and Redis outages. A controlled live website test exercised custom crawler requests, source-claim rejection, shared execution counters and refill around a real public-document read. The website test alone does not establish real crawler acceptance, search visibility or Meta previews. Local database results do not establish customer-scale capacity.

Validate every rendering/API/background path, cache behavior, actual query cancellation, store recovery and shopper latency under representative peak load before opening access broadly.

Related: [Quickstart](./QUICKSTART.md), [SDK reference](./SDK-REFERENCE.md), [Agent identity](./IDENTITY.md), [Errors](./ERRORS.md).


## Live dashboard analytics

Pro and Enterprise accounts can connect installed runtimes to **Dashboard → Valiron Crawler Protection**. It shows reported decisions by provider, verification state, route group and actual/proposed action; a 24-hour traffic timeline; gateway/work/cache metrics; shared database, render and gateway hold counts; reporting instances, configuration versions and modes; monitoring alerts; and a JSON export. It refreshes every 30 seconds and supports site, provider and route-group selection. Empty, stale and unavailable telemetry states are explicit. Configuration export alone does not connect or activate a deployment.

For a standalone gateway, set these **server-side** variables alongside your deployment configuration:

```text
CRAWLER_TELEMETRY_URL=https://valiron-edge-proxy.onrender.com/operator/crawlers/telemetry
VALIRON_API_KEY=<operator API key from Dashboard → API Keys>
# Optional CRAWLER_TELEMETRY_API_KEY overrides the activation key for reporting.
CRAWLER_SITE_ID=<stable site label, such as storefront>
```

Use the account-scoped tenant from your validated configuration export (`operator_<account ID>`). A key from another account cannot report that tenant. Keep API keys out of frontend code and configuration downloads. Site labels must contain 1–100 letters, digits, underscores or hyphens. Leave telemetry variables unset to keep reporting local. Each running process creates its own random instance ID; do not share an instance ID across replicas or counter resets.

For in-process integrations:

```typescript
import { createCrawlerTelemetry } from '@valiron/crawler-protection';
const telemetry = createCrawlerTelemetry({
  runtime,
  endpoint: 'https://valiron-edge-proxy.onrender.com/operator/crawlers/telemetry',
  apiKey: process.env.VALIRON_API_KEY!,
  site: 'storefront',
  onError: () => console.error('Crawler telemetry unavailable'),
});
await telemetry.flush(); // Initial counter baseline, outside the request path.
// On application shutdown: telemetry.stop();
```

Reports run in the background every 30 seconds, with a five-second HTTP timeout, no redirect following and no overlapping uploads. Reporting failures cannot grant access or release work. Cumulative counters recover measurements after transient upload failure while the collector baseline survives. The first accepted sample establishes a baseline and adds no traffic counts. Missing baselines after prolonged outages likewise start a new baseline. There is no disk spool: unsent measurements can be lost when a process exits. Reports exclude raw IPs, request URLs, user agents, sessions, request IDs and credentials. Verified-provider labels reflect runtime source checks; operator-authenticated uploads are customer-reported measurements, not independent crawler attestation.

The hosted collector retains rolling aggregates and last-seen instances for approximately 24 hours in Redis; it is not a permanent audit archive. It bounds each account to 16 sites, each site to 64 reporting instances per 24 hours and 512 aggregate metric series, and each upload to 256 series / 256 KiB. Exceeding a bound rejects the report; runtime reporting overflow is surfaced in the dashboard. Polling/ingestion require a current Pro or Enterprise account; revoked API keys or sessions fail authentication. Uploads are accepted no more frequently than every ten seconds per instance. Duplicate or older sequences are ignored atomically, and decreasing counters require a new instance ID.

The displayed window groups deltas by **server report receipt time**, not individual request time. Delayed uploads can appear as a later spike. Reports cover origin guard decisions and instrumented work; they exclude CDN hits and early ingress rejections. Unknown visitors are not classified as humans. Observe-mode proposals are not actual denials. Rejected callbacks are not measured SQL queries saved. Private work API metrics count permit admissions/rejections, not SQL completion or duration; use application/DB monitoring for those outcomes. Node callback wrappers report actual wrapper outcomes and duration.

Capacity is the latest shared admission ledger reading, counted once rather than added across replicas. Database, render and gateway pools have separate limits. Retained holds include uncertain work awaiting confirmed recovery; a hold is not proof that a query is currently executing. A short spike between reports can be missed. After two minutes without a report, capacity is hidden and status becomes stale; after retention expires, the site no longer appears. A reporting instance proves telemetry connectivity, not correct ingress placement, complete DB integration or genuine crawler acceptance.

Monitoring alerts are displayed in the dashboard for stale reports, unavailable admission storage, paused admission, policy/configuration mismatch, open/probe catalog circuits, full work pools, long-held work, reporting overflow, and reported admission/origin failures. They do not send email, Slack or webhook notifications and do not automatically change policy or reclaim holds. Use existing infrastructure alarms for unattended incident paging.

Hosted endpoints (separate from the private per-deployment work API):

- `POST /operator/crawlers/activation`: server-side operator API key and current Pro/Enterprise plan; `{ tenant }` must match its owner. Returns a five-minute feature/tenant/plan lease, without issuing or deploying policy.
- `POST /operator/crawlers/telemetry`: server-side operator API-key authentication; bounded cumulative snapshots with account-bound tenant, site/instance identity, increasing sequence, configuration/mode, metric series and capacity.
- `GET /operator/crawlers/analytics`: operator dashboard session authentication; account-scoped rolling metrics and alerts. Responses use `Cache-Control: private, no-store`; unavailable storage returns 503 rather than fabricated zero traffic.