Quickstart

Website setup

Open Dashboard → Valiron Crawler Protection (/dashboard/crawler-protection). Free accounts see the Pro upgrade path. Pro and Enterprise accounts can copy integration instructions, validate a schema-v1 site configuration with an optional measured baseline, review rollout blockers and download the validated configuration scoped to their account. The server checks the current authenticated plan; a client-supplied plan does not unlock validation.

Validation and export do not save or deploy policy, connect to your Redis/database, or turn protection on. Install the integration package with npm install @valiron/crawler-protection. Your team installs the runtime and follows the deployment/recovery procedure in its own infrastructure. Optional runtime telemetry powers hosted crawler analytics. Automatic policy activation remains unavailable. A passing configuration check does not prove production capacity or genuine Meta acceptance.

Start here

1. Sign up, upgrade to Pro (or use Enterprise), and create an operator API key in Dashboard → API Keys. Open Dashboard → Valiron Crawler Protection and install @valiron/crawler-protection. Keep the key in the server environment as VALIRON_API_KEY; never put it in browser code or the site configuration JSON.
2. Choose the standalone gateway for an HTTP website, or an in-process adapter for a Node application. Inventory the pages, APIs and expensive operations you need to protect.
3. Validate and export your configuration, then deploy the gateway/runtime with shared Redis in your infrastructure. Route protected traffic through it and prevent direct-origin bypass.
4. For query-level protection, wrap every protected database operation with the private work API or in-process admission. Start work only after admission succeeds; release capacity only after the operation actually stops.
5. Test observation mode, hard workload caps, application authorization and recovery on your own environment before enabling broader crawler access.

Pro API-key activation (SDK 0.2.0+)

The standalone gateway requires VALIRON_API_KEY. In-process integrations call await activateCrawlerProtection({ apiKey, tenant }), then pass the returned activation to createCrawlerRuntime() or CrawlerGuard. Use the operator_<account ID> tenant from the dashboard export. Missing credentials, an invalid/revoked key, a Free account or a foreign tenant cannot activate the supported runtime.

Startup checks the current account through POST /operator/crawlers/activation. A valid lease lasts at most five minutes and refreshes every minute in the background. Admission checks use local memory, without an HTTP request per visitor or database query. A 401/403 refresh response invalidates the lease immediately; temporary network/5xx/429 failures preserve only its remaining lifetime. After expiry, new protected requests and operations fail closed with 503/errors. Existing operations may settle and release capacity; private release/recovery tools and explicitly bypassed static routes remain usable. Call activation.stop() on shutdown. --check validates configuration/credential syntax offline; it does not prove the account is paid.

The account API must be deployed before upgrading to the requiring SDK. Startup needs network connectivity; keys and leases are not written to disk. Activation is independent of optional analytics reporting. Public SDK checks can be bypassed in modified code, and 0.1.0 cannot be retroactively gated; hosted Pro endpoints enforce the account plan server-side.

Website compatibility

Use the standalone HTTP gateway in front of an HTTP(S) origin, regardless of its frontend framework or backend language. The gateway service runs on Node.js 22+, while the website application can run on PHP, Python, Ruby, Java, Go or another HTTP-capable stack. In-process Node middleware and a Nuxt/Nitro adapter are also available.

The npm package includes gateway/service.cjs, a Dockerfile, a Compose template and deployment instructions in gateway/README.md. Route origin-bound website traffic through the gateway, configure allowed public hosts and a fixed origin, authenticate gateway-to-origin requests, and restrict direct-origin access. Protect every relevant origin and backing API; traffic sent to an unprotected origin remains outside this deployment. A hosted platform must permit those routing/origin controls; there is no universal bypass of a platform's hosting restrictions. HTTP WebSocket upgrades, CONNECT tunnels and Expect/Continue are rejected by this gateway.

The gateway applies source checks, route/rate budgets and shared concurrent origin-request limits before forwarding. It streams HTTP responses and preserves application cookies and authorization. Viewer forwarding/internal headers are replaced; internal response headers are stripped. It does not follow origin redirects. Incomplete responses, origin timeouts and uncertain failures retain nonexpiring holds for confirmed-drain recovery. Browser disconnects do not release still-running origin work.

Database work is a separate boundary. Other backend languages call the private work API before starting each protected operation, then release only after confirmed settlement. The gateway forwards a signed request capability scoped to the original method/path/query, tenant/policy and expiry; the API authenticates the application and applies shared parent fanout and work limits. Its acquire/release protocol and included standard-library Python client are documented in gateway/README.md. No Node SDK is required inside the non-Node website. Gateway-only installation bounds HTTP requests, not every query or background job they may launch.

The work API has no public administrative or automatic hold-reset endpoint. Keep it private and authenticated. Releases work across broker replicas and remain possible after request expiry; missing/ambiguous release retains capacity. Existing authorization, connection-pool and query safeguards remain necessary.

Placement

Crawler/browser -> CDN/WAF
                       |-> eligible public cache hit
                       +-> gateway or middleware -> protected handler
                                                       |
                                              operation admission -> database

Place server-side admission before protected rendering, data fetching and API handlers. Client-side navigation checks run too late to protect server work. Wrap each expensive database operation, including parallel fanout, in work admission. For the included adapters, use Nuxt/Nitro server middleware before rendering and Node middleware before API handlers.

CDN cache hits never reach this guard and are not charged as origin work. Keep your CDN/WAF, existing authorization, bounded connection pools, query timeouts and cancellation controls. Unwrapped queries and routes that bypass middleware remain outside its protection. Bound SSR work independently.