Skip to documentation
Docs/Crawler Protection/Policies & capacity
Browse documentation

Crawler Protection

Policies & capacity

Set shared budgets and reserve capacity for customers.

Shared policy

RedisAdmission atomically checks/debits all related rate buckets using Redis server time. A failed compound check does not partially debit the other buckets. Verified IPs share provider/class budgets, and crawler classes share a parent provider ceiling. Unknown traffic has both aggregate and per-IP budgets.

GuardPolicy requires:

FieldPurpose
version, expiresAtPolicy identifier and absolute epoch-millisecond expiry.
globalRate, unknownRate, perIpRate{ capacity, refillPerSecond } token buckets.
providersParent provider rate and explicitly allowed classes, each with its own rate.
routesNamed groups with a positive cost and rate; middleware cost must match.
globalWorkLimit, unknownWorkLimit, providerWorkLimitShared work-slot ceilings.
localWorkLimitConcurrent operations per guard instance.
maxOperationsPerRequestMaximum operations a context may attempt to start.
maxContextAgeMsContext lifetime from 1 to 60,000 milliseconds.

Unknown budgets cannot exceed configured provider/class budgets; route costs must fit burst capacities. Choose limits from measured peak workload and database headroom, not daily averages. Preserve parameters that change content, such as filters, pagination, currency and locale.

Use bounded Redis command timeouts, no offline queue and no automatic command retries. Eviction or lost state after failover can invalidate shared limits. Stop admission during uncertain recovery until prior work is drained/reconciled; retain independent pool/process limits. Automatic failover reconciliation is not implemented.

Shopper isolation and catalog cache

HEAD requests can reuse an existing cached GET representation, with body handling left to the HTTP adapter. A HEAD cache miss runs the admitted loader without storing its result; a stale HEAD never starts regeneration. This prevents an empty HEAD response from replacing the HTML served to GET requests.

An optional customer policy reserves shopper and checkout work capacity and separates their rate budgets. Priority requires a trusted application callback that validates a server-side shopping/session/cart grant; arbitrary cookies, caller flags and user-agents do not qualify. Checkout priority is limited to explicitly configured checkout route groups. Use separate bounded DB pools as well. Anonymous shoppers without a validated grant still share conservative traffic limits; measure false throttles.

CatalogCache stores only eligible public HTML. Explicit meaningful query keys form the cache key; personalized/authenticated requests and unknown query parameters bypass it. Denials, private responses, Set-Cookie and Vary responses are never stored. The application must ensure public representations depend only on configured keys. Shared regeneration holds deduplicate a key and bound concurrent cold misses across distinct keys. Stale refreshes require an explicit background-lifetime hook and the same admitted loader. A busy cold miss returns an uncached 503.