Gateway & SDK integration

Reusable site configuration

The integration package now provides compileCrawlerConfiguration() and createCrawlerRuntime(). The same engine supports site-specific configuration for ecommerce and content routes; company names, URL discovery and database credentials are not part of the engine.

Schema version 1 contains tenant, policy, explicit inventory, rollout, optional crawlerAccess and optional public cache settings. Inventory entries declare IDs, HTTP methods, paths, protected groups or explicit bypasses. Path patterns support literal segments, :parameter and terminal **. GET entries include HEAD. Unknown fields, overlaps, missing groups, expired policy and ambiguous/unmapped targets reject. Every route group must have an inventory entry.

import { activateCrawlerProtection, createCrawlerRuntime } from '@valiron/crawler-protection';

const activation = await activateCrawlerProtection({
  apiKey: process.env.VALIRON_API_KEY!,
  tenant: reviewedSiteConfiguration.tenant,
});
const runtime = createCrawlerRuntime({
  activation,
  configuration: reviewedSiteConfiguration,
  redis: sharedAdmissionRedis,
  clientIp: trustedClientIp,
  customer: validateServerSideShoppingGrant,
});
app.use(runtime.nodeMiddleware);
// In protected handlers:
const product = await runtime.withDatabaseAdmission(
  req.valironCrawler!, () => queryProduct(req.params.id),
);

The named configuration, Redis, trusted ingress, shopping-grant validation and query functions are application dependencies. Runtime construction does not initialize or activate shared policy. Keep application authorization on checkout/account/write routes; matching an inventory entry never grants those permissions. Nuxt uses defineEventHandler(runtime.nuxtMiddleware); wrap actual SSR with runtime.withRenderingAdmission().

rollout.mode = 'observe' evaluates candidate crawler rates/access in separate buckets and records proposed decisions. It still enforces configured safety rates, customer rates, actual SSR/DB caps, fanout, expiry, trusted ingress and dependency failures. Existing CDN/WAF restrictions stay active. In enforce mode, only explicit enforceGroups enforce candidate policy; remaining catalog groups stay in observation. Work caps cannot be disabled by observation.

Optional crawler rules select provider, crawlerClass, route groups and action: 'budget' | 'deny'. Only verified catalog classifications qualify; validated customer grants keep their separate policy. These are origin-admission decisions: eligible public cache hits remain outside classification. Apply edge restrictions too if a class must be denied cached public content.

runtime.publicResponse(request, admittedLoader, background?) authenticates ingress, resolves inventory and applies configured public cache groups. The loader must admit before SSR and wrap every expensive operation. runtime.report.snapshot() returns bounded process-local origin aggregates for actual/proposed request decisions, wrapped work outcomes and cache events. It excludes raw IPs, URLs, UA, sessions and request IDs. Proposed throttles are not blocked requests; rejected callbacks are not a measured count of SQL queries saved. Reports cover guard-evaluated requests and wrapped work/cache events; early adapter rejections and edge traffic require existing ingress logs. Optional authenticated reporting connects these aggregates to the dashboard, as described below. Hosted policy deployment controls remain unavailable; Pro API-key activation authorizes SDK use, not Redis policy activation.

planCrawlerRollout() validates supplied pool/headroom/checkout measurements and returns rollout blockers; missing measurements do not generate guessed production limits. createEcommerceConfiguration() builds the same schema from operator-supplied routes and policy. Budgets, verified ingress and actual grant integration remain deployment responsibilities. Your team installs and operates the runtime in its own infrastructure.

Node integration

These exports belong to the integration package, not @valiron/sdk. Construct one CrawlerGuard with the started activation handle, a policy, CrawlerRegistry, BingVerifier and shared admission store. The integration uses ManagedRedisAdmission with an explicitly activated immutable policy. The remaining dependencies below belong to your application.

import {
  AdmissionError, createNodeCrawlerMiddleware, sendDenial,
} from '@valiron/crawler-protection';

app.use(createNodeCrawlerMiddleware({
  guard,
  clientIp: trustedClientIp,
  route: mapProtectedRoute, // { group, cost }, or undefined for unprotected paths
}));

app.get('/products/:id', async (req, res, next) => {
  try {
    const product = await guard.withDatabaseAdmission(
      req.valironCrawler!,
      () => queryProduct(req.params.id),
    );
    res.json(product);
  } catch (error) {
    if (error instanceof AdmissionError) return sendDenial(res, error.decision);
    next(error);
  }
});

Use the exported GuardedRequest type or application request augmentation in TypeScript. Register this handler only on routes covered by the mapping. The query promise must settle after the operation stops and releases its connection. An HTTP timeout does not by itself terminate database work.

createCloudFrontClientIpResolver() supports an authenticated distribution-to-origin path with configured trusted CIDRs and additional proxy-hop count. Other platforms need their own trusted-ingress adapter; never infer a trusted source from arbitrary client headers.

Nuxt integration

// server/middleware/crawler.ts
import { createNuxtCrawlerMiddleware } from '@valiron/crawler-protection';
import { guard, trustedClientIp, mapProtectedRoute } from '../utils/crawlerGuard';

export default defineEventHandler(createNuxtCrawlerMiddleware({
  guard, clientIp: trustedClientIp, route: mapProtectedRoute,
}));

Admitted context is attached to event.context.valironCrawler; a denied response is ended before the adapter returns. The structural event adapter is tested, while actual Nitro cache ordering/render termination requires validation against the application's installed version.

Contexts belong to the guard instance that created them. Managed mode supports InternalAdmission: Nuxt issues a signed, short-lived child through guard.forward(), and Node uses guard.acceptInternal() on a protected internal route. Tokens bind issuer/audience, policy, method, exact path/query and raw body. The page quota is charged once; every DB operation still consumes shared fanout and work admission. A replay cannot execute another DB operation. Keep signing credentials and tokens on servers; a forwarded alreadyAdmitted flag never grants access.

Use withRenderingAdmission(context, renderAndFetch) to bound SSR separately from database operations, holding capacity until all actual work settles. Keep direct public API routes under ordinary request admission.