Live dashboard analytics

Pro and Enterprise accounts can connect installed runtimes to Dashboard → Valiron Crawler Protection. It shows reported decisions by provider, verification state, route group and actual/proposed action; a 24-hour traffic timeline; gateway/work/cache metrics; shared database, render and gateway hold counts; reporting instances, configuration versions and modes; monitoring alerts; and a JSON export. It refreshes every 30 seconds and supports site, provider and route-group selection. Empty, stale and unavailable telemetry states are explicit. Configuration export alone does not connect or activate a deployment.

For a standalone gateway, set these server-side variables alongside your deployment configuration:

CRAWLER_TELEMETRY_URL=https://valiron-edge-proxy.onrender.com/operator/crawlers/telemetry
VALIRON_API_KEY=<operator API key from Dashboard → API Keys>
# Optional CRAWLER_TELEMETRY_API_KEY overrides the activation key for reporting.
CRAWLER_SITE_ID=<stable site label, such as storefront>

Use the account-scoped tenant from your validated configuration export (operator_<account ID>). A key from another account cannot report that tenant. Keep API keys out of frontend code and configuration downloads. Site labels must contain 1–100 letters, digits, underscores or hyphens. Leave telemetry variables unset to keep reporting local. Each running process creates its own random instance ID; do not share an instance ID across replicas or counter resets.

For in-process integrations:

import { createCrawlerTelemetry } from '@valiron/crawler-protection';
const telemetry = createCrawlerTelemetry({
  runtime,
  endpoint: 'https://valiron-edge-proxy.onrender.com/operator/crawlers/telemetry',
  apiKey: process.env.VALIRON_API_KEY!,
  site: 'storefront',
  onError: () => console.error('Crawler telemetry unavailable'),
});
await telemetry.flush(); // Initial counter baseline, outside the request path.
// On application shutdown: telemetry.stop();

Reports run in the background every 30 seconds, with a five-second HTTP timeout, no redirect following and no overlapping uploads. Reporting failures cannot grant access or release work. Cumulative counters recover measurements after transient upload failure while the collector baseline survives. The first accepted sample establishes a baseline and adds no traffic counts. Missing baselines after prolonged outages likewise start a new baseline. There is no disk spool: unsent measurements can be lost when a process exits. Reports exclude raw IPs, request URLs, user agents, sessions, request IDs and credentials. Verified-provider labels reflect runtime source checks; operator-authenticated uploads are customer-reported measurements, not independent crawler attestation.

The hosted collector retains rolling aggregates and last-seen instances for approximately 24 hours in Redis; it is not a permanent audit archive. It bounds each account to 16 sites, each site to 64 reporting instances per 24 hours and 512 aggregate metric series, and each upload to 256 series / 256 KiB. Exceeding a bound rejects the report; runtime reporting overflow is surfaced in the dashboard. Polling/ingestion require a current Pro or Enterprise account; revoked API keys or sessions fail authentication. Uploads are accepted no more frequently than every ten seconds per instance. Duplicate or older sequences are ignored atomically, and decreasing counters require a new instance ID.

The displayed window groups deltas by server report receipt time, not individual request time. Delayed uploads can appear as a later spike. Reports cover origin guard decisions and instrumented work; they exclude CDN hits and early ingress rejections. Unknown visitors are not classified as humans. Observe-mode proposals are not actual denials. Rejected callbacks are not measured SQL queries saved. Private work API metrics count permit admissions/rejections, not SQL completion or duration; use application/DB monitoring for those outcomes. Node callback wrappers report actual wrapper outcomes and duration.

Capacity is the latest shared admission ledger reading, counted once rather than added across replicas. Database, render and gateway pools have separate limits. Retained holds include uncertain work awaiting confirmed recovery; a hold is not proof that a query is currently executing. A short spike between reports can be missed. After two minutes without a report, capacity is hidden and status becomes stale; after retention expires, the site no longer appears. A reporting instance proves telemetry connectivity, not correct ingress placement, complete DB integration or genuine crawler acceptance.

Monitoring alerts are displayed in the dashboard for stale reports, unavailable admission storage, paused admission, policy/configuration mismatch, open/probe catalog circuits, full work pools, long-held work, reporting overflow, and reported admission/origin failures. They do not send email, Slack or webhook notifications and do not automatically change policy or reclaim holds. Use existing infrastructure alarms for unattended incident paging.

Hosted endpoints (separate from the private per-deployment work API):

POST /operator/crawlers/activation: server-side operator API key and current Pro/Enterprise plan; { tenant } must match its owner. Returns a five-minute feature/tenant/plan lease, without issuing or deploying policy.
POST /operator/crawlers/telemetry: server-side operator API-key authentication; bounded cumulative snapshots with account-bound tenant, site/instance identity, increasing sequence, configuration/mode, metric series and capacity.
GET /operator/crawlers/analytics: operator dashboard session authentication; account-scoped rolling metrics and alerts. Responses use Cache-Control: private, no-store; unavailable storage returns 503 rather than fabricated zero traffic.

Analytics & monitoring

Live dashboard analytics