AI crawler tracking

Install the Clics server-side crawler package to measure verified AI crawler requests from ChatGPT, Claude, Perplexity, Google, and more.

Clics tracks AI crawlers on the server, where the original request IP and user agent are available. An event is recorded only when both values match a crawler operator’s confirmed identity and published IP ranges.

Before you start

1. Create or open a production project in the Clics dashboard
2. Open Configure → Crawler tokens
3. Create a crawler token for that project
4. Keep the token server-side; never expose it in browser code or replace it with a workspace API key

Add the project ID and crawler token to your server environment:

CLICS_PROJECT_ID=your_project_id
CLICS_CRAWLER_TOKEN=your_crawler_token

Install the package

npm install @clicsdev/crawler

You can also use pnpm add @clicsdev/crawler, yarn add @clicsdev/crawler, or bun add @clicsdev/crawler.

Next.js

Add request-only tracking to proxy.ts in Next.js 16, or middleware.ts in earlier versions:

import { trackClicsCrawlerRequest } from "@clicsdev/crawler"
import {
  NextResponse,
  type NextFetchEvent,
  type NextRequest,
} from "next/server"

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackClicsCrawlerRequest(request, event, {
    projectId: process.env.CLICS_PROJECT_ID!,
    token: process.env.CLICS_CRAWLER_TOKEN!,
  })

  return NextResponse.next()
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
}

Passing event uses waitUntil() so tracking does not delay the response. Keep crawler-facing files such as robots.txt, llms.txt, and sitemaps inside the matcher.

TanStack Start

Cloudflare Workers

When the TanStack Start application runs on Cloudflare Workers, create a custom src/server.ts entrypoint:

import { createCloudflareHandler } from "@clicsdev/crawler"
import handler from "@tanstack/react-start/server-entry"
import { env } from "cloudflare:workers"

function isCrawlerFacingFile(pathname: string) {
  return (
    pathname === "/robots.txt" ||
    pathname === "/llms.txt" ||
    pathname === "/llms-full.txt" ||
    (pathname.includes("sitemap") && pathname.endsWith(".xml"))
  )
}

const fetch = createCloudflareHandler(
  (request) => {
    const pathname = new URL(request.url).pathname

    if (isCrawlerFacingFile(pathname)) {
      return env.ASSETS.fetch(request)
    }

    return handler.fetch(request)
  },
  {
    projectId: env.CLICS_PROJECT_ID,
    token: env.CLICS_CRAWLER_TOKEN,
  }
)

export default { fetch }

Open the wrangler.jsonc file at the root of the project. Add the ASSETS binding and these selective routes inside the existing assets object. Keep the existing asset directory and other settings unchanged:

{
  // Keep your existing Wrangler configuration.
  "assets": {
    // Keep your existing directory and other asset options.
    "binding": "ASSETS",
    "run_worker_first": [
      "/robots.txt",
      "/llms.txt",
      "/llms-full.txt",
      "/sitemap.xml",
      "/*sitemap*.xml",
    ],
  },
}

The final pattern also covers other sitemap XML files, including names such as /sitemap-1.xml, /post-sitemap.xml, and nested sitemap paths. Other static assets keep Cloudflare’s normal asset-first behavior.

Cloudflare calls the exported fetch(request, env, context) handler with its real execution context. The Clics wrapper runs the selected static asset or TanStack handler, reads the final response status, and passes the tracking request to context.waitUntil() before returning the unchanged response.

You do not create waitUntil(), call the tracker elsewhere, or add await. Do not also register crawler tracking in src/start.ts, because that would submit the same request twice.

If you already have a custom src/server.ts, keep its existing behavior and wrap its current fetch handler with createCloudflareHandler.

Long-running Node.js server

For a persistent Node.js, Docker, or VPS deployment, wrap the TanStack server entry:

import { withCrawlerTracking } from "@clicsdev/crawler"
import handler from "@tanstack/react-start/server-entry"

const fetch = withCrawlerTracking((request) => handler.fetch(request), {
  projectId: process.env.CLICS_PROJECT_ID!,
  token: process.env.CLICS_CRAWLER_TOKEN!,
})

export default { fetch }

The Node.js process remains alive after returning the response, so no waitUntil() setup is required. The wrapper starts tracking after TanStack produces the response and returns that response unchanged.

For another serverless TanStack deployment, do not assume that TanStack’s middleware context contains the platform execution context. Use the platform’s documented background-task API when available. If the platform has no equivalent of waitUntil(), await response tracking when guaranteed delivery matters; this can add latency.

Hono

Track after await next() so the final response status is included:

import { trackClicsCrawlerResponse } from "@clicsdev/crawler"
import { Hono } from "hono"

const app = new Hono()

app.use("*", async (c, next) => {
  await next()

  trackClicsCrawlerResponse(c.req.raw, c.res, c.executionCtx, {
    projectId: process.env.CLICS_PROJECT_ID!,
    token: process.env.CLICS_CRAWLER_TOKEN!,
  })
})

c.executionCtx lets the tracking request finish in the background when the runtime supports waitUntil().

Express

import { createExpressMiddleware } from "@clicsdev/crawler"

app.use(
  createExpressMiddleware({
    projectId: process.env.CLICS_PROJECT_ID!,
    token: process.env.CLICS_CRAWLER_TOKEN!,
  })
)

The middleware calls next() immediately and submits the event after the response finishes. On a server reached directly, Express normally exposes the remote address through request.ip.

If Express runs behind Cloudflare, Nginx, or another reverse proxy, configure Express trust proxy for the proxy topology you control. This lets request.ip contain the crawler address instead of the proxy address. Do not trust forwarding headers from arbitrary clients.

Cloudflare Workers and Pages

import { createCloudflareHandler } from "@clicsdev/crawler"

export default {
  fetch: createCloudflareHandler(
    (request, env, context) => app.fetch(request, env, context),
    (env) => ({
      projectId: env.CLICS_PROJECT_ID,
      token: env.CLICS_CRAWLER_TOKEN,
    })
  ),
}

The adapter uses context.waitUntil() and reads Cloudflare’s trusted client IP header automatically.

Generic Request and Response handlers

If your backend uses standard Request and Response objects, wrap the handler so Clics can include the response status:

import { withCrawlerTracking } from "@clicsdev/crawler"

export const handler = withCrawlerTracking(
  (request, context) => app.fetch(request, context),
  {
    projectId: process.env.CLICS_PROJECT_ID!,
    token: process.env.CLICS_CRAWLER_TOKEN!,
  }
)

If only the incoming request is available, use request-only tracking:

import { trackClicsCrawlerRequest } from "@clicsdev/crawler"

trackClicsCrawlerRequest(request, context, {
  projectId: process.env.CLICS_PROJECT_ID!,
  token: process.env.CLICS_CRAWLER_TOKEN!,
})

Request-only tracking identifies the requested page. Response-aware tracking additionally records the HTTP status.

Internal hostnames and reverse proxies

Most runtimes expose the public request URL automatically. If your application receives an internal origin such as http://localhost:3000 or http://0.0.0.0, provide the public origin:

trackClicsCrawlerRequest(request, context, {
  projectId: process.env.CLICS_PROJECT_ID!,
  token: process.env.CLICS_CRAWLER_TOKEN!,
  publicOrigin: "https://example.com",
})

Only the protocol, hostname, and port are replaced. The full pathname and query string are preserved.

Client IP detection

Clics uses the crawler’s original client IP to verify the request against the crawler operator’s published IP ranges. The SDK detects the IP from trusted platform headers in this order:

1. cf-connecting-ip
2. true-client-ip
3. fastly-client-ip
4. fly-client-ip
5. x-vercel-forwarded-for
6. x-forwarded-for

When a header contains a comma-separated proxy chain, Clics uses the first IP address.

If your runtime exposes the client IP through another trusted source, provide it explicitly with getClientIp:

trackClicsCrawlerRequest(request, context, {
  projectId: process.env.CLICS_PROJECT_ID!,
  token: process.env.CLICS_CRAWLER_TOKEN!,
  getClientIp(request) {
    return request.headers.get("your-trusted-client-ip-header")
  },
})

Use getClientIp only for a header or runtime field that your own infrastructure controls.

Background delivery: no extra setup required

For every framework integration shown above, copy the example as written. You do not need to create a waitUntil() function, configure background jobs, or add await around the Clics tracker.

The package chooses the correct delivery behavior from the framework arguments you already pass:

When would you manually await?

Only consider a manual await when all three conditions apply:

1. You are calling trackClicsCrawlerRequest or trackClicsCrawlerResponse directly without passing a framework context
2. Your serverless runtime can stop execution immediately after returning the response
3. That runtime does not provide waitUntil(), but you require the tracking request to finish before returning

In that uncommon case, await the context-free form:

await trackClicsCrawlerResponse(request, response, {
  projectId: process.env.CLICS_PROJECT_ID!,
  token: process.env.CLICS_CRAWLER_TOKEN!,
})

This can add tracking latency to the application response. Do not use this form in the Next.js, TanStack Start on Cloudflare, long-running Node.js, Hono, Cloudflare, Express, or wrapped Fetch examples above.

Use onError when failures should be sent to your server logger:

trackClicsCrawlerRequest(request, event, {
  projectId: process.env.CLICS_PROJECT_ID!,
  token: process.env.CLICS_CRAWLER_TOKEN!,
  onError(error) {
    logger.warn("Clics crawler tracking failed", { error })
  },
})

Verify the installation

Deploy the middleware to your production domain, then open AI analytics → AI crawlers. A crawler-looking user agent sent from your own IP will not create an event: Clics accepts only requests whose user agent and source IP both match a supported crawler.

Troubleshooting

• No crawler events: confirm the middleware runs on the server and receives the original client IP. Use getClientIp when your runtime exposes the IP through a custom trusted field.
• Works without a proxy but not behind one: verify the proxy preserves the client IP and that Express trust proxy is configured narrowly.
• Wrong hostname appears: set publicOrigin to the production origin.
• 401 or 403: create or rotate the crawler token for the same project used by CLICS_PROJECT_ID.