AI Crawlers
Read verified AI crawler analytics with the Clics TypeScript SDK.
The AI Crawlers namespace reads verified crawler activity for a project. It uses the same production data as the Clics AI Crawlers dashboard.
Clics records a crawler only when the request user agent and source IP match the ranges published by the crawler operator. Use this endpoint to report which AI providers read your site, which crawler did it, which pages were requested, and which HTTP statuses were returned.
Method
Access the method via clics.aiCrawlers.
Categories
| Category | Meaning |
|---|---|
answer_fetch |
User-triggered answer fetchers, such as ChatGPT or Claude fetching a page for a response |
search_index |
Crawlers used to discover or index content for AI search |
training |
Crawlers used for model-training or corpus-building pipelines |
Supported providers and crawlers
| Provider | Crawlers |
|---|---|
OpenAI |
ChatGPT-User, OAI-SearchBot, GPTBot |
Anthropic |
Claude-User, Claude-SearchBot, ClaudeBot |
Perplexity |
Perplexity-User, PerplexityBot |
Google |
Google-Agent, Google-GeminiNotebook, Google-NotebookLM, Google-Read-Aloud, Google-InspectionTool, Googlebot, GoogleOther, Google-CloudVertexBot |
Microsoft |
Bingbot, msnbot |
Mistral |
MistralAI-User, MistralAI-Index |
Amazon |
Amzn-User, Amzn-SearchBot, Amazonbot |
DuckDuckGo |
DuckAssistBot |
Apple |
Applebot |
Moonshot AI |
Kimi-User, Kimi-SearchBot, KimiBot |
Common Crawl |
CCBot |
Filters
The endpoint supports the dashboard filters: category, provider, crawler, and status.
Use providerOp, crawlerOp, and statusOp with is or is_not when you need inclusion or exclusion. Pass multiple values as comma-separated strings.
Related
- AI crawler tracking: install the server-side crawler package
- Dashboard AI analytics: view AI referrals and crawlers in the UI
- API reference: request and response schemas