robots.txt & llms.txt AI Crawler Analyzer & Generator API
Audit which AI crawlers your robots.txt allows and generate a valid llms.txt file.
Detect AI training crawlers, RAG bots, headless browsers, and device/OS details from User-Agent. Parse User-Agent strings and client hints to identify browsers, OS, devices, search engine crawlers, and AI training/RAG bots (GPTBot, ClaudeBot, PerplexityBot).
{
"ok": true,
"data": {
"user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot",
"is_bot": true,
"is_ai_bot": true,
"category": "ai_crawler",
"bot": {
"name": "GPTBot",
"operator": "OpenAI",
"category": "ai_crawler",
"intent": "training",
"docs_url": "https://platform.openai.com/docs/bots",
"rule_id": "ai_openai_gptbot"
},
"client": {
"browser": {
"name": null,
"version": null
},
"os": {
"name": null,
"version": null
},
"engine": null,
"device_type": "bot"
},
"rules_version": "2026.11.1"
},
"meta": {
"request_id": "req_ua_01",
"ms": 1
}
}
curl -X POST "https://api.tinplate.dev/v1/ua-bot-parser" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TINPLATE_KEY" \
-d '{"user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"}'import { TinplateClient } from "@tinplate/sdk";
const tinplate = new TinplateClient({
apiKey: process.env.TINPLATE_API_KEY!,
});
const { data, meta } = await tinplate.call("/v1/ua-bot-parser", {
"user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"
});
console.log(`[${meta.ms} ms]`, data);import requests
response = requests.post(
"https://api.tinplate.dev/v1/ua-bot-parser",
headers={
"Content-Type": "application/json",
"Authorization": "Bearer YOUR_TINPLATE_KEY",
},
json={
"user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"
}
)
print(response.json())const response = await fetch("https://api.tinplate.dev/v1/ua-bot-parser", {
method: "POST",
headers: {
"Content-Type": "application/json",
"Authorization": "Bearer YOUR_TINPLATE_KEY",
},
body: JSON.stringify({
"user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"
}),
});
const result = await response.json();
console.log(result);{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "tinplate_ua_bot_parser",
"arguments": {
"user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"
}
}
}POST /v1/ua-bot-parser)| Parameter | Type & Constraints | Description |
|---|---|---|
user_agent required |
stringminLen: 1, maxLen: 2048 |
Raw HTTP User-Agent header value (max 2048 characters). |
data Object Fields (200 OK)| Field | Type | Description |
|---|---|---|
user_agent |
string |
Field 'user_agent' |
is_bot |
boolean |
Field 'is_bot' |
is_ai_bot |
boolean |
Field 'is_ai_bot' |
category |
"human" | "ai_crawler" | "ai_assistant" | "search_crawler" | "social_preview" | "seo_crawler" | "monitoring" | "http_client" | "headless_browser" | "unknown_bot" |
Field 'category' |
bot |
object | null |
Field 'bot' |
client |
object |
Field 'client' |
rules_version |
string |
Field 'rules_version' |
Release: v1.0.39 ·
Verified: 16 November 2026 ·
Portfolio Median: 0.318 ms in-Worker V8 compute (add 15–65 ms HTTPS RTT) ·
See Full Portfolio Benchmarks →
Direct Polar All-Access Keys unlock all 42 APIs (49 endpoints), /v1/pipeline, and /mcp on api.tinplate.dev:
Pro ($29/mo · 120 req/min · 100k/mo · 128 KB) or Team ($79/mo · 300 req/min · 500k/mo · 256 KB).
| Plan | Monthly Price | Included Requests | Rate Limit | Max Payload | Best For |
|---|---|---|---|---|---|
| BASIC | $0.00 / mo | 500 / month | 10 req / min | 64 KB | Evaluation & CI smoke tests |
| PRO | $9.99 / mo | 15,000 / month | 60 req / min | 128 KB | Solo devs & production apps |
| ULTRA | $29.99 / mo | 100,000 / month | 300 req / min | 256 KB | Growing SaaS & multi-agent pipelines |
| MEGA | $79.99 / mo | 500,000 / month | 1,000 req / min | 256 KB | High-volume enterprise workloads |
Yes! For example, it separates OpenAI's GPTBot (training) from OAI-SearchBot and ChatGPT-User (search/user browsing).
Yes, use POST /v1/robots-llms-analyzer to audit and generate your site's AI crawler rules.
Typically under 1 ms at the edge.
Stateless in-memory User-Agent parsing.
Worker Zero-Logging Guarantee: Cloudflare Workers Observability and invocation logging are disabled (observability.enabled = false, head_sampling_rate = 0, invocation_logs = false). Requests via RapidAPI or Zyla traverse marketplace proxies; use a Direct Bearer Key for zero-intermediary routing.
Audit which AI crawlers your robots.txt allows and generate a valid llms.txt file.
Block disposable emails, alias abuse, and bot signups with a 0–100 fraud risk score.
Audit CSP, HSTS, and browser security headers with an A+ to F grade and fix snippets.