Reproducible Benchmarks & Boundaries
Benchmarks & Honest Limitations
Security and compliance buyers deserve reproducible numbers, public test corpora, and clear engineering boundaries—not marketing claims of “100% protection”.
Measurement Methodology & Public Corpora
How We Measure Latency & Accuracy (Compute Time vs. Network RTT)
- In-Worker V8 Compute Time vs. End-to-End Network RTT: All sub-millisecond figures below measure in-Worker V8 compute time across the full Hono Gateway pipeline (request-ID, streaming size cap, constant-time auth, Zod schema validation, domain logic, JSON serialization). In production over the public internet, add 15–65 ms for client HTTPS network RTT to the nearest Cloudflare edge PoP (or
45–120 msvia a RapidAPI/Zyla proxy hop). - Reproducible Benchmark Scripts: Verified weekly in CI (
.github/workflows/weekly-benchmarks.yml):scripts/benchmark-prompt-guard.ts— 104 labeled prompts across 7 attack classes and 10 languages.scripts/benchmark-secret-scanner.ts— 60 labeled cases across 28 secret categories + benign UUID/SHA/SRI resistance.scripts/benchmark-hallucination-diff.ts— 25 RAG source-answer pairs for numeric, date, currency, entity, and negation checks.scripts/benchmark-all-apis.ts&scripts/perf-probe-all.ts— 49-endpoint Gateway benchmark + 256 KB pathological ReDoS/CPU probes.
Published Benchmark · 16 November 2026 (v1.0.39)
1. Portfolio-Wide 42-API In-Worker Compute Benchmark (49 POST Endpoints)
Evaluated via scripts/benchmark-all-apis.ts across all 42 APIs (49 POST endpoints, 100 measured iterations each after 10 warmup iterations):
| Portfolio Metric | Measured Value (16 Nov 2026) | Notes |
|---|---|---|
| HTTP 200 Success Rate | 100.0% (49 / 49 endpoints) | All 42 APIs and multi-route endpoints verified end-to-end |
| Portfolio Median P50 Compute | 0.318 ms (Min: 0.151 ms, Max: 1.294 ms) | In-Worker V8 compute time (add 15–65 ms client HTTPS RTT) |
| Portfolio Median P99 Compute | 1.159 ms (Min: 0.527 ms, Max: 3.932 ms) | 100% of endpoints complete well under 4 ms P99 compute; 256 KB pathological probes pass < 100 ms CPU guard |
Published Benchmark · 16 November 2026 (v1.0.39)
2. Prompt Injection Detection API (
2. Prompt Injection Detection API (/v1/prompt-guard v2)
Evaluated via scripts/benchmark-prompt-guard.ts across 104 prompts (64 adversarial + 40 benign) in deterministic mode:
| Metric | Measured Value | Notes |
|---|---|---|
| Detection Recall (TPR) | 93.8% (60 / 64 attacks flagged; 89.1% strict block) | Direct overrides, role hijacks, prompt extraction, Base64/Hex/ROT13/Leetspeak, Unicode tags, 10 languages |
| False Positive Rate (FPR) | 0.0% (0 / 40 benign prompts flagged) | Benign SQL/Python discussions, security coursework, multilingual support text |
| Precision / F1 Score | 100.0% Precision / 96.8% F1 | Zero false alarms on the benign benchmark corpus |
| Direct Logic Compute | P50: 0.017 ms · P99: 2.829 ms | Gateway P50: 0.550 ms; optional use_ai_judge: true adds ~150–450 ms |
What Prompt Injection Detection Does Not Catch
- Purely semantic social engineering: Creative storytelling without override phrasing, role markers, or encoding tricks may score below threshold in deterministic mode.
- Custom ciphers & steganography: Layer 1 decodes Base64, Hex, URL encoding, ROT13, Leetspeak, and Plane 14 tags, not arbitrary user-invented ciphers.
Published Benchmark · 16 November 2026 (v1.0.39)
3. Log Redact & Secret Scanner (
3. Log Redact & Secret Scanner (/v1/secret-scanner)
Evaluated via scripts/benchmark-secret-scanner.ts across 60 labeled cases (40 secrets across 28 types + 20 benign logs/UUIDs/SHAs):
| Metric | Measured Value | Notes |
|---|---|---|
| Detection Recall (TPR) | 100.0% (40 / 40 secrets detected) | 28 vendor/entropy categories (AWS, GitHub, Stripe, Slack, OpenAI, Anthropic, PEM, JWT, DB URIs) |
| False Positive Rate (FPR) | 0.0% (0 / 20 benign cases flagged) | Zero false alarms on UUIDv4s, Git SHAs, SHA-512 SRI hashes, stack traces |
| Precision / F1 Score | 100.0% Precision / 100.0% F1 | Deterministic prefix + structural rules with context-gated Shannon entropy |
| Direct Logic Compute | P50: 0.044 ms · P99: 8.129 ms | Gateway P50: 0.227 ms |
- No Live Credential Verification: Never contacts AWS, GitHub, Stripe, or external providers to check whether a key is active.
Published Benchmark · 16 November 2026 (v1.0.39)
4. Deterministic Hallucination Diff (
4. Deterministic Hallucination Diff (/v1/hallucination-diff)
Evaluated via scripts/benchmark-hallucination-diff.ts across 25 RAG source-answer pairs:
| Metric | Measured Value | Notes |
|---|---|---|
| Human-Judgment Agreement | 96.0% (24 / 25 pairs) | Agrees with human labels on 24 of 25 cases |
| Factual Number & Entity Recall | 100.0% (13 / 13 fabrications caught) | Unsupported numbers, percentages, currencies, dates, proper nouns |
| Grounded Precision | 100.0% (10 / 10 grounded passed) | Handles comma-formatted numbers without false alarms |
| Direct Logic Compute | P50: 0.037 ms · P99: 2.363 ms | Gateway P50: 0.272 ms; enable use_ai_judge: true for pure negation contradictions |
Spec Conformance & Statutory Checksums
5. Tool-Call, MCP, Tax, Invoice & Crypto Suites
- Agent Tool-Call Validator & MCP Linter: Tested against pinned fixtures for OpenAI Chat & Responses API, Anthropic, Gemini, and MCP
2024-11-05/2025-03-26. Static linting does not execute the MCP server binary. - EN 16931 / Peppol & Corporate/Tax IDs: Verified against EU27 VAT rules, ISO 13616 IBAN MOD-97, LEI, EORI, SIREN/SIRET, Steuer-ID, Codice Fiscale, NIF/CIF, NIP/PESEL, BSN, HU Adószám, CRN, and EIN. Offline validation checks syntax/check-digits; live registration requires
verify_vies: trueon/v1/eu-vat-iban/vat. - JWT / JWS / JWK Inspector: Verified against RFC 7515 / 7517 / 7518 / 8725 test vectors.
Verified v1.0.39 Security & Runtime Boundaries
6. OWASP Top 10, SSRF Fetcher & Continuous CI Verification
- Automated Test & Stress Coverage: Gated across 981 Node Vitest tests, 168 Cloudflare
workerdruntime tests, 12 Python SDK tests, 42/42 256 KB adversarial CPU-budget probes (perf-probe-all.ts),npm audit signatures, and CodeQL SAST. - SSRF & DNS-Rebinding Boundaries (
edgeguard-fetcher): Full 8-hextet IPv6 expansion (100::/64, NAT64, ULA, loopback, link-local,fec0::/10,3fff::/20,192.88.99.0/24, AS112, Teredo, zone IDs), local/rebinding domains (.localdomain,.onion,.nip.io,.sslip.io,localtest.me,lvh.me), fail-closed DoH IP checks, and 10s / 1 MB / manual redirect limits. - Payload, Depth & Sandbox Caps: Canonical receipt serialization caps depth at
256; public/free/*,/demo/*, and/verifyenforce UTF-88 KBcaps and distributed IP limits; keylessPOST /mcpenforces a16 KBJSON-RPC cap and max2JSON-RPC items (1tools/call). - Support & Security Contact:
support@tinplate.dev·security@tinplate.dev(Trust Center).