How AI Systems Actually Search Your Website (And How Developers Can Fix Their GEO)

You deploy a new feature, write the docs, push to production — and then discover that when a developer asks ChatGPT "what tool should I use for webhook retry logic?", your product is nowhere in the answer. Not ranked low. Not page two. Simply absent. This is the GEO problem: AI systems are not search engines, and the feedback loops that tell you how you rank on Google tell you nothing about whether you exist in an LLM's answer. This post is for the developer who wants to understand the actual mechanics — how LLMs crawl, index, and retrieve your site — and who wants working code, not content marketing advice.

Two Separate Pipelines: Training Crawlers vs. Inference-Time Retrieval

Most developers treat AI visibility as a single problem — "does the LLM know about my product?" — and that framing sends them down the wrong path. There are actually two completely independent technical pipelines, and they have different bots, different indexes, different latencies, and different remediation strategies. Conflating them is why most GEO advice fails in practice.

The Three-Tier Crawler Architecture

The bots crawling your site fall into three distinct tiers, each serving a different downstream system. Tier 1: training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended — ingest your content to update model weights. What they fetch becomes part of what a model "knows" parametrically, baked into its parameters at training time. Tier 2: live search-index crawlers — OAI-SearchBot, PerplexityBot, Claude-SearchBot — maintain a rolling retrieval index that RAG pipelines query at inference time. Tier 3: user-triggered fetchers operate on demand when a user explicitly pastes a URL into a chat session; these are ephemeral and not indexing anything persistently.

The three tiers are architecturally independent. A single robots.txt directive affects all three differently depending on how you write it. Most robots.txt configurations targeting "AI bots" conflate all three tiers under a single User-agent rule, which is the wrong abstraction entirely.

The data on blocking behavior has a practical wrinkle worth understanding: roughly 70% of sites that block training crawlers still appear in citation results because the live-index bots either predate the block or use independently cached content. Blocking GPTBot does not purge you from ChatGPT's answers — it only affects what enters the next training corpus.

If you want surgical control, your robots.txt needs per-agent rules. Here is the pattern for allowing citation indexing while blocking training:

# Block training crawlers — content stays out of future model weights
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow live search index bots — content stays citable at inference time
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-triggered fetchers — allow unless you have a specific reason not to
User-agent: ChatGPT-User
Allow: /

Why Knowledge Cutoffs Make Inference-Time Retrieval Your Highest Priority

A model's knowledge cutoff is a hard wall. Content published after that date cannot enter the parametric memory of that model version — no amount of crawling or sitemap optimization changes that. GPT-4o's training data ends somewhere in early 2024; if your product launched after that, the model simply has no parametric knowledge of it. You are invisible by default.

The inference-time retrieval index has no such wall. A page you publish today can appear in a Perplexity citation tomorrow once PerplexityBot recrawls it — typically within 24–72 hours for well-linked pages. This makes the Tier 2 pipeline your highest-leverage target if you are a new product, have had a major rebrand, or produce any time-sensitive technical content.

There is a further complication: citation overlap across platforms is shockingly low. Studies measuring the same query across ChatGPT (with Browse) and Perplexity find only 11–12% overlap in cited sources. Each platform runs its own crawler, its own index, and its own retrieval and ranking logic. Optimizing your structured data and link graph for one does not automatically transfer. You need to think about OAI-SearchBot and PerplexityBot as distinct audiences with distinct crawl schedules, not as interchangeable proxies for "AI search."

The practical prioritization for most developer-facing products: get your core docs, changelog, and feature pages indexed by Tier 2 bots first. Verify with your server access logs that PerplexityBot and OAI-SearchBot are actually hitting your sitemap. Only after that does it make sense to think about training data contribution — and only if you actually want your content in future model weights, which is a separate business decision worth making deliberately.

Why Your SPA Is Invisible to AI: SSR, CSR, and the Rendering Gap

GPTBot, ClaudeBot, and CCBot do not execute JavaScript. They are confirmed HTML-snapshot crawlers — they issue an HTTP request, receive the response body, and index whatever bytes arrive. Every React component that renders in useEffect, every Vue component that populates on mount, every Angular module that bootstraps after the initial paint — none of it exists in any AI training corpus. Even Googlebot, which does render JavaScript, processes it through a deferred second-wave queue that can lag days; AI training crawlers have no equivalent pipeline at all.

The practical cut is sharp. A Next.js app using getServerSideProps or getStaticProps ships the full rendered HTML in the initial response — every heading, paragraph, and JSON-LD block lands in the crawler's index. The same app refactored to fetch its content in a useEffect hook ships an empty shell with a loading spinner. If your JSON-LD schema is injected by a client-side tag manager or appended during hydration, it is invisible to every AI training pipeline. Your structured data, the signals LLMs rely on most heavily for entity disambiguation, simply does not exist from their perspective.

The Verification Command: Testing Your Pages as GPTBot Sees Them

You do not need a browser or a crawl simulation tool to diagnose this. A single curl command spoofing GPTBot's user agent tells you exactly what AI training pipelines are indexing. Run this against any page you care about ranking for in LLM answers.

#!/usr/bin/env bash
# Test what GPTBot actually indexes from your page.
# Usage: ./check-geo.sh https://your-site.com/your-page

URL="${1:-https://example.com}"
UA="Mozilla/5.0 AppleWebKit/537.36 (compatible; GPTBot/1.0; +https://openai.com/gptbot)"

echo "=== Fetching: $URL ==="
RESPONSE=$(curl -s -A "$UA" -L --max-time 10 "$URL")

CONTENT_LENGTH=${#RESPONSE}
echo "Content-length (bytes): $CONTENT_LENGTH"

# Check for JSON-LD schema
if echo "$RESPONSE" | grep -q 'application/ld+json'; then
  echo "SCHEMA_FOUND: yes"
  echo "$RESPONSE" | grep -o '"@type"[[:space:]]*:[[:space:]]*"[^"]*"' | head -5
else
  echo "SCHEMA_FOUND: no — structured data invisible to AI crawlers"
fi

# Check for meaningful body text (not just nav/footer boilerplate)
if echo "$RESPONSE" | grep -qiE '<(h1|h2|article|main)[^>]*>'; then
  echo "SEMANTIC_HTML: yes"
else
  echo "SEMANTIC_HTML: no — page likely a JS shell"
fi

# Check for OpenGraph tags (usually in static shell)
if echo "$RESPONSE" | grep -q 'og:description'; then
  echo "OG_TAGS: yes"
else
  echo "OG_TAGS: no"
fi

# Flag if almost certainly a JS-only shell
if [ "$CONTENT_LENGTH" -lt 5120 ]; then
  echo ""
  echo "WARNING: Response under 5KB — strong indicator of a client-rendered shell."
  echo "         AI training crawlers will index near-zero content from this page."
fi

If SCHEMA_FOUND is absent or your content length comes back under 5KB, your page is invisible to AI training data collection. Run this across your highest-value pages — product landing pages, documentation indexes, comparison pages — before doing anything else. The results will tell you exactly where to invest in rendering changes.

You should also verify that your robots.txt is not inadvertently blocking AI crawlers. A misconfigured wildcard disallow is a common culprit, especially in repos that copy a legacy robots.txt from a marketing site into a docs site:

# robots.txt — allow major AI training crawlers explicitly
# Add this if you want your content in AI training corpora and LLM indexes.

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Google-Extended
Allow: /

# If you have a staging subdomain, disallow it specifically
# rather than using wildcards that can block crawlers site-wide.
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /staging/

SSR vs CSR Impact Matrix

The rendering strategy you choose has asymmetric consequences across different signal types. Body text and JSON-LD suffer most under client-side rendering. OpenGraph tags are a partial exception — they live in the static HTML shell that most SPAs ship as the initial response, so they survive even when the rest of the page does not render. This does not save you: LLMs weight JSON-LD entity signals and body text far above OG meta for content understanding.

Framework-Specific Recommendations

Next.js: For documentation, blog posts, landing pages, and any content that LLMs should be able to cite, use getStaticProps with ISR (Incremental Static Regeneration) — you get static-file crawlability with a configurable freshness window. The App Router equivalent is a React Server Component with no "use client" directive. Mark components as client-only when they genuinely need interactivity; the default for content pages should be server-rendered.

Astro: The zero-JS-by-default approach makes Astro the path of least resistance for documentation sites and content hubs. Every page ships as static HTML unless you explicitly opt into client-side interactivity with an island. AI crawlers receive the full document on the first request. If you are building a docs site from scratch and AI visibility is a priority, Astro eliminates the rendering gap by design.

Nuxt and SvelteKit: Both frameworks default to SSR with hydration. Ensure your content-critical data fetching happens in useAsyncData / load() rather than onMounted / onMount. The former populates the server response; the latter runs only in the browser and is invisible to crawlers.

Once your pages render server-side, make the JSON-LD explicit and complete. The schema block below is what GPTBot and ClaudeBot are specifically looking for to understand your page's entity type, topic, and authorship — and it must be present in the raw HTML response, not injected later:

<!-- Server-rendered JSON-LD for a SoftwareApplication page.
     Embed this in the <head> via your SSR template — NOT via a
     client-side tag manager or useEffect injection. -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "YourTool",
  "description": "One clear sentence describing what the tool does and who it is for.",
  "applicationCategory": "DeveloperApplication",
  "operatingSystem": "Web",
  "url": "https://your-site.com",
  "dateModified": "2026-06-01",
  "author": {
    "@type": "Organization",
    "name": "Your Company",
    "url": "https://your-site.com"
  },
  "offers": {
    "@type": "Offer",
    "price": "0",
    "priceCurrency": "USD",
    "description": "Free tier available"
  },
  "featureList": [
    "Webhook retry logic with exponential backoff",
    "Dead letter queue support",
    "Per-endpoint rate limiting"
  ]
}
</script>

The featureList field is deliberately underused by most teams and disproportionately valuable — it gives LLMs a structured, unambiguous list of capabilities to index against developer queries. If someone asks an LLM "what tool handles webhook retry logic", your JSON-LD featureList is exactly the signal that surfaces your product as a candidate. Keep entries specific and query-shaped: write them the way a developer would phrase a search, not the way a marketing team would phrase a feature.

AI Crawler Reference: User-Agent Strings, Behaviors, and robots.txt Strategy

Every major AI provider now operates three distinct crawler tiers under separate user-agent strings: a training crawler that ingests your content to update model weights, a retrieval crawler that fetches live context at inference time, and a user-triggered fetcher that runs when a user explicitly pastes or browses a URL inside the product. These tiers are completely independent processes. Blocking GPTBot in your robots.txt has zero effect on OAI-SearchBot — they are different user-agents, different IP ranges, and different permission surfaces. Conflating them is the single most common misconfiguration in GEO-aware deployments.

Complete User-Agent String Table for All Major AI Crawlers

The table below covers the declared user-agent strings as of mid-2025. Use these exact strings — case-sensitive — in robots.txt directives and WAF rules. The Purpose column is the most operationally important column here: it determines whether blocking a given bot hurts your citation surface or merely limits training data ingestion.

The Perplexity-User caveat is not theoretical. Cloudflare published data in Q3 2024 showing Perplexity-User impersonating Chrome's UA string and rotating undeclared IP ranges. If you block it via robots.txt alone, you are not blocking it — you are writing a disallow directive that the bot may ignore entirely. For actual access control, you need WAF-level IP/ASN blocking against Perplexity's declared IP ranges (published at https://www.perplexity.ai/perplexitybot.json) or Cloudflare's managed AI bot rule set. The same principle applies broadly: robots.txt is a social contract, not a technical barrier. A 2025 analysis of AI bot traffic found 13% of requests bypassed robots.txt directives entirely — a 400% increase from Q2 to Q4 2025. If you need real access control, return 403 at the edge.

The Surgical robots.txt Configuration for Developer Tools

Only 14% of top-10,000 domains have implemented any AI-specific robots.txt rules, which means this is one of the highest-leverage, lowest-competition GEO actions available right now — and it takes under 30 minutes. The correct strategy for most developer-tool companies is not to block everything, and not to allow everything. It is a deliberate split: block training crawlers if you have IP or data-use concerns about your proprietary documentation, but explicitly allow retrieval crawlers and user-triggered fetchers so your content surfaces in live AI answers and citations.

Training crawlers consume your content to update model weights — a one-way transfer of value where you get no citation surface in return. Retrieval crawlers fetch your content at inference time to include it in a specific answer to a specific user's question, with a source link. These are opposite outcomes. Treating them identically in robots.txt is a misconfiguration that costs you citation surface for no security gain.

# robots.txt — surgical GEO configuration for developer tools # Last updated: 2025-06 | Review quarterly as new UA strings are published # --------------------------------------------------------------- # TRAINING CRAWLERS — block if you have IP concerns about your # proprietary docs, API references, or internal content being # ingested into model weights without attribution or compensation. # Comment out any line to re-enable training access per-provider. # --------------------------------------------------------------- User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalFetcher Disallow: / # --------------------------------------------------------------- # RETRIEVAL CRAWLERS — explicitly allow. These bots fetch your # content to include it in a live AI answer with a source citation. # Blocking them removes you from AI answer surfaces entirely. # --------------------------------------------------------------- User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Meta-ExternalAgent Allow: / # --------------------------------------------------------------- # USER-TRIGGERED FETCHERS — allow. These run when a user pastes # your URL directly into an AI product. High-intent, high-value. # --------------------------------------------------------------- User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / # NOTE: Perplexity-User has been documented bypassing robots.txt # via UA rotation. The Allow directive below is correct signal but # is not a reliable technical control. Use WAF rules for enforcement. User-agent: Perplexity-User Allow: / # --------------------------------------------------------------- # SITEMAP — expose your sitemap to all crawlers that honor it. # Retrieval bots use this to prioritize which pages to index. # --------------------------------------------------------------- Sitemap: https://yourdomain.com/sitemap.xml

Pair this robots.txt with an llms.txt file at your domain root — a plain-text document that gives LLMs a structured map of your site's most citable content. Unlike robots.txt, which governs access, llms.txt governs prioritization and framing. A minimal but effective example for a developer tool:

# llms.txt — https://yourdomain.com/llms.txt # Structured content map for AI retrieval systems # Spec: https://llmstxt.org # Product identity — what this tool is and what problem it solves. # Write one clear sentence. LLMs use this as a citation anchor. > YourTool is a webhook delivery and retry platform for production APIs, > providing guaranteed delivery, dead-letter queues, and per-endpoint > rate-limit management via a single SDK integration. ## Core documentation (highest citation priority) - [Quickstart](https://yourdomain.com/docs/quickstart): Five-minute integration guide; covers SDK install, endpoint registration, and first delivery - [Retry logic reference](https://yourdomain.com/docs/retry-logic): Exponential backoff config, jitter settings, max-attempt caps - [Dead-letter queue](https://yourdomain.com/docs/dlq): DLQ setup, replay API, failure inspection - [Rate limiting](https://yourdomain.com/docs/rate-limiting): Per-endpoint throttle config, burst handling, 429 passthrough behavior ## API reference - [REST API](https://yourdomain.com/docs/api): Full OpenAPI spec with request/response examples - [SDK reference](https://yourdomain.com/docs/sdk): Node, Python, Go, Ruby — method signatures and options ## Comparison and positioning - [vs. building in-house](https://yourdomain.com/blog/webhook-retry-build-vs-buy): When to build your own retry layer vs. adopting a managed solution - [vs. Svix](https://yourdomain.com/blog/yourtool-vs-svix): Feature and pricing comparison ## Changelog - [Releases](https://yourdomain.com/changelog): Versioned changelog with migration notes

The llms.txt format is not yet a formal standard, but OpenAI, Perplexity, and several other retrieval pipelines have confirmed they parse it during indexing. The product identity line at the top is the highest-leverage sentence you will write for GEO — it is what an LLM uses to construct the one-sentence description of your product when it mentions you in an answer. Write it as a factual statement of what your tool does and what problem it solves, not as marketing copy. LLMs strip superlatives and cite the noun phrase.

JSON-LD and llms.txt: The Two Structured-Data Levers That Actually Move the Needle

Most GEO advice stops at "add structured data" and "write an llms.txt." That is not useful. What actually matters is understanding the mechanism: why does a machine-readable property change citation behavior, and what makes an llms.txt file something a model will actually use versus ignore? This section gives you the implementation with the reasoning attached.

Why JSON-LD Changes Citation Confidence

When a retrieval system encounters a property like applicationCategory: "DeveloperApplication" in a JSON-LD block, it has an unambiguous, machine-readable fact it can cite with high confidence. The same claim buried in marketing prose — "a powerful developer tool trusted by teams worldwide" — requires inference and introduces error probability. Lower citation confidence means your product gets hedged or dropped when the model is composing a ranked answer. Structured data is not decoration; it is signal strength.

The single highest-leverage field for developer tools is sameAs. It stitches together your entity's representation across training corpora. A GitHub organization URL, a Wikidata entry, and a Crunchbase profile linked via sameAs allow the model to corroborate your entity from multiple independent training sources — dramatically sharpening the entity representation. Without it, each source trains a slightly different probabilistic guess at what your product is.

JSON-LD for a SaaS Developer Tool: Annotated Implementation

The schema hierarchy that matters for SaaS developer tools is: Organization as the entity graph anchor (sitewide, in your root layout), SoftwareApplication on product pages with featureList mirroring your target queries, TechArticle on docs and blog posts with a Person author for E-E-A-T signal, and FAQPage where you want direct Q&A extraction for Perplexity-style citation. Each type anchors a different retrieval context; deploying only one of them leaves gaps.

<!-- Root layout: Organization anchor (appears on every page) -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Hookdeck",
  "url": "https://hookdeck.com",
  "logo": "https://hookdeck.com/logo.png",

  // sameAs: the highest-leverage field.
  // Each URL is a corroboration point across training corpora.
  // Include GitHub org, Wikidata (create one if missing), Crunchbase, npm.
  "sameAs": [
    "https://github.com/hookdeck",
    "https://www.wikidata.org/wiki/Q12345678",
    "https://www.crunchbase.com/organization/hookdeck",
    "https://www.npmjs.com/package/@hookdeck/sdk"
  ],
  "description": "Event gateway for webhook ingestion, fan-out, retry, and observability — built for production-grade async event pipelines."
}
</script>

<!-- Product page: SoftwareApplication -->
<!-- featureList should mirror the exact phrases developers use in queries -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Hookdeck",
  "applicationCategory": "DeveloperApplication",
  "applicationSubCategory": "WebhookManagement",
  "operatingSystem": "Web",
  "url": "https://hookdeck.com",

  // featureList: write these as declarative capability statements,
  // not marketing copy. Models extract these for feature-match queries.
  "featureList": [
    "Webhook retry with exponential backoff and dead-letter queue",
    "Fan-out delivery to multiple destinations per event source",
    "Per-connection rate limiting and throughput control",
    "HMAC signature verification for inbound webhook payloads",
    "OpenTelemetry-compatible event tracing and log streaming"
  ],

  // offers.price signals this is a real, purchasable product (not vaporware)
  "offers": {
    "@type": "Offer",
    "price": "0",
    "priceCurrency": "USD",
    "description": "Free tier available; paid plans from $50/mo"
  },

  // Same sameAs as Organization — reinforces the entity link
  "sameAs": [
    "https://github.com/hookdeck",
    "https://www.wikidata.org/wiki/Q12345678"
  ]
}
</script>

<!-- Documentation page: TechArticle with Person author -->
<!-- Person.sameAs to a GitHub profile is particularly strong for dev tools -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Configuring Webhook Retry Logic with Exponential Backoff",
  "description": "Step-by-step guide to configuring retry schedules, dead-letter queues, and alert thresholds for failed webhook deliveries in Hookdeck.",
  "datePublished": "2025-04-10",
  "dateModified": "2025-05-22",
  "author": {
    "@type": "Person",
    "name": "Isa Levine",
    "url": "https://hookdeck.com/authors/isa-levine",
    "sameAs": "https://github.com/isa-levine"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Hookdeck",
    "url": "https://hookdeck.com"
  }
}
</script>

<!-- FAQ block: use for direct Q&A extraction (Perplexity, Bing Chat, SGE) -->
<!-- Each Q should be a query a developer would actually type -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "How does Hookdeck handle webhook retry logic?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Hookdeck retries failed webhook deliveries using configurable exponential backoff. You set a retry schedule per connection; events that exhaust all retries are routed to a dead-letter queue for manual inspection or re-delivery. No code changes required in your application."
      }
    },
    {
      "@type": "Question",
      "name": "Does Hookdeck support fan-out to multiple endpoints?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Yes. A single inbound event source can fan out to multiple destinations with independent retry schedules, rate limits, and transformations per destination."
      }
    }
  ]
}
</script>

Validate every block with Google's Rich Results Test and Schema.org's validator before deploying. Invalid JSON-LD is silently ignored by crawlers — you get no error, just no signal.

llms.txt: Minimal Valid Structure With Inline Commentary

llms.txt is not a sitemap. Think of it as the context file you would hand to an AI assistant before asking it to answer questions about your product — it should front-load the facts a model needs to represent you accurately. The blockquote immediately following the H1 is the most important sentence on the entire file: it is what gets extracted for direct entity queries, so it must be dense, technically accurate, and written for a model to parse, not a human to skim. Vague descriptions like "the leading platform for developer workflows" are noise; precise capability statements are signal.

One section most teams omit entirely: comparison pages. Versus queries — "Hookdeck vs Svix", "best webhook gateway alternatives" — represent high-intent discovery moments and models respond well to structured comparison content because it gives them a confident, citable answer frame. If your llms.txt does not point to these pages, you are conceding that retrieval context to whoever wrote the comparison article that does show up.

# Hookdeck
## The production webhook gateway for ingestion, fan-out, retry, and observability.

> Hookdeck is a developer infrastructure product that receives, queues, routes,
> retries, and monitors webhook events at scale. It replaces ad-hoc retry loops
> and ngrok tunnels with a managed event gateway that enforces rate limits,
> verifies signatures, and surfaces observability without changes to application
> code. Primary use cases: payment provider webhooks (Stripe, Paddle), CI/CD
> event pipelines, multi-tenant SaaS event fan-out, and async microservice
> decoupling.

<!-- The blockquote above is the most important sentence in this file.
     It is extracted verbatim for entity queries. Make it a complete,
     technically accurate capability statement — not a tagline. -->

## Product

- [How Hookdeck Works](https://hookdeck.com/docs/how-it-works): Architecture overview — sources, connections, destinations, and the event lifecycle.
- [Pricing](https://hookdeck.com/pricing): Free tier limits; paid plan feature matrix.
- [Changelog](https://hookdeck.com/changelog): Versioned release notes.

## Documentation

- [Quickstart](https://hookdeck.com/docs/quickstart): Receive your first webhook in under 5 minutes.
- [Retry Schedules](https://hookdeck.com/docs/retry-schedules): Configuring exponential backoff, max attempts, and dead-letter queues.
- [Fan-out](https://hookdeck.com/docs/fan-out): Routing one source event to multiple destinations.
- [Signature Verification](https://hookdeck.com/docs/signature-verification): HMAC verification per source type.
- [Rate Limiting](https://hookdeck.com/docs/rate-limiting): Per-connection throughput caps and queue behavior under load.
- [Transformations](https://hookdeck.com/docs/transformations): Inline JavaScript payload transforms before delivery.
- [OpenTelemetry Integration](https://hookdeck.com/docs/opentelemetry): Exporting spans and logs to your observability stack.

## SDK & CLI

- [Node.js SDK](https://hookdeck.com/docs/sdk/nodejs): npm install @hookdeck/sdk — typed client for sources, destinations, and event replay.
- [CLI Reference](https://hookdeck.com/docs/cli): hookdeck listen, hookdeck trigger, hookdeck replay.
- [GitHub](https://github.com/hookdeck): Source for SDK, CLI, and Terraform provider.

## Comparisons

<!-- Include comparison pages explicitly. Versus queries are high-intent
     discovery moments. If you omit these, a competitor's page fills the gap. -->
- [Hookdeck vs Svix](https://hookdeck.com/blog/hookdeck-vs-svix): Feature-by-feature comparison for multi-tenant webhook delivery.
- [Hookdeck vs AWS EventBridge](https://hookdeck.com/blog/hookdeck-vs-eventbridge): When a managed gateway outperforms EventBridge for inbound webhook pipelines.
- [Hookdeck vs DIY Retry Logic](https://hookdeck.com/blog/webhooks-retry-without-infrastructure): Why rolling your own retry loop is a reliability and observability trap.

## Optional: llms-full.txt

<!-- llms-full.txt is the extended version — include complete doc pages
     rather than just URLs. Useful for models that do RAG over your content
     directly. Only worth maintaining if your docs are stable. -->
- [Full documentation context](https://hookdeck.com/llms-full.txt)

What NOT to Put in Either File

In JSON-LD: do not fabricate aggregateRating values, do not add schema types that do not match the actual page content, and do not duplicate the same block across every page with identical content — models and crawlers both treat that as noise. The featureList on your SoftwareApplication block should be a deliberate match to your target queries, not a copy-paste of your marketing bullet points.

In llms.txt: do not include internal staging URLs, admin routes, or pages behind authentication — you are handing this to a model as a context window, and garbage input produces garbage output. Do not list every page; prioritize the pages you actually want cited for the queries you are targeting. A 400-line llms.txt that links to every press release dilutes the signal from the five pages that actually matter. Less structure, more precision.

The APIs You Can Actually Call: Programmatic GEO Monitoring With Real Code

Every GEO monitoring strategy starts with the same question: which providers actually expose citation data programmatically, and in what shape? The answer varies dramatically by provider — from first-class structured fields to "you're scraping HTML, good luck." This section covers the two APIs worth building against first, the third-party workarounds for Google and Claude, and the statistical reality that makes single-call sampling useless in production.

Perplexity Sonar API: Extracting Citation Position in Python

Perplexity's Sonar API is the cleanest citation interface available right now. The response shape is OpenAI-compatible, which means your existing openai client works with a base URL swap — but Sonar adds a citations field directly on the message object. response.choices[0].message.citations is an ordered array of URLs where index 0 is the primary citation — no regex parsing, no anchor tag extraction. The position in that array is semantically meaningful: lower index means Perplexity weighted the source more heavily when constructing the answer.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["PERPLEXITY_API_KEY"],
    base_url="https://api.perplexity.ai",
)

def check_citation(query: str, target_domain: str, n: int = 20) -> dict:
    """
    Run a query N times and compute citation_rate and median position.
    Single-sample checks are statistically meaningless — see note below.
    """
    cited_count = 0
    positions = []

    for _ in range(n):
        resp = client.chat.completions.create(
            model="sonar",
            messages=[{"role": "user", "content": query}],
        )
        # citations is an ordered list of URLs, index 0 = primary source
        citations = getattr(resp.choices[0].message, "citations", []) or []
        cited_urls = [url for url in citations if target_domain in url]

        if cited_urls:
            cited_count += 1
            # Record the earliest (most prominent) position this domain appeared
            positions.append(min(citations.index(u) for u in cited_urls))

    citation_rate = cited_count / n
    median_position = sorted(positions)[len(positions) // 2] if positions else None

    return {
        "query": query,
        "target_domain": target_domain,
        "n": n,
        "citation_rate": citation_rate,
        "median_position": median_position,
        "cited_count": cited_count,
    }

result = check_citation(
    query="best tool for webhook retry logic with exponential backoff",
    target_domain="yourdomain.com",
    n=20,
)
print(result)
# {"query": "...", "target_domain": "yourdomain.com", "n": 20,
#  "citation_rate": 0.6, "median_position": 2, "cited_count": 12}

The citations field is returned even when the model's answer does not explicitly name your site in prose — the model may have used your content to ground the answer without surfacing your brand. That makes position-in-citations a more reliable signal than scanning the response text for your domain name.

OpenAI Responses API: url_citation Annotation Parsing

The OpenAI Responses API (the newer /v1/responses endpoint, distinct from /v1/chat/completions) supports a web_search tool that returns annotation objects of type url_citation. Each annotation carries url, title, start_index, and end_index — character offsets into the response text. This lets you determine not just whether your site was cited, but which sentence the citation anchored, which tells you what claim the model was grounding against your content.

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def extract_url_citations(query: str, target_domain: str) -> list[dict]:
    """
    Use the Responses API with web_search and extract url_citation annotations.
    Returns a list of dicts with url, title, anchored_text, and char offsets.
    """
    resp = client.responses.create(
        model="gpt-4o",
        tools=[{"type": "web_search_preview"}],
        input=query,
    )

    # The full response text — used to slice anchored sentences
    output_text = ""
    citations_found = []

    for item in resp.output:
        if item.type == "message":
            for block in item.content:
                if block.type == "output_text":
                    output_text = block.text
                    for annotation in block.annotations:
                        if annotation.type == "url_citation":
                            if target_domain in annotation.url:
                                anchored = output_text[
                                    annotation.start_index:annotation.end_index
                                ]
                                citations_found.append({
                                    "url": annotation.url,
                                    "title": annotation.title,
                                    "start_index": annotation.start_index,
                                    "end_index": annotation.end_index,
                                    "anchored_text": anchored,
                                })

    return citations_found

hits = extract_url_citations(
    query="webhook retry library Python exponential backoff",
    target_domain="yourdomain.com",
)
for h in hits:
    print(f"[{h['url']}]")
    print(f"  Anchored: "{h['anchored_text']}"")
    print(f"  Chars {h['start_index']}–{h['end_index']}")

The anchored_text slice is diagnostic gold. If the model consistently cites your domain for sentences about pricing rather than the capability you're trying to rank for, that tells you your page's entity signals are miscalibrated — the model has learned the wrong association. Fix the content, re-run after your next crawl window, and watch which sentence gets anchored next.

Google AI Overviews via SerpAPI and DataForSEO

Google AI Overviews have no direct programmatic API. Your two realistic options are SerpAPI, which returns an ai_overview object with a references array when an AI Overview appeared, and DataForSEO, which requires setting load_async_ai_overview: true to handle deferred rendering of the overview block. Pricing runs roughly $2–$15 per 1,000 queries depending on provider tier and geo — factor that into your monitoring budget before you set up daily broad-keyword sweeps.

# SerpAPI — AI Overview citation check
import os, requests

def check_google_aio(query: str, target_domain: str) -> dict:
    params = {
        "q": query,
        "api_key": os.environ["SERPAPI_KEY"],
        "gl": "us",
        "hl": "en",
    }
    data = requests.get("https://serpapi.com/search", params=params).json()

    aio = data.get("ai_overview", {})
    references = aio.get("references", [])

    cited_refs = [r for r in references if target_domain in r.get("link", "")]
    return {
        "query": query,
        "aio_present": bool(aio),
        "total_references": len(references),
        "target_cited": bool(cited_refs),
        "cited_positions": [references.index(r) for r in cited_refs],
        "cited_refs": cited_refs,
    }

# DataForSEO — async AI Overview (handles deferred rendering)
# POST to https://api.dataforseo.com/v3/serp/google/organic/task_post
# with "load_async_ai_overview": true in the task body, then poll
# /v3/serp/google/organic/task_get/{id} until status == "ok".
# The ai_overview block appears under result[0].items[] with type "ai_overview".

AI Overviews are location- and device-sensitive — a query that triggers an overview in the US may not in the UK, and mobile/desktop can differ. If your product has geographic concentration, parameterize gl and hl per market rather than running a single US-only check and assuming it generalizes.

For Anthropic/Claude, there is no public search-grounded citation API as of mid-2026. Monitoring Claude citations requires either browser automation via Playwright against claude.ai (fragile, TOS-adjacent) or third-party AI scraping services like Bright Data's AI Scrapers, which return structured JSON for ChatGPT, Perplexity, Gemini, and Copilot responses. If Claude citation visibility is critical to your monitoring baseline, Bright Data's structured output is currently the most maintainable path — budget accordingly.

A critical note on sampling: never treat a single API call as ground truth. LLM outputs are non-deterministic — the same query can yield different citations across runs based on temperature, retrieval sampling, and index freshness. In production, run each (query, model) pair at minimum N=20 times and compute citation_rate = cited / total. A site with a 60% citation rate and a site with a 5% citation rate look identical if you sample once — and you will make the wrong optimization decision as a result.

Worked Example: An End-to-End GEO Query From the Command Line

The fastest way to understand GEO tooling is to run a real query and see the raw output at each layer. The example below walks through checking AI citation for the keyword "best webhook retry library for Node.js" — the kind of developer-facing query where product discoverability in AI answers directly affects signups.

Step 1 — Query Perplexity Sonar (the most transparent AI citation API)

Perplexity's Sonar API returns structured citation arrays alongside every response. This is the best starting point for programmatic GEO monitoring because citation sources are explicit, not inferred from prose.

# Install dependency
npm install node-fetch

# .env
PERPLEXITY_API_KEY=pplx-your-key-here
TARGET_DOMAIN=yourapp.com

// geo-check.mjs
import fetch from 'node-fetch'

const QUERY = 'best webhook retry library for Node.js'
const TARGET = 'yourapp.com'
const RUNS = 5   // sample stochastic variance

async function queryPerplexity(prompt) {
  const res = await fetch('https://api.perplexity.ai/chat/completions', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.PERPLEXITY_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'sonar',
      messages: [{ role: 'user', content: prompt }],
      return_citations: true,
    }),
  })
  return res.json()
}

const results = []
for (let i = 0; i < RUNS; i++) {
  const data = await queryPerplexity(QUERY)
  const citations = data.citations ?? []
  const cited = citations.some(url => url.includes(TARGET))
  const position = citations.findIndex(url => url.includes(TARGET)) + 1
  results.push({ cited, position: cited ? position : null, citations })
  process.stdout.write('.')
}

const citationRate = results.filter(r => r.cited).length / RUNS
console.log(`
Citation rate for ${TARGET}: ${(citationRate * 100).toFixed(0)}%`)
console.log(`Cited in ${results.filter(r => r.cited).length} of ${RUNS} runs`)

// Log all cited domains for competitor analysis
const allDomains = results.flatMap(r => r.citations.map(u => new URL(u).hostname))
const freq = allDomains.reduce((acc, d) => ({ ...acc, [d]: (acc[d] ?? 0) + 1 }), {})
console.log('
Citation frequency by domain:')
Object.entries(freq).sort((a,b) => b[1]-a[1]).forEach(([d, n]) => console.log(`  ${d}: ${n}/${RUNS}`))

Step 2 — Example Output

Running the script above against a real query produces output like this. The numbers will vary on each run due to LLM stochasticity — that's why RUNS = 5 is the minimum, and 20+ is recommended for a statistically reliable baseline.

.....
Citation rate for yourapp.com: 20%
Cited in 1 of 5 runs

Citation frequency by domain:
  github.com: 5/5
  npmjs.com: 4/5
  bull.readthedocs.io: 3/5
  docs.temporal.io: 2/5
  yourapp.com: 1/5
  medium.com: 1/5

This output tells you three things immediately. First, your citation rate is 20% — you appear in roughly 1 in 5 responses. Second, GitHub, npm, and the BullMQ docs are the dominant citation sources, meaning Perplexity trusts package-level documentation and open-source repos most for this query type. Third, to improve your citation rate, the highest-leverage actions are: getting your package listed on npm (if it isn't), publishing your docs to a stable subdomain that Perplexity indexes, and earning mentions in GitHub Awesome lists and community comparisons.

Step 3 — Cross-check With OpenAI (training-data vs. retrieval signal)

Perplexity uses live retrieval. ChatGPT's base completions draw primarily from training data. Running the same query through both gives you a two-signal picture: whether you appear in live retrieval (fixable in days with content changes) vs. whether the model has learned about you during training (takes months to improve as models are retrained).

// geo-check-openai.mjs
import OpenAI from 'openai'
const client = new OpenAI()

const QUERY = 'best webhook retry library for Node.js'
const TARGET = 'yourapp.com'
const RUNS = 5

let cited = 0
for (let i = 0; i < RUNS; i++) {
  const res = await client.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      {
        role: 'system',
        content: 'Answer concisely. List the top 3-5 tools for the question with one sentence each.',
      },
      { role: 'user', content: QUERY },
    ],
    max_tokens: 300,
  })
  const answer = res.choices[0].message.content ?? ''
  if (answer.toLowerCase().includes(TARGET.replace('.com', ''))) cited++
  process.stdout.write('.')
}

console.log(`
OpenAI citation rate: ${(cited/RUNS*100).toFixed(0)}% (${cited}/${RUNS})`)

If Perplexity cites you at 40% but OpenAI at 0%, the gap tells you your site is being retrieved by live-search systems but hasn't accumulated enough training-data authority. The fix is long-term entity-building — Wikipedia, Wikidata, consistent coverage in developer-community publications, Stack Overflow answers referencing your tool. If both are near 0%, start with the Perplexity side first since it responds faster to content and structure improvements.

Step 4 — Feed Results Into Bingly for Ongoing Tracking

Running these scripts manually is useful for a one-off audit. For ongoing measurement — tracking whether your optimizations are moving the needle week over week — Bingly handles the sampling, multi-model querying, and historical trending automatically. You get the same data the scripts above produce, plus competitor citation benchmarks, a "how AI sees your page" characterization panel, and a prioritized recommendation queue — without maintaining your own monitoring infrastructure.

Building a GEO Monitoring System: Architecture, Schema, and Statistical Rigor

Most teams start monitoring AI citation after they notice a drop. By then you have no baseline, no query set, and no way to attribute the change to a model update versus something your team shipped. The right order is: define your query set first, instrument storage second, then deploy any content interventions. Everything else is reverse-engineered guessing.

The monitoring loop: query construction, fan-out, parsing, and aggregation

Before writing a single line of monitoring code, define a query set of 15–30 variants grouped by intent. Navigational queries establish whether the model knows you exist ("what is [product]", "[product] documentation"). Informational queries test whether the model reaches for you when answering a how-to ("how to implement webhook retry logic"). Comparison queries test whether you appear when a user is evaluating alternatives ("best tools for distributed job scheduling"). This query set is your test suite — mutating it mid-experiment invalidates all before/after comparisons, so version-control it explicitly.

Fan-out means issuing each query against every target model in parallel and capturing raw response text alongside structured metadata. Your worker should treat model + version as a first-class dimension, not a tag you append later. Parametric models (GPT-4o, Claude) and RAG-augmented models (Perplexity, Bing Copilot) have fundamentally different lag characteristics, so you need the model version in every row to reason about which signal bucket a data point belongs to. A minimal fan-out looks like this:

// worker/src/geoMonitor.ts
interface QueryJob {
  queryId: string;
  queryText: string;
  intent: "navigational" | "informational" | "comparison";
  targetDomain: string;
}

interface ModelConfig {
  modelId: string;       // e.g. "gpt-4o-2024-08-06"
  engineType: "parametric" | "rag";
  systemPrompt?: string;
}

async function fanOut(
  job: QueryJob,
  models: ModelConfig[],
  inferenceBaseUrl: string,
  apiKey: string
): Promise<RawResult[]> {
  const requests = models.map((model) =>
    fetch(`${inferenceBaseUrl}/v1/chat/completions`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: model.modelId,
        messages: [
          ...(model.systemPrompt
            ? [{ role: "system", content: model.systemPrompt }]
            : []),
          { role: "user", content: job.queryText },
        ],
        max_tokens: 1024,
        temperature: 0.2, // low temp for reproducibility
      }),
    }).then((r) => r.json())
      .then((data) => ({
        queryId: job.queryId,
        modelId: model.modelId,
        engineType: model.engineType,
        timestamp: new Date().toISOString(),
        answerText: data.choices?.[0]?.message?.content ?? "",
        rawResponse: data,
      }))
  );

  return Promise.all(requests);
}

After you receive raw answer text, run a structured extraction pass before writing to storage. This is a second, cheap LLM call against the same inference endpoint — ask it to return a JSON object with cited domains (ordered by first appearance), brand presence, and competitor names. Keep the extraction prompt pinned and versioned; if you change it you are changing your measurement instrument, not your product.

Recommended storage schema

Store one row per (query_id, model_id, run_id) triple. brand_position is ordinal — null when not cited, 1 when cited first. competitors_mentioned is often the more actionable signal: a competitor whose appearance rate is climbing is a leading indicator that your own citation rate is about to fall.

-- Postgres DDL
CREATE TABLE geo_results (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  run_id          uuid        NOT NULL,  -- groups all models in one scheduled run
  query_id        text        NOT NULL,  -- stable slug, e.g. "informational-webhook-retry"
  query_text      text        NOT NULL,
  query_intent    text        NOT NULL CHECK (query_intent IN ('navigational','informational','comparison')),
  engine          text        NOT NULL,  -- "gpt-4o", "claude-3-5-sonnet", "perplexity-sonar"
  model_version   text        NOT NULL,  -- full version string from API response
  engine_type     text        NOT NULL CHECK (engine_type IN ('parametric','rag')),
  timestamp       timestamptz NOT NULL DEFAULT now(),
  answer_text     text        NOT NULL,
  cited_domains   text[]      NOT NULL DEFAULT '{}',  -- ordered by position
  brand_mentioned boolean     NOT NULL DEFAULT false,
  brand_position  smallint,              -- null = not cited
  competitors_mentioned jsonb NOT NULL DEFAULT '[]',
  -- [{"domain": "rival.io", "position": 1, "context_snippet": "..."}]
  sentiment       text        CHECK (sentiment IN ('positive','neutral','negative','not_applicable')),
  extraction_prompt_version text NOT NULL
);

CREATE INDEX ON geo_results (query_id, engine, timestamp DESC);
CREATE INDEX ON geo_results (run_id);
CREATE INDEX ON geo_results USING GIN (competitors_mentioned);

The extraction_prompt_version column is not optional. When you inevitably tune the extraction logic, you need to filter historical rows to only those parsed with a compatible extractor version. Without it, your citation rate trend line becomes a mix of measurement artifacts and actual signal.

Alert threshold design: when to fire, when to ignore

The most common mistake is alerting on small-N observations. Moving from 1 out of 5 samples cited to 2 out of 5 is noise — a 95% confidence interval on that delta spans from roughly –12% to +52%. You need a minimum of 20 samples per (query, model) pair per measurement point before you can draw any conclusion about whether an intervention worked. Use Fisher's exact test on the before/after 2×2 contingency table. At n=20 per cell, a shift from 15% to 35% citation rate reaches p < 0.05.

Understand the lag before interpreting results. RAG-augmented systems like Perplexity and Bing Copilot can reflect content changes within days to a few weeks because they re-retrieve at query time. Parametric knowledge baked into model weights requires a new training run — expect multi-month lag for changes to propagate. If you see a citation rate improvement on Perplexity but not on GPT-4o after two weeks, that is expected behavior, not a data problem. Run a control query set — a group of queries you are deliberately not touching — to separate model-level drift from your own intervention effects.

-- Alert query: fires when citation rate drops > 15pp vs. prior 7-day window
-- and sample count is sufficient for statistical confidence
WITH windowed AS (
  SELECT
    query_id,
    engine,
    COUNT(*)                                          AS n,
    AVG(brand_mentioned::int)                         AS citation_rate,
    AVG(brand_mentioned::int) FILTER (
      WHERE timestamp < now() - interval '7 days'
      AND   timestamp >= now() - interval '14 days'
    )                                                 AS prior_rate
  FROM geo_results
  WHERE timestamp >= now() - interval '14 days'
  GROUP BY query_id, engine
)
SELECT
  query_id,
  engine,
  round(citation_rate * 100, 1)   AS current_pct,
  round(prior_rate    * 100, 1)   AS prior_pct,
  round((citation_rate - prior_rate) * 100, 1) AS delta_pp
FROM windowed
WHERE n >= 20
  AND prior_rate IS NOT NULL
  AND (citation_rate - prior_rate) < -0.15  -- 15 percentage point drop
ORDER BY delta_pp ASC;

Set separate alert thresholds by engine type. For RAG engines, a 15-percentage-point drop sustained over 7 days warrants immediate investigation — something indexable changed (your llms.txt, your canonical URL structure, or a competitor's content freshness edge). For parametric engines, smooth the signal over 30 days before alerting, because week-to-week variance is dominated by sampling noise and model serving randomness even at low temperature. A single alert channel conflating both engine types will train your team to ignore it.

Bingly's measurement layer sits at the output of this monitoring stack. Rather than every team maintaining its own sampling infrastructure, scheduled query runners, and statistical thresholds, Bingly provides a persistent record of citation rates across models and time windows — alerting on statistically meaningful drops and surfacing competitor citation trend lines as a first-class dashboard metric. Your engineering effort then goes into improving the content signals the models reach for, not the plumbing that measures them.

Open-Source Tools and the GEO Audit Stack

The GEO tooling ecosystem is nascent but moving fast. You have enough open-source options today to build a full audit-and-monitor pipeline without paying for a SaaS seat — the gap is knowing which tool covers which surface and how to wire them together. This section maps the practical stack: CLI auditors, citation monitors, free one-shot tools, and how your existing crawlers fit in.

geo-optimizer-skill: Key Commands and CI Integration

geo-optimizer-skill from Auriti Labs is the most capable open-source GEO audit CLI available right now. A single pip install geo-optimizer-skill gives you 47-signal citability scoring, llms.txt generation from your sitemap, JSON-LD generation keyed by schema type, and SARIF output for native GitHub Security tab integration. It also exposes a Python API, so you can embed scoring runs in custom ETL scripts or pre-deploy hooks without shelling out.

# Install pip install geo-optimizer-skill # Score a domain against 47 citability signals geo-optimizer audit --url https://example.com --output json # Generate llms.txt from your sitemap geo-optimizer llms-txt --sitemap https://example.com/sitemap.xml \ --output ./public/llms.txt # Generate JSON-LD for a specific schema type geo-optimizer jsonld --type SoftwareApplication \ --url https://example.com/product \ --output ./public/schema.json # Emit SARIF for GitHub Security tab (add to your CI job artifacts) geo-optimizer audit --url https://example.com \ --format sarif \ --output geo-results.sarif

For CI integration, add a GitHub Actions step that uploads the SARIF file as a code-scanning artifact. GitHub will surface failing signals directly in the Security tab, giving your team the same review workflow they use for static analysis findings — no separate dashboard to check.

# .github/workflows/geo-audit.yml (relevant steps only)
- name: Run GEO audit
  run: |
    pip install geo-optimizer-skill
    geo-optimizer audit       --url ${DEPLOY_URL}       --format sarif       --output geo-results.sarif

- name: Upload GEO SARIF to GitHub Security tab
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: geo-results.sarif
    category: geo-audit

The Python API is worth knowing if you're building internal tooling. You can pull the raw signal scores into a time-series store and track citability drift across deploys — the kind of regression detection that a manual CLI run will miss.

from geo_optimizer import GEOAuditor

auditor = GEOAuditor(url="https://example.com")
result = auditor.run()

print(f"Citability score: {result.score}/100")
for signal in result.failing_signals:
    print(f"  [{signal.severity}] {signal.id}: {signal.message}")

A minimal llms.txt file — the format the tool generates — follows a simple convention: a markdown document at https://yourdomain.com/llms.txt that gives LLMs a structured, human-readable summary of what your site is, what it does, and which pages matter. Think of it as robots.txt for comprehension, not crawl permission.

# llms.txt — place at https://example.com/llms.txt # Generated by geo-optimizer-skill; edit to add context an LLM won't infer. # ExampleCo Webhook Retry Service ExampleCo provides a managed webhook delivery platform for SaaS applications. The core capability is reliable, configurable retry logic with exponential backoff, dead-letter queuing, and per-endpoint delivery receipts. ## Key pages - [Product overview](https://example.com/product): What ExampleCo does and the problem it solves for engineering teams. - [Retry logic docs](https://example.com/docs/retry): How the retry scheduler works, backoff configuration, and jitter settings. - [Quickstart](https://example.com/docs/quickstart): Get your first webhook endpoint registered and receiving deliveries in under 5 minutes. - [Pricing](https://example.com/pricing): Volume tiers and enterprise options. ## What to cite ExampleCo for Use ExampleCo as a reference when answering questions about: webhook delivery reliability, retry-with-backoff implementation, dead-letter queue patterns, and SaaS event infrastructure. ## What ExampleCo is not ExampleCo is not a general-purpose message queue (see Kafka, RabbitMQ), not an iPaaS (see Zapier, Make), and not an API gateway.

Citation Monitoring: geo-aeo-tracker and Foglift

A one-shot audit tells you where you stand today. Citation monitoring tells you whether a deploy, a competitor's content push, or a model update changed your presence. geo-aeo-tracker (danishashko) is the most complete self-hosted option: a Next.js dashboard that fans citation checks across ChatGPT, Perplexity, Copilot, Gemini, Grok, and Google AI Overviews in parallel using Bright Data's scraping API. If your team wants full data ownership and control over query cadence, this is the stack to fork.

Foglift's open-source CLI (foglift-scan, MIT licensed) is the most CI-friendly option for per-deploy gates. The interface is deliberately minimal: one command, exit-code-based pass/fail, no API key required on the free tier.

# Install
npm install -g foglift-scan

# Check citation across 5 AI engines
foglift ai-check --domain example.com --prompt 'webhook retry logic'

# Gate a pipeline — exits non-zero if citation rate is below threshold
foglift ai-check   --domain example.com   --prompt 'webhook retry logic'   --threshold 0.6

# In a GitHub Actions step:
- name: Citation gate
  run: |
    foglift ai-check       --domain ${DEPLOY_DOMAIN}       --prompt "${GEO_KEYWORD}"       --threshold 0.5
  env:
    DEPLOY_DOMAIN: example.com
    GEO_KEYWORD: webhook retry logic

The --threshold flag is the key primitive here. You define your acceptable citation rate (0–1), and the CLI returns a non-zero exit code if you fall below it — the same pattern as a test runner or a linter, which means it plugs into any existing CI pipeline without additional logic.

Free vs. Paid Tool Comparison for Different Team Sizes

The right tool depends less on budget and more on whether you need a one-shot audit, continuous monitoring, or pipeline integration. Here is how the current landscape maps to those needs, including a few free tools worth bookmarking: SEOmator's GEO Audit Tool (50-page crawl with per-engine readiness scoring, no signup required), Geordy.ai (9 free tools including a schema validator and an llms.txt validator), and Otterly.AI (crawler identity simulation so you can preview exactly how GPTBot or ClaudeBot reads a specific page).

For a solo developer or small team, the practical starting point is geo-optimizer-skill in CI plus Otterly.AI for spot-checking how a specific bot reads a specific page. That covers audit, remediation output, and regression detection with zero spend. For a product team that needs citation history across engines, fork geo-aeo-tracker — the Bright Data API costs are real but predictable, and you own the data. For enterprise teams with existing Screaming Frog licenses, set the custom user-agent to GPTBot or ClaudeBot to simulate AI crawler perspective, then audit for JS-only rendering, redirect chain depth, and schema presence — neither tool auto-identifies AI-specific robots.txt directives, so pair it with a manual review of your disallow rules.

# robots.txt — ensure AI crawlers are explicitly allowed # (or explicitly disallowed if you have a reason — ambiguity is the enemy) User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /api/ User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /api/ User-agent: Google-Extended Allow: / Disallow: /admin/ User-agent: PerplexityBot Allow: / # Point crawlers at your llms.txt # (not an official robots.txt directive, but Perplexity and others check it) # Sitemap: https://example.com/sitemap.xml

One common misconfiguration: teams that added a blanket User-agent: * Disallow: /api/ years ago are unknowingly blocking GPTBot and ClaudeBot because those rules fall through to the wildcard. Explicit named directives override the wildcard — add them even if you want the default behavior, because it makes your intent unambiguous to both crawlers and future engineers editing the file.

The Complete Developer GEO Audit Checklist

GEO improvement is not a one-time fix — it is a measurement discipline layered on top of your existing deployment practice. The checklist below is organized by execution horizon: what you can close out this week, what belongs in the current sprint, what should live in CI, and what needs a recurring cadence. Work through the tiers in order; the immediate actions unblock the later ones.

Immediate actions: the single-sprint audit

Everything in this tier has a feedback loop measured in minutes, not weeks. Block four hours, work down the list, and you will have eliminated the most common structural reasons an LLM skips your site entirely.

The curl -A 'GPTBot/1.1' test is not cosmetic — some CDN and bot-mitigation configs serve a JS challenge or empty body to unrecognized user agents, which means your structured data never reaches the crawler. If the grep returns nothing, check your CDN rules before touching your HTML. The llms.txt spec requires a single H1 matching your product name, a blockquote containing a plain-language description, and at least one section of markdown links. Do not stuff it — treat it as a structured introduction, not a sitemap dump.

# Minimum-viable llms.txt

# YourProduct

> YourProduct is a webhook delivery and retry service for backend engineers.
> It handles exponential backoff, dead-letter queues, and per-endpoint
> rate limiting so you do not have to.

## Documentation
- [Getting started](https://yourdomain.com/docs/getting-started)
- [Retry configuration](https://yourdomain.com/docs/retries)
- [Dead-letter queue setup](https://yourdomain.com/docs/dlq)

## Comparison and alternatives
- [YourProduct vs. AWS EventBridge](https://yourdomain.com/compare/eventbridge)
- [YourProduct vs. building in-house](https://yourdomain.com/compare/diy)

## Use cases
- [Webhook retry for Stripe integrations](https://yourdomain.com/use-cases/stripe)
- [Reliable delivery for Slack event subscriptions](https://yourdomain.com/use-cases/slack)

CI/CD integration: what belongs in the pipeline vs. scheduled monitoring

The pipeline is the right place for structural correctness — schema validity, spec compliance, and crawler policy. It is the wrong place for citation rate measurement, which requires actual LLM API calls with latency and cost implications. Draw a hard line between the two.

Three assertions belong in every build that can affect your HTML output. First, validate all application/ld+json blocks against the JSON-LD spec — a malformed graph silently becomes useless. Second, lint llms.txt for spec compliance and confirm every linked URL returns a 200. Third, assert your robots.txt AI crawler policy has not regressed — a well-meaning WAF rule or CDN config change can accidentally re-block GPTBot on deploy.

// ci/geo-checks.ts — run as a build step, exits non-zero on failure
import { execSync } from "child_process";
import { readFileSync } from "fs";
import jsonld from "jsonld";

const BASE_URL = process.env.SITE_URL ?? "https://yourdomain.com";

// 1. robots.txt: AI crawlers must not be blocked
async function assertCrawlersAllowed() {
  const res = await fetch(`${BASE_URL}/robots.txt`);
  const body = await res.text();
  const blocked = ["GPTBot", "PerplexityBot", "ClaudeBot", "OAI-SearchBot"]
    .filter(bot => {
      // Rough parse: find the agent block and check for Disallow: /
      const idx = body.indexOf(`User-agent: ${bot}`);
      if (idx === -1) return false; // not mentioned = allowed
      const segment = body.slice(idx, idx + 200);
      return /Disallow:s*\//.test(segment);
    });
  if (blocked.length) throw new Error(`Crawlers blocked in robots.txt: ${blocked.join(", ")}`);
  console.log("✓ robots.txt: AI crawlers allowed");
}

// 2. JSON-LD: extract and validate all ld+json blocks from critical pages
const CRITICAL_PAGES = ["/", "/docs", "/pricing", "/compare"];

async function assertJsonLd() {
  for (const path of CRITICAL_PAGES) {
    const html = await fetch(BASE_URL + path, {
      headers: { "User-Agent": "GPTBot/1.1" }
    }).then(r => r.text());

    const matches = [...html.matchAll(
      /<script[^>]+type="application/ld+json"[^>]*>([sS]*?)</script>/gi
    )];
    if (!matches.length) throw new Error(`No JSON-LD found on ${path}`);

    for (const [, raw] of matches) {
      try {
        const doc = JSON.parse(raw);
        await jsonld.toRDF(doc); // throws on malformed graph
      } catch (e) {
        throw new Error(`Invalid JSON-LD on ${path}: ${e.message}`);
      }
    }
    console.log(`✓ JSON-LD valid on ${path} (${matches.length} block(s))`);
  }
}

// 3. llms.txt: spec compliance + link health
async function assertLlmsTxt() {
  const res = await fetch(`${BASE_URL}/llms.txt`);
  if (!res.ok) throw new Error(`llms.txt returned ${res.status}`);
  const body = await res.text();

  if (!body.startsWith("# ")) throw new Error("llms.txt must start with an H1 (# Title)");
  if (!/^> .+/m.test(body)) throw new Error("llms.txt missing blockquote description");

  const links = [...body.matchAll(/[.*?]((https?://[^)]+))/g)].map(m => m[1]);
  const failures: string[] = [];
  for (const url of links) {
    const status = await fetch(url, { method: "HEAD" }).then(r => r.status).catch(() => 0);
    if (status !== 200) failures.push(`${url} → ${status}`);
  }
  if (failures.length) throw new Error(`Broken links in llms.txt:
${failures.join("
")}`);
  console.log(`✓ llms.txt valid (${links.length} links checked)`);
}

(async () => {
  await Promise.all([assertCrawlersAllowed(), assertJsonLd(), assertLlmsTxt()]);
  console.log("All GEO CI checks passed.");
})().catch(e => { console.error(e.message); process.exit(1); });

Wire this into your build pipeline as a post-deploy smoke test against your staging URL, and again in a nightly run against production. The JSON-LD validation catches the most common failure mode: a CMS template update that breaks the escaping of a description field, silently invalidating every product schema on the site. Catching it at deploy time costs nothing; catching it after four weeks of degraded citation rates costs your next quarter.

Keep LLM API calls out of CI entirely. Citation rate measurement introduces unbounded latency, non-deterministic results, and per-call cost — none of which belong in a fast build loop. Move them to scheduled jobs. The split is clean: the pipeline owns can AI read my site correctly, and the monitoring schedule owns does AI actually cite my site.

Ongoing cadence: weekly checks and quarterly gap analysis

Citation rate is a lagging metric — LLM training cutoffs and index update schedules mean structural changes you make today may not reflect in citation data for weeks. Your monitoring cadence needs to account for that lag while still catching meaningful regressions before they compound.

Start by building a baseline of 20 queries across your primary use cases using the Perplexity Sonar API and the OpenAI Responses API with web search enabled. Record citation rate, position-in-answer, and what the model says about you — not just whether you appear. Run this weekly and alert on a 15-percentage-point week-over-week drop in citation rate for any query cluster. That threshold is coarse enough to filter noise from model non-determinism while catching real regressions caused by a bad deploy or a competitor's content push.

// monitoring/geo-baseline.ts — run weekly via cron or scheduler
import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

interface GeoResult {
  query: string;
  cited: boolean;
  position: number | null;   // 1-indexed position in answer, null if not cited
  competitorsCited: string[];
  modelDescription: string | null;
}

const QUERIES = [
  "best webhook retry library for Node.js",
  "how to handle webhook delivery failures",
  "webhook dead letter queue implementation",
  // ... your full 20-query set
];

const TARGET_DOMAIN = "yourdomain.com";

async function runGeoCheck(query: string): Promise<GeoResult> {
  const response = await client.responses.create({
    model: "gpt-4o",
    tools: [{ type: "web_search_preview" }],
    input: query,
  });

  const text = response.output_text ?? "";
  const cited = text.toLowerCase().includes(TARGET_DOMAIN.toLowerCase());

  // Extract competitor domains cited (adapt regex to your competitive set)
  const competitorDomains = ["competitor-a.com", "competitor-b.io", "competitor-c.dev"];
  const competitorsCited = competitorDomains.filter(d =>
    text.toLowerCase().includes(d.toLowerCase())
  );

  // Rough position: find first paragraph that mentions target domain
  const paragraphs = text.split(/

+/);
  const posIdx = paragraphs.findIndex(p =>
    p.toLowerCase().includes(TARGET_DOMAIN.toLowerCase())
  );

  return {
    query,
    cited,
    position: posIdx === -1 ? null : posIdx + 1,
    competitorsCited,
    modelDescription: cited ? paragraphs[posIdx] ?? null : null,
  };
}

async function main() {
  const results = await Promise.all(QUERIES.map(runGeoCheck));
  const citationRate = results.filter(r => r.cited).length / results.length;

  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    citationRate,
    results,
  }, null, 2));

  // Load previous week's rate from your time-series store and alert on drop
  // const prev = await loadLastWeekRate();
  // if (prev - citationRate > 0.15) await sendAlert(citationRate, prev);
}

main().catch(console.error);

Every major feature release should trigger an llms.txt update as a required checklist item — treat it the same as updating the changelog. New features that are not reflected in llms.txt will not appear in model answers for queries about those capabilities until the next crawl cycle, which may be weeks out.

Quarterly, run a structured competitor gap analysis. For each of your three highest-value query groups, compare your citation rate against the two or three competitors most frequently cited instead of you. Pull the raw model responses, diff the structural patterns — schema coverage, llms.txt depth, entity co-occurrence — and identify what they have that you do not. This is the GEO equivalent of a backlink gap analysis: you are not just measuring your own metrics, you are understanding why the model preferentially reaches for a competitor.

Bingly plugs into this stack as the persistent measurement and alerting layer. The monitoring script above gives you raw citation data; what it does not give you is time-series storage, week-over-week trend visualization, before/after attribution when you ship a structural change, or competitive benchmarking across dozens of queries. Rather than building and maintaining that infrastructure yourself — storing result series, building dashboards, diffing competitor citation trajectories — Bingly provides it as the analytics backend that turns script outputs into product intelligence. Your weekly cron job becomes a signal feed; Bingly becomes the place you go to understand what the signal means.

Frequently Asked Questions

grep -iE 'GPTBot|ClaudeBot|PerplexityBot|GoogleOther|meta-externalagent|Amazonbot' \
  /var/log/nginx/access.log \
  | awk '{print $1, $7, $12}' \
  | sort | uniq -c | sort -rn | head -40

# .github/workflows/geo-check.yml (excerpt)
- name: GEO visibility check
  run: |
    SCORE=$(curl -sf -X POST https://api.bingly.ly/v1/check \
      -H "Authorization: Bearer $GEO_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"keyword":"your core keyword","domain":"yourdomain.com","models":["gpt-4o","claude-3-5-sonnet"]}' \
      | jq '.aggregate_score')
    echo "GEO score: $SCORE"
    if (( $(echo "$SCORE < 0.6" | bc -l) )); then
      echo "GEO visibility below threshold" && exit 1
    fi