AI agents & crawlers

This site is built to be read by agents, not merely tolerated by them.

The indexes

| Surface | Format | For | |---|---|---| | /llms.txt | Markdown | A curated editorial index for LLM crawlers. | | /llms.json | JSON | The structured variant, for modern agents. | | /sitemap.xml | XML | Derived — MDX content, marketplace, agents. | | /robots.txt | text | The crawler allowlist. |

/llms.txt is hand-curated, not generated from the sitemap. That is the point: a crawler following it lands on the pages worth reading rather than on every URL that happens to exist.

Allowed crawlers

robots.txt explicitly allows, among others:

• GPTBot
• ClaudeBot
• anthropic-ai
• PerplexityBot
• Google-Extended

This is a deliberate allowlist, not an oversight. We want these docs in answer engines.

Structured data

Pages carry JSON-LD appropriate to their type: WebPage and Article throughout, FAQPage where there is an FAQ, HowTo on the tutorials and the per-client connection guides, Product with AggregateRating and Offer on marketplace listings, Event on community events.

Breadcrumbs are emitted as BreadcrumbList on every documentation page.

What is deliberately not crawlable

Three surfaces are noindex and out of the sitemap on purpose:

• Intelligence records at /intelligence/<category>/<slug>, because they describe third-party stores.
• Creative Drop deliveries at /drop/<token>, public by token, with a Disallow in robots.txt on top of the noindex. The token is the access control.
• Embeds at /embed/bulletin and /embed/widget, the only two paths where the site's global frame-deny is lifted.

Reading these docs as an agent

Every page under /community/docs/ is a single MDX document rendered server-side, with no client-side hydration required to read the content. Fetching the URL gets you the full text.

The two trunks — product and developer — are derived from front-matter, so the hub at /community/docs is a complete index of what exists. There is no hidden section.

Our own agent profile

When BoostEcom acts as a client against a Shopify store, it publishes its capabilities at /.well-known/ucp-agent. Every store we query reads it to negotiate before answering. It declares catalogue read access and contains no secrets.