Which search APIs respect robots.txt?

Exa, You.com, Valyu and Linkup say their crawlers follow robots.txt. Brave, Perplexity and Parallel document exceptions or make no explicit promise, Tavily publishes nothing, and Grounder's Allowed sources only setting filters results by each site's robots.txt.

Last checked: 20 October 2026. Disclosure: Grounder (grounder.net) publishes this page and sends searches to Exa, Brave, Perplexity, Parallel, You.com, Valyu, Firecrawl, Tavily and Octen. We tested no crawler; every statement here comes from the vendor's own pages.

Quick pick

What robots.txt does and does not do

robots.txt is a text file at the root of a site that tells crawlers which paths they may fetch. The Robots Exclusion Protocol is IETF RFC 9309 (2022). It says its rules "are not a form of access authorization", so it is a request that well-behaved crawlers follow, not a lock. It says what may be fetched, not what may be done with the text.

A crawler builds an index ahead of time. A user-triggered fetcher loads one page because a person asked, and some vendors say robots.txt does not apply to it. Perplexity says this about Perplexity-User. Parallel says Shap-User is not managed through automated controls. Exa and Brave both say robots.txt controls crawling, not indexing: to remove an indexed page, use a noindex tag.

Some sites add a line to robots.txt stating whether their content may be used for search, for AI answers or for AI training. These lines are preferences, not technical blocks, and we found no vendor page cited here that commits to honor them. This page gives no legal conclusion.

The IETF AI Preferences working group is drafting a shared vocabulary (train-ai, ai-use, search). The newest draft read, draft-ietf-aipref-vocab-08 (14 September 2026), is not an RFC and says its contents do not reflect working group consensus.

What each provider publishes

Exa

• What it publishes: ExaSearchBot "respects the Robots Exclusion Protocol", checks its own group first and then the rules for major search engines or *. Requests are signed with HTTP Message Signatures so owners can verify them. It does not bypass logins, paywalls or CAPTCHAs.
• How a site opts out: robots.txt for crawling; a noindex tag or X-Robots-Tag header to remove indexed pages.
• Limits: The page covers ExaSearchBot only.
• Pick it when: you want a crawler whose requests a site owner can verify.

Brave

• What it publishes: Its crawler "does not advertise a differentiated user agent". "robots.txt is not used to prevent a page from being indexed." Brave will not crawl a page that Googlebot cannot.
• How a site opts out: noindex, then a re-fetch.
• Limits: The page names no Brave-only robots.txt token.
• Pick it when: you accept Googlebot access rules as the rule for what Brave crawls.

Perplexity

• What it publishes: PerplexityBot "is designed to surface and link websites in search results" and "is not used to crawl content for AI foundation models". Perplexity-User supports user actions and "generally ignores robots.txt rules" because a user asked for the page.
• How a site opts out: A robots.txt rule for PerplexityBot; changes take up to 24 hours.
• Limits: On 4 August 2025 Cloudflare alleged Perplexity used undeclared crawlers to evade no-crawl rules; Perplexity disputed the claim. This page does not track the dispute since.
• Pick it when: you accept that user-triggered fetches may ignore robots.txt.

Parallel

• What it publishes: ShapBot builds Parallel's search index and has the token ShapBot. Shap-User identifies itself when fetching on behalf of users, has no token, and is "intended to provide visibility rather than to be managed through automated controls".
• How a site opts out: robots.txt for ShapBot.
• Limits: The page does not say outright that Parallel honors robots.txt.
• Pick it when: you are content with a token for the index crawler and no explicit compliance promise.

Tavily

• What it publishes: Nothing found. No crawler page appears in its docs, site or privacy policy. Third-party bot directories list a TavilyBot user agent and disagree about its behavior.
• How a site opts out: Not published.
• Limits: Tavily's privacy policy says query data may go to third-party search index providers when its own index cannot retrieve the content.
• Pick it when: you do not need a documented crawler policy. None was published by 20 October 2026.

Others and model-native tools

• You.com: YouBot "fully respects robots.txt directives", honors crawl-delay and caches robots.txt for 30 minutes.
• Valyu: Says robots.txt is "always honored" by HenkBot, and crawl-delay is respected.
• Linkup: "Crawling respects robots.txt"; it indexes only publicly available content.
• Firecrawl: Reads robots.txt before it crawls a site, matching FirecrawlAgent or *. Its docs state this for crawling only.
• Octen: No crawler documentation found.
• Model-native tools: OpenAI says OAI-SearchBot respects robots.txt. Anthropic says Claude-SearchBot, ClaudeBot and Claude-User honor it. Google says Vertex AI grounding does not use pages that disallow Google-Extended. Microsoft says Web IQ is designed to honor supported publisher controls, including applicable robots.txt directives.

Where Grounder fits

Grounder (grounder.net) is a web search API and MCP server for AI agents. One search goes across 10 web search providers, results are deduplicated, and each result is marked with the provider that found it.

Allowed sources only is a filter on that result list, and it is off by default. On, Grounder searches as before but returns a result only from a site whose robots.txt allows the provider that found it and does not opt out of search or AI answers. A disallowed result is replaced by the same page from another provider that is allowed, else by Ozone's publisher content from the same site, else by a result from another provider's search. A result that could not be checked in time is left out, so a search can return fewer results. Ozone's publisher content and the specialized APIs are not crawls of the open web, so they are always allowed. The response then carries an allowed list with one entry per provider (provider, allowed_pct, checked), and every result has provenance.allowed set to true. allowed reports; it is not part of any score.

Turn it on on the account page (Routing, Sources) or for one search with routing_settings and allowed_only set to true.

How it checks: Grounder fetches https://<host>/robots.txt with its own user agent, sends the site nothing else (no query, account or prompt data), waits at most 1.5 seconds, and keeps the rules for 24 hours.

What it does not do: it compares a site's rules with the documented crawler token of the provider that found the result. That does not show how the provider actually crawled the page, and it cannot see user-triggered fetches. It filters results; it makes no statement about any vendor's crawler. No benchmark on Grounder's site measures this setting.

Grounder's own robots.txt allows search, AI answers and AI training on its public pages and disallows /account.

How we compared

We looked for a documented crawler name, a robots.txt statement, separate user-triggered fetchers, and any opt-out beyond robots.txt. Sources are each vendor's own crawler, docs and policy pages, read on 20 October 2026, plus the IETF, RFC 9309 and press reports on the Perplexity dispute. "Not published" means we found nothing on the vendor's own pages. Grounder publishes this page and has a commercial interest in filtered search; its benchmark is at /benchmarks. See also the best search API for AI agents.

FAQ

Do AI search APIs respect robots.txt?

Some say so. Exa, You.com, Valyu and Linkup state it on their own pages. Perplexity says its user-triggered fetcher generally ignores robots.txt, and Brave says robots.txt does not control its index. Tavily and Octen publish no statement. These are vendor claims, not test results.

Does Exa respect robots.txt?

Exa says ExaSearchBot respects the Robots Exclusion Protocol, signs its requests so owners can verify them, and does not bypass logins, paywalls or CAPTCHAs.

Does Tavily respect robots.txt?

Tavily has published no crawler page or robots.txt statement that we could find on 20 October 2026. Third-party directories list a TavilyBot user agent and disagree about it, so this page does not say either way.

Does the Brave Search API respect robots.txt?

Brave says its crawler has no distinct user agent, that robots.txt is not used to prevent indexing (use noindex), and that it will not crawl pages Googlebot cannot.

Does Perplexity respect robots.txt?

For PerplexityBot, Perplexity's docs recommend allowing it in robots.txt. For Perplexity-User, which fetches a page at a user's request, it says robots.txt is generally ignored.

Does robots.txt remove a page from search results?

No. It controls crawling, not indexing. Exa and Brave both point to a noindex tag for removal.

How can an API filter out sites that disallow it?

Grounder's Allowed sources only setting returns a result only if the site's robots.txt allows the provider that found it. It is off by default and reports each provider's allowed_pct.

Next step

Your first 1,000 Grounder searches are free, up to 100 a day. No card needed. Add https://api.grounder.net/mcp to your client and sign in with Google, GitHub or Apple, or read the Docs.