Every AI answer that mentions a law firm began with a crawler reading a page. OpenAI, Google, Anthropic, Perplexity, and Microsoft each operate named bots, each governed by your robots.txt — and each potentially blocked by layers most firms never inspect: a CDN's bot-protection defaults, a security plugin's blanket rules, a template shipped during the 2023 scraping panic with every AI agent denied. Firms spend thousands on content strategy while a two-line configuration quietly makes them invisible to the engines they are optimizing for. The fix starts with knowing who the crawlers are and what each one actually feeds.
The cast of crawlers, and what each one feeds
The names matter because the consequences differ. GPTBot gathers training data for OpenAI's models; OAI-SearchBot feeds ChatGPT's live search index — block it and you vanish from ChatGPT answers with citations. ClaudeBot crawls for Anthropic; PerplexityBot builds Perplexity's index. Google-Extended is subtler: it is not a separate bot but a robots.txt token controlling whether ordinary Googlebot crawling may also serve Gemini's training — AI Overviews, however, ride on normal Google Search indexing, so blocking Google-Extended does not remove you from AI Overviews. Bingbot quietly matters twice, feeding both Bing and Copilot. A firm deciding its policy is really making five or six separate decisions, not one.
The audit: three layers where blocking hides
- robots.txt — read it today; note every Disallow and which user-agents it names
- CDN and firewall — Cloudflare-class services ship AI-bot blocking as a toggle, and 'verified bot' rules can still challenge AI crawlers; check the bot-management dashboard, not just robots.txt
- Security plugins and rate limiters — aggressive rules return 403s to crawlers your robots.txt welcomes
Then verify from the outside in: request key pages with each crawler's user-agent string and confirm a 200 with full HTML, and check your server logs for the bots' actual visit patterns. The commonest finding is disagreement between layers — robots.txt says welcome, the CDN says 403 — and the crawler experiences whichever layer says no. The second commonest: JavaScript-rendered content, where the bot receives a technically successful page that contains none of your substance. Most AI crawlers do not execute JavaScript; if your practice-area text is not in the raw HTML response, access was never really granted.
The policy question firms actually face
Is there a case for blocking? For publishers selling content, perhaps. For a law firm, the trade is lopsided: your content exists to attract clients, AI engines are where a growing share of clients ask their questions, and blocking the crawlers removes you from those answers while protecting nothing you monetize. The training-versus-search distinction deserves one nuance — some firms allow search-serving bots (OAI-SearchBot, PerplexityBot) while blocking pure training bots (GPTBot) on principle. That stance is coherent, but understand the cost: models trained without your content know your competitors instead, and the "principle" mostly donates mindshare. The pragmatic policy for a firm that wants AI-era visibility is simple: allow the named AI crawlers sitewide, disallow only genuinely private paths, and revisit annually.
A sensible robots.txt, spelled out
The welcome list, explicitly: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, and Bingbot alongside Googlebot — each with an explicit Allow, because explicitness protects you from a future template update that changes the default. Keep Disallow rules for admin paths, staging, and thank-you pages. Pair the file with a current sitemap reference, and if you publish an llms.txt map, the two files work together: one grants access, the other provides orientation. After any site migration or security change, re-run the outside-in check — access regressions after migrations are so common they should be a standing checklist item.
The migration trap: how access breaks silently
Most blocking is never decided; it is inherited. A site migration copies a staging robots.txt that disallowed everything. A security review enables the CDN's "block AI scrapers" toggle during an unrelated incident and nobody revisits it. A plugin update resets bot rules to a new, stricter default. Because the pages keep working for humans, the breakage is invisible — the only symptoms are a slow fade from AI answers and a referral line that quietly flattens, both easy to misattribute to content or competition. The defence is procedural: put the outside-in crawler check on the launch checklist for every migration, redesign, and security change, and diff your robots.txt in version control so any change is a decision someone made on purpose rather than a default someone shipped by accident. Five minutes of verification per launch is the entire cost of never losing a quarter of AI visibility to a checkbox nobody remembers ticking.
What changes when access opens
Firms that discover and remove a block see a characteristic sequence: crawler visits resume within days, pages surface in the affected engine's citations over the following weeks, and referral traffic from that engine appears in analytics shortly after. The sequence is also the measurement plan — log the unblock date, watch the engine's citation behaviour on your monthly visibility checks, and confirm the referral line wakes up. Access is the least glamorous layer of an AI-ready website, with the best ratio in the discipline: an hour of configuration against every future answer the engine might have cited you in. Content strategy determines whether you deserve citations; access determines whether you are even in the room.
Frequently Asked Questions
Grow your AI Websites for Law Firms practice with AI
Lexscale.ai builds AI search visibility, websites, and intake systems for ai websites for law firms firms across North America. Book a free strategy call to see what would move the needle for your practice.
Book a Free Strategy Call →