cff-version: 1.2.0
message: "If you use this dataset, please cite it as: Probative Growth, US law-firm websites and AI crawlers: crawler-file rules, text before JavaScript, snippet limits and Spanish pages, by state and market, 2026, https://doi.org/10.5281/zenodo.24015038. Source: Probative, US law-firm websites and AI crawlers: crawler-file rules, text before JavaScript, snippet limits and Spanish pages, by state and market, version 1.0.0, https://doi.org/10.5281/zenodo.24015038, licensed CC BY 4.0."
type: dataset
title: "US law-firm websites and AI crawlers: crawler-file rules, text before JavaScript, snippet limits and Spanish pages, by state and market"
authors:
  - name: "Probative Growth"
    website: "https://probativegrowth.com"
version: "1.0.0"
date-released: 2026-11-06
doi: 10.5281/zenodo.24015038
url: "https://probativegrowth.com/data/law-firm-ai-readiness/"
license: CC-BY-4.0
abstract: "What US law-firm websites show AI crawlers, measured from the public files and pages of the websites themselves: which of eight named AI crawlers and controls (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, meta-webindexer) each site's crawler file (the plain-text file at its root that tells crawlers what they may visit) names and whose access to the home page it blocks, whether the apex and www host names lead to one site, how much home-page text a reader gets before JavaScript runs, and how their Spanish pages present themselves. It also counts FAQPage and Organization structured data and nosnippet and max-snippet tags on the same pages. Aggregates by state and market only; no firm or website is named or scored."
keywords:
  - "law firm AI crawler rules"
  - "law firms blocking GPTBot"
  - "law firm website JavaScript AI crawlers"
  - "law firm FAQ schema share"
  - "law firm Spanish website"
  - "law firm websites AI crawlers data"