# VinAssessment robots.txt # Allow crawling of all marketing pages; block private/admin areas. # Block one aggressive content scraper that ignores crawl-rate etiquette. User-agent: meta-externalagent Disallow: / # AhrefsBot (backlink index crawler) walked ~58k VDPs/day for 10.4 GB of # egress (2026-09-09; steady 48-58k/day all week) and returns nothing: its # index is built by crawling OTHER sites that link here, so blocking it does # not hide our backlinks from Ahrefs or from competitors. It does fetch # robots.txt (4x on 2026-09-09), so this is enforced without a WAF rule. User-agent: AhrefsBot Disallow: / # DataForSeoBot (DataForSEO backlink/data crawler): resells crawl data to # third-party SEO tools, sends us no visitors. Owner decision 2026-10-10; # also blocked at the Cloudflare WAF (robots.txt stays reachable). User-agent: DataForSeoBot Disallow: / # MochiImageCrawler (unknown image crawler on AWS EC2): bulk-downloads # vehicle photos, sends us no visitors. Owner decision 2026-10-18; # also blocked at the Cloudflare WAF (robots.txt stays reachable). User-agent: MochiImageCrawler Disallow: / # Baidu (Baiduspider + Baiduspider-render), Alibaba (YisouSpider — Shenma/UC # search, AliyunSecBot — Alibaba Cloud scanner), Reflectionbot (reflection.ai # training crawler), AionBot, GolfHunter (inventory-history harvester): send # us no visitors. Owner decision 2026-10-22; Baiduspider, Reflectionbot, # AionBot and GolfHunter are also blocked at the Cloudflare WAF (robots.txt # stays reachable). Same list as vehisales.com and vehileaks.com. User-agent: Baiduspider User-agent: Baiduspider-render User-agent: YisouSpider User-agent: AliyunSecBot User-agent: Reflectionbot User-agent: AionBot User-agent: GolfHunter Disallow: / # AhrefsSiteAudit (the crawler behind our own Site Audit reports) is a # different token, so the AhrefsBot block above does not touch it. It deliberately has # NO group of its own: a named group does not inherit "User-agent: *", and an # "Allow: /"-only group would lift every private-area Disallow below for it. # ClaudeBot (Anthropic training crawler) hammered imgproxy images at ~11 r/s # (2026-08-17); nginx now enforces 2 r/s with 429 + Retry-After. Crawl-delay # is the polite channel asking it to self-throttle. Training stays deliberately # allowed. A named group does NOT inherit "User-agent: *", so the effective # rules from that group are restated here (Allow: / covers the redundant # per-path Allows; only the overrides that must beat a Disallow are repeated). User-agent: ClaudeBot Crawl-delay: 1 Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /users/sign_up Allow: /inventory_analytics Disallow: /admin Disallow: /sidekiq Disallow: /backend Disallow: /inventory Disallow: /users Disallow: /rails/ Disallow: /api/v1/ Disallow: /demo/vin_lookup Disallow: /vehicle_map # ShapBot: kept off the per-VIN pages (/vehicle/...) — VIN-level pricing, # history and listings are licensed data, not crawl material (2026-10-07). # A named group does NOT inherit "User-agent: *", so the "*" rules are # restated (same pattern as ClaudeBot above) so ShapBot gets no MORE access # than before. "Allow: /vehicle/" from "*" is deliberately NOT copied: it # ties "Disallow: /vehicle/" on length and Allow wins ties. User-agent: ShapBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /users/sign_up Allow: /inventory_analytics Disallow: /vehicle/ Disallow: /admin Disallow: /sidekiq Disallow: /backend Disallow: /inventory Disallow: /users Disallow: /rails/ Disallow: /api/v1/ Disallow: /demo/vin_lookup Disallow: /vehicle_map # ExaSearchBot (Exa AI search crawler): same treatment as ShapBot — no # per-VIN pages. Owner decision 2026-10-07: Exa brings 0 visitors; it is # also blocked at Cloudflare. Rules mirror the ShapBot group above. User-agent: ExaSearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /users/sign_up Allow: /inventory_analytics Disallow: /vehicle/ Disallow: /admin Disallow: /sidekiq Disallow: /backend Disallow: /inventory Disallow: /users Disallow: /rails/ Disallow: /api/v1/ Disallow: /demo/vin_lookup Disallow: /vehicle_map # Everyone else — including AI assistant crawlers (GPTBot, ClaudeBot, # PerplexityBot, Google-Extended, Amazonbot, Applebot-Extended, CCBot, # Bytespider, etc.) — is ALLOWED on the public site so VinAssessment can be # discovered, indexed, and cited in AI search answers (ChatGPT, Claude, # Perplexity, Gemini, Google AI Overviews). Content-Signal explicitly opts in # to search indexing, AI answer grounding, and model training (being in # training data is deliberate: it teaches future models the VinAssessment # brand and product line). User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /products/ Allow: /data Allow: /api Allow: /mcp Allow: /pricing Allow: /demo Allow: /about_us Allow: /contact_us Allow: /sample_reports/ Allow: /prices Allow: /market Allow: /depreciation Allow: /research Allow: /vehicle/ Allow: /free-tools/ # Param variants of /contact_us (?origin=...) and the homepage (?for=shopper) # are intentionally NOT disallowed: blocking them left GSC stuck on "Blocked # by robots.txt" (failed validation 7/25/26). Since 2026-08-18 the ?origin= # CTAs are deferred-query (clean hrefs in HTML), but Google still recrawls # the known URLs from memory for weeks — they self-canonicalize to the clean # URL, which Google can only see if it is allowed to crawl them. # Private / non-indexable areas. # /users/sign_up must stay crawlable (longest-match beats "Disallow: /users") # so Google can follow its 301 to /pricing instead of reporting the ?plan=... # links from the old /pricing as "Blocked by robots.txt". (Until 2026-08-19 # this comment claimed the URL served a "noindex, follow" meta — it never # did: registrations are skipped in routes.rb, so the URL was a hard 404 and # GSC filed ?plan=mcp-growth under "Not found (404)".) Allow: /users/sign_up # /inventory_analytics is a public 301 to /products/pricing-intelligence, but # "Disallow: /inventory" is a prefix match and swallowed it — Google reported # it as "Blocked by robots.txt" instead of following the redirect. Allow: /inventory_analytics Disallow: /admin Disallow: /sidekiq Disallow: /backend Disallow: /inventory Disallow: /users Disallow: /rails/ Disallow: /api/v1/ Disallow: /demo/vin_lookup Disallow: /vehicle_map Sitemap: https://vinassessment.com/sitemap.xml