# Groups mirror the tiers in thchunter/middleware.py (BlockIPMiddleware). # Crawl-delay is kept below each tier's enforced per-minute limit so compliant # crawlers never receive 429s. Crawlers obey only their most specific group. # Search engines: not rate limited. Partials stay crawlable because pages load # content through HTMX and both engines render JavaScript. User-agent: Googlebot User-agent: Bingbot Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ # AI fetchers answering a live user question. No Crawl-delay: they fetch a few # pages per question. Enforced: 240/min overall, 90/min heavy, 40/min county. User-agent: ChatGPT-User User-agent: Perplexity-User User-agent: Claude-User Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /partials/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ # AI search and training crawlers (GEO). Enforced: 120/min overall, # 40/min on deal/city/dispensary/company pages, 30/min on county pages. User-agent: OAI-SearchBot User-agent: GPTBot User-agent: PerplexityBot User-agent: Claude-SearchBot User-agent: ClaudeBot User-agent: Applebot Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /partials/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ Crawl-delay: 2 # SEO tools. Enforced: 120/min overall, 30/min heavy pages, 20/min county pages. User-agent: AhrefsBot User-agent: SemrushBot Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /deal/ Disallow: /partials/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ Crawl-delay: 3 # Site audits. Enforced for unverified IPs: 10/min overall, 5/min county pages. User-agent: AhrefsSiteAudit Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /partials/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ Crawl-delay: 12 User-agent: ShapBot Disallow: /deal/ Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /partials/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ Crawl-delay: 30 User-agent: Bytespider User-agent: TikTokSpider Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /deal/ Disallow: /partials/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ Crawl-delay: 30 User-agent: MJ12bot User-agent: DotBot User-agent: Barkrowler User-agent: PetalBot Disallow: / # Everyone else. Enforced for unlisted bots: 60/min overall, 20/min heavy pages, # 10/min county pages. User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /reviews/write/ Disallow: /partials/ Disallow: /search-location/ Disallow: /search-location-json/ Disallow: /search_location/ Crawl-delay: 10 Sitemap: https://thchunter.com/sitemap.xml