Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.bunny.net/llms.txt

Use this file to discover all available pages before exploring further.

To prevent abuse, Bunny Shield verifies the authenticity of every bot request before allowing it through.

​
Verification Process

Not all traffic claiming to be a search engine bot is genuine. Many malicious actors spoof user agents to bypass protections. To solve this, Bunny Shield verifies the authenticity of SEO bots using reverse DNS lookups (PTR records):
  1. The bot’s IP address is resolved to its PTR hostname.
  2. The hostname is checked against our whitelist of trusted domains.
  3. If the hostname matches, Bunny Shield performs a forward DNS lookup and confirms the hostname resolves back to the same IP range.
Only when these checks pass is the bot classified as legitimate and allowed through.

​
Supported Verified Bots

The following crawlers and preview agents are verified and automatically allowed across all Bunny Shield layers.

​
Search Engine Crawlers

  • Google: .googlebot.com, .google.com
    • Googlebot
    • Googlebot-Image
    • Googlebot-Video
    • Google-InspectionTool
    • Google-Site-Verification
    • GoogleProducer
    • APIs-Google
    • Storebot-Google
    • Google-AdWords-Instant
    • Google-AdWords-Express
  • FeedFetcher-Google: .googleusercontent.com, .google.com
  • AdsBot-Google: .googlebot.com, .google.com
    • AdsBot-Google
    • AdsBot-Google-Mobile
  • AdsBot-Google-Mobile-Apps: .google.com
  • Mediapartners-Google: .google.com
  • Yahoo! Slurp: .crawl.yahoo.net
  • Bingbot (Microsoft): .search.msn.com
  • Yandex: .yandex.ru, .yandex.net, .yndx.net, .yandex.com
  • Baiduspider: .crawl.baidu.com, .baidu.jp
  • Applebot: .applebot.apple.com
  • MojeekBot: .mojeek.com
  • QwantBot: .qwant.com
  • PinterestBot: .pinterest.com
  • PrerenderGateway
    • Used by our Optimizer HTML Prerender service for SEO and JavaScript rendering.

​
Performance & Auditing Tools

  • Chrome Lighthouse: .googleusercontent.com, .google.com
  • Facebookexternalhit: .facebook.com, .fbsv.net
  • Twitterbot / X Preview Bot: .twttr.com, .twitter.com, .x.com
  • LinkedInBot: .linkedin.com
  • Pinterest: .pinterest.com
  • WhatsApp: .whatsapp.net
  • Tumblr: .tumblr.com
  • TelegramBot: .telegram.org

​
Messaging & Collaboration Preview Bots

  • SkypeUriPreview: .skype.com, .skype.net

​
Why Verification Matters

  • Prevents abuse: Many malicious crawlers disguise themselves with fake user-agent strings. PTR verification means only real search and social bots get access.
  • Preserves SEO: Legitimate crawlers are never blocked by WAF rules, rate limits, or bot detection challenges.
  • Improves security: Fake bots are denied, reducing the chance of scraping, spam, and credential stuffing attempts.
Legitimate SEO bots are vital for site visibility and usability. With Bunny Shield, you can rest assured that authentic crawlers pass freely while fakes are stopped at the edge.