Skip to main content

Block web crawlers

The Bravura Security Fabric allows you to block the web crawling by adding web crawlers to the robots.txt file located in the samples directory. To block web crawlers, place robots.txt file in C:\inetpub\wwwroot.

By default robots.txt comes with blocking following web crawlers:

# Block all but AdsBot crawlers
User-agent: *
   
Disallow: /
# Block Google AdsBots
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile-Apps
Disallow: /
In this section: