Journal ·
robots.txt FileTo exclude all Robots From The Entire Site
Written in 2015. Some steps may have changed since.
If you don’t want to let your sites get indexed on search engines, you can disable access via robots.txt file.
Create a file named robots.txt and upload it to your root directory of domain.
User-agent: *
Disallow: /
To exclude all robots from certain directories robots.txt:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/