To disallow any bot from crawling your site, add the following to robots.txt robotUser-Agent: * Disallow: /