To disallow any bot from crawling your site, add the following to robots.txt

robotUser-Agent: *
Disallow: /