despens: This is already possible ⚫ via robots.txt https://support.google.com/webmasters/answer/156449?hl=en ⚫ and meta tags in HTML https://support.google.com/webmasters/answer/93710?hl=en It is important that this convention will be respected in the future.
Brett O'Connor: Even with properly configured robots.txt it seems a website can appear in Google. For example: - My robots.txt http://negatendo.net/robots.txt - Google results: https://www.dropbox.com/s/bzv3fo1mzg17vvb/googled.png "How do I keep a robot off my server?" http://www.robotstxt.org/faq/away.html
despens: This is already possible ⚫ via robots.txt https://support.google.com/webmasters/answer/156449?hl=en ⚫ and meta tags in HTML https://support.google.com/webmasters/answer/93710?hl=en It is important that this convention will be respected in the future.
Brett O'Connor: Even with properly configured robots.txt it seems a website can appear in Google. For example: - My robots.txt http://negatendo.net/robots.txt - Google results: https://www.dropbox.com/s/bzv3fo1mzg17vvb/googled.png "How do I keep a robot off my server?" http://www.robotstxt.org/faq/away.html