Sounds strange doesn’t it? “keep a website out of the search engines”?? Why would anyone want to do that? Well there may be several reasons. For example, the site is in development and you don’t want it indexed before its ready to be indexed. Maybe you have a private member site that is for a small group like your family or church group and you don’t want it showing up all over Google. There could be a number of reasons you would want to do this. Well the good news is, you can — and its easy!

Important: It makes a difference if your site is already indexed in Google or Yahoo or where ever.

Method 1 - For a brand new website that has never been indexed at all…
1. First check to make sure it really hasn’t been indexed. You may think to yourself, “I just bought this and set it up on the hosting yesterday, it can’t be indexed already!”. But you’d be surprised how fast it can get picked up these days. Make sure it is NOT indexed at all! (in Google type “site:yourdomain.com” — without any quotes and change the domain to your domain. You should see a message

“Your search – site:yourdomain.com – did not match any documents.”

This is good, you may proceed to step 2. If it shows some indexed pages and not that message above, move to Method 2 below.

2. Create a robot.txt file with the following code:
# go away
User-agent: *
Disallow: /

3. Upload to the root of your website. DONE!

Now if Google, Yahoo, Bing or any other legitimate search agent tries to get it, it will be rejected right at the gate.

Method 2 - If your site is already indexed you must NOT put that robot.txt file there. “Why?” you ask. Because if its already indexed, the search engines cannot learn to de-index it because they cannot get in. Your existing indexed pages will remain indexed. So if your pages are showing up in the search results do the following:
1. Remove any robot.txt file (for now, you can put it back later when its all clear)
2. Add the following meta-tag to every page on the site… EVERY PAGE! <meta name=”robots” content=”noindex,nofollow” />
3. Upload all pages to the live server and you are done.

Now you can just wait it out. It may take a few weeks for the search engines to catch on. If your site was in existence for some time and there were a large amount of pages, it will take quite a while to remove it all. For removing a dead site out of existence, see my article How to Remove an Entire Website from the Internet

Another step you can use if you have Google Webmaster Tools account is to “remove URL” feature. In Google Webmaster Tools under the domain you are working with select Site Configuration>> Crawler Access >> Remove URL (tab). If you can get your hands on that, it will remove the site from Google pretty quick. You have to have verified ownership of the account, so don’t try this with your competitors website, it won’t work. However, this will not remove it from other Search Engines like Yahoo. For that you will have to wait for the noindex, nofollow tag to take effect.

Also note that bad bots (naughty evil search agents) will still ignore these directives and follow and index all the day long. So this works to get your site out of the important nice search engines, not the naughty ones.

Apparently this is nothing new as there are lots of other posts about it. So, I have been receiving letters from Domain Registry of America notifying me that certain domains I own are expiring. Sounds like a thoughtful, service-oriented company doesn’t it? That might be true, except, I don’t have any domains registered with Domain Registry of America.