Search engines: Friends or foes?

By Published on .

Question for IM What's your name? What's your email? Where do you work? Do you have a question for Inquiring Minds? Something you've always wanted to know but never known who to ask? Let us know, and we'll do our best to answer it. Our Web site used to reside within another company's domain. We went ahead and acquired our own domain name and placed it with all the search engines we knew, but every time we go to look for our site, we also find the old site and when we select the old site, it brings us right back to the company we used to be listed under! How can we get our old Internet address off all the various search engines?

One little question, so many issues. Let's dive right in then, shall we?

First off, you clearly read the September issue of NetMarketing, which told you how to register your own domain, and the October issue, which told you how to register your site with search engines. Good job! Dealing with the aftermath is going to get a little technical.

Let's start with how the search engines work. The enormity of the Web means that putting together a database of Web sites must be automated. So the search engines have developed programs that go around the Web and catalog Web sites automatically. These programs are called robots, or spiders.

According to several of the major search engines, (HotBot, AltaVista, Lycos, InfoSeek, and Excite) old addresses get expunged from the database periodically if the site no longer exists. Different sites have different refresh rates, usually every couple of weeks.

Yahoo!, because it's more of an index than a search engine, has a quick form for changing listings on its site.

You do have other alternatives. Your best bet is to work out something with your Web provider when you first launch your site (if you don't have your own domain). Ask if it will leave a page at your old address for some period of time with your new address in case you ever decide to part company.

Another option is to have your old site include a special file called robots.txt. This file tells robots which files they should not catalog. Not all engines support it, nor is it a standard, but many of the major engines do support it and it's simple to set up.

To learn more, read the "official" robots page.

If you have your own domain, all of this is moot because you can just move it to a new provider. If you plan to be on the Web for a while, and can afford it, it's worth the registration fee and the slightly-higher monthly fees that most ISPs charge for maintaining your own domain.

OK, Mr. IM, how can I make search engines catalog my site the way I want?

Many of the major search engines support the use of "META tags." METAs are headers to the html documents on your site. These tags can do a lot of different things but one task is to set up key words for your site.So if I wanted to set up this column on the NetMarketing site the header might look like this:

You can also set the description that a search engine returns to searchers about your site:

Ask your Web developer if you use them, or even if you just want more information about them. Keep in mind, there is no standard here. For instance, Infoseek doesn't support META keywords because there is nothing to stop people from putting whatever they want in that section -- including things that are false or not relevant to the actual content of the site -- to get their pages to come up under more searches.

The bottom line is that every little bit helps and none of this hurts.

Most Popular
In this article: