Search Engines and Directories
How to get a listing
NOTE: Whilst a lot of the information here is still true and relevant it was writen in 1996 and needs updating. There is however an excellent site which has more information than I can ever convey at http://searchenginewatch.internet.com which you should visit if you are serious about getting a good ranking with Search Engines. (8th July 2000). This page has now been superceeded by a Search page in the Small Firms section (21st October 2003)
Introduction
Now we have got our site organised to the point where we are not
ashamed to have visitors and have a rationale for why we should
maintain our site (Mission Statement)
the time has come to get it listed on the various Search Engines.
At present our friends and some of our contacts hear about it
by word of mouth or because it is on the bottom of our letters
or mentioned in our Email. There are many others who are not aware
of its existence and it is important to us to maintain our "networks"
of contacts, especially when it comes to keeping up to date and
offering consultancy services.
Choosing Search Engines and Directories
There are several thousand Search Engines and Directories in existence
but fortunately the number where we need to initially inform of
the sites existence is much smaller. The main Browsers and service
providers give a simple entry point to searches with a page with
perhaps 5 or 6 searches available and sometimes a search of the
day. There is considerable overlap in the favorite searches they
suggest and that is where we will first put our efforts. The pages
I have looked at are - Microsoft Internet Explorer's All in One Search,
Netscape Navigators Net Search
and CompuServe. These feature:
- Yahoo! with 13 Million
hits per day
- Lycos
with 10 million sites (circa 91% of the Internet)
- Infoseek claims 5
million accesses per day
- AltaVista which
has very sophisticated indexing and searches of pages
- Magellan Indexes over
400,000 sites every day
- Excite Has indexed over
1.5 million sites and has detailed reviews on 35,000.
- HotBot which claims to
have indexed 54 million pages.
I also found there are a number of services for registering your
URL with a large number of Search Engines. Submit It!:
has many useful tips for submitting an URL to Search Engines and
Directories. This combined with a careful look at the various
help and FAQ pages for registration on the above services made
me realise that I had to do a considerable amount of preparation
to optimise the homepage and control what would be indexed and
how our site would be catagorised before I could go any further.
How do search Engines and Directories work
The following is some of the information I gleaned from my investigations:
- Search Engines utilize indexing programs to constantly
search the Web for new or updated pages going from URL to URL
until, in theory, they visit every Web site which is referenced
on the Internet. When visiting a Web site they record the full
text of every page (home and sub-pages) within the site then continue
on to visit all external links. Following links allows them to
find sites regardless of whether registered. Submitting a URL
only speeds up the process. An easy way to tell whether a Web
index is a search engine as opposed to a directory is by the information
it requires when adding your URL. A true search engine will only
need the Web address. The indexing agent takes care of the rest
- Directories are quite different. They do not
make use of indexing software so a directory can not list your
URL unless you or someone else registers it with them. The registration
form needs much more information than a URL. Directories are usually
subdivided into categories and you have to submit your URL under
the most appropriate heading - in some cases one is constrained
and in the case of YAHOO all personal home pages must go into
the Entertainment/People category. YAHOO needs a 15-20 word description
of the site and the keywords are derived from the URL, the Title
and this short description.
Changes required to the site
Optimising a site for Search Engines requires attention to:
- The Title: It is suggested that one uses keywords in
the <TITLE> of your home page and makes it as descriptive
as possible as the robots go first to the <TITLE> . This
is not the first HTML heading that shows up on your it is what
a browser will display in its title and is the text located between
the <TITLE> and </TITLE> tags.
- The use of <META> tags. Using these allows one
to provide more detail about your Web pages and to gain control
over how your pages are indexed. Not all search engines, however,
make use of <META> tags, but adding these tags to ones pages
will make them more accessible to the search engines that do.
<META> tag codes are inserted within the <HEAD> and
</HEAD> tags. Two tags are important.
- "Description" Tag: The <META name="description"
content=" "> will determine what appears as the summary
of a Web page and will be displayed after the title of your document
in the index listing.
- "Keyword" Tag: The <META name="keywords"
content=" "> allows provision of extra information
about your page to the search engines without it being visible
to the reader. Search engines usually add these keywords to the
those found in the contents of the page.
- Are Meta Tags essential? Not in most cases as the robots
default to the text in the page - usually that at the top for
the description. It is however currently essential to use <META>
tags if one has Netscape Frames or Javascript at the top of the
page. Some indexing software places preference on complete, punctuated
sentences.
INFOSEEK - an example Search Engine
Infoseek provides a good and well documented example of
what a Search Engine requires and is the model I have used in
preparing my META tags. It supports both the keywords and description
tags. The description can include up to 200 characters of text
and the keywords can include up to 1000 characters of text. One
is warned not to repeat versions of a keyword more than seven
times or InfoSeek will disregard the entire keyword list. If one
does not make use of the description <META> InfoSeek's agent
will use the first 200 characters after the <BODY> tag as
the web page description. I assume this to be common practice
so have taken care to make the first part of the site an accurate
description and I have also included plenty of key words in the
top part.
Changes to our Homepage
The careful use of META tags and choice of title has enabled our
site to have an open and a hidden agenda. Most of our friends,
colleagues and other professional contacts who visit the site
will learn about it from communications from us and other people
on our personal "network" and will go straight to the
home page. We, however, also carry out consultancy services and
those seeking services are more likely to come to the site via
a Search Engine so the key word list is long and features many
of our skills and background to ensure plenty of hits. The first
contact with the site from a Search will be the text from the
<META name="description" content="Consultancy
built on experience in UK government service and research to advise
on policy, strategy and tactical approaches. Specialities: longer
term innovation, space, earth observation and quality.">.
First Registrations
At the first couple of runs I registered
- Infoseek - via an Email of the URL
- Altavista - which just needs URL. It immediately accessed
the site and promised to add within 1 day
- HotBot - which just needed the URL and confirmed.
- Lycos which took the URL and said it would take 6 weeks
- Yahoo - which needed the 20 word summary which I had carefully
prepared in advance and had on the clipboard ready.
- and Magellan - which needed a 75 word summary and 4 key words which I had to stop and prepare
Yahoo had a list of other sites for registration which produced
a few which seemed worth looking at and I registered on the fly
(saving bookmarks of where they were of course) at:
- WebCrawler - only the
URL was needed (in fact it will take 9 URLs at a time so it may
be worth adding individual pages latter)
- WWW Worm
- this needed a short description so I provided the 20 words I
had already prepared for Yahoo prefixed with the homepage title.
An Audit trail
Netscape bookmarks are in an HTML file (Bookmark.htm in the Netscape
Folder) which can be accessed and the URLs copied into documents
such as this for documentation. This page was written and updated
as I went along and serves as the documentation for what I have
done so far. The hot links to the services were pasted in from
the Netscape Bookmarks after I had found the sites and then formed
the links I used to register. Notes were made and this page updated
as soon as I left the Browser after each registration.
Back to diary item | Back to the home page
Copyright ©; Peter Curtis
Most recent revision: 27th August, 1996
Notes added: 8th July 2000 and 21st October 2003