Search Engine Coverage Tool

This freeware tool will allow you to see at a glance how well indexed your web pages are by major search engines. The easiest way to understand what this tool does is to look at the example report below. You configure the tool with a list of your web pages. You run the tool and a few minutes later you'll get a report showing which pages are indexed by which search engines.

	Google Aol	Yahoo AllTheWeb Altavista	MSN Search	WebSquash	Exalead
http://www.searchenginecoverage.co.uk/	yes	yes	yes	no	yes
http://www.searchenginecoverage.co.uk/page1.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/page2.htm	yes	yes	yes	no	yes
http://www.searchenginecoverage.co.uk/contactus.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/about.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/profile.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/page3.htm	yes	no	yes	no	yes
http://www.searchenginecoverage.co.uk/page4.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/page5.htm	yes	no	yes	yes	yes
http://www.searchenginecoverage.co.uk/page6.htm	yes	yes	yes	yes	no
http://www.searchenginecoverage.co.uk/page7.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/page8.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/page9.htm	yes	no	yes	yes	yes
http://www.searchenginecoverage.co.uk/pgae10.htm	yes	yes	yes	no	yes
http://www.searchenginecoverage.co.uk/page11.htm	yes	yes	yes	yes	yes
http://www.searchenginecoverage.co.uk/page12.htm	yes	yes	no	no	yes
http://www.searchenginecoverage.co.uk/page13.htm	yes	no	yes	yes	no
http://www.searchenginecoverage.co.uk/page14.htm	yes	yes	yes	yes	yes
	Produced by SearchCoverage www.searchenginecoverage.co.uk

Usage

Create a 'WEBPAGEFILE', a text file containing the URLs of your web site. Edit the SearchCoverage.conf file to point to it. Then double click on SerachCoverage.exe. A few minutes later (or more if you enter a lot of URLs) the program will exit and you will have a report.html in the same directory.

Easy way to produce the 'WEBPAGEFILE'

You can produce the list of web pages however you like. This is one possible way.

Open a DOS window. Press 'Start' -> 'Run' and then 'cmd' will do it.

CD to the directory containing your web pages. e.g.

f:
cd f:\Work\Website\html\

Type 'dir /s/b > webpagefile.txt'. (Or 'dir /s/b *.htm*' to restrict it to HTML files if you have images in the directory as well). Open the newly created 'webpagefile.txt' which is in the current directory. It will look something like this.

F:\Work\Website\html\advancedhtml.htm
F:\Work\Website\html\animbuttons.htm
F:\Work\Website\html\basictags.htm
F:\Work\Website\html\colours.htm
F:\Work\Website\html\dictionary.htm
F:\Work\Website\html\faq.htm
F:\Work\Website\html\favicon.htm
F:\Work\Website\html\fframe.htm
F:\Work\Website\html\frame.htm
F:\Work\Website\html\framebutton.htm

Do a find and replace of the directory name for your web site URL. e.g. in this case I replace 'F:\Work\Website\html\' with 'http://www.searchenginecoverage.co.uk/'.

Limitations

SearchCoverage will only work with small to medium sized sites. I've used it succesfully on sites from around 10 web pages, up to a 100. Many search engines place restrictions on the number of results they return so there is no way it will work on your 10,000 page site!
SearchCoverage will keep retrieving results from the search engine until it reaches a page with no more results. Therefore it will only work properly if you include all your pages in the WEBPAGEFILE. Including 10 out of your 30 pages will not work correctly.
It is unlikely this script will work with dynamically produced pages. e.g. ones with the '?' symbol and then a dynamic part after it. Most search engines don't index these pages anyway.
The speed of this script is deliberatelly throttled so that the search engines don't get bombed with queries.
If your root page is contained in the ROOTPAGE section of the configuration file then it will only produce a match for the http://www.bbc.co.uk/ style URL and not the http://www.bbc.co.uk/index.html style URL if both are included in the WEBPAGEFILE.

Download

SearchCoverage 0.4 (642kb)

Installation

Simply unzip the files into whatever directory you want to install the program to.

Future Features

Statistics for the % coverage and your count of covered pages for each search engine.
Support for more search engines. I would especially like to include Ask Jeeves but it does not allow you to get a list of indexed pages in a domain - even though its help pages tell you it can.

Contact Us

If you have any comments or suggestion then feel free to get in contact with me using the form on the Contact Us page. If you have set up any custom search engines in the configuration file then I'd really like to recieve the settings so they can be included in a future version of SearchCoverage.

History

0.4 19/02/06
First version which is actually suitable for public viewing! It is still far from perfect so don't expect miracles.

0.3 - 05/02/06
Now able to cope with the fact that search engines may index your home page differently. e.g. 'www.bbc.co.uk/' could be indexed as 'www.bbc.co.uk/index.html' or 'www.bbc.co.uk/index.php'.

0.2 - 30/01/06
Improved to work with more search engines. Can now work with pages from more than one domain.

0.1 - 29/01/06
Search coverage result are shown in a HTML table format which is easy to understand.

0.0 - 28/01/06
First prototype version completed. It does the basic job of printing a list of which pages are listed in Google, MSN and Yahoo.