Important Search Engines
Search directories are a hugely important part of the web. It is truly impressive what you can turn up by searching for keywords on just about any subject. There are really two types of web index, the search spiders or worms which crawl over the web tracing and indexing URL's they find starting from those submitted, and the structured indices where each URL submitted is checked, and classified.
Despite constant updating, all of these directories index only a small fraction of the whole web which is growing at a phenomenal rate. The spiders seem to have a hard time just keeping up with the URL's which are submitted to them. I used to put a lot of effort into submitting the URL's of my pages to these directories and following how they were taken up. It always takes a few weeks for them to come round and even after a few months I had only managed to get about half my pages indexed on any of them. Then one weekend almost all my pages disappeared from some of the larger directories. I suppose that the CompuServe web page server must have been down when they visited so the URL's got wiped off.
Maintaining these directories must require a huge effort so those which work deserve applaud. When the web was new several of these services popped up. Now they have almost all either gone commercial or fallen into disuse. The best of what remains are listed here. Perhaps market forces will be sufficient to encourage the invention of a technique to make these directories more complete.
Comments :
Post a Comment