[Axiom-developer] search engines accessing MathAction websites

[Bill Page]

On December 1, 2005 10:29 PM I wrote:
> Subject: [Axiom-developer] RE: address@hidden: 
> [RHH#FKL-86268-984]: move Axiom to shared dedicated server]
> 
> ... 
> I have also cut-back the robots.txt to be more selective
> about what we allow the search engines to do.
> ...

After thinking about this a little more I realized that one of
the things that causes a higher access rate for the robot search
engines and spiders is that we now have multiple virtual urls for
the same MathAction web sites:

http://page.axiom-developer.org  (wiki and portal)
http://www.axiom-developer.org  (wiki and portal)
http://wiki.axiom-developer.org 
http://portal.axiom-developer.org

Since the robots can not tell which of these urls map to which
web sites, it is a result it is possible that we are being
scanned 3 times for the same content!

So today, I made url specific urls that block all robots from the
first two urls but allow full indexing of both the wiki and the
portal (except for bibliography and SandBox pages):

User-agent: *
Disallow: /SandBox
Disallow: /bibliography
Disallow: /refs/axiombib
Disallow: /refs/redbib

-------

I also changed

http://www.axiom-developer.org and
http://axiom-developer.org

to map directly to http://wiki.axiom-developer.org

So far, this seems to have resulted in significantly lower rate
of access by these robots.

As a result, the performance of the MathAction websites now
seems to me to be very fast. When you have some time, please
try the Axiom wiki and portal sites and comment on how fast
(or slow) they seem to you. Your comments on these changes are
welcome.

Regards,
Bill Page.