[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Axiom-developer] search engines accessing MathAction websites

From: Bill Page
Subject: [Axiom-developer] search engines accessing MathAction websites
Date: Fri, 2 Dec 2005 14:50:41 -0500

On December 1, 2005 10:29 PM I wrote:
> Subject: [Axiom-developer] RE: address@hidden: 
> [RHH#FKL-86268-984]: move Axiom to shared dedicated server]
> ... 
> I have also cut-back the robots.txt to be more selective
> about what we allow the search engines to do.
> ...

After thinking about this a little more I realized that one of
the things that causes a higher access rate for the robot search
engines and spiders is that we now have multiple virtual urls for
the same MathAction web sites:  (wiki and portal)  (wiki and portal)

Since the robots can not tell which of these urls map to which
web sites, it is a result it is possible that we are being
scanned 3 times for the same content!

So today, I made url specific urls that block all robots from the
first two urls but allow full indexing of both the wiki and the
portal (except for bibliography and SandBox pages):

User-agent: *
Disallow: /SandBox
Disallow: /bibliography
Disallow: /refs/axiombib
Disallow: /refs/redbib


I also changed and

to map directly to

So far, this seems to have resulted in significantly lower rate
of access by these robots.

As a result, the performance of the MathAction websites now
seems to me to be very fast. When you have some time, please
try the Axiom wiki and portal sites and comment on how fast
(or slow) they seem to you. Your comments on these changes are

Bill Page.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]