[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Axiom-developer] RE: search engines accessing MathAction websites
From: |
Bill Page |
Subject: |
[Axiom-developer] RE: search engines accessing MathAction websites |
Date: |
Fri, 2 Dec 2005 14:54:45 -0500 |
I wrote:
>
> So today, I made url specific urls that block all robots from the
> first two urls but allow full indexing of both the wiki and the
> portal (except for bibliography and SandBox pages):
>
> User-agent: *
> Disallow: /SandBox
> Disallow: /bibliography
> Disallow: /refs/axiombib
> Disallow: /refs/redbib
>
I meant:
So today, I made url specific robots.txt files that block all
robots from the ...