[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: lynx-dev Web robots
From: |
Philip Webb |
Subject: |
Re: lynx-dev Web robots |
Date: |
Mon, 7 Sep 1998 17:11:02 -0400 (EDT) |
980907 Jonathan Lawson wrote:
> does anyone know about web robots and how to write/implement them?
my personal & ever-growing file of useful URLs tells me:
Agents www.opensesame.com/agents/
Agents: I/Net intelligence www.cs.umbc.edu/agents/
FAQ & listserv www.ee.mcgill.ca/~belmarc/agent_faq.html
User-agent string www.idrc.ca/cgi-bin/bc.cgi
ditto sailor.lib.md.us/cgi-bin/find-agents
Robot searchers : info.webcrawler.com/mak/projects/robots/robots.html
-/-/-/-/norobots.html
> 'They came forth from unholy darknesses ...
> and were driven back by the rage of Angels'
robot spammers, maybe?
--
========================,,============================================
SUPPORT ___________//___, Philip Webb : address@hidden
ELECTRIC /] [] [] [] [] []| Centre for Urban & Community Studies
TRANSIT `-O----------O---' University of Toronto