Re: lynx-dev Non-interactive lynx

[Patrick]

In "lynx-dev Non-interactive lynx"
[17/Mar/2001Sat 13:13:00]
Ilya Zakharevich wrote:

> Re prohibiting lynx from visiting sites due to misusage of
> unattended-operation mode.  What about prefixing "Non-interactive " to
> the default user-agent string for non-interractive robot-like runs of
> lynx?

I think it would still provoke those who spend time and consideration
on which of their files have;
        <META NAME="robots" CONTENT="all/none/nofollow/noindex">
and so forth.  Also bear in mind that no robot can read copyright
notices in the body of a page.

Just wondered: how easy/hard would it be to make Lynx obey robot
exclusion protocols in non-interactive mode?  This is also done
with HTTP headers?


                             Patrick
                   <mailto:address@hidden>
                <http://www.island.net/~pboylan/>

; To UNSUBSCRIBE: Send "unsubscribe lynx-dev" to address@hidden