[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: lynx-dev Give me a help!!!

From: David Woolley
Subject: Re: lynx-dev Give me a help!!!
Date: Sat, 17 Mar 2001 09:36:54 +0000 (GMT)

> How can I use lynx for grabbing web sites--as "Web Copier", "Whacker"...
> If possible, advice me scripts or tools that moves the sites to own disk.

Do not use Lynx for this purpose.  Many sites consider this a violation
of their conditions of use and will block users and programs that do
it (e.g. the International Movie Database).  You are less likely to
get blocked if you obey the rules for web robots, which allow sites
to exclude them from all or part of a site, and require a 30 second
delay between accesses.  wget is a program designed for the purpose,
which does obey the rules, but Lynx's web crawling capabilities were
only intended for internal use and it does not obey the rules.

Using Lynx for this purpose may well result in all users of Lynx being
barred from some of the sites accessed.

; To UNSUBSCRIBE: Send "unsubscribe lynx-dev" to address@hidden

reply via email to

[Prev in Thread] Current Thread [Next in Thread]