Re: lynx-dev The traversal limitation

[David Combs]

RE "well-behaved crawlers" vs "not well-behaved"

For those of us who know nothing much about robots, and
whose experience with this idea is that alias "treelynx",
which seems to do what it advertises -- could someone
say some more about what makes wget so different?

(I have not seen any book with robots as subject -- do
you know of any?  Or with this general subject in mind.)

Vocab: would lynx in treelynx-mode a "robot"?



by treelynx, I mean:

{netcom9:276} alias treelynx
/usr/local/nuglops/bin/lynx -show_cursor -vikeys -cache=20 lynx_bookmarks.html  
-traversal -crawl -realm  > /dev/null 
{netcom9:277} 


Thanks.  Am sure there lots of us who wouldn't mind knowing
some more about this subject!

David