Re: LYNX-DEV Since Lynx won't, what will?

[Jim Dennis]

>> My hope is that a number of  webmasters and sysadmins will use
>> Lynx   as  their prefered   "web  automation"  tool  and  will
>> consequently put furhter pressure  on their designers  to have
> 
> Please don't even think of marketing anything like this to this market 
> unless it fully supports robots.txt (even if there are non-default
> options to defeat it).  Your proposed market is not going bother
> with enforcing such restrictions manually.

        As Fote points out -- my market is for the sysadmins and webmasters
        hitting their own site (from their own LAN or even localhost).

        One nice thing about using -traversal is that you get
        "orphan detection" -- you can find any of your files that
        have not links to them (with a little filtering and a diff).

        Another nice thing is you can prepare a ToC.html (table
        of contents) which is a list of every link  that's anywhere
        on your site -- and another one that lists ever link that
        exits your site.

        In any event, I agree that it should have support for 
        robots.txt by default -- with a command line switch to 
        over-ride that (and maybe a global configuration option 
        to over-ride *that* over-ride and mail a nastygram to 
        the SA when a user attempts to do so).

--
Jim Dennis  (800) 938-4078              address@hidden
Proprietor, Starshine Technical Services:  http://www.starshine.org
        PGP  1024/2ABF03B1 Jim Dennis <address@hidden>
        Key fingerprint =  2524E3FEF0922A84  A27BDEDB38EBB95A 
;
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.
;