|
From: | Patrick Vervoorn |
Subject: | Re: [Lynx-dev] Viewing Dave's Trailer page with Lynx 2.8.7dev.5: error |
Date: | Thu, 16 Apr 2009 15:02:18 +0200 (CEST) |
User-agent: | Alpine 2.00 (DEB 1167 2008-08-23) |
On Thu, 16 Apr 2009, Chuck Houpt wrote:
Hi Patrick,It looks like the website is browser sniffing, and rejecting anything with 'libwww' or 'libcurl' in the User-Agent field. Here are some tests with curl that show the blocking of libwww/libcurl, but not 'foobar':$ curl -A 'libwww' -I http://www.davestrailerpage.co.uk/ HTTP/1.1 406 Not Acceptable $ curl -A 'libcurl' -I http://www.davestrailerpage.co.uk/ HTTP/1.1 406 Not Acceptable $ curl -A 'foobar' -I http://www.davestrailerpage.co.uk/ HTTP/1.1 200 OKPossibly the site is trying to deter rogue robots and scrapers, but filtering based on the User-Agent field won't be very effective, since it is easily changed (for example from the Lynx options page).
Aha, I did not know that. I changed the User-Agent field in my lynx.cfg to just plain 'Lynx' and the site is displaying again.
I also forwarded your comments to Dave himself, perhaps he can do something about it. This limitation was introduced when he recently moved the site to a new host, so perhaps it's something which was introduced by his new hoster.
Thanks for the explanation in any case. As to the other response: my apologies for not searching through the archives before posting (emailing) my question; too late for that now though...
Cheers - Chuck http://chuck.habilis.net
Regards, Patrick.
On Apr 16, 2009, at 4:06 AM, Patrick Vervoorn wrote:Hello,My apologies for sending this 'out of the blue', since I did not subscribe to the lynx-dev list.However, I'm seeing a problem which is rather Lynx-specific, since viewing the same URL, on the same machine, but using Lynx, works rather ok.The full story is in the email below which I sent to Dave, but since w3m works fine, I'm not sure it's a problem with the site...Regards, Patrick. ---------- Forwarded message ---------- Date: Sun, 12 Apr 2009 17:01:51 +0200 (CEST) From: Patrick Vervoorn <address@hidden> To: address@hidden Subject: Viewing Trailer page with Lynx: error Hi there,When I try to view your site after the latest move with Lynx, I invariably get the following error-message:(This is when viewing both the main site at: http://www.davestrailerpage.co.uk/ but also happens when I try to view one of the sub-pages directly, like: http://www.davestrailerpage.co.uk/about.html)Not Acceptable An appropriate representation of the requested resource / could not be found on this server. Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.When using something like w3m, it is displayed. Using a graphical browser like Firefox also works fine.Is there anything you can do from your side to be able to view your Trailer page using Lynx?Best regards, Patrick. _______________________________________________ Lynx-dev mailing list address@hidden http://lists.nongnu.org/mailman/listinfo/lynx-dev
[Prev in Thread] | Current Thread | [Next in Thread] |