lynx-dev internal cache proposal

[Leonid Pauzner]

Lynx internal cache implementation proposal.

    - the source of every document now cached for greatly speed operations
      which require a re-parse, including '\' view-source,
      '^V' other-DTD, '*' image-URLs, '[' pseudo-alts,
      '"' soft-dquotes, '`' and "'" comment-parsing,
      '@' raw-mode, and changes in assumed document character set.
      This also work for downloading document from the History page
      and would probably usefull for future table implementation (if happend).

    - simple HTTP 1.1 cache implemented, e.g. if you request http:// document
      for purposes other than listed above, the document's source
      already in lynx internal cache and the requested method is GET -
      a "conditional GET" request follows (one include additional headers
      If-Modified_Since and/or If-None-Match against cached value Last-Modified
      and ETag). If the remote server (or proxy) detect no change and returns
      with 304 (Not Modified) status - our internally cached entry used.
      Proccessing the other statuses have not been changed.
      The header of cached entry updated with every responce.
      We also change lynx request header from "HTTP/1.0" to "HTTP/1.1"
      to allow server send us ETag for best caching validation
      (does we lost something that SHOULD be implemented in HTTP 1.1 client?).
      The expiration status calculated by LYinternal_cache_stale() function.
      The documents we sure as not expired will be loaded from internal cache
      immediately (no DNS search, no HTTP request).
      No methods other than GET http:// cached (except for internal use like 
^V).

      It should be stressed separately that according to HTTP 1.1 and 1.0,
      the expired or "no-cache" document may also be cached but
      the conditional GET request SHOULD be proccessed for validation
      by the remote server, not an intermediate proxy nor ourself
      (so "no-cache" directive act on validation, but data may be valid).

    - Implementation details: cache updated in LYAddVisitedLink(),
      e.g. when the document already received successfully,
      there should not be any problem from redirections.
      Klaus's internal (#fragment) links now obsolete and will be removed soon.

    - Possible problems/future domain:
      LYinternal_cache_stale() currently assume any document expired for sure,
      will be fixed soon. Method POST may be not cached for internal use
      like changing ^V etc. SSL? Redirections?
      Cache recycling mechanism?
      Persistent cache can be implemented easily when everything else done.

    - Questions:
      * If we interrupt transfer with 'Z' - the result will be cached
      for "internal use" but assumed as already expired for normal 
operations(?).
      Probably it should not be forced as expired but not persistent for sure.
      * Should any responces besides 200 (OK) status be cached?
      * #fragments?
      * compressed files?