Re: [Pan-users] cache article inconsistent

[Duncan]

Thufir Hawat posted on Tue, 26 Mar 2013 02:06:39 +0000 as excerpted:

> Are all articles equally "cache"-able?

In my experience, yes... with one exception that's server-side, thus out 
of pan's control (plus another couple that would be local admin issues, 
not under pan's control).  See below...

> http://www.zimagez.com/zimage/screenshot-03252013-070130pm.php
> 
> I selected all read articles, cached them ok.  The I selected more
> articles, the unread articles you see in the screenshot, and selected to
> cache them.
> 
> Now, I have a cache set to 100MB but might not have that much RAM free.
> In this case, cache means to RAM or hdd?  I tried setting 1000MB but
> still see this, so brought cache back to 100MB.

The cache is definitely filesystem storage.  I've been using a multi-gig 
cache for years, back long before I had multi-gig memory!  (For my pan 
binary instance, I have pan's cache symlinked to a dedicated 12-gig 
partition, with pan set to use it all.  For my text instance, pan is set 
to ~5 gigs in my home dir, with all servers set unexpiring and an actual 
usage after several years of under a gig, and several "private" groups 
from a no-longer existing ISP-server archived.  While I have a 16-gig RAM 
machine now, IIRC when I set this up it was half a gig, and back with pre-
C++-rewrite pan 0.14.x, I remember using a 4-gig cache with a quarter gig 
of RAM, possibly less.)

At present, I believe pan keeps the overviews (basically the headers seen 
in the headers pane, tho that's not all headers) for all not deleted 
messages in memory at once, thus its scaling issues on extremely large 
groups, particularly for 32-bit systems where a single app's RAM can't 
exceed 2 GB with a normal kernel, 4 GB with the "big-mem" kconfig options 
applied.  Additionally, pan reloads and re-threads all those messages at 
startup, so with my unexpiring text instance, for instance, initial 
(system level) cold filesystem cache loading takes several minutes -- I 
start it with my kde/X session and let it churn, and a few minutes later 
it's ready.  (Once the system has warmed up its filesystem cache, 
restarting pan is still practically instantaneous; it's the first cold-
cache load that takes the time.  Also note that a file-level backup and 
restore does make the load time dramatically shorter, as the files are 
defragged as they're copied back.  I imagine load times would be near 
half an hour here now, almost definitely at least 15 minutes, if I hadn't 
periodically done that over the years.  Of course if I had the files on 
SSD instead of "spinning rust"...)

> In any event, why weren't the read articles unchached so that the unread
> articles could be cached?
> 
> This is not an isolated instance, just a good screenshot.  Some articles
> get cached, others don't.  Why/not?

In the screenshot, it looks like they're all cached but one.  I'd guess 
that one was server error when pan tried to fetch it.  You can click it 
and see if pan loads it on demand.  In my experience it seldom can, as 
the server has either expired it, or removed it due to message cancel, or 
simply lost track of it due to a bug, or whatever.  It's one of the older 
but not the oldest message in the screenshot and I don't know your 
server's expiration policy, but it's worth noting that servers typically 
don't expire strictly by posted date, more often by received date (which 
you won't see), and that when they expire, it's bunches at a time, so as 
expiration time nears, more and more posts will be unavailable, but 
you'll often see posts older (by posting date) than the unfetchable ones 
that fetch just fine.

Meanwhile, just because pan has a set cache size doesn't mean there's 
actually that much room in the cache partition to save the posts, and for 
systems with user quotas or the like, it could be the quota that your 
hitting, not partition size limits.  Finally, filesystem damage can have 
some curious effects, so if you've not fscked the filesystem in awhile, 
try that too.

But for a single post like that, I'd assign about 98% probability to it 
being the server's issue -- the server still had the message in its 
overviews when they were fetched, but when you went to fetch the message 
itself a bit later, it couldn't find it, and returned a "no such message" 
error.  There's nothing pan can do about that.


Meanwhile, your question asking why pan didn't uncache the read articles 
in ordered to cache the unread articles suggests a misunderstanding of 
how pan's caching works, as well.

Pan was originally designed for either text-message use, or for binaries, 
direct-download-and-save, basically no pre-caching, at least for 
binaries.  This is quickly evident with the default 10 MB cache size if 
you try downloading to cache a bunch of binary articles.  I remember 
being caught out by this early in my pan experience, probably early in 
2002, as I settled into Linux after switching from MS and MSOE in late 
2001 and early 2002.  I set pan to cache a bunch of messages and went 
away to do something else (probably work or sleep), expecting to come 
back and find them all downloaded and ready to process locally.  But when 
I came back, most of them were missing!  After grumbling and perhaps 
swearing a bit, I set about redownloading, knowing some of them would be 
expired now and thus unavailable.  As I watched, pan downloaded a few, 
and deleted others that I hadn't read yet right in front of my eyes!

Of course it was running into the 10 MB default cache limit, and deleting 
the oldest, still unread, cached messages. =:^(  Once I figured that out, 
I set a nice big (for the time) 4 GB cache. (I had to edit the config 
file directly to do it as at the time pan's max cache size setting in the 
GUI was 1 GB.  A request to bump that was one of the first bugs I filed 
and Charles responded by bumping the GUI-max to a then unheard of 20 GB!)

You're running into the opposite effect.  Having set pan's cache size to 
100 MB while apparently doing only text, pan has plenty of room to cache, 
and won't be deleting messages, read or unread, from cache for quite some 
time.  (I don't know how your text-group usage compares to mine, but as I 
said, I have several years of text-group usage cached, and last I 
checked, my usage was still under a gig, nowhere near the 5 gig cache 
size limit I set.  So if your text group usage is comparable to mine and 
assuming you don't go doing a bunch of binaries with the same pan 
instance, thus hogging the cache with binaries, a 100 MB cache may be 
something like a year's worth of messages.)

Of course if that bothers you, you can return the cache to something 
closer to its default 10 MB, or even reduce it to say 2 MB.  But for text 
groups anyway, most people like to be able to go back and reread the 
thread context once in awhile, and local-caching few extra MB of read 
messages doesn't hurt on today's terabyte-scale disks, so...

Of course a cache size larger than needed for your expire time doesn't 
help... but doesn't really hurt, either.

But do be aware that the number of headers/overviews in combination with 
having them all cached does dramatically affect startup time.  On my pan 
binary instance, I have no-expire set as well, and as I said, a 12-gig 
dedicated partition cache (actually, I think I upped it to 15 gig the 
last time I upgraded disks...).  But I multi-select and delete messages 
as soon as I'm done with them, in ordered to help keep the load time 
down.  On the text instance, I accumulate, but I start pan with X/KDE and 
let it churn away, too, so it's ready a few minutes later when I'm 
finished with checking mail, maybe playing a game of solitaire or 
whatever, etc.  And I normally suspend-to-ram rather than shutting down 
the system as well, with the system-level cache staying intact, and pan 
is either still running or a quick restart with hot cache is near 
instant, so I don't really have to worry about my pan text instance cold-
cache-load-time that often.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman