[Pan-users] Re: freeze, high CPU getting new headers

[Duncan]

David Kelly <address@hidden> posted
address@hidden, excerpted below, on  Fri,
01 Aug 2008 23:48:25 -0500:

> On Aug 1, 2008, at 10:01 PM, Greg Lee wrote:
> 
>> That happens to me.  I think it's a bug.  When loading a large number
>> of headers, about 3/4 of the way through, the GUI loses it, and I just
>> have to wait until the headers from that group are loaded.  Pan will
>> not respond until the headers are all loaded.  I have to wait around 3
>> minutes, at most (not 20), but I have 3G of ram, which probably helps.
>> It doesn't freeze my whole system -- I can go to another window and do
>> something else -- it just affects Pan.
> 
> 
> Expect Linux has similar but FreeBSD has per-process limits on memory.
> Default is 512MB. So unless one has a number of processes drawing 512MB
> having 3GB doesn't do a single application much good. However the OS
> will use the extra for disk caching.

On Linux this will be set by the distribution (which on the BSDs, FreeBSD 
is effectively its own distribution, so I guess it's sort of the same 
there).  The command to check and set various per-process resource limits 
is "ulimit".  This is a shell built-in, at least for bash, which has way 
more options than the relatively bare POSIX standard.

In a bash shell, type "help ulimit" for a listing of the options.  There 
are "soft" limits which can be reset if necessary, and "hard" limits 
which cannot be raised except from a privileged process (root or the 
like) but may be lowered.  In particular and among others, -a reports all 
current limits, -m is the maximum physical memory a program may use, and 
-v is the maximum virtual memory it is allowed to use.

A distribution will often set some of these limits in the profile and/or 
bashrc scripts.  Here, I have some (including -v and -m) set (-H, hard) 
at the system level, and some set stricter in my normal user's scripts.  
Depending on how you launch X, they may or may not apply to it (startx, 
they will, as it's a child of the shell that started it, a system level 
*dm, maybe not, unless it is scripted to source and inherit the usual 
system environmental variables and limits).

> When all of the headers from a group have been downloaded pan spends
> some time doing housekeeping at the expense of GUI updates. Guess its
> reconciling the list against the list stored on disk, and saving to
> disk. When the GUI freezes I notice about 8 MB/sec of disk activity.

This is correct and what I was referring to in the previous post when I 
mentioned that pan does a whole lot more than "get headers" when that 
action is initialized.

In old-pan, every time a group loaded, pan would re-thread everything 
from scratch.  One of the speedups in new-pan is that it keeps the thread 
state and saves it to disk when you exit a group (or quit pan, I think it 
saves part of it when you fetch messages too, but not all state is saved 
as if pan crashes before you exit the group, it forgets at least part of 
it), then reads the current threading back off of disk when you reenter a 
group, so it only has to figure it out once. However, it must still 
dynamically figure out where to plug in all those new posts when it 
downloads headers.

>From the debug as mentioned in a different post, the last action pan 
reports is "trying to add articles to tree", indicating it's doing this 
threading.  Something's apparently going wrong there.  However, it could 
be with the threading itself, or with related processes, the string 
management I mentioned earlier, etc.  (I suspect it's NOT the scoring 
now, as presumably that would be shown in the debug if it was.)

It's worth noting that threading isn't always exactly precise, and 
sometimes pan has to guess.  In particular, there have been cases where a 
post forming part of the upthread itself references a post apparently 
downthread from it.  Pan has code specifically to break this illogical 
endless loop, doing the best it can.  (I came across it investigating 
another bug... the code and comments therein handling this are... quite 
interesting to read, even if I don't fully understand C++.)  However I 
don't believe it's a problem with that triggering (or failing to trigger) 
in this case, as if it triggers, it /should/ do so quickly, while if it 
fails to trigger, pan probably wouldn't wakeup at all.  I'm simply 
pointing out how complex the problem can be, and the sort of hoops pan is 
jumping thru at this point to sort it all out. =8^S

So it could be an inefficiency in that tree widget as David mentions he 
suspects, or it could be a loop in pan due to strange threading, or some 
inefficiency in decoding some i18n portion or another (it's worth noting 
here that the way this is handled in headers is a pile of hacks in the 
first place, and there's some disagreement in implementations, given no 
accepted standard and the fact that the original RFCs covering the 
material were working with 7-bit-ANSI/ASCII only).

Anyway, looks like it's gdb time -- well, after that clean install you 
said you were going to try first.  Hope it fixes the problem!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman