[Pan-devel] Re: Pan2 Very Slow

[Duncan]

"John W. M. Stevens" <address@hidden> posted
address@hidden, excerpted below, on 
Sun, 15 Apr 2007 19:03:54 -0600:

> On Sun, Apr 15, 2007 at 12:37:59PM -0600, John W. M. Stevens wrote:
>> 
>> After a recent upgrade to Debian 4.0 (Etch), I decided to try to
>> upgrade from pan 0.14.2.91 to pan 0.127.  To my surprise, downloading
>> new headers from news groups has slowed down by almost a factor of
>> fifty.
> 
> At a guess, this is a difference in what is being reported, right?
> 
> I'm guessing that pan is now reporting number of "parts", not number of
> article headers downloaded.  Good guess, or bad?

AFAIK, that's correct -- pan now sorts once on download, combining 
headers into parts (using your terminology, as it can get rather 
confusing trying to keep the multiple levels straight and figure out what 
is being referenced in a particular instance) on the fly, then keeping 
/that/ in memory.  Author names, subjects, etc, have identical strings 
combined where possible, thus saving memory and allowing pan to scale far 
better than before, but at the expense of somewhat more processing on the 
front-end.

In addition to the memory advantages, pan now stores the sorted/combined 
state rather than the raw headers, thus not needing to re-thread every 
time it loads, as the old C version (0.14.x) did.  That speeds up reload.

However, the additional work at the front-end /might/ be an issue on 
older/slower systems, under certain circumstances where CPU speed is 
significantly more constrained than memory size.  The bonus however, as I 
said, is that it does it only once, then plugs new headers in on updates 
but doesn't re-sort existing headers.

>> The GUI is a lot less responsive than 0.14, and it regularly hangs for
>> long periods of time (20 to thirty seconds).  On doing some reading,
>> I've found some emails that suggest that the new version is not
>> multi-threaded.  I would suggest a minimum of three threads: One for
>> the GUI, one for the encoder/decoder task, and one for data up/down
>> load. Experiments with the news readers I've written show this as a
>> nice balance between responsivness and network bandwidth usage.
> 
> This is still true, and pan-0.127 crashed five times in my attempts to
> work my way through the process of figuring out where it's fragile.

The new version is indeed rather less threaded.  However, in theory it 
shouldn't be that much worse as the code is far cleaner and less complex, 
and it shouldn't be bottlenecking due to single-threading anyway.

Pan now only multithreads in two instances -- it hatches worker threads 
to initiate the connections (then feeds the now-open connections back to 
the main thread, and the worker threads die), and just being added, pan 
will hatch threads for the decode and save process after message 
download.  (The latter was being worked on last I knew, after some 
profiling indicated it would help in some instances -- the decode was 
sometimes taking long enough that the network buffers filled up, thus 
slowing the networking, which should be the bottleneck, down.  I'm not 
sure if it's released yet or not.)

There has been some discussion (mainly my suggestion) of possibly 
hatching a dedicated GUI thread, but so far, it hasn't been demonstrated 
necessary, and Charles is reluctant to do it due to the multi-threading 
complexity it would add.  The new code is extraordinarily clean and 
simple (so everyone says, I'm not a coder, tho I speak the language well 
enough to follow discussions as long as they don't get too code concrete, 
much of the time), and Charles hopes to keep it that way if at all 
possible.  Multithreading the main program wouldn't help in that regard.

However, if you can satisfactorily profile the existing code and 
demonstrate the need, pan will likely do it, eventually.  So far, 
however, the problems have turned out to be bugs that could be resolved 
without multithreading the main program.  Hopefully, it will stay that 
way.  Meanwhile, Charles already replied, suggesting your profiling 
efforts may be of help

> Do not, under any circumstances, try to delete the contents of a group
> while the header download is in progress.  Badness ensues.

This has been mentioned before, by someone else in similar 
circumstances.  Check the bugs.  It may be that this is being worked on, 
or it may be that it still needs a sanity check added somewhere, to avoid 
this sort of problem.  I just got a sanity check added for a crash I was 
running into, for 0.127, I believe, tho mine was relatively trivial.  So 
yes, places where such sanity checks are needed are still being 
discovered.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman