[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Pan-devel] Re: Pan2 Very Slow
From: |
Duncan |
Subject: |
[Pan-devel] Re: Pan2 Very Slow |
Date: |
Tue, 17 Apr 2007 09:24:43 +0000 (UTC) |
User-agent: |
Pan/0.126 (Demon Sweat) |
"John W. M. Stevens" <address@hidden> posted
address@hidden, excerpted below, on
Sun, 15 Apr 2007 19:03:54 -0600:
> On Sun, Apr 15, 2007 at 12:37:59PM -0600, John W. M. Stevens wrote:
>>
>> After a recent upgrade to Debian 4.0 (Etch), I decided to try to
>> upgrade from pan 0.14.2.91 to pan 0.127. To my surprise, downloading
>> new headers from news groups has slowed down by almost a factor of
>> fifty.
>
> At a guess, this is a difference in what is being reported, right?
>
> I'm guessing that pan is now reporting number of "parts", not number of
> article headers downloaded. Good guess, or bad?
AFAIK, that's correct -- pan now sorts once on download, combining
headers into parts (using your terminology, as it can get rather
confusing trying to keep the multiple levels straight and figure out what
is being referenced in a particular instance) on the fly, then keeping
/that/ in memory. Author names, subjects, etc, have identical strings
combined where possible, thus saving memory and allowing pan to scale far
better than before, but at the expense of somewhat more processing on the
front-end.
In addition to the memory advantages, pan now stores the sorted/combined
state rather than the raw headers, thus not needing to re-thread every
time it loads, as the old C version (0.14.x) did. That speeds up reload.
However, the additional work at the front-end /might/ be an issue on
older/slower systems, under certain circumstances where CPU speed is
significantly more constrained than memory size. The bonus however, as I
said, is that it does it only once, then plugs new headers in on updates
but doesn't re-sort existing headers.
>> The GUI is a lot less responsive than 0.14, and it regularly hangs for
>> long periods of time (20 to thirty seconds). On doing some reading,
>> I've found some emails that suggest that the new version is not
>> multi-threaded. I would suggest a minimum of three threads: One for
>> the GUI, one for the encoder/decoder task, and one for data up/down
>> load. Experiments with the news readers I've written show this as a
>> nice balance between responsivness and network bandwidth usage.
>
> This is still true, and pan-0.127 crashed five times in my attempts to
> work my way through the process of figuring out where it's fragile.
The new version is indeed rather less threaded. However, in theory it
shouldn't be that much worse as the code is far cleaner and less complex,
and it shouldn't be bottlenecking due to single-threading anyway.
Pan now only multithreads in two instances -- it hatches worker threads
to initiate the connections (then feeds the now-open connections back to
the main thread, and the worker threads die), and just being added, pan
will hatch threads for the decode and save process after message
download. (The latter was being worked on last I knew, after some
profiling indicated it would help in some instances -- the decode was
sometimes taking long enough that the network buffers filled up, thus
slowing the networking, which should be the bottleneck, down. I'm not
sure if it's released yet or not.)
There has been some discussion (mainly my suggestion) of possibly
hatching a dedicated GUI thread, but so far, it hasn't been demonstrated
necessary, and Charles is reluctant to do it due to the multi-threading
complexity it would add. The new code is extraordinarily clean and
simple (so everyone says, I'm not a coder, tho I speak the language well
enough to follow discussions as long as they don't get too code concrete,
much of the time), and Charles hopes to keep it that way if at all
possible. Multithreading the main program wouldn't help in that regard.
However, if you can satisfactorily profile the existing code and
demonstrate the need, pan will likely do it, eventually. So far,
however, the problems have turned out to be bugs that could be resolved
without multithreading the main program. Hopefully, it will stay that
way. Meanwhile, Charles already replied, suggesting your profiling
efforts may be of help
> Do not, under any circumstances, try to delete the contents of a group
> while the header download is in progress. Badness ensues.
This has been mentioned before, by someone else in similar
circumstances. Check the bugs. It may be that this is being worked on,
or it may be that it still needs a sanity check added somewhere, to avoid
this sort of problem. I just got a sanity check added for a crash I was
running into, for 0.127, I believe, tho mine was relatively trivial. So
yes, places where such sanity checks are needed are still being
discovered.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman