[Pan-users] Re: Pan and the 2.6 kernels

[Duncan]

Rich posted <address@hidden>, excerpted below,  on Sat, 15
Jan 2005 12:11:05 -0400:

> I've been using the 2.6 kernels for a few months now. Started with 2.6.7,
> and have recently installed 2.6.10, which is MUCH faster and appears to
> have had some issues addressed.
> 
> However, pan runs somewhat better in 2.6.10, but still grinds the system
> to a standstill for seconds at a time.  It used to be minutes with prior
> kernel.
> 
>>From what I can see, if I delete all the articles in pan, and then get
> new articles. I do not see the system halted. When I do not delete the
> articles and get new, the systems starts processes spike and everything
> freezes. Sometimes I was able to get at a text prompt and kill pan, which
> the system them resumed as normal.
> 
> It seems to happen more on the larger newsgroups.
> 
> Is there anything I can do to stop the system hangs?

In general, no.  PAN is a resource hog on large groups, due to the way it
handles things.  If you have less than a gig of RAM, PAN will likely be
swapping like mad on large groups, as it tries desperately to get enough
memory to chew thru all those overviews (often incorrectly called
headers, incorrect, because the overviews are only a limited subset of
the actual message headers) it's attempting to sort.

The reason 2.6 works better than 2.4 is because 2.6 manages scheduling and
multi-tasking better, particularly under heavy load such as when there's a
process eating up all available real memory and throwing the system deep
into swap, as PAN will be doing under these circumstances.  While it /is/
possible to tweak the 2.6 scheduler and swap mechanisms, and that would
possibly get you slightly better performance, the real problem is that PAN
is using GTK+ widgets that simply weren't designed to scale to hundreds of
thousands or even millions of entries, and at present, PAN doesn't really
make it much easier for them, as it keeps everything in memory (altho it's
/far/ better than it was, by about a factor of 10, managing a million or
so overviews as opposed to the 100k it could manage before, without
choking, on a 1/2-3/4 gig memory system).

That's why Charles and others are currently working on switching PAN to
use a real database back-end library (SQLite), as it should scale far
better than the current widgets, which while they have some data
functionality were really intended for display, and simply don't scale
well in data mode.  As well, while they are doing the back-end db rewrite,
they are making additional memory saving improvements, such as only saving
the common elements of a subject line once, then using a much smaller (4
byte, 32-bit, on x86_32) token to refer to it for each message with the
same general subject line, instead of saving the entire thing once for
/each/ message or message-part, as PAN does currently.  Another trick they
are using is only keeping in actual memory the data for a few thousand
overviews at a time, using new and more efficient database to store the
rest of the overviews for the group on disk.  Thus, when it's all done,
PAN's scaling should be virtually unlimited, while instead of using
hundreds of megabytes of memory and /still/ running out of resources on
the largest groups, it'll only use tens of megabytes of memory, and
continue to be responsive (both it and the system) even with groups with
millions or tens of millions of overviews.

However, that's a fairly major rewrite of the back-end code, and isn't an
instant process.  Depending on how much time Charles has to integrate the
work others have already been experimenting with, I wouldn't expect a beta
of this to be out until late Q2 or Q3 at the earliest, and that's likely
to have a significant number of bugs still.  We may be looking at Q42005
or into 2006 before it's stable, even if there's reasonable time for
development.  However, it /is/ being worked on, now, which is good as that
particular project has been put off for some time, because it /is/ such a
major project.

In the mean time, if you haven't yet, look up Klibido, a
new-this-past-summer KDE binary news harvester application.  Being so new,
its interface is rather rough still, and it lacks features such as
automated filtering/scoring entirely, but it's extremely efficient at what
it /does/ do, manage multiple connections to multiple servers (if you have
them), automatically downloading parts from whatever server they are found
on, sucking down news binaries with a speed and overall system resource
efficiency that frankly left me (and some others I've talked to about it)
in awe.  It is KDE based so you'll need to have at least the KDE basics
(qt, kde-libs, kde-base) installed, which might be a negative for some,
and as I said, it's new and obviously unfinished and a bit rough, but the
thing works with an efficiency I simply hadn't realized was possible! 
Being so new, your distribution likely won't have it except in
testing/unstable, but there are various types of packages and a tarball
for those who wish to or need to compile it manually, on the site. 
(Gentoo, my distrib of choice, already has it in portage, altho it is
still keyworded ~arch, which means unstable.)

http://klibido.sourceforge.net/

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin