Re: [Pan-users] Can creating task list be optimized?

[Darren]

Duncan wrote:
> Try this with 0.109.
> 
> Go to a group with many single-part binaries, say a picture group, a few
> tens of thousands of posts.  Here, I tried one with 23,820 unread posts
> according to pan, plus some downloaded/read already.
> 
> Select-all headers and try to download.
> 
> Here, pan sits for over an hour, figuring out and adding a single post at
> a time to tasks.nzb, less than ten a second.  This is on Gentoo/amd64,
> dual Opteron 242 w/ 8 gig of memory.  Pan's memory use is actually rather
> modest, ~80 meg, the entire time.  Little disk activity beyond the
> periodic write of the updated tasks.nbz from cache to disk.  Pegging one
> CPU at 100% (single thread, other CPU nearly idle), 80-ish percent of the
> single CPU is pan application mode usage, 15-ish percent system mode usage.


I feel strange answering something that Duncan posts  ;-)

This was filed as a bug and resolved for 0.110
http://bugzilla.gnome.org/show_bug.cgi?id=352170


> Unfortunately... pan is apparently single-threaded on the download as
> well, and with all those little posts, is /again/ pegging a single CPU,
> while the other one sits idle and the download limps along at a measly
> 150Kbyte/sec (6Mbps ~768KByte/sec) or less (that's the top of the graph).
> 
> nzb is xml format.  It's nice to have the nzb files, but xml is definitely
> not known for its efficiency in parsing, and it shows!  BIG TIME it shows!
> 
> Looks like pan can be pretty efficient on large multipart posts,
> presumably several MB per nzb entry, and it does OK on text posts as there
> aren't so many of them, but get several tens of thousands of small
> binaries, and it just doesn't work well at all, with the current tasks.nzb
> anyway.

Interesting, I guess most of my downloads haven't been a large number of
smaller binaries and I generally don't add much to the Queue so I have
never noticed this but it is obviously a problem.  Right now I am
downloading at almost my full 10meg speed.

Charles fixed the other issue so I suspect that this one is just a
matter of optimization as well.

> 
> Additionally, pan needs to be multithreaded, at least UI in one thread,
> while everything else is in another, giving the UI back some
> responsiveness.  With dual CPUs, I'm not /used/ to having a non-responsive
> app unless it's crashed, any more, at least not unless the system is in
> i/o-wait, not only running the disk momentarily every few seconds.
> 

I agree with this, but I can imagine that it would be a lot of work at
this point.

> Maybe I better remerge klibido, if it's going to be doing /this/.

<cringe>