Re: [Pan-users] pan 0.139 crash with large groups

[Bob]

On Thu, 06 Sep 2012 01:46:24 +0000, Duncan wrote:

> Bob posted on Wed, 05 Sep 2012 17:57:10 +0000 as excerpted:
> 
>> I am seeing the same thing with 0.139 on Mint 13 (mate), but the lockup
>> seems to be heat related.  Binary group has 108,000+ entries, upon
>> click to enter the group processor core temp rises 16 to 18 degrees F. 
>> If downloads are in progress, tasks are stopped and the screen info
>> goes away. Closing the application (pan), waiting a few seconds for the
>> temp to drop and restarting pan continues the downloads, as if nothing
>> happened.
>> 
>> This is happening with an AMD Phenom II x 4 965 @3.4 Ghz with 8 gig of
>> memory.
> 
> Ouch!
> 
> For large binaries in particular, it's worth noting that there's a
> couple ways of splitting them; binaries over ~ 100 MB are often split
> both ways.  There's the traditional news client built-in attachment
> splitting, done during the post itself, and "pre-splitting" the file
> into parts before posting, for reassembly after download (tho some of
> the automated posters can handle this automatically as well).
> 
> The first kind, attachment splitting, divides single files as posted
> into a series of individual posts (as appearing on the server) of
> approximately the same size.  The reason this is significant is that PAN
> NORMALLY RECOMBINES THESE TRANSPARENTLY, AND SHOWS ONLY ONE ARTICLE THAT
> MAY IN FACT BE 100+ individual parts!
> 
> So your 108k of entries could easily be 10 million+ individual parts (as
> appearing on the server)!  If indeed that's the case, no WONDER pan
> heats up the CPU and uses a LOT of memory, potentially hitting the
> 32-bit 4-gig-
> per-app barrier!
> 
> Presumably with 8 gigs RAM you're running 64-bit (altho you could do it
> with the 64-gig mem option on 32-bit, but that adds another level of
> indirection so is less efficient), so it shouldn't be 32-bit issues. 
> Tho of course you could be running a 64-bit kernel with 32-bit userland
> (either the traditional way or the new x32), and there pan could still
> run into the 32-bit 4-gig-per-app barrier even if the machine has 8 gig
> of RAM.  But with full 64-bit, the 4-gig per-app barrier disappears
> (unless of course you have a per-app limit set locally, via ulimit or
> /etc/security/limits.* or the like), then it's whatever you set).
> However, pan *IS* known for pressing the limits on 10-million-plus post
> groups, even with 8-gig-plus RAM.  There has long been talk of
> converting pan to use a post database so it doesn't need to keep all
> that info in actual memory, but it hasn't happened yet.
> 
> 
> That's the memory side.  CPU-temp side, if your system's going unstable
> when it hits 100% CPU for too long, it's time to examine your cooling
> solution, and/or get a CPU upgrade and/or turn down your overclocking
> (note that if the system was overclocked previously, it may have gotten
> to the point where it can't even maintain rated clocking with any
> stability, and you may have to underclock).
> 
> FWIW, I run gentoo here, so /regularly/ run flat-out while building
> updates like the kde 4.9.1 that came out yesterday, or the job-uncapped
> kernel builds that fill my 16 gigs RAM and run up 500+ one-minute load
> averages. =:^)  (AMD "bulldozer" fx6100 6-core rated 3.3 GHz but
> marginally overclocked to 3.6 GHz w/ stock cooling.  At full load, stays
> right at the rated 95W rating.)
> 
> If you're having overheat/stability problems due to CPU usage, like I
> said, you better investigate!

I know what you mean about the splits on files.  I do regularly look at
the .nzb files and they show the split files required to make split binaries.

The phenom 965 was for several years the top of the line for 
AMD.  The price on newegg has been dropping almost weekly.  This one
is a 125 watt  with the standard non/oc speed of 3.4 Ghz.  When I built this
system the 965 costed out at around $165.  I got it about a month ago for
$99 with free shipping.  Noticed that yesterday it was for sale for $89.
This for the Boxed Set of processor and Fan/heatsink.

We have a cold front coming through this weekend forecasting daily high temps
in the low to mid 80's and night time temps down to 61 Saturday and Sunday.  
Much better than the 101 to 104 we have had this week with 50%+ humidity.

I think I will pull the processor/fan/heatsink and check it out.  I just may 
have put
too much thermal paste on the interface between the processor and heatsink.
If nothing else, I may re-install the phenom 2 x 4 925 which I originally used 
to
build the system. It is rated at 2.8Ghz but will oc to 3ghz with stock cooling.

I do hope that the database project comes about, it would probably be a real
help.

Bob