[Pan-users] Re: Reproducible crash retrieving headers in million-header group

[Duncan]

K. Haley posted on Fri, 30 Jul 2010 00:43:04 -0600 as excerpted:

> On 7/29/2010 10:51 AM, Wayne E. Nail wrote:
>> If this log is useless please tell me, and I will run Pan again under
>> the same conditions but with whatever modifications you advise.
>>
>> I am pretty sure I have had Pan download *all* headers from a
>> well-over-one-million-headers newsgroup in the past.
>>   
> According to the log Pan is crashing while trying to allocate a mere
> 5KB. This implied that it's hitting a limit to the amount of memeory it
> can allocate.  I know other users regularly view such large groups so
> I'm fairly certain it's not a bug.  At this point all I can suggest is
> checking how much memory Pan is using when it crashes.

Also, as the user that runs pan, check your resource limits.  You can do 
that from bash using its ulimit builtin, as "ulimit -a" (-a = all, give 
you the whole list.  In particular, you're interested in max memory size
(-m), and max virtual memory (-v).

There's actually two limits per setting, a hard limit and a soft limit.  
By default, you get (or set) the soft limit.  You can also get/set the 
hard limit using -H, so -Hv to get/set the hard virtual memory limit.  The 
soft limits are what is normally used, but you can reset your soft limit 
up to the hard limit.  The hard limit can be reduced but not increased, 
unless you're root, tho many are unlimited by default, so unless they're 
set by the system, you can choose whatever value you want the first time, 
or simply leave them unlimited.

But it's good to set some sort of limit, to prevent a fork-bomb taking out 
the entire system.  With a limit set, it'll simply run into that limit, 
and crash, instead of taking out the entire system.  Thus, some 
distributions set a limit of some kind, perhaps related to the amount of 
memory you have, by default.

FWIW, for my regular username, with 6 gigs of RAM and 20 gigs swap[1], I 
set a soft limit of a gig of memory, two gigs of virtual memory.  Hard 
limits are double that, 2 gigs physical RAM, 4 gigs virtual.  My admin 
username (the one I give full password-less sudo access, which my regular 
username doesn't get, and from which I run all my Gentoo compiles) has 
somewhat higher limits, IIRC 2 gigs normal RAM, as there's an app or two 
that I run, that require over a gig to compile.  Before I bumped it up for 
them, I was running into the ulimits trying to compile those programs.  
(The one I remember was kmail, from kde3.  Of course I'm running kde4 now, 
and it seems better in that regard, but I've kept the higher settings, 
just in case.)

In addition to being able to get/set ulimits in your login shell (just 
setting them in any old shell won't do any good, except for programs run 
from that shell), the pam system can control resource limits using the 
pam_limits.so module.  Those settings should be in /etc/limits.conf or a 
similar file (simply /etc/limits, here on Gentoo), and in /etc/security/
limits.d/*.  See the pam_limits (8) and limits.conf (5) manpages for 
information on those.
___
[1] 20 gigs of swap because: It's 4x 5-gig swap partitons, one on each of 
four different drives, set to the same priority, so the kernel stripes 
them for 4X the normal swap speed.  I chose 5 gigs per partition as I 
normally run 8 gigs RAM, but a stick died that I've not replaced, so I 
only have six at present.  A mainline kernel suspend image can be up to 
half the size of physical RAM, so that would be 4 gig.  The suspend image 
uses a swap partition, but it can only use one, so I setup five gig swap 
partitions so there'd be plenty of room for the 4 gig suspend image on one 
of them.  The other three are the same size so the partition layout of 
each disk remains identical, so with four disks each with a 5 gig swap 
partition, I end up with the otherwise rather insane total of 20 gigs of 
swap! =:^/

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman