pan-users
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Pan-users] Re: Reproducible crash retrieving headers in million-header


From: Wayne E . Nail
Subject: [Pan-users] Re: Reproducible crash retrieving headers in million-header group
Date: Fri, 30 Jul 2010 14:57:09 +0000 (UTC)
User-agent: Loom/3.14 (http://gmane.org/)

Duncan <address@hidden> writes:

> 
> K. Haley posted on Fri, 30 Jul 2010 00:43:04 -0600 as excerpted:
> 
> > On 7/29/2010 10:51 AM, Wayne E. Nail wrote:
> >> If this log is useless please tell me, and I will run Pan again under
> >> the same conditions but with whatever modifications you advise.
> >>
> >> I am pretty sure I have had Pan download *all* headers from a
> >> well-over-one-million-headers newsgroup in the past.
> >>   
> > According to the log Pan is crashing while trying to allocate a mere
> > 5KB. This implied that it's hitting a limit to the amount of memeory it
> > can allocate.  I know other users regularly view such large groups so
> > I'm fairly certain it's not a bug.  At this point all I can suggest is
> > checking how much memory Pan is using when it crashes.
> 
> Also, as the user that runs pan, check your resource limits.  You can do 
> that from bash using its ulimit builtin, as "ulimit -a" (-a = all, give 
> you the whole list.  In particular, you're interested in max memory size
> (-m), and max virtual memory (-v).
> 
> There's actually two limits per setting, a hard limit and a soft limit.  
> By default, you get (or set) the soft limit.  You can also get/set the 
> hard limit using -H, so -Hv to get/set the hard virtual memory limit.  The 
> soft limits are what is normally used, but you can reset your soft limit 
> up to the hard limit.  The hard limit can be reduced but not increased, 
> unless you're root, tho many are unlimited by default, so unless they're 
> set by the system, you can choose whatever value you want the first time, 
> or simply leave them unlimited.
> 
> But it's good to set some sort of limit, to prevent a fork-bomb taking out 
> the entire system.  With a limit set, it'll simply run into that limit, 
> and crash, instead of taking out the entire system.  Thus, some 
> distributions set a limit of some kind, perhaps related to the amount of 
> memory you have, by default.
> 
> FWIW, for my regular username, with 6 gigs of RAM and 20 gigs swap[1], I 
> set a soft limit of a gig of memory, two gigs of virtual memory.  Hard 
> limits are double that, 2 gigs physical RAM, 4 gigs virtual.  My admin 
> username (the one I give full password-less sudo access, which my regular 
> username doesn't get, and from which I run all my Gentoo compiles) has 
> somewhat higher limits, IIRC 2 gigs normal RAM, as there's an app or two 
> that I run, that require over a gig to compile.  Before I bumped it up for 
> them, I was running into the ulimits trying to compile those programs.  
> (The one I remember was kmail, from kde3.  Of course I'm running kde4 now, 
> and it seems better in that regard, but I've kept the higher settings, 
> just in case.)
> 
> In addition to being able to get/set ulimits in your login shell (just 
> setting them in any old shell won't do any good, except for programs run 
> from that shell), the pam system can control resource limits using the 
> pam_limits.so module.  Those settings should be in /etc/limits.conf or a 
> similar file (simply /etc/limits, here on Gentoo), and in /etc/security/
> limits.d/*.  See the pam_limits (8) and limits.conf (5) manpages for 
> information on those.
> ___
> [1] 20 gigs of swap because: It's 4x 5-gig swap partitons, one on each of 
> four different drives, set to the same priority, so the kernel stripes 
> them for 4X the normal swap speed.  I chose 5 gigs per partition as I 
> normally run 8 gigs RAM, but a stick died that I've not replaced, so I 
> only have six at present.  A mainline kernel suspend image can be up to 
> half the size of physical RAM, so that would be 4 gig.  The suspend image 
> uses a swap partition, but it can only use one, so I setup five gig swap 
> partitions so there'd be plenty of room for the 4 gig suspend image on one 
> of them.  The other three are the same size so the partition layout of 
> each disk remains identical, so with four disks each with a 5 gig swap 
> partition, I end up with the otherwise rather insane total of 20 gigs of 
> swap! =:^/
> 

Dang, I thought Duncan had likely nailed it, but...

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64402
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

2.6.32.16-141.fc12.i686.PAE #1 SMP Wed Jul 7 04:41:17 UTC 2010 i686
athlon i386 GNU/Linux

I have only 2GB swap on this 8GB RAM Fedora 12 box, but gkrellm always shows
nearly all the swap free and more than 6GB RAM free, seemingly without much
regard for what I'm doing.

In addition to Pan which runs often but not always, I am normally running
Evolution mail, xchat, two or three konqueror filenmanager instances, exaile
music player, kjots, kate, transmission, two firefox instances with 10+
tabs open each, freenet, and a terminal instance with five to ten tabs open.
None of this appears to tax the system, nor is this environment different
from when Pan has in the past pulled multi-million-header groups without 
complaint.

Here's a wrinkle though; I run Pan in stunnel with news server set to
localhost:119 and the secure NSP url stored in /etc/stunnel/stunnel.conf.
Might this setup be hitting a memory wall Pan by itself would not?

_________________________

Unrelated question: How does one avoid the 'You have lines longer than 80
characters. Fix that.' error in gmane composition when the stupid thing
lets you exceed that in its own compose box?






reply via email to

[Prev in Thread] Current Thread [Next in Thread]