[Pan-users] Re: Forcing an expire?

[Duncan]

Ron Johnson <address@hidden> posted
address@hidden, excerpted below, on  Thu, 09 Jul 2009 11:48:18
-0500:

> After reading this:
>    http://www.westnet.com/~gsmith/content/linux-pdflush.htm
> 
> and especially this thread:
> http://lkml.org/lkml/2009/1/7/303
> 
> I boosted dirty_ratio from 10 to 30 and dirty_background_ratio from 5 to
> 10.
> 
> Now tasks.nzb.tmp seems to get written 3x faster.
> 
> Thanks!!!

Cool! =:^)

BTW, I don't know what kernel you are on, but on the leading edge, 
uncovered and stop-gap fixed in kernel 2.6.29, fixed more correctly in 
2.6.30, and possibly settle-in fixes to in-development 2.6.31 (I'm not 
sure on that), there have been a number of some would say way too late 
changes to the way ext3 handles such things.  This was in large part 
rooted in a MAJOR kerflukle with ext4 development, that uncovered a 
number of badly chosen defaults and policy decisions and implementations 
in existing ext3 and the under development ext4.

This isn't the proper context to go into the full detail and others have 
covered it way better than I could in any case, but long story short (ha, 
in /my posts??? =:^), ext3 should be rather better performing at defaults 
in 2.6.30 and beyond.  At the same time, those absolutely paranoid about 
data reliability may want to have a look as well, as the default ext3 
journaling choice switched from data=ordered to a now *FAR* more reliable 
data=writeback.  This is actually how I came upon all these tuning 
parameters in the first place -- they among other things came up in the 
discussion.  FWIW, however, I run reiserfs(3) here, not ext3.  However, 
reiserfs uses many of the same vm settings and basic ideas, tho reiserfs 
is implemented enough differently and at a late enough stage in its 
lifecycle than they aren't doing the same level of changes to it.

But that LKML thread you linked was right at the beginning of it and 
deals with some of the issues involved, as does the older 2007 info you 
linked as well, so you should now already have a good understanding of 
the history.  I'd suggest the coverage at LWN if you want to groke the 
newer developments.  I follow it rather religiously every week, tho at 
the week delay for non-subscribers, as I subscribed for a time but 
dropped it due to philosophical issues[1].  It has covered the issue and 
resulting changes rather well on its weekly kernel page.

Here's links to the two big LWN stories on the subject:

ext4 and data loss (March 11, 2009):
http://lwn.net/Articles/322823/

That massive filesystem thread (March 31, a followup to the above):
http://lwn.net/Articles/326471/

There have been further developments since then, of course.  Both ext3 
and ext4 now have rename and open-for-write-and-truncate disk sync 
priorities bumped to the head of the line, and as a result of the 
reliability improvements from that and other changes, ext3 now defaults 
to something less than data=ordered (IDR whether it's data=writeback or 
the proposed intermediate option).  If you wish, you can read the above, 
then follow the weekly kernel pages forward from there thru the release 
of 2.6.30 anyway, to see where things are now, as best I know.  2.6.31 
hasn't had any serious changes to that stuff that I'm aware of, but I 
admit I don't always get the significance of git commit log entries.  I'm 
not a C dev let alone a kernel C dev, so while I understand part of it, I 
know I miss at least as much as I understand, probably way more.  But it 
should also be obvious I'm way more in tune with kernel issues than a 
good 90% of Linux self-sysadmins, and that's being conservative I'm sure.

.....

[1] Philosophical issues: Refusal to open the site code.  Long promised.  
ATI and others the site has long lambasted for their failure to open 
specs and code, have done so.  Meanwhile, LWN still has the same promise 
to open its site code "real soon now" that it did when I first checked 
the site, 2002-ish.  I have a simple rule that keeps my conscience 
clear.  I do not and cannot support closed source with my purchasing 
dollar, including purchasing subscriptions to otherwise flagship open 
source sites I'm otherwise a huge fan of.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman