Re: Very large folderTo:

[George Michaelson]

Its always been my belief that large folders cause multi level directory block chaining in traditional UNIX fs. This itself incurs costs and consequences on how the cross-system file buffer cache works. Basically, any operation which requires all the directory blocks to be walked in sequence flood kernel file buffers. It has impacts on other uses of the OS. 

It is likely more modern FS like ZFS handle this differently but I don't know, I've never seen an analysis. 

Your system has cronjobs doing things like find . -type f -mtime <x> which may run slower, you may be causing general systems slowdowns.

I think it would make sense to filter out the things you want.

I Share your problem, mails from now dead relatives it is exquisitely painful for me to read but I am unwilling to delete, and the thought of having to write filters to find and file them doesn't fill me with joy. On the other hand, I have replicated the data because you have other risks: disk media is fragile.

Don't have only one copy of these mails. A cloud mail provider like Google might be a good backup, and has filter, search and tag options.

Cheers

G

On Sun, 6 Jun 2021, 7:10 am , <norm@dad.org> wrote:

Starting in late 2014 I have stopped deleting messages, putting them in a
directory, +gone, which now contains 465,147 messages and uses about 17
gigabytes. The bulk of these messages were of transitory or of less interest
to me. But they include 1,702 messages from my daughter. They were almost all
of no interest or use to me within a day or two of when she sent them. But she
recently died (the worst thing by far that's ever happened to me). Now every
byte she ever wrote is precious to me. So I am glad that I stopped deleting
messages that I no longer care about.

In practice this large folder has little impact on performance. For example,
whenever I do a pick which is, or in a script which might be, +gone, I give it
an argument like last:100000. I could, if necessary split +gone into several
smaller folders, but I would rather not. But I'm concerned that a bug in nmh
might cause a problem. For example, some kind of a buffer overflow.

So, what is the likelihood of such a bug? Does anybody have any experience
dealing with such large folders?






    Norman Shapiro

----------
Starting in late 2014 I have stopped deleting messages, putting them in
a directory, +gone, which now contains 465,147 messages and uses
about 17 gigabytes. The bulk of these messages were of transitory or of less
interest to me.  But they  include 1,702 messages from my daughter.  They
were almost
all of no interest or use to me within a day or two of when she sent them.
But she recently died (the worst thing by far that's ever happened to me).
Now every byte she ever wrote is precious to me. So I am glad that I stopped
deleting messages that I no longer care about.

In practice this large folder has little impact on performance. For example,
whenever I do a pick which is, or in a script which might be, +gone I give
it an argument like last:100000. I could, if necessary split +gone into
several smaller folders, but I would rather not. But I'm concerned that a bug
in nmh might cause a problem. For example, some kind of a buffer overflow.

So, what is the likelihood of such a bug? Does anybody have any experience
dealing with such large folders?
such a large folder might






    Norman Shapiro