RE: [Ifile-dev] Adding some extra switches

[Paul Chambers]

> -----Original Message-----
> From: Jason Rennie [mailto:address@hidden 
> 
> Jeremy Brown ripped out some of the bloat that made ifile not 
> handle lots of messages very well in a patch for 1.1.0,

I'll take a look - thanks for the reference.

> but it looks like there may still be a leak somewhere (I
> regularly do runs w/ up to 2000 messages and see memory
> usage increase during the run

Hmmm... messages or files? Later I tried runnning each maildir through
separately, to see if that worked around the issue. Some still got
killed, even though they have only a few thousand messages. I don't see
the pattern yet.

> One question---how do your patches get around this problem?  
> Seems like ifile would suck up all the memory whether you
> pass files via the command-line or via stdin...

They don't - at least not directly. I didn't realize that what I was
attempting would be a problem until I tried it :) Potentially it could
make it worse, since it's easy to use find to pass a whole maildir
subtree into ifile (using the -n switch to identify the destination
folder on a file-by-file basis). 

But it could help indirectly. What started be down this road in the
first place is the goal of incrementally updating the database in a cron
job. I use maildirs/imap, so was planning to automatically insert all
the messages that had been received, read and possibly moved (if
misfiled) since the last time the cron job had run.

So the theory is that it'll help to keep the number of files processed
in a given batch down, especially if I do it semi-frequently. My
problems occurred when I tried to initially build my database, and pump
my (large) existing maildirs through it. I'd also have to do this if I
restructured my folders (which happens occasionally).

> Sorry for all the pestering---just don't want to be adding cmd
> line options too freely :)

I agree, there are quite enough already :) I just thought I'd offer,
since I thought them to be general enough that others may find them
useful.

Paul