Re: memory usage in cp -a

[Steven Augart]

OK, I'll tackle this one since nobody else has.

First, Mr. Diesler's problem was that he was misinterpreting the memory 
consumption information displayed by graphical utilities such as "xosview" 
and by the command-line tool "free".  He did not realize that the memory 
allocated to buffers and to cache is, in fact, available for use by the 
operating system.
"cp" was behaving normally.

Second of all, it sounds to me as if you were unable to fix the damage to 
your ext3 file system.  I assume you never got "fsck" on /backup to fully 
fix the filesystem.

If you mount a filesystem containing errors, you risk having the kernel 
panic, or (worse) having the system fail mysteriously.  The Linux kernel 
filesystem code assumes that the file system laid out on disk is correct.

Third, you do not know the cause of the failure of the file system.  It is 
possible that you have flaky hardware (other than the drive itself) which 
led to the file system being corrupted.  I'd advise you to remove the 
/backup disk and try to fix it on a known good machine.  At any rate, this 
is clearly beyond the scope of the bug-coreutils mailing list.

Fourth, this description does not give us a reproducible test case.

For these reasons, I do not think it would be profitable to do any work to 
hunt down the real cause of this bug; I expect  that it will never appear 
on working hardware with a consistent file system. 

Good luck resolving your problem.  I'm sorry that we can't make good use 
of this bug report.  If you find a problem that you can regularly 
reproduce on an otherwise working system, please do write back.

Sincerely Yours,

--Steven Augart






KAMOSAWA Masao <address@hidden>
Sent by: address@hidden
11/27/2003 03:39 AM
 
        To:     address@hidden
        cc: 
        Subject:        memory usage in cp -a


Hi,

I occured to similar situation with Mr. Thomas Diesler in message
00876 on this list.

http://www.mail-archive.com/address@hidden/msg00876.html

I'm using a backup tool "pdumpfs", takes and maintains everyday
snapshots of target directories by copying it (updated or newly
created files) and making hard links (unchanged files).
(http://www.namazu.org/~satoru/pdumpfs/index.html.en)
I have enjoyed it and happily taken snapshots of whole tree of 
my Linux web/mail/db server everyday for this 7 months. 

Then, the filesystem of my backup drive has corrupted.

That is, when I tried to delete some largest files after 
backed them up to CD-R, the drive space still not freed, 
after I removed all of the links of those files. 
And fsck returned error code and could not repair the 
filesystem (ext3).

I decided to rescue "all" the filetree, but the tools for this
purpose are very limited. For tar, it could not handle hardlinks
correctly (as far as I read the man pages). For dump and restore,
they could dump and restore the corrupted filesystem faithfully.
Only the tool I could use was "cp -a", for handling capability
of hardlinks and it preserves all attributes.

The filesystem cotains about 200day's snapshots of my whole
system include mirrored indexed "web cache", so thousands 
 (How I could get the number?) of files (mostly hardlinks) 
and directory entries are in each day's dir.

My command was:

cp -a /backup/2003/* /backup2/2003

The box was stopped after about 3 hours, kernel was running 
(virtual terminals switchable) but I could not logged in, 
for a message continuously repeated that killing an process 
of httpd. I judged that the kernel could not start new processes
and it runnawayed (no Ctrl-Alt-Del valid).

After hard-reset and rebooted, the copy was done about 4 days 
of backups. Then I rerun the command and observed it on top(1).
It revealed that the process growed up and exceeded the memory 
space on the system (386MB RAM + 128MB swap).
I added swap files some time and when the process reach to
3GB, it left message "memory exhausted" and stopped.

I thought it could be a memory leakage or design flaw. 
I searched Google and come across the above messages.

I think this is worth to be reported, isn't it...?

KAMOSAWA Masao


_______________________________________________
Bug-coreutils mailing list
address@hidden
http://mail.gnu.org/mailman/listinfo/bug-coreutils