Look up "overcommit". Linux by default will let malloc (really sbrk or mmap) succeed even when the request exceeds the amount of available virtual memory, in hopes that not all of it will actually be used, or that more memory will be available by the time it is. You get a pointer to unmapped memory, and memory is allocated by the page fault handler when you actually write to the page. If no memory is available, the OOM killer fires up in an attempt to get some. But if your process is the chosen victim, it gets killed. This happens "asynchronously" so there isn't really any good way for the process to handle it more gracefully. On Jul 12, 2016, at 14:41, Mark Grandi via Duplicity-talk < address@hidden> wrote: I guess i'm just confused on the behavior here, if the system was out of memory, wouldn't python throw a MemoryError when a malloc() fails? why would the linux OOM killer terminate python/duplicity before it terminates itself due to an uncaught MemoryError?
~Mark
On Jul 12, 2016, at 02:49, Chris Vanden Berghe via Duplicity-talk < address@hidden> wrote:
So, indeed, looks like an out-of-memory issue... slightly surprisingwith 1.4GB of free memory, no?# free -h total used free shared buffers cachedMem: 1.9G 737M 1.2G 117M 19M 169M-/+ buffers/cache: 549M 1.4GSwap: 0B 0B 0BDid memory usage increase from 0.6.x to 0.7.x?Thanks,Chris.---On 12/07/2016 11:37, Remy van Elst via Duplicity-talk wrote:You might want to check your syslog (/var/log/messages or
/var/log/syslog) and dmesg output to see if you are having memory
issues (RAM/out of memory killer)
https://raymii.org
On Tue, Jul 12, 2016 at 3:41 AM, Chris Vanden Berghe via
Duplicity-talk <address@hidden
<mailto:address@hidden>> wrote:
Hi,
Since the recent upgrade to Duplicity 0.7.08 I'm having some
problems of
the process being killed spontaneously.
I start Duplicity as follows:
nice /usr/bin/duplicity --verbosity 9 --full-if-older-than 1M
--archive-dir=/root/Backup/cache/ --include-globbing-filelist
/root/Backup/backup.list --exclude '**' / gs://bobo-backup/
The output is as follows:
Using archive dir: /root/Backup/cache/01f81df3eb37282f76c93fb89a3708f5
Using backup name: 01f81df3eb37282f76c93fb89a3708f5
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named
dropbox
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Reading globbing filelist /root/Backup/backup.list
Main action: inc
================================================================================
duplicity 0.7.08 ($reldate)
Args: /usr/bin/duplicity --verbosity 9 --full-if-older-than 1M
--archive-dir=/root/Backup/cache/ --include-globbing-filelist
/root/Backup/backup.list --exclude ** / gs://bobo-backup/
Linux bobodioulasso 3.16.0-44-generic #59-Ubuntu SMP Tue Jul 7
02:07:39
UTC 2015 x86_64 x86_64
/usr/bin/python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2]
================================================================================
Using temporary directory /tmp/duplicity-2G5OwC-tempdir
Registering (mkstemp) temporary file
/tmp/duplicity-2G5OwC-tempdir/mkstemp-9wghI1-1
Temp has 280009195520 available, backup will use approx 272629760.
Listed
gs://bobo-backup//duplicity-full-signatures.20160704T151139Z.sigtar.gpg
Listed gs://bobo-backup//duplicity-full.20160704T151139Z.manifest.gpg
<SNIP>
Listed
gs://bobo-backup//duplicity-full.20160704T151139Z.vol97.difftar.gpg
Listed
gs://bobo-backup//duplicity-full.20160704T151139Z.vol98.difftar.gpg
Listed
gs://bobo-backup//duplicity-full.20160704T151139Z.vol99.difftar.gpg
Local and Remote metadata are synchronized, no sync needed.
Listed
gs://bobo-backup//duplicity-full-signatures.20160704T151139Z.sigtar.gpg
Listed gs://bobo-backup//duplicity-full.20160704T151139Z.manifest.gpg
<SNIP>
Listed
gs://bobo-backup//duplicity-full.20160704T151139Z.vol97.difftar.gpg
Listed
gs://bobo-backup//duplicity-full.20160704T151139Z.vol98.difftar.gpg
Listed
gs://bobo-backup//duplicity-full.20160704T151139Z.vol99.difftar.gpg
256 files exist on backend
11 files exist in cache
Extracting backup chains from list of files:
[u'duplicity-full-signatures.20160704T151139Z.sigtar.gpg',
u'duplicity-full.20160704T151139Z.manifest.gpg',
u'duplicity-full.20160704T151139Z.vol1.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol10.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol100.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol101.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol102.difftar.gpg',
<SNIP>
u'duplicity-full.20160704T151139Z.vol92.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol93.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol94.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol95.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol96.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol97.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol98.difftar.gpg',
u'duplicity-full.20160704T151139Z.vol99.difftar.gpg']
File duplicity-full-signatures.20160704T151139Z.sigtar.gpg is not part
of a known set; creating new set
Ignoring file (rejected by backup set)
'duplicity-full-signatures.20160704T151139Z.sigtar.gpg'
File duplicity-full.20160704T151139Z.manifest.gpg is not part of a
known
set; creating new set
Processing local manifest
/root/Backup/cache/01f81df3eb37282f76c93fb89a3708f5/duplicity-full.20160704T151139Z.manifest
(161677527)
Found manifest volume 1
Found manifest volume 2
Found manifest volume 3
<SNIP>
Found manifest volume 252
Found manifest volume 253
Found manifest volume 254
Found 254 volumes in manifest
File duplicity-full.20160704T151139Z.vol1.difftar.gpg is part of
known set
File duplicity-full.20160704T151139Z.vol10.difftar.gpg is part of
known set
File duplicity-full.20160704T151139Z.vol100.difftar.gpg is part of
known set
<SNIP>
File duplicity-full.20160704T151139Z.vol97.difftar.gpg is part of
known set
File duplicity-full.20160704T151139Z.vol98.difftar.gpg is part of
known set
File duplicity-full.20160704T151139Z.vol99.difftar.gpg is part of
known set
Found backup chain [Mon Jul 4 17:11:39 2016]-[Mon Jul 4 17:11:39
2016]
Last full backup date: Mon Jul 4 17:11:39 2016
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/Backup/cache/01f81df3eb37282f76c93fb89a3708f5
Found 0 secondary backup chains.
Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Mon Jul 4 17:11:39 2016
Chain end time: Mon Jul 4 17:11:39 2016
Number of contained backup sets: 1
Total number of contained volumes: 254
Type of backup set: Time: Num
volumes:
Full Mon Jul 4 17:11:39 2016
254
-------------------------
No orphaned or incomplete backup sets found.
Registering (mktemp) temporary file
/tmp/duplicity-2G5OwC-tempdir/mktemp-nXSXjQ-2
Deleting /tmp/duplicity-2G5OwC-tempdir/mktemp-nXSXjQ-2
Forgetting temporary file
/tmp/duplicity-2G5OwC-tempdir/mktemp-nXSXjQ-2
Processing remote manifest
duplicity-full.20160704T151139Z.manifest.gpg
(161677527)
Found manifest volume 1
Found manifest volume 2
Found manifest volume 3
<SNIP>
Found manifest volume 251
Found manifest volume 252
Found manifest volume 253
Killed
Any idea what could cause this? If not, what is the best way to debug?
Thanks,
Chris.
_______________________________________________
Duplicity-talk mailing list
address@hidden <mailto:address@hidden>
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________Duplicity-talk mailing listaddress@hiddenhttps://lists.nongnu.org/mailman/listinfo/duplicity-talk
|