I have a 64bit Ubuntu 9.04 system here that I'd be happy to
attempt to
reproduce it over the weekend. Is it basically a backup that involves
lots of incrementals with many volumes that's causing this behavior?
On Tue, May 12, 2009 at 1:51 PM, Ian Barton <address@hidden
<mailto:address@hidden>> wrote:
I have got a couple of 64 bit Ubuntu 9.1 (Jaunty) computers. I
can
try running a test on my system if you tell me what you would
like
me to do.
FWIW my main computer for running is Debian Lenny 64bit and I
have
never seen the problem on it.
Ian.
Thanks for the help!
I'm not sure at this point. I built a 64bit Ubuntu 9.04
system
and it
seems to be working fine. Debian may be different, but
not much.
As to what's broke, it could even be at the system level,
but my
bet is
on gpg.
...Ken
address@hidden <mailto:address@hidden> wrote:
I am gonna install a 64bit debian in a virtual machine
and gonna
doublecheck that ... but what do you think is buggy?
python,
gpg ?
..ede
The only real change to GnuPGInterface is a shift
right
instead of a
shift left when printing the error code after an
exception. He's not
getting any exceptions, so he's not hitting the
change.
Other than
that, its the same code from years ago.
My best guess is still that its a 32-bit versus
64-bit
issue.
...Ken
Edgar Soldin wrote:
true they close every time .. the only thing i
see
is that the amount of
gpg processes grows with the amount of
incrementals ...
@yang: how many incrementals does your backup
hold?
the other thing is, maybe gpg itself is buggy ..
@yang: could you try another gpg version?
and the third thing
@ken: is it really impossible for 0.5.17 to
use an
oldGnuPGInterface
version, that might be laying around on the
system?
How could yang look
for it?
.. 2cents again :) ..ede
--
I tried this with tiny volume sizes of 1M
over a
2GB backup (2K
volumes)
and did a count of the gpg processes (live,
defunct) every 2 seconds
and
never had more than 5 (3 live, 2
defunct). This
was on a very fast
link
between local VMs. Tried both backup and
restore so it would test both
directions. By the end, all were closed.
It's possible to have some defunct while the
task scheduler is
updating,
but Yang is showing hundreds that are holding
files open. That's the
confusing part.
...Ken
Edgar Soldin wrote:
i really have to be fast (a window of 10s
let's say) and it helps to
have a slow backup connection with
lots of
incrementals... but I
may be
seeing ghosts here. Can it that it is
perfectly normal for
duplicity to
gpg decrypt something per each
incremental
but close all processes
just
a bit later? .. just a guess
..ede
I cannot reproduce this on a 32bit
machine at all. I'll be able
to test
on a 64bit machine sometime this week
and, hopefully, find the
problem.
...Ken
address@hidden
<mailto:address@hidden> wrote:
i seem to get closer to it ...
the more incrementals
(regardless if
file: or ftp: backend) the more
defunct gpg processes appear
.. but
dissappear again while the
backup is
still running
I did
initial backup of 3GB
touched a new test file to have a
change & did a 2nd backup
touched a new test2 file to
have a
change & did a 3nd backup
...
and since the 3rd backup the
defunct
gpg processes keep appearing
essentially I say: Take any more
than 3 incrementals in size
backup and
do a new incremental and you will
see those zombie gpg processes ..
while I am not sure that this is
what happens on the ulimit
problem I
thought I should report these
observations ... ede
PS:
here a 'ps xf -u user' shortly
after
starting the sixth run .. the
number of gpg zombies seems to be
number of sets (full + incr's)
minus 2 ..
9122 pts/0 S+
0:00 \_ /bin/bash
/srv/www/user//release/ftplicity_1.4.2/ftplicity
test backup
19128 pts/0 S+
0:00 \_ /bin/bash
/srv/www/user//release/ftplicity_1.4.2/ftplicity
backup
19193 pts/0 S+
0:00 \_ /bin/bash
/srv/www/user//release/ftplicity_1.4.2/ftplicity
backup
19194 pts/0 R+
0:03 \_
/usr/bin/python
/srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
--verbosity 4
--encrypt-key 00000000--sign-key
00000000--gpg-options=--always-trust
--volsize 1
--exclude-globbing-filelist
/srv/www/user//.ftplicity/test/excl
19196 pts/0 SL+
0:00
\_ gpg
--logger-fd 4
--passphrase-fd 8 --batch
--no-tty
--default-key 00000000--recipient
00000000--no-secmem-warning
--always-trust --encrypt --sign
19197 pts/0 SL+
0:00
\_ gpg
--status-fd 6
--passphrase-fd 11 --logger-fd 5
--batch --no-tty --default-key
00000000--no-secmem-warning
--always-trust --decrypt
19199 pts/0 Z+
0:00
\_ [gpg]
<defunct>
19200 pts/0 Z+
0:00
\_ [gpg]
<defunct>
19201 pts/0 Z+
0:00
\_ [gpg]
<defunct>
19202 pts/0 Z+
0:00
\_ [gpg]
<defunct>
19203 pts/0 SL+
0:00
\_ gpg
--logger-fd
23 --passphrase-fd 28 --batch
--no-tty --default-key
00000000--recipient
00000000--no-secmem-warning
--always-trust --encrypt --sign
hmm .. you are right these
processes are zombie (they
appear in
top as
such) .. later on the
processes
disappear, while duplicity
is still
running ..
but I don't feel this is
normal.. or is it?
just to be more sure, I'll
try a
1MB chunk full backup of
my 3GB
home
dir .. let's see with how
many
zombies I'll end up ...
interestingly my ulimit =
unlimited ... seems to be
default in
the old
SUSE 10.2, btw also on my
debian
5.0 box both 32bit
can't remember to ever have
touched this setting
..ede
--
These are not open
processes, they are
zombie
processes, so no
real
resources taken. That
simplifies it a bit.
What
normally
causes this
is the failure of a
parent
process to properly
retrieve its
exit status,
but this is not the
case or
more people would have
this
problem.
As far as I can tell, the
only systems this is
happening on are
64bit,
and not even all of
those.
Some of my systems are
64bit and
they don't
show this, so I'm
wondering
if this could be
limited to a
particular
version of GnuPG, or
what.
Edgar, from your note I
could not tell, is this
happening to
you too?
...Ken
address@hidden
<mailto:address@hidden>
wrote:
just my quick
observation .. a
simple
incr backup opens and
leaves over
20 gpg processes
open. I
don't think this is
healthy and also can
imagine that this
multiplies if the
volumes get smaller
(mine are
50MB).
regards ede
address@hidden:~> ps
-u user xf
PID TTY
STAT TIME COMMAND
32731 ?
S 0:00 sshd:
address@hidden/0
32732 pts/0
Ss 0:00 \_ -bash
1167 pts/0 S
0:00 \_
/bin/bash
/usr/local/bin/ftplicity
profile backup
1168 pts/0 S
0:00 | \_
/bin/bash
/srv/www//release/ftplicity_1.4.2/ftplicity
profile backup
1174 pts/0 S
0:00 | \_
/bin/bash
/srv/www//release/ftplicity_1.4.2/ftplicity
backup
1239 pts/0 S
0:00
| \_
/bin/bash
/srv/www//release/ftplicity_1.4.2/ftplicity
backup
1240 pts/0 S
0:05
| \_
/usr/bin/python
/srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
--verbosity 4
--encrypt-key XXXXXXX
--sign-key B59ECD99
--gpg-options=--always-trust
--full-if-older-than 1M
--volsize 50
--exclude-globbing-filelist
/srv/www/...
1249 pts/0
SL 0:00
|
\_ gpg
--logger-fd 4
--passphrase-fd 8
--batch --no-tty
--default-key
XXXXXXXX
--recipient
XXXXXXXX
--no-secmem-warning
--always-trust
--encrypt
--sign
1265 pts/0
SL 0:00
|
\_ gpg
--status-fd 6
--passphrase-fd 11
--logger-fd 5 --batch
--no-tty
--default-key
XXXXXXXX
--no-secmem-warning
--always-trust
--decrypt
1267 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1272 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1275 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1278 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1282 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1286 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1289 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1292 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1296 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1299 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1302 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1305 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1308 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1311 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1314 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1317 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1320 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1324 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1327 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1330 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1333 pts/0 Z
0:00
|
\_ [gpg]
<defunct>
1334 pts/1 Ss+
0:00
| \_
/usr/bin/ncftp -u
*******
backup.server.de
<http://backup.server.de>
1335 pts/0
R+ 0:00 \_
ps -u user xf
_______________________________________________
Duplicity-talk
mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
------------------------------------------------------------------------
_______________________________________________
Duplicity-talk mailing
list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
------------------------------------------------------------------------
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
------------------------------------------------------------------------
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
------------------------------------------------------------------------
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden
<mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
------------------------------------------------------------------------
_______________________________________________
Duplicity-talk mailing list
address@hidden <mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden <mailto:address@hidden>
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
------------------------------------------------------------------------
_______________________________________________
Duplicity-talk mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/duplicity-talk