Re: [Duplicity-talk] Duplicity 1.0.1 Released

[Kenneth Loafman]

Hi Scott,

I would say that all your errors are from not being able to build _librsync.so properly.  There is one thing that may help in lines 65-76 of setup.py.  You can supply LIBRSYNC_DIR as an env variable, or on the command line as --librsync-dir=.  I have export LIBRSYNC_DIR=/usr/local in my .bash_profile.  That came from homebrew installing into /usr/local instead of /usr.  Seems none of these installers care much for standards.

Try exporting LIBRSYNC_DIR and let me know how it goes after that.

...Ken

On Sun, Oct 9, 2022 at 4:46 PM Scott Hannahs <sth@p-hall.net> wrote:

So there are two major failures in testing.  One is four failures in the tahoe backend, (which I don’t use).

testing/unit/test_backend_instance.py ..............................FF.. [ 10%]

FF........................ssssssssss                                     [ 18%]

This shows a “Key Error” so is something wrong with my GPG version?

self = <duplicity.backends.tahoebackend.TAHOEBackend object at 0x10f58deb0>
commandline = 'tahoe cp /var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/local var:folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output/file-a'

    def subprocess_popen(self, commandline):
        u"""
        Execute the given command line with error check.
        Returns int Exitcode, string StdOut, string StdErr
    
        Raise a BackendException on failure.
        """
        import shlex
    
        if isinstance(commandline, (list, tuple)):
            logstr = u' '.join(commandline)
            args = commandline
        else:
            logstr = commandline
            args = shlex.split(commandline)
    
        logstr = self.munge_password(logstr)
        log.Info(_(u"Reading results of '%s'") % logstr)
    
        result, stdout, stderr = self.__subprocess_popen(args)
        if result != 0:
            try:
>               ignores = self.popen_breaks[args[0]]
E               KeyError: '/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/overrides/bin/tahoe'

/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:522: KeyError

During handling of the above exception, another exception occurred:

The second testing error is more general, so many of the tests pass and then starting at the “bad upload” test it all goes south.

testing/functional/test_badupload.py F                                   [ 70%]

testing/functional/test_cleanup.py FFF                                   [ 71%]

testing/functional/test_final.py FFFFFFFFFFFFFFFFFFFFFFFFFFF             [ 77%]

testing/functional/test_log.py F                                         [ 77%]

testing/functional/test_rdiffdir.py F                                    [ 77%]

testing/functional/test_replicate.py FFF                                 [ 78%]

testing/functional/test_restart.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF    [ 86%]

testing/functional/test_selection.py .FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 94%]

FFFFssFFFFFFFssss                                                        [ 98%]

This test has a long back trace, But I think that the basic problem is that duplicity cannot import “librsynch”???  The end of the traceback has the text "b"ImportError: cannot import name '_librsync' from 'duplicity' (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)"

...return_val: 1"

/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:527: BackendException

_______________________ BadUploadTest.test_missing_file ________________________

self = <testing.functional.test_badupload.BadUploadTest testMethod=test_missing_file>

    @pytest.mark.slow

    def test_missing_file(self):

        u"""

        Test basic lost file

        """

        try:

>           self.backup(u"full", u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"])

/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:44: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testing.functional.test_badupload.BadUploadTest testMethod=test_missing_file>

type = 'full'

input_dir = '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1'

options = ['full', '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', 'file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output', '--volsize', '1', '--skip-volume=1']

kwargs = {}, before_files = set(), now = 1664937609.016365

    def backup(self, type, input_dir, options=[], **kwargs):  # pylint: disable=redefined-builtin

        u"""Run duplicity backup to default directory"""

        options = [type, input_dir, self.backend_url, u"--volsize", u"1"] + options

        before_files = self.get_backend_files()

    

        # If a chain ends with time X and the next full chain begins at time X,

        # we may trigger an assert in dup_collections.py.  If needed, sleep to

        # avoid such problems

        now = time.time()

        if self.last_backup == int(now):

            time.sleep(1)

    

>       self.run_duplicity(options=options, **kwargs)

/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:189: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testing.functional.test_badupload.BadUploadTest testMethod=test_missing_file>

options = ['full', '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', 'file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output', '--volsize', '1', '--skip-volume=1']

current_time = None, fail = None, passphrase_input = []

    def run_duplicity(self, options=[], current_time=None, fail=None,

                      passphrase_input=[]):

        u"""

        Run duplicity binary with given arguments and options

        """

        # We run under setsid and take input from /dev/null (below) because

        # this way we force a failure if duplicity tries to read from the

        # console unexpectedly (like for gpg password or such).

    

        # Check all string inputs are unicode -- we will convert to system encoding before running the command

        for item in options:

            if sys.version_info.major == 2:

                assert not isinstance(item, str), u"item " + unicode(item) + u" in options is not unicode"

    

        for item in passphrase_input:

            assert isinstance(item, u"".__class__), u"item " + unicode(item) + u" in passphrase_input is not unicode"

    

        if platform.platform().startswith(u'Linux'):

            cmd_list = [u'setsid']

            if self._setsid_w:

                cmd_list.extend([u"-w"])

        else:

            cmd_list = []

        basepython = os.environ.get(u'TOXPYTHON', None)

        if basepython is not None:

            cmd_list.extend([basepython])

        run_coverage = os.environ.get(u'RUN_COVERAGE', None)

        if run_coverage is not None:

            cmd_list.extend([u"-m", u"coverage", u"run", u"--source=duplicity", u"-p"])

        cmd_list.extend([u"{0}/bin/duplicity".format(_top_dir)])

        cmd_list.extend(options)

        cmd_list.extend([u"-v0"])

        cmd_list.extend([u"--no-print-statistics"])

        cmd_list.extend([u"--allow-source-mismatch"])

        cmd_list.extend([u"--archive-dir={0}/testfiles/cache".format(_runtest_dir)])

        if current_time:

            cmd_list.extend([u"--current-time", current_time])

        cmd_list.extend(self.class_args)

        if fail:

            cmd_list.extend([u"--fail", u"".__class__(fail)])

        cmdline = u" ".join([u'"%s"' % x for x in cmd_list])

        cmdline = u" ".join([u'"%s"' % x for x in cmd_list])

    

        if not passphrase_input:

            cmdline += u" < /dev/null"

    

        # The immediately following block is the nicer way to execute pexpect with

        # unicode strings, but we need to have the pre-4.0 version for some time yet,

        # so for now this is commented out so tests execute the same way on all systems.

    

        # if parse_version(pexpect.__version__) >= parse_version("4.0"):

        #     # pexpect.spawn only supports unicode from version 4.0

        #     # there was a separate pexpect.spawnu in 3.x, but it has an error on readline

        #     child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline], timeout=None, encoding=sys.getfilesystemencoding())

        #

        #     for passphrase in passphrase_input:

        #         child.expect(u'passphrase.*:')

        #         child.sendline(passphrase)

        # else:

    

        # Manually encode to filesystem encoding and send to spawn as bytes

        # ToDo: Remove this once we no longer have to support systems with pexpect < 4.0

        if sys.version_info.major > 2:

            child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline], timeout=None)

        else:

            child = pexpect.spawn(b'/bin/sh', [b'-c', cmdline.encode(sys.getfilesystemencoding(),

                                                                     u'replace')], timeout=None)

    

        for passphrase in passphrase_input:

            child.expect(b'passphrase.*:')

            child.sendline(passphrase)

    

        # if the command fails, we need to clear its output

        # so it will terminate cleanly.

        child.expect_exact(pexpect.EOF)

        lines = child.before.splitlines()

        child.wait()

        child.ptyproc.delayafterclose = 0.0

        return_val = child.exitstatus

    

        if fail:

            self.assertEqual(30, return_val)

        elif return_val:

            print(u"\n...command:", cmdline, file=sys.stderr)

            print(u"...cwd:", os.getcwd(), file=sys.stderr)

            print(u"...output:", file=sys.stderr)

            for line in lines:

                line = line.rstrip()

                if line:

                    print(line, file=sys.stderr)

            print(u"...return_val:", return_val, file=sys.stderr)

>           raise CmdError(return_val)

E           testing.functional.CmdError: 1

/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:175: CmdError

During handling of the above exception, another exception occurred:

self = <testing.functional.test_badupload.BadUploadTest testMethod=test_missing_file>

    @pytest.mark.slow

    def test_missing_file(self):

        u"""

        Test basic lost file

        """

        try:

            self.backup(u"full", u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"])

            self.fail()

        except CmdError as e:

>           self.assertEqual(e.exit_status, 44, str(e))

E           AssertionError: 1 != 44 : 1

/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:47: AssertionError

----------------------------- Captured stderr call -----------------------------

...command: "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity" "full" "/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1" "file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output" "--volsize" "1" "--skip-volume=1" "-v0" "--no-print-statistics" "--allow-source-mismatch" "--archive-dir=/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/cache" < /dev/null

...cwd: /private/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T

...output:

b'Traceback (most recent call last):'

b'  File "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity", line 37, in <module>'

b'    from duplicity.dup_main import main'

b'  File "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_main.py", line 50, in <module>'

b'    from duplicity import commandline'

b'  File "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/commandline.py", line 43, in <module>'

b'    from duplicity import backend'

b'  File "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py", line 44, in <module>'

b'    from duplicity import dup_temp'

b'  File "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_temp.py", line 34, in <module>'

b'    from duplicity import path'

b'  File "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/path.py", line 49, in <module>'

b'    from duplicity import librsync'

b'  File "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/librsync.py", line 36, in <module>'

b'    from . import _librsync'

b"ImportError: cannot import name '_librsync' from 'duplicity' (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)"

...return_val: 1

On Oct 9, 2022, at 11:26 AM, Kenneth Loafman <kenneth@loafman.com> wrote:

Ok, sounds good.  Any progress?

...Ken

On Thu, Oct 6, 2022 at 3:15 PM Scott Hannahs <sth@p-hall.net> wrote:

I will try to reinstall Xcode and the command line tools.  I had been using py38 which is installed in both the standard and an updated version in the non-standard location.

On Oct 6, 2022, at 14:57, Kenneth Loafman <kenneth@loafman.com> wrote:

OK,

I could not remember what version of Python 3 you were running, so py3x meant for you to fill it in.  I'm guessing either py38 or py39 for Python 3.8 and Python 3.9 respectively.  Others follow the same model, py27,py35,...py310.

I am not sure why it's trying to build for anything but x86_64 if you have Intel silicone and not Apple silicone.

Here's what mine looks like for py38:

ken@dione:~/workspace/duplicity-origin$ python3.8 ./setup.py build_ext

Unable to get SCM version: defaulting to 1.0.1

running build_ext

Building extension for librsync...

building 'duplicity._librsync' extension

creating build/temp.macosx-12-x86_64-cpython-38

creating build/temp.macosx-12-x86_64-cpython-38/duplicity

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/usr/local/include -I/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c duplicity/_librsyncmodule.c -o build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o

creating build/lib.macosx-12-x86_64-cpython-38

creating build/lib.macosx-12-x86_64-cpython-38/duplicity

clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-12-x86_64-cpython-38/duplicity/_librsync.cpython-38-darwin.so

ld: warning: -undefined dynamic_lookup may not work with chained fixups

copying build/lib.macosx-12-x86_64-cpython-38/duplicity/_librsync.cpython-38-darwin.so -> duplicity

It looks like your xcode libs are out of date, and it also looks like you're not using the standard locations for software.  That's gonna complicate matters.

I'm on Monterey 12.6 with the current Xcode from Apple, all in standard places.

...Ken

On Thu, Oct 6, 2022 at 1:13 PM Scott Hannahs <sth@p-hall.net> wrote:

Ken,

I will leave fink out, but it should be doing exactly the same thing!  :-)

the following doesn’t even build duplicity correctly, so I don’t get to the tox command.

% tar xf duplicity-1.0.1.tar 

% cd duplicity-1.0.1 

% % C_INCLUDE_PATH=/opt/sw/include LIBRARY_PATH=/opt/sw/lib python3 ./setup.py build_ext > build.out.txt 2>&1

This gives the following output.

Unable to get SCM version: defaulting to 1.0.1

running build_ext

Building extension for librsync...

building 'duplicity._librsync' extension

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/include/python3.9 -c duplicity/_librsyncmodule.c -o build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o

clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -Wl,-headerpad,0x1000 build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o -lrsync -o build/lib.macosx-10.9-universal2-3.9/duplicity/_librsync.cpython-39-darwin.so

ld: warning: ignoring file /opt/sw/lib/librsync.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

ld: warning: dylib (/opt/sw/lib/librsync.dylib) was built for newer macOS version (12.4) than being linked (10.9)

copying build/lib.macosx-10.9-universal2-3.9/duplicity/_librsync.cpython-39-darwin.so -> duplicity

I am building on an X86 machine and my libraries are x86 and I am not trying to build a universal arm 64 system.   But with warnings it tries to build.

tox does not like the py3x value

% tox -epy3 

py3 create: /Users/sth/Downloads/duplicity-1.0.1/.tox/py3

py3 installdeps: -rrequirements.txt

and I ran out of time this AM waiting for more output.

-Scott

On Oct 5, 2022, at 11:54 AM, Kenneth Loafman <kenneth@loafman.com> wrote:

Could you try just building and running the tests in a 'normal' user environment?  Leave Fink out of the picture.

do this:

1. $ tar xf duplicity-1.01.tar.gz
2. $ ./setup.py build_ext
3. $ tox -e py3x

...Thanks,

...Ken