[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] support xz compression
From: |
edgar . soldin |
Subject: |
Re: [Duplicity-talk] support xz compression |
Date: |
Sun, 21 Oct 2018 19:33:38 +0200 |
User-agent: |
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 |
On 10/21/2018 18:52, Radim Tobolka via Duplicity-talk wrote:
> Hi,
> I'd like to add XZ (LZMA) compression to Duplicity. I'd like to discuss, how
> to go about it, so that the solution is aligned with your high level plan for
> the project.
>
> It would be based on lzma module in Python 3 and it's backport in Python 2.
>
> The feature could be be activated with new option,
> --compression-xz=<on|off|<preset>>
> on - turn on, use default preset 6
> off - turn off, give user chance to override, if specified multiple times (in
> scripts)
> <preset> - 0-9[e], turn on with given preset with meaning and effect as
> documented in lzma package and xz(1) man page
ideally this syntax should be transferred to gzip compression as well. but
while at, why not using some that would be easily extendable in the future with
different algo's like
--compression=<algo> and
--compression-params="" or --compression-level=<level>
not sure what to do w/ the current default to compress via gzip unless
'--no-compression' is given. could be kept, meaning --no-compression would
translate to --compression="", but maybe we should deactivate the automatic
compression?
> Upon archiving, if active with encryption on:
> - adjust archive files' extensions in file_naming to have .xz.gpg extension
> - in gpg.GPGWriteFile, turn off default gpg compression, run data through
> lzma compressor before feeding it to gpg
> if not encrypting:
> - adjust archive files' extensions in file_naming to have .xz extension
> - output to file obtained by means of lzma.open() in gpg.GzipWriteFile()
don't like reusing anything named Gzip** handling xz. how about eventually
merging
PlainWriteFile()
GzipWriteFile()
to a clean
WriteFile() supporting different encryptions via parameter (eg. derived from
'--compression=<algo>')?
> Upon restoring (the option need not be present, the feature could be
> autodetected):
as that should be the case already with gzip compression, i see no obstacle
there.
> - detect XZ compression in file_naming.parse, set_encryption_or_compression
> function and perhaps set new flag on the ParseResults object
> - activate XZ decompressor in path.DupPath.filtered_open() based on above flag
same here
> There may be issues with accuracy of --volsize feature, because lzma uses
> larger buffers. Let's see during testing.
>
> I'm successfully running PoC along these lines for few months now, albeit
> with piping to external xz process.
>
> And of course, I'll add battery of tests and entry in manpage.
always a good idea!
> Any ideas, comments, feedback?
above ;) and please document everything in the man page.
> Best,
> Radim
dito and regards.. ede/duply.net