Re: Discussion about file format for the future

[Patrik Dufresne]

As mentioned by Robert searching for metadata is complex because you need
to scan multiple file to actually find the right value. instead of having a
query if we were using a database.

Obviously performance-wise it's not great either because we need to scan
multiple file.

The only thing I hate about that  is lake of visibility as a compromise
maybe we can find the most common database and add layer on top using
command line to search in this database? To let users be autonomous.
SQLite is probably one of those very popular and simple database.

On Thu., Jun. 4, 2020, 11:03 p.m. Robert Nichols, <
rnicholsNOSPAM@comcast.net> wrote:

> On 6/4/20 11:43 AM, Patrik Dufresne wrote:
> > But two cent on the subject is, should we really keep this filebase ? For
> > rdiffweb, scanning the metadata files is a nightmare. When I just need a
> > subset of the data to be displayed to the user. I always thought a
> database
> > could be better fit for the job. Something like a key store or similar.
>
> +1 from me
>
> The way rdiff-backup stores metadata is its worst feature, in my opinion.
> Keeping the metadata in various text files makes analysis unnecessarily
> complex and searches very inefficient. Inode data for hard-linked files
> is replicated in the mirror_metadata file, except for the checksum, which
> is stored just on the first entry for that inode, so you have to go
> hunting for it, and make sure it is always in the right place when
> that linking changes. That sort of thing just screams to be stored in
> a database.
>
> --
> Bob Nichols     "NOSPAM" is really part of my email address.
>                  Do NOT delete it.
>
>
>