Re: Package file indexing

[Christopher Baines]

Pierre Neidhardt <address@hidden> writes:

> Christopher Baines <address@hidden> writes:
>
>> So, to elaborate a bit more on the architecture I've had in mind for
>> dealing with the actual nars…
>>
>> I see the scope of the Guix Data Service extending as far as what nars
>> are available for outputs, and what outputs are associated with each
>> revision, but I don't think it should store the actual nar files.
>>
>> What you could have is another service, which subscribes to the Guix
>> Data Service to find out about new revisions and nars (from build
>> servers). When this new service finds out about Guix revisions, it would
>> ask this Guix Data Service for all the outputs, and store this away in a
>> database. When it finds out about nars, it would download them, and
>> maybe extract out the list of files.
>>
>> I think this setup would allow this new service to construct a file
>> containing information about all files in all the outputs for a
>> revision, which it has nars available for. This file could then be
>> downloaded, and searched through when you want to find which output
>> contains a file.
>
> Tell me if I understood you correctly: in this scenario we would modify
> the Guix derivation process to store the file list in the nars.  Is this 
> correct?

Not quite. As Ludo mentioned, you can trivially extract out the file
list from nar files already (like guix archive -t). So this new service
I'm thinking about which stores the nar files, would be able to read the
list of files from the nar.

> Question about the Guix Data Service: I suppose that the information about 
> the outputs
> of a given revision is built incrementally, i.e. as they get published
> by the build farm.  Is this correct?

So the information about what outputs the derivations produce is
completely available once the revision has been loaded.

However, nar files for those outputs become available as build farms
build them, and the Guix Data Service hopefully finds out about these
soon after. Also, since the build could be non-deterministic, it's
possible for multiple different nar files to be generated for the same
output (maybe even containing different files in the extreme case!).

> If so, then the file index service needs to update the database
> incrementally as well.  So we need some entry point to fetch the
> information delta between now and the last time we fetch the information.

Yeah, you might fetch the file list database, and then another
derivation is built, revealing which files are in the associated
outputs. A new database can then be constructed containing that
additional information.

In the trivial case, the new file could be downloaded to replace the old
one. This could maybe be optimised by just downloading the changes,
maybe by using something like xdelta.

> Please correct me if I got it all wrong! :D

All great questions, hopefully I've managed to clear things up!

signature.asc
Description: PGP signature