Re: [gwl-devel] support for containers

[zimoun]

Hi Ricardo,

On Tue, 29 Jan 2019 at 10:54, Ricardo Wurmus <address@hidden> wrote:
> Ricardo Wurmus <address@hidden> writes:
>
> > the GWL could already support execution in containers with this patch:
> […]
> > The directories to be mounted in the container are derived from the
> > declared inputs and outputs.  The only problem is that inputs are
> > read-only in this implementation.  I like it this way, actually, but it
> > means that the extended example workflow won’t work as it tries to
> > delete its inputs.
> >
> > Should data inputs be declared as (mutable-file …) or (file …) instead
> > of being plain strings?
>
> Taking a step back I think it is worth remembering that ultimately we’d
> like to have output caching via an immutable data store.  I think
> declaring files as mutable would be a mistake.  Garbage collection is
> better than modifying output files.

I agree that Garbage Collection is a better design, from my opinion.
However, as an user I would like to garbage collect only the
inputs/outputs of a specific workflow. Not all inside the store.


> Some thoughts on how the data store should work: it’s easy to add stuff
> to the store (we just hash the inputs leading up to the output,
> excluding the output itself); but how do we make store items available
> to a process?  When using containers we can bind mount the file to the
> declared input location; when not using containers we need to link or
> copy the file from

By store, do you the Guix store? or another store? as GWL store?

By inputs, do you mean data-inputs and package-inputs?


I do not know if below makes sense.
What happens if a kind of record managing the data-inputs is added?
I mean, there are: package, process, workflow and we add data.
Then the data is manage as other symbols, with define-public, etc.
However, these data need to live a special store, not in the store of packages.

>
> I also want to have finer control over inputs.  Only declared input
> *files* should be available in the container, not whole *directories*.

Hum? I make sense.
Especially for references used by aligners.
Even if the size of the container will significantly growth. :-)


All the best,
simon