Re: Managing data files in workflows

[Konrad Hinsen]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> This works for me correctly:

Thanks for looking into this! For me, your change makes no difference.
Nor should it, because in my setup the "data" directory already exists.
I still get an error message about the already existing file.

Maybe it's time to switch to the development version of GWL!

> It skips the process because the output file exists and the daring
> assumption we make is that outputs are reproducible.
>
> I would like to make these assumptions explicit in a future version, but
> I’m not sure how.  An idea is to add keyword arguments to “file” that
> allows us to provide a content hash, or merely a flag to declare a file
> as volatile and thus in need of recomputation.

Declaring a file explicitly as volatile or reproducible sounds good. I
am less convinced about adding a hash, except for inputs external to the
workflow.

In my example, the file I download changes on the server once per week,
so I'd mark it as volatile. I'd then expect it to be re-downloaded at
every execution of the workflow. But I a also OK with doing this
manually, i.e. deleting the file if I want it to be replaced. Old make
habits never die ;-)

> I also wanted to have IPFS and git-annex support, but before I embark on
> this I want to understand exactly how this should behave and what the UI
> should be.  E.g. having an input that is declared as “IPFS-file” would
> cause that input file to be fetched automatically without having to
> specify a process that downloads it first.  (Something similar could be
> implemented for web resources as in your example.)

Indeed. An extended version of "guix download" for workflows.

However, what I had in mind with my question is the management of
intermediate results in my workflow, especially in its development
phase. If I change my workflow file, or a script that it calls,
I'd want only the affected steps to be recomputed. That's not much
of an issue for my current test case, but I have bigger dreams for
the future ;-)

Cheers,
  Konrad.