gwl-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RFC: Support for DRMAA backend


From: Ricardo Wurmus
Subject: RFC: Support for DRMAA backend
Date: Mon, 03 May 2021 11:24:35 +0200
User-agent: mu4e 1.4.15; emacs 27.2

Hello there,

I pushed a new branch “wip-drmaa”, which lets GWL use an HPC scheduler via DRMAA. Set the GWL_USE_DRMAA variable and use the default execution engine. (In the future this variable will disappear once we have agreed on a nicer way to select DRMAA execution.)

I only tested this with a local Slurm cluster on my laptop, but it seems to work fine. I’d be happy to hear test reports!

Before this can make its way into the main branch we have to figure out if the concept of an execution engine still makes sense the way it is implemented.

Execution engines currently are only used to build custom scripts (or script wrappers) — but using DRMAA does not require any new kind of script. The old Grid Engine backend builds the usual workflow scripts and a set of job scripts that call those scripts after passing Grid Engine settings. For DRMAA we only need run-time support.

I wonder if we should scrap execution engines as they currently exist; we would only ever generate the same kind of workflow scripts and move all execution-related work (like submitting jobs to a scheduler) to, well, execution time. This also means that the workflow scripts would not need to be modified when picking a different execution backend, which I think is good.

Comments?

--
Ricardo



reply via email to

[Prev in Thread] Current Thread [Next in Thread]