[myexperiment-hackers] RE: myExperiment and Trident workflows

[Roger Barga]

Thanks Jits,

We will go through this and discuss if there are any questions.

roger

-----Original Message-----
From: Jiten Bhagat [mailto:address@hidden 
Sent: Monday, April 20, 2009 3:58 AM
To: Roger Barga
Cc: David De Roure; Trident Core Team; address@hidden
Subject: Re: myExperiment and Trident workflows

Hi Roger,

Apologies for the delayed response. Hope you're well too.

Thanks very much for the info. We've looked this over with the team here and 
the whole scenario looks fine, and for the most part very doable.

Comments:

- For stage 2 in the diagram: we currently have two mechanisms to upload 
workflow packages:
  - via the HTTP/HTML interface, for authenticated users only (as you 
mentioned).
  - via the RESTful API (HTTP/XML based; which currently requires HTTP Basic 
Auth for authentication of a user's account to allow the upload under their 
myExp user account. We're looking to make this SSL enabled soon). See: 
http://wiki.myexperiment.org/index.php/Developer:API#POST_workflow for 
documentation on how to do this. The "content_type" identifier for Trident OPC 
packages is "trident_opc" and for Trident XOML files alone it's 
"application/xaml+xml" (let us know if you would like any of these changed). 
For the latter I suspect we won't be able to parse any metadata out of the XOML 
file, but this can still be provided in the HTML form and/or the RESTful API.

- I should also mention that it's possible to have full myExperiment search and 
tag cloud functionality (and more) within the Trident system, through the 
RESTful API. See: 
http://wiki.myexperiment.org/index.php/Developer:Taverna_plugin for an example 
of a plugin I developed for Taverna 1 (which we are improving for Taverna 2, 
with more write abilities). For API reference: 
http://wiki.myexperiment.org/index.php/Developer:API (this is maintained by Don 
Cruickshank <address@hidden> who is able to provide support).

- The running of Trident workflows from within the myExp interface might be 
something that we cannot do just yet and maybe something we should plan for 
further down the line?

- In terms of parsing metadata from the OPC packages to display in 
myExperiment, the list you provided is very comprehensive and exactly what we 
need, thanks. Few comments on this:
  - The workflow description can have most HTML elements in it (allowing for 
rich text).
  - Currently, we use sequential version numbers in myExp that the system 
automatically generates. What we can do if a 'version' field is provided is to 
store that as a special "version label".
  - We require just one preview image, which we will resize to the various 
different sizes (for thumbnails etc).
  - If you could also provide an SVG version of the preview image, in the 
package, that would be great (but not compulsory).
  - We support the following license options (if you need to support different 
licenses, let us know):
    - "by-nd" - Creative Commons Attribution-NoDerivs 3.0 License
    - "by-sa" - Creative Commons Attribution-Share Alike 3.0 License
    - "by" - Creative Commons Attribution 3.0 License
  - For credits: could you make these either names of people, or full URLs to 
people's myExperiment profiles? This way we can match what we get to existing 
users on myExp. We're not sure about actually creating users based on new data. 
In a worst case scenario, we'll try and store these as names/URLs directly so 
they still show up under credits.
  - For attributions: these are about attributing existing workflows/files/etc 
that were used to help build the workflow (and it's code and so on). So these 
would be URLs to existing workflows/files/etc, much like for credits, above, 
but rather than being about people it's about things.

- In terms of downloading a workflow, this is either possible through a direct 
download URL (http://www.myexperiment.org/workflows/{id}/download. Note: only 
users who have download permissions will be able to get a file back from this
URL) or via the RESTful API.

- Speaking of permissions, when uploading a workflow through the RESTful API, 
default credits, sharing and license* settings are applied (see attached 
image). The user would then need to go the workflow manage page to change these 
(http://www.myexperiment.org/workflows/{id}/edit).

* If credits and license are not provided.

In terms of taking this forward, one of the things required from the 
myExperiment side is the development of a "workflow processor" (in Ruby), that 
acts as the bridge between the Trident OPC package and the myExp system. The 
interface/skeleton of this processor is here: 
http://myexperiment.rubyforge.org/svn/trunk/lib/workflow_processors/interface.rb
and an example is the Taverna 1 processor here: 
http://myexperiment.rubyforge.org/svn/trunk/lib/workflow_processors/taverna_scufl.rb

You will notice that this interface does not currently support all the metadata 
elements you have mentioned in your list. However this isn't a problem and we 
can easily add new methods to this interface to cater for the new metadata 
elements. (We'll then write the appropriate routines in the create workflow 
process to add these to our database).

At this stage, would it be possible for one of your devs to write an initial 
version of this workflow processor so we have something to build on? I am at 
hand to provide info/support (especially since the interface isn't fully 
documented yet).

Hope this is useful.

Kind regards,
Jits


Roger Barga wrote:
>
> Hi David, Jits:
>
>  
>
> I hope all is going well for you both.  Below is a first draft of our 
> view of publishing Trident workflows to myExperiment.  We would very 
> much appreciate your feedback on this and especially on metadata 
> provided with the OPC package.  Once we have this in hand we would 
> like to begin creating these packages to test out the end-to-end 
> implementation.
>
>  
>
> Roger
>
>  
>
> *From:* Dean Guo (DI)
> *Sent:* Thursday, April 09, 2009 5:26 PM
> *To:* Roger Barga
> *Subject:* myExperiment and Trident workflows
>
>  
>
> Roger,
>
>  
>
> Here is the overview of publishing Trident workflows to myExperiment 
> and downloading Trident workflows from it.    Please review it and 
> modify it as needed.  More details below the diagram.
>
>  
>
>  
>
> Currently Trident can export a workflow package (a zip file) with core 
> entities required to run Trident workflows.   We are working on 
> importing a package back to Trident to complete the round trip. 
>
>  
>
> We will add more data to a Trident workflow package specifically 
> targeted for myExperiement.    Trident UI design will support data 
> entries for it.   We need to get more information from myExperiment in 
> terms of the key meta data needed.    Here is a preliminary list of 
> data elements that we have identified for myExperiment:
>
>  
>
> *         Workflow category
>
> *         Workflow name
>
> *         Workflow description
>
> *         Keywords
>
> *         Version
>
> *         Type: Trident (XOML)  (Windows Workflow MIME type)
>
> *         WF Preview image
>
> *         WF Thumbnail image
>
> *         License (Open source license name)
>
> *         Credits (People or groups)
>
> o   Name
>
> o   email
>
> o   Website
>
> o   Location (city, state, and country)
>
> *         Attributions
>
> o   WF XOML
>
> o   WF assemblies
>
> o   WF Input parameters
>
> o   WF output parameters
>
> *         Others
>
> o   Documentation (PDF)
>
>  
>
> We need feedback from myExperiment on this list asap.     Thanks.
>
>  
>
> Dean
>