[myexperiment-hackers] RE: myExperiment and Trident workflows

[Nitin Gautam (Persistent Systems Private Ltd)]

Hi Jits,

I hope you are doing fine. Allow me to introduce myself. My name is Nitin and I 
am a developer working on Trident. I will be working on the integration between 
Trident and my Experiment.

I went through the email exchange that has happened so far and I have come up 
with a small plan about how I would be going about the integration. I am 
attaching the document file for your reference. Please let me know if there is 
any step that I am missing out.

The first thing that I plan to work on is the workflow processor. Previously 
Dean had sent a list of things that would be present inside the package. 
However I think that if the entire OPC package will be uploaded as binary data 
then there is no additional requirement for including any Trident field as part 
of the metadata. So this means that we can live with the current interface and 
not introduce any interface changes. Again there are two ways of sending 
myExperiment data along with trident workflows.
        a. Adding data required by myExperiment to the Trident package. 
        b. Send the my experiment data as part of the rest xml for upload.

Let me know if this makes sense. Also if you can guide me to some reference 
material about how to go about writing this processor it would be great. I have 
never worked on Ruby so I would need some help to get started.

Best Regards,
Nitin


-----Original Message-----
From: Dean Guo (DI) 
Sent: Wednesday, May 06, 2009 1:40 PM
To: Nitin Gautam (Persistent Systems Private Ltd)
Subject: FW: myExperiment and Trident workflows

This is Jiten's response.

-----Original Message-----
From: Jiten Bhagat [mailto:address@hidden 
Sent: Monday, April 20, 2009 3:58 AM
To: Roger Barga
Cc: David De Roure; Trident Core Team; address@hidden
Subject: Re: myExperiment and Trident workflows

Hi Roger,

Apologies for the delayed response. Hope you're well too.

Thanks very much for the info. We've looked this over with the team here and 
the whole scenario looks fine, and for the most part very doable.

Comments:

- For stage 2 in the diagram: we currently have two mechanisms to upload 
workflow packages:
  - via the HTTP/HTML interface, for authenticated users only (as you 
mentioned).
  - via the RESTful API (HTTP/XML based; which currently requires HTTP Basic 
Auth for authentication of a user's account to allow the upload under their 
myExp user account. We're looking to make this SSL enabled soon). See: 
http://wiki.myexperiment.org/index.php/Developer:API#POST_workflow for 
documentation on how to do this. The "content_type" identifier for Trident OPC 
packages is "trident_opc" and for Trident XOML files alone it's 
"application/xaml+xml" (let us know if you would like any of these changed). 
For the latter I suspect we won't be able to parse any metadata out of the XOML 
file, but this can still be provided in the HTML form and/or the RESTful API.

- I should also mention that it's possible to have full myExperiment search and 
tag cloud functionality (and more) within the Trident system, through the 
RESTful API. See: 
http://wiki.myexperiment.org/index.php/Developer:Taverna_plugin for an example 
of a plugin I developed for Taverna 1 (which we are improving for Taverna 2, 
with more write abilities). For API reference: 
http://wiki.myexperiment.org/index.php/Developer:API (this is maintained by Don 
Cruickshank <address@hidden> who is able to provide support).

- The running of Trident workflows from within the myExp interface might be 
something that we cannot do just yet and maybe something we should plan for 
further down the line?

- In terms of parsing metadata from the OPC packages to display in 
myExperiment, the list you provided is very comprehensive and exactly what we 
need, thanks. Few comments on this:
  - The workflow description can have most HTML elements in it (allowing for 
rich text).
  - Currently, we use sequential version numbers in myExp that the system 
automatically generates. What we can do if a 'version' field is provided is to 
store that as a special "version label".
  - We require just one preview image, which we will resize to the various 
different sizes (for thumbnails etc).
  - If you could also provide an SVG version of the preview image, in the 
package, that would be great (but not compulsory).
  - We support the following license options (if you need to support different 
licenses, let us know):
    - "by-nd" - Creative Commons Attribution-NoDerivs 3.0 License
    - "by-sa" - Creative Commons Attribution-Share Alike 3.0 License
    - "by" - Creative Commons Attribution 3.0 License
  - For credits: could you make these either names of people, or full URLs to 
people's myExperiment profiles? This way we can match what we get to existing 
users on myExp. We're not sure about actually creating users based on new data. 
In a worst case scenario, we'll try and store these as names/URLs directly so 
they still show up under credits.
  - For attributions: these are about attributing existing workflows/files/etc 
that were used to help build the workflow (and it's code and so on). So these 
would be URLs to existing workflows/files/etc, much like for credits, above, 
but rather than being about people it's about things.

- In terms of downloading a workflow, this is either possible through a direct 
download URL (http://www.myexperiment.org/workflows/{id}/download. Note: only 
users who have download permissions will be able to get a file back from this
URL) or via the RESTful API.

- Speaking of permissions, when uploading a workflow through the RESTful API, 
default credits, sharing and license* settings are applied (see attached 
image). The user would then need to go the workflow manage page to change these 
(http://www.myexperiment.org/workflows/{id}/edit).

* If credits and license are not provided.

In terms of taking this forward, one of the things required from the 
myExperiment side is the development of a "workflow processor" (in Ruby), that 
acts as the bridge between the Trident OPC package and the myExp system. The 
interface/skeleton of this processor is here: 
http://myexperiment.rubyforge.org/svn/trunk/lib/workflow_processors/interface.rb
and an example is the Taverna 1 processor here: 
http://myexperiment.rubyforge.org/svn/trunk/lib/workflow_processors/taverna_scufl.rb

You will notice that this interface does not currently support all the metadata 
elements you have mentioned in your list. However this isn't a problem and we 
can easily add new methods to this interface to cater for the new metadata 
elements. (We'll then write the appropriate routines in the create workflow 
process to add these to our database).

At this stage, would it be possible for one of your devs to write an initial 
version of this workflow processor so we have something to build on? I am at 
hand to provide info/support (especially since the interface isn't fully 
documented yet).

Hope this is useful.

Kind regards,
Jits

MyExperimentApp.doc
Description: MyExperimentApp.doc