[GMG-Devel] MediaGoblin multi-file upload braindump/plans

[Christopher Allan Webber]

Hello all,

Braindump time!  Complete with ascii art!  It's been a while,
right? :)

A long desired feature for MediaGoblin is a multi-file-upload tool.
There have been a few demonstrations on how to implement
this... pythonsnake (a javascript-only, ajax'y one) and kamikaze-m (a
no-javascript HTML5 example) have both done really good jobs of
implementing demonstrations of how this might work.  I'm really
excited with both of those branches, and I reviewed both of their
work, but I think in order to get a design that everyone's going to be
really happy with for the long haul I needed to sit down and do some
planning.  So here goes.

Let's start with some mockups
-----------------------------

Here's an ascii art mockup I tossed together (this is the 2nd edition
of it):

  http://dustycloud.org/misc/mediagoblin_submit_mockup2.txt

And here's a glow'y demo by Aeva Palecek that implements some (but not
all) of the things... but in a form that gets us pretty close to
seeing what it'll look like!

  http://pirateradiotheater.org/files/uploader_test/articulated.html

Exciting, amiright?!

Okay, there's some things you might notice from this mockup... let's
discuss the general design!

 - There's an area to drag and drop files, and there's also a general
   "click this to browse your filesystem" button.  Once you select
   that file, it should immediately appear in the list of files to be
   uploaded below it.

 - Each file is listed one after each other.  You can't see it in the
   drawing, but my suspicion is that the nicest way to do this is to
   have alternating background colors for each file listed.
   
 - Files are in one of three states (sorry, I'm describing them
   backwards from the order they appear in the mockup):

   - to be uploaded: the bottom two items in the mockup
     (apricot_screenshot.png and sintel_workflow.png) have not been
     updated yet.  There's a "cancel button" that you can press to
     remove them from the list.

   - currently uploading: the middle item, bigbuckbunny.ogv, is
     currently uploading.  This is made clear by the uploading
     progress bar (which might be a lot smaller than this mockup
     suggests, more below).  It also has a cancel button... pressing
     that will both cancel the current upload process and will remove
     it from the list.

   - already uploaded: the first two items in the mockup
     (elephantsdream.webm and blender_manual.pdf) are both in this
     state.  The "cancel" button has moved to a "Uploaded!" notice.

 - The upload bar may not be as suggested.  In fact, here's two
   alternate bits:
   - schendje pointed out that we should probably have a thinner
     upload bar:
     http://i.imgur.com/hTogr7k.png
     ... there's a bit of zeitgeist there, because that's very similar
     to how pythonsnake's upload bar already works, and Aeva's demo!
   - There's some advantages to this, because the bar can "stay there"
     after upload without moving things around and taking up extra
     space or possibly moving the element under someone's mouse
     (nobody wants that!)
   - Some discussion that maybe if we eventually implement processing
     feedback, there can be two colors, one for upload progress, one
     for processing progress.
   - Have you looked at
     http://pirateradiotheater.org/files/uploader_test/articulated.html
     yet???  You should! :)

 - One other thing you might notice is that in the ascii art mockup,
   if you click the arrow to the left of an item, it "expands down".
   In the mockup, blender_manual.pdf is expanded as such.  As shown in
   the demo, you should be able to edit the details of an entry via
   this, such as the title, description, license, or tags of an entry.
   
   Possibly more details may be shown; when uploading, "how many
   kilobytes uploaded of how much remaining may be shown" or some such
   thing, and if uploaded and processing feedback is implemented, we
   may give verbose data here.

 - As you can tell, this is a submission interface that's designed
   with javascript in mind.  If you're not into javascript, there's no
   need to worry, we'll be keeping around a no-js submit tool that
   probably works mostly the same as the present one.
   
 - This also means we'll be doing some refactoring of the submit
   code... this also means that this might be a good opportunity to
   iterate on this a bit.  It also means that we'll probably finally
   get, as part of this work, a long-desired feature: a command-line
   upload tool!
   
 - You see an [X] in the upper right.  As you may have guessed, we
   will probably be moving media submission to a "lightbox" style
   feature.  This way, you never need to leave the page you're on in
   order to submit media.

 - I mentioned it briefly above, but it's not really probably going to
   be covered in this iteration; one thing that would be nice would be
   to provide "currently in-processing" feedback here also.  That can
   happen later, however.


Technical implementation
------------------------

Okay!  Actually I think the details of the implementation are in some
ways self-descriptive by the mockup, but let's go over some of them
anyway.

 - I think we're going to need some custom code for
   this... pythonsnake's previous branch used jquery-upload... that
   might still be usable, though I have some suspicion that it's both
   a bit too heavy (it provides a lot of features, including previews
   of media types, though that's hard to generalize for us) and I'm
   not sure it can do one-by-one uploading of a bunch of items that
   have been queued.  I'm not 100% sure though.
   
   At any rate, we're going to need quite a bit of custom code anyway
   I think, especially so we can have nice ajax'y edit forms and
   friends.

 - https://hacks.mozilla.org/2011/01/how-to-develop-a-html5-image-uploader/
   might be a good general guide to some of this?

 - I suspect that hitting the submit button will go something like the
   following:

   It calls code loaded in multisubmit.js or some such thing, which
   does an ajax request for the "templates" used in submitting, which
   includes both the general box that will be .innerhtml()'ed or
   whatnot into the lightbox's dom, as well as various small templates
   for each "file" display, as well as the "edit" display, etc.
     
   Each of these "template snippets" are rendered versions of jinja2
   templates.  Yes, confusing naming.  What I mean is that on the
   server side you load up various jinja2 templates so you can get
   these pieces translated (and possibly rearranged a bit for
   theming), and they're then passed up to the browser so they can be
   inserted into the DOM.  So you might get something like this back
   after that first request:
   
     {"lightbox": "<div><div class="title">Submit your media:</div>foooo</div>",
      "file_row": "<div><span 
class="filename"></span><button>cancel</button></div>",
      "edit_box": "<div><form><div class="edit_row"><div 
class="edit_row_title">Title</div><div class="edit_widget"><input name="title" 
/><div></div></form></div>"}
      
   (No nitpicking here... I'm not trying to construct real HTML, just
   showing what I mean)

   The javascript should know how to arrange this, as well as what
   segments mean what, probably based on their class names.

 - As soon as you select a file, it should start to upload it.  You
   can hit cancel, of course.  It should probably use the method shown
   in hacks.mozilla.org above to do AJAX'y things.
   
 - Expanding the "edit" section and filling things in should do
   different things depending on what the "state" of the media is.  If
   it hasn't started uploading yet, it's easy: that's part of the POST
   data along with the file giving the proper title and description
   and etc.  If it finished uploading already, it's also not too hard:
   it's an AJAX'y request to edit that data.  However, if it's
   mid-upload, there's no way currently to do either of
   those... instead, it should queue the data for doing an edit AJAX
   request after the actual submission ends.

 - As said, there's going to need to be some refactoring of the submit
   code.  I'm not sure if that should come before the rest of this or
   just happen as part of it.


Lastly
------

As usual, many things in MediaGoblin end up happening due to the
thanks of many people, but in this case, pythonsnake deserves special
thanks.  pythonsnake's hard work on the multi-uploader branch is what
made it clear to me how a multi-upload tool can actually work... I'm
not sure whether or not we'll be using any of that code exactly (we
might or might not!  to be seen!), but it's certainly pythonsnake's
code that provided the general vision for where this is going, and
reading that code made it clearer to me what needed to happen.

Thank you pythonsnake for all your hard work!  And thanks for being
patient with me even though it took some time for me to get to
reviewing it.

Anyway, that's it for now.  What do people think?  Does it make sense?
Feedback is welcome!