Re: [O] Re: Test framework needed

[Eric Schulte]

Suvayu Ali <address@hidden> writes:

> On Wed, 30 Mar 2011 15:42:19 -0600
> "Eric Schulte" <address@hidden> wrote:
>
>> >
>> > This suite should actually be updated with effectively each patch
>> > which introduces new features and run after each patch.
>> >  
>> 
>> Agreed, in a perfect world...
>> 
>> >
>> > So is it only necessary to add meat to this framework?
>> >  
>> 
>> Yes, I believe the best way forward would be to add tests to the
>> existing framework.
>
> I have a possibly completely useless idea regarding "automatically"
> checking regressions. As far I understand the problem now is its not
> very feasible to do automated tests with what ever test suite we have
> (or will have after the improvements) and see the exported results for
> each patch, as at some step it involves human intervention (as in, was
> the export good).
>

I would disagree that we need user interaction in the test suite.  There
are already fully automated tests which (e.g., export to some backend
like html or tex and then programatically check for properties of the
exported results.

It is certainly likely that I am missing something, but I can't think of
a situation or a feature of Org-mode which could not be tested under the
current setup (mainly due to the fact that *every* user action in Emacs
reduces to a series of function calls which could be programatically
recreated).

>
> So maybe we can have a directory on the Worg website (not part of the
> Worg git repo) where every week or so the test suites will publish with
> what ever the org-mode head is at the time for all the supported
> formats. Then us "puny lisp illiterate" users can check up on it over
> the course of the week and report back to the list if there is a
> problem.
>
> Since this way people can look at the export formats they are
> interested in, none of the formats get treated like a step child
> either. Would that be feasible? Or did I completely misunderstand the
> problem at hand?

I'd think that a better way for contributing to the test suite in a
non-lisp manner would be to submit test cases, e.g. "this block of
Org-mode text should export to this but sometimes instead exports to
this", or "when I press this key sequence in this place in this org-mode
text I expect x to happen to the text".

We could even potentially leverage the existing Emacs macro system to
build a *record* method so that users could semi-automatically record
their actions allowing an interactive method of recording tests (or
submitting a re-creatable bug report).  Or at least recording enough
information so that someone with a little bit more elisp-fu could wrap
the recorded actions into a unit test.

Hope this is helpful -- Eric