Re: [Heartlogic-dev] the plan

[William L. Jarrold]

On Wed, 13 Apr 2005, Joshua N Pritikin wrote:

> On Wed, 2005-04-13 at 00:34 -0500, William L. Jarrold wrote:
> > I'm still feeling very unclear about what this is all about.
> >
> > What is the structure of each item?  Is there a scenario cue
> > (i.e. an explicitly overriding goal plus an outcome)?  Is their
> > an emotion?
>
> Shessh, just look at level 'g'.  Imagine level 'g' with items which are
> contributed by participants.
>
> Let's discuss things in more detail after I add some more web pages.  I
> think this is a case of a picture being worth a thousand words (or a web
> page being worth a thousand emails).

Okay.  Add some more web pages -- e.g. 3-5.  When you do, let me know,
tell me to look at them.

As I remember at least some of those items are kind of hard to understand,
needed to be better written.

> > > However, it seems like a useful thing to start exploring and we
> > > can do it in a Hot-or-Not style format without any registration
> > > requirement.
> >
> > Okay fine.  Well, if you want to keep going on this, can I see like 3 to 5
> > proposed items...But if it is an informal, pilot kind of thing, I might
> > want to run, oh, 20 - 30 subjects on it.  Not hundreds.  Hundreds of
> > subjects are valuable things to be used on well thought out research
> > studies.
>
> Actually we won't even know how many subjects are rating the items
> because we haven't asked them for registration.

Huh?  You sent me something that was bzipped.  This thing had bunches
of responses in it.  One line per subject's rating.  Each subject had an
email address.  I could figure out how many subjects rated at least
one item by counting unique occurrences of emails in that bunch of
responses.  Right?  If so, then why do you say we won't even know
how many subjects are rating the items?

....Bill replies to more text below...Then he has a flash of insight...
Oh, when you say "Actually we won'at even know how many subjs are rating
the items bc we haven't asked for reg." what you mean is with what we are
about to do.  You were not referring to data already collected.

> > Maybe I am loosing the fun spirit of Josh White's turning challenge idea.
> > But we need a working AI model before we do that.  Making the AI model
> > live again is a top priority.  BTW, I want to try to get Peter Yeh's
> > matcher in on the action too.
>
> Sure, but those things could be part of real studies.
>
> I am talking about something which is not a real study which can be
> operated without registration, with a single click in Hot-or-Not style.
>
> Also, the level 'g' idea is just one proposal.  We can add more
> single-click, Hot-or-Not style things later on a rotation basis.

Well, I am operating under the assumption that subjects are pretty hard to 
come by.  So, if we are blessed with anyone's time for free we should make
sure that our methodology is going to make the best of their time.

Now, if people get as obessed with this as hot-or-not then I would love
to do the kind of thing you propose.

Just a thought. Not a requirement that we not under all circumstances
do a non-registration thing....I don't know whatever.  Show me the
g5 items.

> > > Don't we need a valid email address to allow participation in a study?

<...>

> > I remember you at least sorta already did this.
>
> It sounds like you are leaning towards requiring registration for real
> studies.  Let's just decide to do that.  It's not an onorous requirement
> and it substantially increases confidence with the integrity of our data
> collection.

Great.  But now that I see what you are talking about, I like the idea.
I think especially for return visitors...Return visitors have a "spoiled 
mind." since they are slightly more clued into the hypotheses.  I believe
most in conclusions made from data in which people have not had to
introspect about the contents of the own theory of mind mechanism.

But yes, higest priority on nice clean well controled studies.  If we can
get lotsa visitors or get return visitors we can def try this kind of 
study or if we just don't need a formal statistical study.  E.g. if we 
are piloting a study.  But again in pilot studies, or informal 
non-statistical studies we'd prefer not to waste virgin subjects.

Can you make the interface as for your email address.  If it knows your
email, then it executes a different set of studies.  Once we clean up
your g5 items then we can pilot them on like 3-5 people.  Then clean them
up more.  THat kinda thing.

> > > > Get clear on the terms/constructs, research questions and hypotheses.
> > > > Develop the experimental method.  Probably write it up to prove we know
> > > > what we are doing.  Plan the statistical analysis -- e.g. seek input from
> > > > UT Statistical Services as necessary.  Pilot he items on 1 - 20 friends
> > > > and tweak based on their feedback.  Run study.  Analyze results.  Write
> > > > up.  Publish.
> > >
> > > OK, and I want to do something like WLJ 2003 modified into a tree /
> > > flowchart.
> >
> > Okay, again, planning this will be a major undertaking.  As I have said
> > before, it would be *very* interesting to use the flowchart technique to
> > address some of the puzzling bits of the data.  I.e. the items that did
> > not live up to the hypothesis.
>
> Heh, then let's get to work!

Yes, if you are feeling idle and want to sink you teeth into something, 
check out those items that did not live up to hypothesis.  See espec 
dicussion section of chapter about study 3.  also conclusion chapter, 
maybe also disucssion section of chapter about study 2.  think about
how to use a flowchart to answer some of the explanations of those items 
that did not live up to hypothesis.  (i can't remember the issues well 
enough, so caveat emptor -- this may be a wild goose chase.  but it
may be cool.)

>
<...>

> Well, OK.  From an implementation point of view, this is a minor change
> so I'm going to postpone it since testing the web site is easier if the
> choice is available.

I am not sure I understand what you mean.  Here is my guess:  You are 
saying that while we are testing the website the user will still be able
to choose between different studies.  You are going to postpone 
implementation of the gizmo that chooses a study at random for the
user to participate in.  Once we make sure everythin is hunky dorey,
then you'll implement the random chooser well test that and once we've
made sure that that works too then we can tell the world to come to our 
website.

Right?

If so, then let me also integrate in this:

This is our "requirements document"

If they are a virgin user, then any study they get will be a "formal 
statistically well designed study."

If they are a virgin user and if there are several "formal statistically 
well designed studies" (aka a "formal study") then the study in which they 
participate wil be randomly chosen.

If they are a veteran user, then they *may* be offered an informal study.
If they are offered a formal study, then their data will be sequestered 
apart from the virgin data.

They *may* be offered a choice of studies.

We can later think about doing anonymous studies.  These are studies in 
which users are not registered (bc we don't collect their email address).
We might be more people to provide data anonymously.  THis is bc it takes
less time and activated less paranoia.

> > > 2. WLJ 2003 replication
> > > 3. WLJ 2003 modified into a flow-chart format
> > > 4. Adding cues and exploring the statistical results of #1
>
> OK, I expect/assume you are more or less in agreement with this
> plan.  ;-)
>
> > By the way, the other day, I tried out your ohl.nirmalvihar.info website
> > and it was just after midnight here and it worked.  I thought you had
> > regular power outages 12 to 3, no?  Maybe you've got those generators
> > hooked up to the school playground equipment!!?!!??!!
>
> There is a lot of variance in the exact timing of outages.

Ah.

>