[Heartlogic-dev] Re: methodology (was RE: OHL v2 alpha test)

[William L. Jarrold]

On Tue, 29 Mar 2005, Joshua N Pritikin wrote:

> On Tue, 2005-03-29 at 00:33 -0600, William L. Jarrold wrote:
> > On Sun, 27 Mar 2005, Joshua N Pritikin wrote:
> > > Sure, but have you thought carefully about what it would mean to have
> > > those three ratings (reason alone, emotion alone, reason and emotion
> > > together)?  Consider your scenario *H3.  The model generates three
> > > appraisals:
> > >
> > > ((:seq TypeIIAppraisal *Happy "At least she gets to get attention from
> > > her mom.")
> > > (:seq TypeIIAppraisal-B *Happy "At least she gets to have fruit.")
> > > (:seq TypeIIIAppraisal *Sad "She did not get what she wanted."))
> > >
> > > What if I don't believe in Type-II appraisals?  Given a Type-II
> > > appraisal, what is the point of asking about the emotion?  This is not a
> > > rhetorical question, try to answer.
> >
> > Perhaps our theory is incomplete.  Perhaps there are reasons that our
> > model that does not generate such that she feels happy.  If so, then
> > people would say the emotion happy was believable but the reason was not
> > believable.  And the emotion and reason together  is not believable.
> >
> > We would want to know about such cases because this would be a clue that
> > we needed other reasons that concluded happy.
> >
> > This is an example of why we would want to ask about the emotion alone,
> > the reason alone, and the two together.
>
> Hrm, one email about rating comments is still pending but I think I'll
> respond briefly here anyway.  Actually I have a very specific proposal
> in mind.  I'll just spell it out.

Okay, so below you give me an example of I guess you might call it a 
flowchart.  Yes, intersting idea.  I suppose we could do somethign like 
that.  The wording of some of the questions needs fixing.

POINT A: Also this drifts away from Josh White's idea of having a live 
model that you can poke and interact with.

POINT B: And I am still interested in doing a simple replication of my 
dissertation just for the sake of being able to claim that conclusions 
made with a WWW sample are equally valid to those made via a traditional 
psychology research study human subjects sample.

In spite of POINT A and POINT B, I do not want to emphatically rain on 
your parade.  If we get enough subjects we can do both your idea, my idea,
Josh's idea and much much more.

Even if we do not do your idea but instead to somethign like a simple
replication of my dissertaiton or a "live AI model" eventually we will 
HAVE to collect data in the matter you propose.  Why?  Because your way
of gathering data allows us to test our hypotheses in different ways.
If you really want to solidy understand your hypothesis you must do it
in different ways.

So, how about this:  A comprimise.  I will help you develop items like 
the below if you help me develop my own different types of items...We can 
randomly assign subjects to your items and randomly assign them to mine.

One more issue: you are proposing a radically different kind of study.  I 
am not sure how to do the statistical analysis or other aspects of the 
research design.  To tease apart these issues will take time.  It takes a 
VERY long time to design a study before it can be implemented...So, I 
suppose that I would very much like to do the kind of study you propose. 
However, I would like to do the replication first because we need to get 
our infrastructure up and running.  Once we do that, we can do all kinds 
of way cool studies.

Yet one more issue:  If you look in the conclusion of my diss. regarding 
future work and also if you look at an analysis of some of the items that 
violated the hypothesis you will see some odd puzzles in the data.  The
kind of flowchart based item generation thhat you have architected below
would probably be an EXCELLENT way of why those particular items behaved
contrary to the hypotheses.

Still one more issue: What we really need is a large remote control boot 
to kick me in the pants to send you the damned items for all the groups in 
Study 3 of the dissertation.

> Toby wants some orange juice.
> At bedtime daddy makes Toby some hot chocolate.
>
> 1. Does Toby believe that Hot Coco is a good substitute for OJ in this
> situation?  If not, skip to #3.

The above is fairly clear and good.  Substitutability is an important 
variable that we need to get a handle on.

But what are the specifics of the design of this item?  Is does the item 
contain a scenario cue plus an appraisal or does the human research 
participant generate their own appraisal (e.g. pick whether Toby feels 
happy or sad).

I also think it would be interesting and important in a completely 
separate study to get people to rate the importance of different goals.
To what extent do these judgements made in such a study predict 
substitutability in items like your #1 above or substitutability as 
measured by a study like my dissertation?  If they are reasonably 
predictive, then that is good.  Theory of Affective Mind is more rational
than it otherwise would be.  It will be easier to model and easier to
gather data for it to build the model.  If not, we'll have to dig
in and find out what in the world is going on.

> 2. Toby feels happy/indifferent/sad when he gets Hot Coco from his dad.
> (ask for Likert rating, as usual)

Unclear.  For each subject do we ask a multiple choice question as well as 
a likert believability rating?  Or do have the flowchart pick one of the 
emotions and then ask for a likert believability rating?  If the 
flowchart picks one of the emotions then the independennt variable of 
reversal is manifest in the research design.

> 3. Does Toby more interested in getting attention from his dad than what
> is the specific kind of drink?  If not, skip to #5.

Your writing is unclear and not grammatical..I am not sure what you 
are after.  Here is my guess as to how this would best be assessed in the 
form of an experimental item:

Toby wants orange juice.
At bedtime daddy makes Toby some hot coco.
Toby feels happy.

Given the above, which of the following, A or B, seems more believable:

(A) Toby feels happy because he got attention from daddy.

(B) Toby feels happy because he got a drink - the exact type of drink does 
not matter so much to him.

Well, I'd like to see your response to the above before I continue on with 
the rest of these.  Some of my same comments apply.

> 4. Toby feels happy/indifferent/sad when he gets attention from his dad.
> (ask for Likert rating, as usual)
>
> 5. Is there any other reason why Toby might feel happy/indifferent/sad
> about getting Hot Coco from his dad?
>
> 6. Is it believable that Toby did not get what he wanted?  If not, skip
> the next question.
>
> 7. Toby feels happy/indifferent/sad because he did not get what he
> wanted. (ask for Likert rating, as usual)
>
> That's how I propose to structure it.  I believe the structure of these
> questions address your concerns and as well as mine.