[Heartlogic-dev] Re: methodology

[Joshua N Pritikin]

On Tue, 2005-04-05 at 00:11 -0500, William L. Jarrold wrote:
> On Tue, 29 Mar 2005, Joshua N Pritikin wrote:
> > Hrm, one email about rating comments is still pending but I think I'll
> > respond briefly here anyway.  Actually I have a very specific proposal
> > in mind.  I'll just spell it out.
> 
> Okay, so below you give me an example of I guess you might call it a 
> flowchart.  Yes, interesting idea.  I suppose we could do somethign like 
> that.  The wording of some of the questions needs fixing.
> 
> POINT A: Also this drifts away from Josh White's idea of having a live 
> model that you can poke and interact with.
> 
> POINT B: And I am still interested in doing a simple replication of my 
> dissertation just for the sake of being able to claim that conclusions 
> made with a WWW sample are equally valid to those made via a traditional 
> psychology research study human subjects sample.
> 
> In spite of POINT A and POINT B, I do not want to emphatically rain on 
> your parade.  If we get enough subjects we can do both your idea, my idea,
> Josh's idea and much much more.
> 
> Even if we do not do your idea but instead to somethign like a simple
> replication of my dissertaiton or a "live AI model" eventually we will 
> HAVE to collect data in the matter you propose.  Why?  Because your way
> of gathering data allows us to test our hypotheses in different ways.
> If you really want to solidly understand your hypothesis you must do it
> in different ways.
> 
> So, how about this:  A compromise.  I will help you develop items like 
> the below if you help me develop my own different types of items...We can 
> randomly assign subjects to your items and randomly assign them to mine.

Yah, fine.

> One more issue: you are proposing a radically different kind of study.  I 
> am not sure how to do the statistical analysis or other aspects of the 
> research design.  To tease apart these issues will take time.  It takes a 
> VERY long time to design a study before it can be implemented...So, I 
> suppose that I would very much like to do the kind of study you propose.

OK

> However, I would like to do the replication first because we need to get 
> our infrastructure up and running.  Once we do that, we can do all kinds 
> of way cool studies.

OK

> Yet one more issue:  If you look in the conclusion of my diss. regarding 
> future work and also if you look at an analysis of some of the items that 
> violated the hypothesis you will see some odd puzzles in the data.  The
> kind of flowchart based item generation that you have architected below
> would probably be an EXCELLENT way of why those particular items behaved
> contrary to the hypotheses.

Yes, I was trying to point that out.

> Still one more issue: What we really need is a large remote control boot 
> to kick me in the pants to send you the damned items for all the groups in 
> Study 3 of the dissertation.

As I run out of other ideas to knock around, that will probably help get
you focused.  ;-)

> > Toby wants some orange juice.
> > At bedtime daddy makes Toby some hot chocolate.
> >
> > 1. Does Toby believe that Hot Coco is a good substitute for OJ in this
> > situation?  If not, skip to #3.
> 
> The above is fairly clear and good.  Substitutability is an important 
> variable that we need to get a handle on.
> 
> But what are the specifics of the design of this item?  Is does the item 
> contain a scenario cue plus an appraisal or does the human research 
> participant generate their own appraisal (e.g. pick whether Toby feels 
> happy or sad).

Only this part:

  [cue here] Toby believes that Hot Coco is a good substitute for OJ.

Then ask for a Likert rating of believability like we do for everything
else.

The difference between this kind of question and your dissertation items
is that here we don't have a hypothesis about which valence of
believability is "correct".  It's merely a branching of the tree.  If
the statement is believable then we go down one branch.  If it's
unbelievable then we go down the other branch.

> I also think it would be interesting and important in a completely 
> separate study to get people to rate the importance of different goals.

This seems relevant to part of your other email which I quote here:

> >> I'm not sure what you men by empirically testing the overriding
> goal?
> >> Are you testing that the goal is one that is believable for kids to
> have?
> >
> > In a sense, yes.
> >
> > Level 'g' is concerned with classifying a participant's goal status
> as #
> > $Goal, #$AntiGoal, or #$NoGoal.  Unsurprisingly, all the kids seem
> to
> > have a #$Goal.  The parents, on the other hand, can be construed in
> more
> > diverse ways.
> 
> So at this level you are trying to get people to rate whether the
> supposed 
> goal is actually somethign kids in general want (#$Goal), do not want 
> (#$AntiGoal), or are agnostic about having or not having (#$NoGoal).
> Is 
> that what you are wanting to do there?

Yes, and I also look at the parent's goal (or lack of goal).

> To what extent do these judgements made in such a study predict 
> substitutability in items like your #1 above or substitutability as 
> measured by a study like my dissertation?  If they are reasonably 
> predictive, then that is good.  Theory of Affective Mind is more rational
> than it otherwise would be.  It will be easier to model and easier to
> gather data for it to build the model.  If not, we'll have to dig
> in and find out what in the world is going on.

Yes, and I expect the goal status to be predictive.  That's why I added
it.

> > 2. Toby feels happy/indifferent/sad when he gets Hot Coco from his dad.
> > (ask for Likert rating, as usual)
> 
> Unclear.  For each subject do we ask a multiple choice question as well as 
> a likert believability rating?

No, I wrote it like that because in a recent email you chastised me for
omitting manipulated affects when I was merely writing something in
short-hand.  I don't remember the details, but anyway, the items are
constructed as similar as possible to your dissertation.

>   Or do have the flowchart pick one of the 
> emotions and then ask for a likert believability rating?

Yes.

>   If the 
> flowchart picks one of the emotions then the independennt variable of 
> reversal is manifest in the research design.

Yes.

> > 3. Does Toby more interested in getting attention from his dad than what
> > is the specific kind of drink?  If not, skip to #5.
> 
> Your writing is unclear and not grammatical..I am not sure what you 
> are after.  Here is my guess as to how this would best be assessed in the 
> form of an experimental item:
> 
> Toby wants orange juice.
> At bedtime daddy makes Toby some hot coco.
> Toby feels happy.
> 
> Given the above, which of the following, A or B, seems more believable:
> 
> (A) Toby feels happy because he got attention from daddy.
> 
> (B) Toby feels happy because he got a drink - the exact type of drink does 
> not matter so much to him.
> 
> Well, I'd like to see your response to the above before I continue on with 
> the rest of these.  Some of my same comments apply.

Again, I am writing short-hand where I'd actually split it into two
separate items which are both statements.  So:

Item 1: [cue here] Toby wants attention from daddy.  (Ask Likert
believability)

Item 2: [cue here] Toby is thirsty.  (Ask Likert believability)

I don't see the advantage of multiple-choice format questions when we
can do all items with Likert believability.  I expect that you will
agree.

-- 
If you are an American then support http://fairtax.org
 (Permanently replace 50,000+ pages of tax law with about 200 pages.)

signature.asc
Description: This is a digitally signed message part