[Heartlogic-dev] Re: design (was Re: statistics)

[William L. Jarrold]

On Tue, 28 Oct 2003, Joshua N Pritikin wrote:

> On Tue, Oct 28, 2003 at 01:17:25AM -0600, William L. Jarrold wrote:
> > On Mon, 27 Oct 2003, Joshua N Pritikin wrote:
> > > Are you confident that you can wave the magic wand and
> > > do robust statistical inference of the highest scientific
> > > standards?
> >
> > Not confident but with a little elbow grease we can work it
> > out....If we want to publish in an AI journal, they won't care.
> > A psych journal will care.  If I re-read my diss (and hopefully
> > even if I don't) it will come back to me...BUT, we can get
> > good important results by departing from statistical stricture.
> > In fact, it is very very likely we will have to.
>
> Erm, well, OK.
>
> I'll follow your lead.

Right.....One way to make sure it is statistically rigorous is to
do somethign like I did for my dissertation. That would mean
we'd want to make sure that each subject saw the same number of items.
That the order in which they view the items is controlled.  So, they
should not be able to go back....(or if they did go back we might want
to know...I *told* my subjects please don't go back on the instructions.
But they could have disobeyed these instructions and I would not know) And
we'd need to be careful about which items we give to which subjects.

A strict replication of the diss study via the www may be publishable
in the methodology area as a means of comparing the effect of www vs
paper and pencil survey techniques.

>
> > But, one of the nearest term mods I can see us wanting is the following:
> > I am also interested in just having people write their own appraisals
> > of situations.  E.g. we might have them select from n emotions and add in
> > their own brief text explanation.
>
> OK
>
> > It would also be cool to just have users rate stuff.  E.g. "How
> > interesting[common sensical, logical, etc] is the following
> > assertion."...I'd like to compare ratings of interestingness (etc)
> > as a funciton of a) was the assertion generated via Cyc or Thought
> > Treasure or Open Mind....But this drifts away from open heart.
>
> "the assertion generated" -- you mean the appraisal or something
> more free-form?

Something more free form.  Remember my paper on Validation of
Intelligence?  Remember the bit about rumination?  I would love to
compare ratings as a function of A) Cyc vs ThoughtTreasure B)
ground fact in the kb vs deduced during rumination.

>
> Is it unreasonable to design Open Heart Logic (OHL) as a framework
> for testing cognitive appraisal theories?  In other words, can
> I insist on (1) scenerio cue (2) appraisal (3) believability
> where any of these three items may be from a human or computer
> generated?

Seems reasonable to me.  I'd say lets go for that as our first
step.  We'll keep in mind other cool ideas like the interestingness
rating stuff as back burner ways of keeping our code generalizable.

To be sure, I've always thought of believability as somethign that human
subjects type in.  It is a likert rating and an optional bit of commentary
text.  Now, as per your OHL framework suggestion, I suppose that we might
even want to have computer generated believability ratings someday.

>
> > > Nov 25.  Let's see what I can do.
> >
> > No guarantee that is what the date is.
>
> Ultimately we want to gather input from the whole WWW.  I'm not
> terribly worried about U of Texas.  Even so, it would be cool
> to get something working soon.

Agreed.  The nice thing about U of Texas is it buys credibility in
academic circles.  It does this for good reason.  If you have a
homogeneous pool of respondants, your variance is more restricted.
If your variance is more restricted, you are less likely to get
weird chancy things happening during your study.  You are also more
likely to get statistical sensitivity bc your noise level is low.
The minus is because your sample is upper middle class white kids
who say "y'all" then your ability to generalize to the entire population
is somewhat limited (depending on what your hypothesis and the rest
of the literature says).

The internet, by contrast, is a freewheeling domain where people
respond at all hours from all walks of life.  Some of them even respond
"at 3am while eating cheetos in their underwear."  Yee gads!

Bill

>
> --
> A new cognitive theory of emotion, http://savannah.nongnu.org/projects/aleader
>