Re: [Bug-gnupedia] A Detailed Proposal - Mk I

[Bryce Harrington]

On Fri, 19 Jan 2001, Simon Cross wrote:
> Peer review:  The idea is that articles are not moderated (in the Slashdot
> / Kuro5hin sense) but rather that a reviewer reads an article and gives it
> his/her approval by digitally signing it.  If the reviewer does not
> approve of the article for some reason or feels the article needs
> corrections then he/she does not sign the article.  Authors will be
> responsible for having people review their articles.  When searching the
> encyclopedia a user can then specify that only articles reviewed by a
> particular set of reviewers (or teams of reviewers) should be searched.  
> Or the user could ask that articles reviewed by a certain set of reviewers
> be shown first.  Ideally the core of the encyclopedia should eventually
> all be reviewed by well-known review teams.

Actually I think this could be improved on...

Consider: I could be judged an "master" on spacecraft propulsion
systems, but only a "journeyman" for spacecraft communciation systems.
Let's say for some very strange reason, someone actually values my
reviewing skills and includes me in his search criteria, since he wants
only the best of the best (stuff he can base multi-million dollar
decisions on).  Now, you can be sure the propulsion articles I've signed
off on are good, but he's also going to get a bunch of inane articles on
communications that I thought were ok, but in truth are just so much
pseudo-science.  

A better approach would be to have each reviewer be rated on each
particular subject area.  Perhaps this could be done in an advogato type
manner.  

Then the reader would be able to specify very clearly and succintly,
that he only wishes to see articles that have been reviewed by "grandest
of masters" (and thereby filter out all of my drivel), and if by some
chance he is interested in spacecraft communications, and there have
been no other reviewers but me, he can drop his standards down to mere
"journeyman", get the articles I've reviewed, and know to take what he
reads with a grain of salt...

> Virtual identities:  Checking that an article has been signed by the GPG
> Private Key associated with a given GPG Public Key is easy.  But how do we
> associate a person with their GPG Public Key?  Checking real identities is
> time consuming and expensive. VeriSign have made lots of money doing
> it.  But we don't need to know who an author or reviewer is, we just need
> to know that the same person who reviewed aritcle A also reviewed articles
> B, C and D.  Perhaps we also need to know that the person who reviewed
> this article is an official member of this-or-that review team (for
> example, the GNUPedia review team).

I may be alone, but I would have to say that if the process required for
being (and proving to be) an editor is too difficult, and I'm not going
to get paid for my troubles, I ain't gonna bother.

A potentially simpler alternative might be to use the approach that
advogato uses, and peer-review the peer reviewers...?

> - Non-HTML content
> 
> Provision will have to be made for images, video clips, sound files and
> other media to be included in encyclopedia articles.  These also need to
> be signed by the authors and reviewers if possible.  Some standard method
> for doing this needs to be divised.  Get cracking. :) If you like the
> proposal so far. :)

Consider allowing submission of "packages"  (e.g., tarballs) containing
both text and images and other stuff.  I might not wish to sign a video
clip unless it also included a transcript, or might not wish to sign a
flow chart diagram that was not also accompanied by a description of
what all the squigly lines mean.

> - Hector's statement that peer review can wait until later:  I don't buy
> it.  We need peer review from the start.  From now.  From yesterday. :)

There are probably more important things than peer review to work out
first, but I would agree it is something that ought be figured out
soonish.  Honestly, you just plain AIN'T going to be able to get enough
qualified reviewers until you've got enough articles to make it seem
worth the bother (_especially_ if you impose severe proof-of-identity
requirements).  So if you require peer review now, you may either have
to dilute the value of your reviewers, or getting into a chicken-and-egg
position.  

Bryce