Budapest Open Access Initiative      

Budapest Open Access Initiative: BOAI Forum Archive

[BOAI] [Forum Home] [index] [options] [help]

boaiforum messages

[BOAI] Crowd-Sourced Peer Review: Substitute or Supplement?

From: Stevan Harnad <amsciforum AT gmail.com>
Date: Thu, 21 Aug 2014 15:18:31 -0400


Threading:      • This Message
             [BOAI] Re: Crowd-Sourced Peer Review: Substitute or Supplement? from dennis.hamilton AT acm.org

--001a11c003da1231410501289680
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Harnad, S. (2014) *Crowd-Sourced Peer Review: Substitute or supplement for
the current outdated system?
<http://blogs.lse.ac.uk/impactofsocialsciences/2014/08/21/crowd-sourced-pee=
r-review-substitute-or-supplement/>
**LSE
Impact Blog* 8/21

EXCERPT:

If, as rumoured
<http://www.wired.co.uk/news/archive/2014-08/13/google-science-would-we-nee=
d-it>,
google builds a platform for depositing unrefereed research papers for
=E2=80=9Cpeer-reviewing=E2=80=9D viacrowd-sourcing
<https://en.wikipedia.org/wiki/Crowdsourcing>, can this create a 
*substitut=
e
<https://www.google.ca/?gws_rd=3Dssl#q=3Dharnad+%22peer+review%22+substitut=
e+supplement>*
for
classical peer-review or will it merely *supplement
<https://www.google.com/webhp?tbm=3Dblg&gws_rd=3Dssl#q=3Dsite:openaccess.ep=
rints.org+%22peer+review%22+substitute+supplement&tbm=3Dblg>*
classical
peer review with crowd-sourcing?

... no one knows whether crowd-sourced peer-review, even if it could work,
would be scaleable or sustainable.

The key questions are hence:










*1. Would all (most? many?) authors be willing to post their unrefereed
papers publicly (and in place of submitting them to journals!)?2. Would all
(most? many?) of the posted papers attract referees? competent experts?3.
Who/what decides whether the refereeing is competent, and whether the
author has adequately complied? (Relying on a Wikipedia-style cadre of
2nd-order crowd-sourcers
<http://www0.cs.ucl.ac.uk/staff/L.Capra/publications/ubicrowd11.pdf> who
gain authority recursively in proportion to how much 1st-order
crowd-sourcing they have done =E2=80=94 rather than on the basis of experti=
se =E2=80=94
 sounds like a way to generate Wikipedia quality, but not peer-reviewed
quality=E2=80=A6)4. If any of this actually happens on any scale, will it b=
e
sustainable?5. Would this make the landscape (unrefereed preprints, referee
comments, revised postprints) as navigable and useful as classical peer
review, or not?*

My own prediction (based on nearly a quarter century of umpiring
<http://users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html> both 
classical
peer review and open peer commentary) is that crowdsourcing will provide an
excellent supplement to classical peer review but not a substitute for it.
Radical implementations will simply end up re-inventing classical peer
review, but on a much faster and more efficient PostGutenberg platform. We
will not realize this, however, until all of the peer-reviewed literature
has first been made open access. And for that it is not sufficient for
Google merely to provide a platform for authors to put their unrefereed
papers, because most authors don=E2=80=99t even put their refereed papers i=
n their
institutional repositories until it is mandated by their institutions and
funders.

http://openaccess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-R=
eview-Substitute-or-Supplement.html

--001a11c003da1231410501289680
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><span 
style=3D"font-weight:bold;color:rgb(51,51,51);f=
ont-family:verdana,arial,helvetica,sans-serif;font-size:13px">Harnad, 
S. (2=
014)=C2=A0</span><b><a 
href=3D"http://blogs.lse.ac.uk/impactofsocialscience=
s/2014/08/21/crowd-sourced-peer-review-substitute-or-supplement/">Crowd-Sou=
rced Peer Review: Substitute or supplement for the current outdated system?=
</a><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,=
sans-serif;font-size:13px">=C2=A0</span></b><em 
style=3D"font-weight:bold;c=
olor:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size=
:13px">LSE Impact Blog</em><span 
style=3D"font-weight:bold;color:rgb(51,51,=
51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">=C2=A08/=
21=C2=A0</span><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial=
,helvetica,sans-serif;font-size:13px"><br>
</span></div><div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,ar=
ial,helvetica,sans-serif;font-size:13px"><br></span></div><div><span style=
=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font=
-size:13px">EXCERPT:</span></div>
<div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica=
,sans-serif;font-size:13px"><br></span></div><div><span style=3D"color:rgb(=
51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">If=
,=C2=A0</span><a 
href=3D"http://www.wired.co.uk/news/archive/2014-08/13/goo=
gle-science-would-we-need-it" 
style=3D"color:rgb(0,51,102);font-family:verd=
ana,arial,helvetica,sans-serif;font-size:13px">as 
rumoured</a><span style=
=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font=
-size:13px">, google builds a platform for depositing unrefereed 
research p=
apers for =E2=80=9Cpeer-reviewing=E2=80=9D via</span><a 
href=3D"https://en.=
wikipedia.org/wiki/Crowdsourcing" 
style=3D"color:rgb(0,51,102);font-family:=
verdana,arial,helvetica,sans-serif;font-size:13px">crowd-sourcing</a><span =
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif=
;font-size:13px">, can this create a=C2=A0</span><em 
style=3D"color:rgb(51,=
51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px"><a hr=
ef=3D"https://www.google.ca/?gws_rd=3Dssl#q=3Dharnad+%22peer+review%22+subs=
titute+supplement" 
style=3D"color:rgb(0,51,102)">substitute</a></em><span s=
tyle=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;=
font-size:13px">=C2=A0for classical peer-review or will it 
merely=C2=A0</sp=
an><em 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,san=
s-serif;font-size:13px"><a 
href=3D"https://www.google.com/webhp?tbm=3Dblg&a=
mp;gws_rd=3Dssl#q=3Dsite:openaccess.eprints.org+%22peer+review%22+substitut=
e+supplement&amp;tbm=3Dblg" 
style=3D"color:rgb(0,51,102)">supplement</a></e=
m><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sa=
ns-serif;font-size:13px">=C2=A0classical peer review with 
crowd-sourcing?</=
span><br>
</div><div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,hel=
vetica,sans-serif;font-size:13px"><br></span></div><div><span style=3D"colo=
r:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13=
px">...=C2=A0</span><span 
style=3D"color:rgb(51,51,51);font-family:verdana,=
arial,helvetica,sans-serif;font-size:13px">no one knows whether 
crowd-sourc=
ed peer-review, even if it could work, would be scaleable or sustainable.</=
span></div>
<br 
style=3D"font-size:13px;color:rgb(51,51,51);font-family:verdana,arial,h=
elvetica,sans-serif"><span 
style=3D"font-size:13px;color:rgb(51,51,51);font=
-family:verdana,arial,helvetica,sans-serif">The key questions are 
hence:</s=
pan><blockquote 
style=3D"font-size:13px;color:rgb(51,51,51);font-family:ver=
dana,arial,helvetica,sans-serif">
<em>1. Would all (most? many?) authors be willing to post their 
unrefereed =
papers publicly (and in place of submitting them to 
journals!)?<br><br>2. W=
ould all (most? many?) of the posted papers attract referees? competent exp=
erts?<br>
<br>3. Who/what decides whether the refereeing is competent, and whether 
th=
e author has adequately complied? (Relying on a=C2=A0<a 
href=3D"http://www0=
.cs.ucl.ac.uk/staff/L.Capra/publications/ubicrowd11.pdf" 
style=3D"color:rgb=
(0,51,102)">Wikipedia-style cadre of 2nd-order 
crowd-sourcers</a>=C2=A0who =
gain authority recursively in proportion to how much 1st-order crowd-sourci=
ng they have done =E2=80=94 rather than on the basis of expertise =E2=80=94=
 =C2=A0sounds like a way to generate Wikipedia quality, but not peer-review=
ed quality=E2=80=A6)<br>
<br><br>4. If any of this actually happens on any scale, will it be 
sustain=
able?<br><br>5. Would this make the landscape (unrefereed 
preprints, refere=
e comments, revised postprints) as navigable and useful as classical peer r=
eview, or not?</em></blockquote>
<span 
style=3D"font-size:13px;color:rgb(51,51,51);font-family:verdana,arial=
,helvetica,sans-serif">My own prediction (based on nearly a quarter 
century=
 of=C2=A0</span><a 
href=3D"http://users.ecs.soton.ac.uk/harnad/Temp/bbs.val=
edict.html" 
style=3D"color:rgb(0,51,102);font-size:13px;font-family:verdana=
,arial,helvetica,sans-serif">umpiring</a><span 
style=3D"font-size:13px;colo=
r:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif">=C2=A0both =
classical peer review and open peer commentary) is that crowdsourcing will =
provide an excellent supplement to classical peer review but not a substitu=
te for it. Radical implementations will simply end up re-inventing classica=
l peer review, but on a much faster and more efficient PostGutenberg platfo=
rm. We will not realize this, however, until all of the peer-reviewed liter=
ature has first been made open access. And for that it is not sufficient fo=
r Google merely to provide a platform for authors to put their unrefereed p=
apers, because most authors don=E2=80=99t even put their refereed papers in=
 their institutional repositories until it is mandated by their institution=
s and funders.</span><br 
style=3D"font-size:13px;color:rgb(51,51,51);font-f=
amily:verdana,arial,helvetica,sans-serif">
<div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica=
,sans-serif;font-size:13px"><br></span></div><div><a href=3D"http://openacc=
ess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-Review-Substitu=
te-or-Supplement.html">http://openaccess.eprints.org/index.php?/archives/11=
21-Crowd-Sourced-Peer-Review-Substitute-or-Supplement.html</a></div>
</div>

--001a11c003da1231410501289680--

        
--      
To unsubscribe from the BOAI Forum, use the form on this page:
http://mailman.ecs.soton.ac.uk/mailman/listinfo/boai-forum

[BOAI] Re: Crowd-Sourced Peer Review: Substitute or Supplement?

From: "Dennis E. Hamilton" <dennis.hamilton AT acm.org>
Date: Sun, 24 Aug 2014 10:27:04 -0700


Threading: [BOAI] Crowd-Sourced Peer Review: Substitute or Supplement? from amsciforum AT gmail.com
      • This Message


As an independent scholar, I have anecdotal evidence in favor of Stevan’s 
prediction.  Two aspects stand out for me:
 
1.  In the evolving delivery of Massively Open Online Courses (MOOCs), some 
subjects do not admit well of machine grading, although multiple-choice quizzes 
are heavily used.  For meaty assignments, peer assessment procedures are used, 
since it is the only way for it to work among thousands of participants.  In 
some technical courses, peer assessment is central and quality of assessments 
is taken seriously by the students.  At the worst (and I am participating in 
one of those at the moment), peer assessments are largely ceremonial and the 
level of returned comments is superficial and banal, so the opportunity to 
learn and improve from the review and perspective of others is lost.  One can, 
of course, learn a great deal from making assessments of the work of other 
students, since it improves one’s own critical understanding and provides 
practice at affirmatively appraising the work of others.  The instilling of 
this spirit is uneven across MOOCs I have been in.  And, where taken seriously 
and guided appropriately, the peer assessment process is invaluable.
   
2.  Scholarly and scientific peer review are a different matter, and it has 
different drivers, including editorial limitations and the availability of 
qualified and interested reviewers.  (In the Coursera MOOCs, a student does not 
receive marks and appraisal of their own assignment without first providing 
blind peer assessments for at least five other students. This is valuable so 
long as the students decline to game the system by simply giving their peers 
high marks and no feedback. There are some who forget that review is supposed 
to be constructive and not ego-tripping, not unique to the MOOC case.)  

Recent experience with EasyChair suggests that on-the-whole reviewers take 
their duties seriously and provide excellent observations.  Here there are 
constraints on length of submissions, norms for the community, time available 
for review, and the fact that not all submissions, regardless of quality, can 
be selected.  In my case, I can still take value from the review process of a 
rejected submission and, if I choose, self-publish the work on one of the 
sites, such as arXive, available for that level of contribution.  It won’t 
have the imprimatur of inclusion in a conference proceeding or professional 
publication, yet I can place the work in public and it will have benefitted 
from the reviews obtained and from subsequent comments by those whose attention 
is drawn to the work, although that will be by informal means.
 
For me, none of this is a bad thing. It serves to make work available, there 
are quality drivers even if not up to peer-review standards of a given field, 
and sometimes it is the best way to have work preserved, available, and an 
invitation for further review and discussion.  I think, in the MOOC case where 
peer-assessment and the cultivation of Community Teaching Assistants (sort of 
like trustees) is well-nurtured, the long-term effect may be profound in the 
promotion of learning.  And the historically-revered edifices will remain at 
the pinnacle of all this.
 
 
-- Dennis E. Hamilton
     <mailto:dennis.hamilton AT acm.org> dennis.hamilton AT acm.org    
+1-206-779-9430
     <https://keybase.io/orcmid> https://keybase.io/orcmid  PGP F96E 89FF 
D456 628A
    X.509 certs used and requested for signed e-mail
 
 
 
 
 
From: boai-forum-bounces AT ecs.soton.ac.uk [mailto:boai-forum-bounces AT 
ecs.soton.ac.uk] On Behalf Of Stevan Harnad
Sent: Thursday, August 21, 2014 12:19
To: ASIS&T Special Interest Group on Metrics
Cc: LibLicense-L Discussion Forum; Lib Serials list
Subject: [BOAI] Crowd-Sourced Peer Review: Substitute or Supplement?
 
Harnad, S. (2014) Crowd-Sourced Peer Review: Substitute or supplement for the 
current outdated system? 
<http://blogs.lse.ac.uk/impactofsocialsciences/2014/08/21/crowd-sourced-peer-review-substitute-or-supplement/>  LSE Impact Blog 8/21 
 
[ … ]
My own prediction (based on nearly a quarter century of  
<http://users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html> umpiring 
both classical peer review and open peer commentary) is that crowdsourcing will 
provide an excellent supplement to classical peer review but not a substitute 
for it. Radical implementations will simply end up re-inventing classical peer 
review, but on a much faster and more efficient PostGutenberg platform. We will 
not realize this, however, until all of the peer-reviewed literature has first 
been made open access. And for that it is not sufficient for Google merely to 
provide a platform for authors to put their unrefereed papers, because most 
authors don’t even put their refereed papers in their institutional 
repositories until it is mandated by their institutions and funders.
 
http://openaccess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-Review-Substitute-or-Supplement.html

ATTACHMENT: message.html!

-- To unsubscribe from the BOAI Forum, use the form on this page: http://mailman.ecs.soton.ac.uk/mailman/listinfo/boai-forum

[BOAI] [Forum Home] [index] [options] [help]

 E-mail:  openaccess@soros.org .