Budapest Open Access Initiative      

Budapest Open Access Initiative: BOAI Forum Archive

[BOAI] [Forum Home] [index] [prev] [next] [options] [help]

boaiforum messages

[BOAI] Crowd-Sourced Peer Review: Substitute or Supplement?

From: Stevan Harnad <amsciforum AT gmail.com>
Date: Thu, 21 Aug 2014 15:18:31 -0400


Threading:      • This Message
             [BOAI] Re: Crowd-Sourced Peer Review: Substitute or Supplement? from dennis.hamilton AT acm.org

--001a11c003da1231410501289680
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Harnad, S. (2014) *Crowd-Sourced Peer Review: Substitute or supplement for
the current outdated system?
<http://blogs.lse.ac.uk/impactofsocialsciences/2014/08/21/crowd-sourced-pee=
r-review-substitute-or-supplement/>
**LSE
Impact Blog* 8/21

EXCERPT:

If, as rumoured
<http://www.wired.co.uk/news/archive/2014-08/13/google-science-would-we-nee=
d-it>,
google builds a platform for depositing unrefereed research papers for
=E2=80=9Cpeer-reviewing=E2=80=9D viacrowd-sourcing
<https://en.wikipedia.org/wiki/Crowdsourcing>, can this create a 
*substitut=
e
<https://www.google.ca/?gws_rd=3Dssl#q=3Dharnad+%22peer+review%22+substitut=
e+supplement>*
for
classical peer-review or will it merely *supplement
<https://www.google.com/webhp?tbm=3Dblg&gws_rd=3Dssl#q=3Dsite:openaccess.ep=
rints.org+%22peer+review%22+substitute+supplement&tbm=3Dblg>*
classical
peer review with crowd-sourcing?

... no one knows whether crowd-sourced peer-review, even if it could work,
would be scaleable or sustainable.

The key questions are hence:










*1. Would all (most? many?) authors be willing to post their unrefereed
papers publicly (and in place of submitting them to journals!)?2. Would all
(most? many?) of the posted papers attract referees? competent experts?3.
Who/what decides whether the refereeing is competent, and whether the
author has adequately complied? (Relying on a Wikipedia-style cadre of
2nd-order crowd-sourcers
<http://www0.cs.ucl.ac.uk/staff/L.Capra/publications/ubicrowd11.pdf> who
gain authority recursively in proportion to how much 1st-order
crowd-sourcing they have done =E2=80=94 rather than on the basis of experti=
se =E2=80=94
 sounds like a way to generate Wikipedia quality, but not peer-reviewed
quality=E2=80=A6)4. If any of this actually happens on any scale, will it b=
e
sustainable?5. Would this make the landscape (unrefereed preprints, referee
comments, revised postprints) as navigable and useful as classical peer
review, or not?*

My own prediction (based on nearly a quarter century of umpiring
<http://users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html> both 
classical
peer review and open peer commentary) is that crowdsourcing will provide an
excellent supplement to classical peer review but not a substitute for it.
Radical implementations will simply end up re-inventing classical peer
review, but on a much faster and more efficient PostGutenberg platform. We
will not realize this, however, until all of the peer-reviewed literature
has first been made open access. And for that it is not sufficient for
Google merely to provide a platform for authors to put their unrefereed
papers, because most authors don=E2=80=99t even put their refereed papers i=
n their
institutional repositories until it is mandated by their institutions and
funders.

http://openaccess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-R=
eview-Substitute-or-Supplement.html

--001a11c003da1231410501289680
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><span 
style=3D"font-weight:bold;color:rgb(51,51,51);f=
ont-family:verdana,arial,helvetica,sans-serif;font-size:13px">Harnad, 
S. (2=
014)=C2=A0</span><b><a 
href=3D"http://blogs.lse.ac.uk/impactofsocialscience=
s/2014/08/21/crowd-sourced-peer-review-substitute-or-supplement/">Crowd-Sou=
rced Peer Review: Substitute or supplement for the current outdated system?=
</a><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,=
sans-serif;font-size:13px">=C2=A0</span></b><em 
style=3D"font-weight:bold;c=
olor:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size=
:13px">LSE Impact Blog</em><span 
style=3D"font-weight:bold;color:rgb(51,51,=
51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">=C2=A08/=
21=C2=A0</span><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial=
,helvetica,sans-serif;font-size:13px"><br>
</span></div><div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,ar=
ial,helvetica,sans-serif;font-size:13px"><br></span></div><div><span style=
=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font=
-size:13px">EXCERPT:</span></div>
<div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica=
,sans-serif;font-size:13px"><br></span></div><div><span style=3D"color:rgb(=
51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">If=
,=C2=A0</span><a 
href=3D"http://www.wired.co.uk/news/archive/2014-08/13/goo=
gle-science-would-we-need-it" 
style=3D"color:rgb(0,51,102);font-family:verd=
ana,arial,helvetica,sans-serif;font-size:13px">as 
rumoured</a><span style=
=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font=
-size:13px">, google builds a platform for depositing unrefereed 
research p=
apers for =E2=80=9Cpeer-reviewing=E2=80=9D via</span><a 
href=3D"https://en.=
wikipedia.org/wiki/Crowdsourcing" 
style=3D"color:rgb(0,51,102);font-family:=
verdana,arial,helvetica,sans-serif;font-size:13px">crowd-sourcing</a><span =
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif=
;font-size:13px">, can this create a=C2=A0</span><em 
style=3D"color:rgb(51,=
51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px"><a hr=
ef=3D"https://www.google.ca/?gws_rd=3Dssl#q=3Dharnad+%22peer+review%22+subs=
titute+supplement" 
style=3D"color:rgb(0,51,102)">substitute</a></em><span s=
tyle=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;=
font-size:13px">=C2=A0for classical peer-review or will it 
merely=C2=A0</sp=
an><em 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,san=
s-serif;font-size:13px"><a 
href=3D"https://www.google.com/webhp?tbm=3Dblg&a=
mp;gws_rd=3Dssl#q=3Dsite:openaccess.eprints.org+%22peer+review%22+substitut=
e+supplement&amp;tbm=3Dblg" 
style=3D"color:rgb(0,51,102)">supplement</a></e=
m><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sa=
ns-serif;font-size:13px">=C2=A0classical peer review with 
crowd-sourcing?</=
span><br>
</div><div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,hel=
vetica,sans-serif;font-size:13px"><br></span></div><div><span style=3D"colo=
r:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13=
px">...=C2=A0</span><span 
style=3D"color:rgb(51,51,51);font-family:verdana,=
arial,helvetica,sans-serif;font-size:13px">no one knows whether 
crowd-sourc=
ed peer-review, even if it could work, would be scaleable or sustainable.</=
span></div>
<br 
style=3D"font-size:13px;color:rgb(51,51,51);font-family:verdana,arial,h=
elvetica,sans-serif"><span 
style=3D"font-size:13px;color:rgb(51,51,51);font=
-family:verdana,arial,helvetica,sans-serif">The key questions are 
hence:</s=
pan><blockquote 
style=3D"font-size:13px;color:rgb(51,51,51);font-family:ver=
dana,arial,helvetica,sans-serif">
<em>1. Would all (most? many?) authors be willing to post their 
unrefereed =
papers publicly (and in place of submitting them to 
journals!)?<br><br>2. W=
ould all (most? many?) of the posted papers attract referees? competent exp=
erts?<br>
<br>3. Who/what decides whether the refereeing is competent, and whether 
th=
e author has adequately complied? (Relying on a=C2=A0<a 
href=3D"http://www0=
.cs.ucl.ac.uk/staff/L.Capra/publications/ubicrowd11.pdf" 
style=3D"color:rgb=
(0,51,102)">Wikipedia-style cadre of 2nd-order 
crowd-sourcers</a>=C2=A0who =
gain authority recursively in proportion to how much 1st-order crowd-sourci=
ng they have done =E2=80=94 rather than on the basis of expertise =E2=80=94=
 =C2=A0sounds like a way to generate Wikipedia quality, but not peer-review=
ed quality=E2=80=A6)<br>
<br><br>4. If any of this actually happens on any scale, will it be 
sustain=
able?<br><br>5. Would this make the landscape (unrefereed 
preprints, refere=
e comments, revised postprints) as navigable and useful as classical peer r=
eview, or not?</em></blockquote>
<span 
style=3D"font-size:13px;color:rgb(51,51,51);font-family:verdana,arial=
,helvetica,sans-serif">My own prediction (based on nearly a quarter 
century=
 of=C2=A0</span><a 
href=3D"http://users.ecs.soton.ac.uk/harnad/Temp/bbs.val=
edict.html" 
style=3D"color:rgb(0,51,102);font-size:13px;font-family:verdana=
,arial,helvetica,sans-serif">umpiring</a><span 
style=3D"font-size:13px;colo=
r:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif">=C2=A0both =
classical peer review and open peer commentary) is that crowdsourcing will =
provide an excellent supplement to classical peer review but not a substitu=
te for it. Radical implementations will simply end up re-inventing classica=
l peer review, but on a much faster and more efficient PostGutenberg platfo=
rm. We will not realize this, however, until all of the peer-reviewed liter=
ature has first been made open access. And for that it is not sufficient fo=
r Google merely to provide a platform for authors to put their unrefereed p=
apers, because most authors don=E2=80=99t even put their refereed papers in=
 their institutional repositories until it is mandated by their institution=
s and funders.</span><br 
style=3D"font-size:13px;color:rgb(51,51,51);font-f=
amily:verdana,arial,helvetica,sans-serif">
<div><span 
style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica=
,sans-serif;font-size:13px"><br></span></div><div><a href=3D"http://openacc=
ess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-Review-Substitu=
te-or-Supplement.html">http://openaccess.eprints.org/index.php?/archives/11=
21-Crowd-Sourced-Peer-Review-Substitute-or-Supplement.html</a></div>
</div>

--001a11c003da1231410501289680--

        
--      
To unsubscribe from the BOAI Forum, use the form on this page:
http://mailman.ecs.soton.ac.uk/mailman/listinfo/boai-forum

[BOAI] [Forum Home] [index] [prev] [next] [options] [help]

 E-mail:  openaccess@soros.org .