|
|
|||||||||
|
|
|
|
|
|||||
|
Budapest Open Access Initiative: BOAI Forum Archive [BOAI] [Forum Home] [index] [options] [help]boaiforum messages[BOAI] Crowd-Sourced Peer Review: Substitute or Supplement?From: Stevan Harnad <amsciforum AT gmail.com>
--001a11c003da1231410501289680 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Harnad, S. (2014) *Crowd-Sourced Peer Review: Substitute or supplement for the current outdated system? <http://blogs.lse.ac.uk/impactofsocialsciences/2014/08/21/crowd-sourced-pee= r-review-substitute-or-supplement/> **LSE Impact Blog* 8/21 EXCERPT: If, as rumoured <http://www.wired.co.uk/news/archive/2014-08/13/google-science-would-we-nee= d-it>, google builds a platform for depositing unrefereed research papers for =E2=80=9Cpeer-reviewing=E2=80=9D viacrowd-sourcing <https://en.wikipedia.org/wiki/Crowdsourcing>, can this create a ↵ *substitut= e <https://www.google.ca/?gws_rd=3Dssl#q=3Dharnad+%22peer+review%22+substitut= e+supplement>* for classical peer-review or will it merely *supplement <https://www.google.com/webhp?tbm=3Dblg&gws_rd=3Dssl#q=3Dsite:openaccess.ep= rints.org+%22peer+review%22+substitute+supplement&tbm=3Dblg>* classical peer review with crowd-sourcing? ... no one knows whether crowd-sourced peer-review, even if it could work, would be scaleable or sustainable. The key questions are hence: *1. Would all (most? many?) authors be willing to post their unrefereed papers publicly (and in place of submitting them to journals!)?2. Would all (most? many?) of the posted papers attract referees? competent experts?3. Who/what decides whether the refereeing is competent, and whether the author has adequately complied? (Relying on a Wikipedia-style cadre of 2nd-order crowd-sourcers <http://www0.cs.ucl.ac.uk/staff/L.Capra/publications/ubicrowd11.pdf> who gain authority recursively in proportion to how much 1st-order crowd-sourcing they have done =E2=80=94 rather than on the basis of experti= se =E2=80=94 sounds like a way to generate Wikipedia quality, but not peer-reviewed quality=E2=80=A6)4. If any of this actually happens on any scale, will it b= e sustainable?5. Would this make the landscape (unrefereed preprints, referee comments, revised postprints) as navigable and useful as classical peer review, or not?* My own prediction (based on nearly a quarter century of umpiring <http://users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html> both ↵ classical peer review and open peer commentary) is that crowdsourcing will provide an excellent supplement to classical peer review but not a substitute for it. Radical implementations will simply end up re-inventing classical peer review, but on a much faster and more efficient PostGutenberg platform. We will not realize this, however, until all of the peer-reviewed literature has first been made open access. And for that it is not sufficient for Google merely to provide a platform for authors to put their unrefereed papers, because most authors don=E2=80=99t even put their refereed papers i= n their institutional repositories until it is mandated by their institutions and funders. http://openaccess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-R= eview-Substitute-or-Supplement.html --001a11c003da1231410501289680 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div><span ↵ style=3D"font-weight:bold;color:rgb(51,51,51);f= ont-family:verdana,arial,helvetica,sans-serif;font-size:13px">Harnad, ↵ S. (2= 014)=C2=A0</span><b><a ↵ href=3D"http://blogs.lse.ac.uk/impactofsocialscience= s/2014/08/21/crowd-sourced-peer-review-substitute-or-supplement/">Crowd-Sou= rced Peer Review: Substitute or supplement for the current outdated system?= </a><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,= sans-serif;font-size:13px">=C2=A0</span></b><em ↵ style=3D"font-weight:bold;c= olor:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size= :13px">LSE Impact Blog</em><span ↵ style=3D"font-weight:bold;color:rgb(51,51,= 51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">=C2=A08/= 21=C2=A0</span><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,arial= ,helvetica,sans-serif;font-size:13px"><br> </span></div><div><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,ar= ial,helvetica,sans-serif;font-size:13px"><br></span></div><div><span style= =3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font= -size:13px">EXCERPT:</span></div> <div><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica= ,sans-serif;font-size:13px"><br></span></div><div><span style=3D"color:rgb(= 51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">If= ,=C2=A0</span><a ↵ href=3D"http://www.wired.co.uk/news/archive/2014-08/13/goo= gle-science-would-we-need-it" ↵ style=3D"color:rgb(0,51,102);font-family:verd= ana,arial,helvetica,sans-serif;font-size:13px">as ↵ rumoured</a><span style= =3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font= -size:13px">, google builds a platform for depositing unrefereed ↵ research p= apers for =E2=80=9Cpeer-reviewing=E2=80=9D via</span><a ↵ href=3D"https://en.= wikipedia.org/wiki/Crowdsourcing" ↵ style=3D"color:rgb(0,51,102);font-family:= verdana,arial,helvetica,sans-serif;font-size:13px">crowd-sourcing</a><span = style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif= ;font-size:13px">, can this create a=C2=A0</span><em ↵ style=3D"color:rgb(51,= 51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px"><a hr= ef=3D"https://www.google.ca/?gws_rd=3Dssl#q=3Dharnad+%22peer+review%22+subs= titute+supplement" ↵ style=3D"color:rgb(0,51,102)">substitute</a></em><span s= tyle=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;= font-size:13px">=C2=A0for classical peer-review or will it ↵ merely=C2=A0</sp= an><em ↵ style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,san= s-serif;font-size:13px"><a ↵ href=3D"https://www.google.com/webhp?tbm=3Dblg&a= mp;gws_rd=3Dssl#q=3Dsite:openaccess.eprints.org+%22peer+review%22+substitut= e+supplement&tbm=3Dblg" ↵ style=3D"color:rgb(0,51,102)">supplement</a></e= m><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica,sa= ns-serif;font-size:13px">=C2=A0classical peer review with ↵ crowd-sourcing?</= span><br> </div><div><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,arial,hel= vetica,sans-serif;font-size:13px"><br></span></div><div><span style=3D"colo= r:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13= px">...=C2=A0</span><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,= arial,helvetica,sans-serif;font-size:13px">no one knows whether ↵ crowd-sourc= ed peer-review, even if it could work, would be scaleable or sustainable.</= span></div> <br ↵ style=3D"font-size:13px;color:rgb(51,51,51);font-family:verdana,arial,h= elvetica,sans-serif"><span ↵ style=3D"font-size:13px;color:rgb(51,51,51);font= -family:verdana,arial,helvetica,sans-serif">The key questions are ↵ hence:</s= pan><blockquote ↵ style=3D"font-size:13px;color:rgb(51,51,51);font-family:ver= dana,arial,helvetica,sans-serif"> <em>1. Would all (most? many?) authors be willing to post their ↵ unrefereed = papers publicly (and in place of submitting them to ↵ journals!)?<br><br>2. W= ould all (most? many?) of the posted papers attract referees? competent exp= erts?<br> <br>3. Who/what decides whether the refereeing is competent, and whether ↵ th= e author has adequately complied? (Relying on a=C2=A0<a ↵ href=3D"http://www0= .cs.ucl.ac.uk/staff/L.Capra/publications/ubicrowd11.pdf" ↵ style=3D"color:rgb= (0,51,102)">Wikipedia-style cadre of 2nd-order ↵ crowd-sourcers</a>=C2=A0who = gain authority recursively in proportion to how much 1st-order crowd-sourci= ng they have done =E2=80=94 rather than on the basis of expertise =E2=80=94= =C2=A0sounds like a way to generate Wikipedia quality, but not peer-review= ed quality=E2=80=A6)<br> <br><br>4. If any of this actually happens on any scale, will it be ↵ sustain= able?<br><br>5. Would this make the landscape (unrefereed ↵ preprints, refere= e comments, revised postprints) as navigable and useful as classical peer r= eview, or not?</em></blockquote> <span ↵ style=3D"font-size:13px;color:rgb(51,51,51);font-family:verdana,arial= ,helvetica,sans-serif">My own prediction (based on nearly a quarter ↵ century= of=C2=A0</span><a ↵ href=3D"http://users.ecs.soton.ac.uk/harnad/Temp/bbs.val= edict.html" ↵ style=3D"color:rgb(0,51,102);font-size:13px;font-family:verdana= ,arial,helvetica,sans-serif">umpiring</a><span ↵ style=3D"font-size:13px;colo= r:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif">=C2=A0both = classical peer review and open peer commentary) is that crowdsourcing will = provide an excellent supplement to classical peer review but not a substitu= te for it. Radical implementations will simply end up re-inventing classica= l peer review, but on a much faster and more efficient PostGutenberg platfo= rm. We will not realize this, however, until all of the peer-reviewed liter= ature has first been made open access. And for that it is not sufficient fo= r Google merely to provide a platform for authors to put their unrefereed p= apers, because most authors don=E2=80=99t even put their refereed papers in= their institutional repositories until it is mandated by their institution= s and funders.</span><br ↵ style=3D"font-size:13px;color:rgb(51,51,51);font-f= amily:verdana,arial,helvetica,sans-serif"> <div><span ↵ style=3D"color:rgb(51,51,51);font-family:verdana,arial,helvetica= ,sans-serif;font-size:13px"><br></span></div><div><a href=3D"http://openacc= ess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-Review-Substitu= te-or-Supplement.html">http://openaccess.eprints.org/index.php?/archives/11= 21-Crowd-Sourced-Peer-Review-Substitute-or-Supplement.html</a></div> </div> --001a11c003da1231410501289680-- -- To unsubscribe from the BOAI Forum, use the form on this page: http://mailman.ecs.soton.ac.uk/mailman/listinfo/boai-forum [BOAI] Re: Crowd-Sourced Peer Review: Substitute or Supplement?From: "Dennis E. Hamilton" <dennis.hamilton AT acm.org>
As an independent scholar, I have anecdotal evidence in favor of Stevan’s ↵
prediction. Two aspects stand out for me:
1. In the evolving delivery of Massively Open Online Courses (MOOCs), some ↵
subjects do not admit well of machine grading, although multiple-choice quizzes ↵
are heavily used. For meaty assignments, peer assessment procedures are used, ↵
since it is the only way for it to work among thousands of participants. In ↵
some technical courses, peer assessment is central and quality of assessments ↵
is taken seriously by the students. At the worst (and I am participating in ↵
one of those at the moment), peer assessments are largely ceremonial and the ↵
level of returned comments is superficial and banal, so the opportunity to ↵
learn and improve from the review and perspective of others is lost. One can, ↵
of course, learn a great deal from making assessments of the work of other ↵
students, since it improves one’s own critical understanding and provides ↵
practice at affirmatively appraising the work of others. The instilling of ↵
this spirit is uneven across MOOCs I have been in. And, where taken seriously ↵
and guided appropriately, the peer assessment process is invaluable.
2. Scholarly and scientific peer review are a different matter, and it has ↵
different drivers, including editorial limitations and the availability of ↵
qualified and interested reviewers. (In the Coursera MOOCs, a student does not ↵
receive marks and appraisal of their own assignment without first providing ↵
blind peer assessments for at least five other students. This is valuable so ↵
long as the students decline to game the system by simply giving their peers ↵
high marks and no feedback. There are some who forget that review is supposed ↵
to be constructive and not ego-tripping, not unique to the MOOC case.)
Recent experience with EasyChair suggests that on-the-whole reviewers take ↵
their duties seriously and provide excellent observations. Here there are ↵
constraints on length of submissions, norms for the community, time available ↵
for review, and the fact that not all submissions, regardless of quality, can ↵
be selected. In my case, I can still take value from the review process of a ↵
rejected submission and, if I choose, self-publish the work on one of the ↵
sites, such as arXive, available for that level of contribution. It won’t ↵
have the imprimatur of inclusion in a conference proceeding or professional ↵
publication, yet I can place the work in public and it will have benefitted ↵
from the reviews obtained and from subsequent comments by those whose attention ↵
is drawn to the work, although that will be by informal means.
For me, none of this is a bad thing. It serves to make work available, there ↵
are quality drivers even if not up to peer-review standards of a given field, ↵
and sometimes it is the best way to have work preserved, available, and an ↵
invitation for further review and discussion. I think, in the MOOC case where ↵
peer-assessment and the cultivation of Community Teaching Assistants (sort of ↵
like trustees) is well-nurtured, the long-term effect may be profound in the ↵
promotion of learning. And the historically-revered edifices will remain at ↵
the pinnacle of all this.
-- Dennis E. Hamilton
<mailto:dennis.hamilton AT acm.org> dennis.hamilton AT acm.org ↵
+1-206-779-9430
<https://keybase.io/orcmid> https://keybase.io/orcmid PGP F96E 89FF ↵
D456 628A
X.509 certs used and requested for signed e-mail
From: boai-forum-bounces AT ecs.soton.ac.uk [mailto:boai-forum-bounces AT ↵
ecs.soton.ac.uk] On Behalf Of Stevan Harnad
Sent: Thursday, August 21, 2014 12:19
To: ASIS&T Special Interest Group on Metrics
Cc: LibLicense-L Discussion Forum; Lib Serials list
Subject: [BOAI] Crowd-Sourced Peer Review: Substitute or Supplement?
Harnad, S. (2014) Crowd-Sourced Peer Review: Substitute or supplement for the ↵
current outdated system? ↵
<http://blogs.lse.ac.uk/impactofsocialsciences/2014/08/21/crowd-sourced-peer-review-substitute-or-supplement/> LSE Impact Blog 8/21
[ … ]
My own prediction (based on nearly a quarter century of ↵
<http://users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html> umpiring ↵
both classical peer review and open peer commentary) is that crowdsourcing will ↵
provide an excellent supplement to classical peer review but not a substitute ↵
for it. Radical implementations will simply end up re-inventing classical peer ↵
review, but on a much faster and more efficient PostGutenberg platform. We will ↵
not realize this, however, until all of the peer-reviewed literature has first ↵
been made open access. And for that it is not sufficient for Google merely to ↵
provide a platform for authors to put their unrefereed papers, because most ↵
authors don’t even put their refereed papers in their institutional ↵
repositories until it is mandated by their institutions and funders.
http://openaccess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-Review-Substitute-or-Supplement.html
[BOAI] [Forum Home] [index] [options] [help]
|
E-mail: