Budapest Open Access Initiative      

Budapest Open Access Initiative: BOAI Forum Archive

[BOAI] [Forum Home] [index] [prev] [next] [options] [help]

boaiforum messages

[BOAI] Re: OA policies and their "weight"

From: Stevan Harnad <harnad AT ecs.soton.ac.uk>
Date: Mon, 19 Jul 2010 09:56:09 -0400


Threading: [BOAI] OA policies and their "weight" from rmelero AT iata.csic.es
      • This Message

Dear Reme, if I may also make an intervention in your exchange with Steve 
Hitchcock about the MELIBEA OA policy evaluator:
http://www.accesoabierto.net/politicas/ 

The MELIBEA service is extremely timely and promising, and could be potentially 
useful and even influential in shaping OA mandates -- but that makes it all the 
more important to get it right, rather than releasing MELIBEA prematurely, when 
it still risks increasing confusion rather than providing clarity and 
direction.

You are right to point out that -- unlike the CSIC's University Ranking and the 
Repository Ranking -- the policy evaluator is not really a ranking. But you 
have set up the composite algorithm and the graphics to make it a ranking just 
the same. 

You are also point out, correctly, that the policy criteria for institutions 
and funders are not (and should not be) the same. Yet, with the MELIBEA coding 
as well as the algorithm, they are treated the same way. 

You also point out, rightly, that gold OA publishing policy is not central to 
institutional OA policy making, yet there it is, as part of the MELIBEA 
algorithm.

You also point out that the color code has nothing to do with the 
"green" OA coding -- yet there it is, competing with the widespread 
use of green to designate self-archiving, and thereby causing confusion, both 
overt and covert.

I would be more than happy to give you feedback on every aspect of MELIBEA -- 
it could be a useful and natural complement to the ROARMAP registry of OA 
policies. 

But as it is designed now, I can only agree with Steve Hitchcock's points and 
conclude that consulting MELIBEA today would be likely to induce confusion and 
would not help in bringing the all-important focus and direction to OA 
policy-making that I am sure CSIC, too, seeks, and seeks to help bring about.

Here are just a few prima facie points:

(1) Since MELIBEA is not, and should not be construed as a ranking of OA 
policies -- especially because it includes both institutional and funder 
policies -- it is important NOT to plug it into an algorithm until and unless 
the algorithm has first been carefully tested, with consultation, to make sure 
it weights policy criteria in a way that optimizes OA progress and guides 
policy-makers in the right direction.

(2) For this reason, it is more important to allow users to generate separate 
flat lists of institutions or funders on the various policy criteria, 
considered and compared independently, rather than on the basis of a 
prematurely and arbitrarily weighted joint algorithm.

(3) This is all the more important since the data are based on less then 200 
institutions, whereas the CSIC University Rankings are based on thousands. 
Since the population is still so small, MELIBEA risks having a disproportionate 
effect on initial conditions and hence direction-setting; all the more reason 
NOT to amplify noise and indirection by assigning untested initial weights 
without carefully thinking through and weighing the consequences.

(4) A potential internal cross-validator of some of the criteria would be a 
reliable measure of outcome -- but that requires much more attention to 
estimating the annual size and growth-rate of each repository (in terms of OA's 
target contents, which are full-text articles), normalized for institution size 
and annual total target output. Policy criteria (such as request/require or 
immediate/delayed) should be cross-validated against these outcome measures 
(such as percentage and growth rate of annual target output).

(5) The MELIBEA color coding needs to be revised, and revised quickly, if there 
is to be an algorithm at all. All those arbitrary colors in the display of 
single repositories as ranked by the algorithm are both unnecessary and 
confusing. The objective should be to order and focus clearly and intuitively. 
Whatever is correlated with more green OA output (such as a higher level or 
faster growth rate in OA's target content) should be coded as darker or bigger 
shades of green. The same should be true for the policy criteria, separately 
and jointly: in each case, request/require, delayed/immediate, etc., the 
greenward polarity is obvious and intuitive. This should be reflected in the 
graphics as well as in any comparative rankings.

(6) If you include repositories with no OA policy at all (i.e., just a 
repository and an open invitation to deposit) then all you are doing is 
duplicating ROAR and ROARMAP, whereas the purpose, presumably, of MELIBEA, is 
to highlight, weigh and compare specific policy differences among (the very 
few) repositories that DO have policies.

(7) The sign-up data -- 
http://www.accesoabierto.net/politicas/nueva.php?directorio=politicas -- are 
also rather confusing; the criteria are not always consistent, relevant or 
applicable. The sign-up seems to be designed to make a funder mandate the 
generic option, whereas this is quite the opposite of reality. There are far 
more institutions and institutional repositories and policies than funders. 
There should be separate criterial lists for institutional policies and for 
funder policies; they are not the same. There is also far too much focus on 
gold OA policy and payment. If included at all, this should only be at the end, 
as an addendum, not the focus at the beginning, and on a par with green OA 
policy. 

(8) There is also potential confusion on the matter of "waivers": 
There are two aspects of a mandate. One concerns whether or not deposit is 
required (and if so, whether that requirement can be waived) and the other 
concerns whether or not rights-reservation is required (and if so, whether that 
requirement can be waived). These two distinct and independent 
requirements/waivers are completely conflated in the current version of 
MELIBEA.

I hope there will be substantive consultation and conscientious redesign of 
these and other aspects of MELIBEA before it is can recommended for serious 
consideration and use.

Stevan Harnad


On 2010-07-19, at 5:18 AM, Remedios Melero wrote:

> Dear Steve,
> 
> I apologize for the delay in my response, but I will try to give some 
explanations to make clear some issues you raised in your message (my comments 
are in capital letters, to distinguish them from yours)
> 
> 
> 
> El 15/07/2010 11:22, Steve Hitchcock escribió:
>> Reme,    Thank you for bringing this new service to our attention. OA 
policies are vitally important to the development of institutional 
repositories, and services that can highlight and bring attention to this 
development can be valuable.
>> 
>> There are a few aspects of the validation aspects of the new MELIBEA 
service that confuse, and possibly trouble, me. The first is the main 
indicator, %OAval, which is the most visible result for a policy. What do you 
expect this will tell people about a given policy? I randomly selected a couple 
of policies, one of which was for my own school, to find they each scored about 
50%. I would expect these to be among the leaders in terms of OA policies, so 
this seems a surprisingly unhelpful score.
>> 
>> So what's the explanation? Note that the objects being evaluated are 
institutional OA policies; they are effectively being presented in relation to 
institutional repositories when the policy specifies where to archive is an IR 
with a URL. It seems that the scores include ratings for OA publication policy, 
libre vs gratis OA, publisher pdf, sanctions (score if Yes), incentives (score 
if Yes), etc., some of which an institution might specify but which might not 
apply to an IR 
>> http://www.accesoabierto.net/politicas/politicas_estructura.php
>> . However you weight these factors they are still contributors to the 
overall score, so a policy that is specific to an IR is immediately 
handicapped, or appears to be unless there is more context to understand the 
scores.
>> 
>>   
>> 
> AS I WROTE BEFORE THIS IS NOT A RANKING, IT IS NOT THE AIM OF MELIBEA  BUT 
TO HAVE A KIND OF REFERENCE ON WHAT TOPICS, ISSUES OR MATTERS TO BE INCLUDED IN 
AN OA POLICY. WE ARE TALKING ABOUT INSTITUTIONAL POLICIES OF DIFFERENTE NATURE, 
NOT ABOUT REPOSITORIES POLICIES. IF THE POLICY ONLY TALKS ABOUT THE REQUIREMENT 
TO DEPOSIT IN A REPOSITORY, IT SHOULD  SPECIFY WHAT, WHEN AND UNDER WHAT 
CONDITIONS, IF ANY. IT IS NOT THE SAME TO SAY WHAT DOCUMENTS  AND WHAT VERSIONS 
AND WHEN THAN SIMPLY SAY " ANY" OR "AS SOON AS POSSIBLE" 
(this could be a month after publication or years after publication, depending 
on one's criteria). GOLD ROUTE, NEVER IS REQUIRED ACCORDING OUR APPROACH 
("Gold  (Recommended in OA journals") AND NOT ALL OA JOURNALS ARE 
SUPPORTED BY  SAME ECONOMIC MODEL.
> 
> 
>> Which leads me to another question on the visualisation of the 
validator, and its use of green, gold (and red) in the meter. Do the green and 
gold refer the the classic OA colours? This would be quite convenient, since it 
would appear that the green repository policies I mentioned above are achieving 
almost full scores in the green zone of the meter. However, I suspect this 
cannot be the case, because it would assume that institutions must have a green 
AND gold policy, but not simply gold (whatever argument could be put for that). 

>>   
>> 
> COLORS DO NOT MEAN THAT, WE WANTED JUST TO DISTINGUISH ZONES LIKE IT WERE 
A SPECTRA. 
> 
>> It is important that new services should help reveal and promote OA 
policies, as you seek to do, but at the same time not to prejudice the 
development of such policies by mixing and not fairly separating the 
contributing factors, especially where these relate to different types of OA.
>>   
>> 
> I DO NOT THINK WE ARE MIXING, IN FACT THERE TWO MODELS, ONE FOR UNIV. AND 
RESEARCH CNETRES AND ANOTHER FOR FUNDERS AND GOV. INSTITUTIONS AND THE 
QUESTIONS FOR THEM ARE DIFFERENT, for instance, FOR A FUNDER THE QUESTION ABOUT 
DEPOSIT O THESIS IS NOT APPLICABLE.
> IN SUMMARY, OUR MODEL COULD NOT BE "PERFECT" BUT I IS ONE, WHICH 
COULD DETECT DIFFERENCES BETWEEN REQUEST AND REQUIRE, WHO, WHAT , WHEN IF THERE 
ARE ANY INCENTIVES OR SANCTIONS (  this has not to be a negative point but to 
remember we should  assume  reponsible attitudes).
> 
> However we will revise the model to see if we can make any improvement to 
make it clear, we are working also in a graph interface to show some data in 
graphical form.
> Best wishes
> Reme
> 
>> 
>>> R. Melero 
>>> IATA, CSIC 
>>> Avda Agustín Escardino 7, 46980 Paterna (Valencia), Spain 
>>> TEl +34 96 390 00 22. Fax 96 363 63 01 
>>> E-mail 
>>> rmelero AT iata.csic.es
>>>  
>>> 
>>> http://www.accesoabierto.net
>>> 
>>> 
>>> --      
>>>     
>>> 
> 
> 
> --      
> To unsubscribe from the BOAI Forum, use the form on this page:
> http://www.soros.org/openaccess/forum.shtml?f


        
--      
To unsubscribe from the BOAI Forum, use the form on this page:
http://www.soros.org/openaccess/forum.shtml?f


[BOAI] [Forum Home] [index] [prev] [next] [options] [help]

 E-mail:  openaccess@soros.org .