Methods for accrediting publications to authors or countries: Consequencesfor evaluation studies

One aim of science evaluation studies is to determine quantitatively the co ntribution of different players (authors, departments, countries) to the wh ole system. This information is then used to study the evolution of the sys tem, for instance to gauge the results of special national or international programs. Taking articles as our basic data, we want to determine the exac t relative contribution of each coauthor or each country. These numbers are then brought together to obtain country scores, or department scores, etc. It turns out, as we will show in this article, that different scoring meth ods can yield totally different rankings. In addition to this, a relative i ncrease according to one method can go hand in hand with a relative decreas e according to another counting method. Indeed, we present examples in whic h country (or author) c has a smaller relative score in the total counting system than in the fractional counting one, yet this smaller score has a hi gher importance than the larger one (fractional counting). Similar anomalie s were constructed for total versus proportional counts and for total versu s straight counts. Consequently, a ranking between countries, universities, research groups or authors, based on one particular accrediting method doe s not contain an absolute truth about their relative importance. Different counting methods should be used and compared. Differences are illustrated w ith a real-life example. Finally, it is shown that some of these anomalies can be avoided by using geometric instead of arithmetic averages.