Critical Notes on the Abuse of Bibliometric Data

I have recently been informed by Catuscia Palamidessi that the editorial board of the journal Mathematical Structures in Computer Science has put together a critical note on the (ab)use of bibliometric data, which will appear in the issue 19.1 of that journal. The note has been written by the editor in chief, Giuseppe Longo, and subscribed by all the members of the editorial board of that journal.

The note expresses the worries of the scientists in the board about

 

• the way the evaluation of research activity is evolving in many countries,
• the general trend to use criteria purely based on numbers and citation indexes in judging the quality of researchers and
• the fact that the management of the data used in the numerical evaluations is entrusted to private agencies, whose methodologies and software might be rather dubious or cannot be subjected to scrutiny by the research community.

Did you know that

“The first journal according to ISI (...) is the 195th according to CiteSeer; the 2nd according to ISI does not appear in CiteSeer; the 6th for ISI is 958th for CiteSeer... Conversely, the 1st for CiteSeer (...) is 26th for ISI; the 4th for CiteSeer (...) is 122nd for ISI”

(See this document, in French.) I did not, and the fluctuation in the data is worrying, to say the least.

What is the situation regarding the use of citation indexes and impact factors in Iceland? Are we already abusing them?

 

Similar concerns have been raised by others. See, e.g., the talk "Bibliometric Evaluation of Computer Science - Problems and Pitfalls" by Friedemann Mattern (Institute for Pervasive Computing, Department of Computer Science, ETH Zurich).

There is also a very interesting joint report from three mathematical boards, which is definitely worth reading.

Finally, the "Sector Overview Report from the Computer Science and Informatics Sub-Panel (UoA 23)" after the British nationwide "Research Assessment Exercise 2008" (available at http://www.rae.ac.uk) includes the following passage:

We frequently found that citation counts were poorly correlated with the sub-panel’s assessment of the impact of the work examined. Citations also varied widely between research areas. For instance, much of the highly significant theoretical research, in which the UK is world leading, typically attracts low citation counts. Despite these low citations, the work is often found to have profound long-term impact on practical aspects of the field.

(The emphasis is mine and is not present in the original text.)

I hope that some of you will find these contributions interesting. I thank Catuscia Palamidessi and Vladimiro Sassone for pointing them out to me. Feel free to distribute this post as widely as you see fit.