Qualitative Evaluation of Scientific Output

In one of the comments to this post, Indriði Indriðason wrote:

"Somehow I can't help but smile when scientists advocate the use of qualitative measures.   Though I suspect the disagreement is about the kind of bean counting that should take place. "

I added a further comment to Indriði's comment, which I reiterate below in case any reader of this blog is interested in it. (I wonder how many people read comments to posts .) 

The reason why I keep boring readers (sorry about that) with this issue is that it is all too easy to read too much in the figures and the conclusions of the "Skyrsla". People with knowledge of academic work (should) know better, but the leaders and articles appearing in the newspapers over the last couple of days are just hype.

Anyway, for what it's worth, here is my comment. Enjoy!

------------------------------------------

Indriði, 

Of course, there is no perfect way of measuring academic quality and the path is littered with those who have tried to come up with the "right equation" to do so. (If you want to see an elaborate evaluation system, look at the way departments are evaluated in the British Research Assessment Exercise 2008. You will be able to find specific evaluation plans for each discipline at http://www.rae.ac.uk/panels/.)

At the very least, however, I would like to see a system in which the numerical value of a paper is weighted in some way by the impact of the outlet in which it appears, the citations it has had etc. It makes no sense to me to give X points both to a paper that appears in the "Mickey Mouse Journal of Computer Science" and in the "Journal of the ACM" (one of the most prestigious journals in computer science).

According to pure bean counting, a researcher who writes 41 "mickey mouse papers" brings more points than, say, Timothy Gowers (Cambridge mathematician who won the Fields Medal---the most important prize in mathematics for mathematicians under 40) because Gowers has written fewer than 40 papers, or so I believe. A system which gives this kind of result cannot be one that we want to use. 

Anyway, I think that it is really worthwhile  for any department to undertake a research evaluation every five years, say. A lot of the gain is in the process of looking back at one's work, put it into perspective, evaluate whether one has achieved what one wanted to achieve, and decide what one wants to do in the future. An outside opinion on the performance, the structure and the aims of a department can only help it to grow scientifically, improve and set itself goals that are both ambitious and achievable. 

At the end of the day, this is much more important than any numerical value assigned to a department.

 

 

Athugasemdir

[Indridi Indridason (IP-tala skráð) 15.6.2007 kl. 17:00]

Luca,

just to be clear - I couldn't agree with you more & I have frequently made the same argument in my department and elsewhere.  But I will still maintain it is bean counting - we would just like to take account of the fact that some beans are bigger than others.  

That said, I am glad that someone is counting something.  That, at least, is a sign that attention is being paid to what gets produced.  Sadly it has been the case that some "scholars" have gotten away without producing anything and any sort of counting is helpful in at least establishing the expectation that you need to do your job.  And at UI, this sort of bean counting has actually been very helpful in energizing the faculty.  It is, I agree, not perfect but it is better than nothing and hopefully this is only the start.

Unfortunately I also sense a lot of resistance to the idea of taking account of quality in the social science and some other fields.  Rather, I should say, there is very little consensus on what constitutes a quality publication.  To a certain extent this is understandable because, for example, it is arguable much more difficult to publish a piece on Icelandic politics than U.S. politics in the top political science journals.  I imagine fields, such as computer science, don't face problems such as these.  Scholars of "Icelandic things", therefore, often resort to publish their work in Iceland where most of their audience is but in doing so their work wouldn't count as "quality publication" even if it is first rate.  The problem is that there are no mechanisms to evaluate "local" publications.  We don't have a range differently reputable publishers and, in general, peer reviews are not employed.  Thus, there is no simple way to say whether a book is of high or low quality - especially if the only person who could have prevented the book's publication is the publisher's editor, who presumably is not an expert in the given field. 

I'm afraid I have gotten a bit of tangent here but my general point is that although assessing quality is very important it is not always a very simple matter.  While personally I would favor a system like described in the post it may also not be very fair towards some fields of study (social sciences, humanities, law) since the UI has an obligation to study Icelandic "things", which, if I'm correct, doesn't extend to other universities.  This is only one example of how the government's higher education has been misguided - if it wants competition to work its wonders it really should create a level playing field.

But Luca, thanks for raising the issue of quality of research - it certainly deserves attention.

[Indridi Indridason (IP-tala skráð) 15.6.2007 kl. 17:25]

Luca,

I also think your assessment of the RAE is too positive. This is taken from the UOA 23, Computer Science and Informatics:

"In arriving at an overall assessment of research quality the sub-panel will use its professional judgement rather than applying a rigid or formulaic method of assessing research quality. It will not use a formal ranked list of outlets, nor impact factors, nor will it use citation indices in a formulaic way."

It seems to me that the evaluating panel will have a lot of reading to do! (Actually, they do state that they will really only give 1/4 of the papers a read).  In addition, each researcher is only required to submit four "outputs", which will surely bias the results.  But, yes, still better than what we have.

[Luca Aceto, 16.6.2007 kl. 14:41]

Indridi,

Thanks a lot for your thoughtful input to this discussion. I think that we actually agree on most, I'd even say all, points. 

Fields like computer science, mathematics and physics are international enterprises, and there are several, more or less refined, forms of "weighted bean counting" that most people use to evaluate candidates for positions and the relative standing of academic researchers (e.g., citations, h-index, g-index, papers in certain types of outlets etc.). There are also on-line bibliographic data bases that collect publications in the outlets that "matter". Bibliometric data in these fields are extremely important. I am using them all the time myself

You are right in pointing out that different fields have different traditions, and therefore alternative approaches need to be used in evaluating research quality. That is the only reason why I pointed out the RAE in the UK. Different evaluation panels use different sets of ground rules, and they only rank the quality of research carried out within their specific field.

I agree with you in that the RAE has problems. It is the oldest research evaluation enterprise I am aware of, but it has probably got out of hand. 

Overall, thanks again for the pleasant and stimulating discussion on a topic that is close to my heart. You will see me publish bibliometric data on computer science departments in Reykjavik soon. I am being forced to do so by what I see published in MBL on a daily basis, alas.

We should not leave the evaluation of results on academic research in the hands of  journalists.