Bean Counting Cannot be Used to Evaluate the Quality of a Department

I just had a brief look at Skýrslan „Kostnaður, skilvirkni og gæði háskólakennslu“ í heild and the accompanying newspaper article available at mbl.is. 

Let me start by saying that it is a very good idea to evaluate university departments. I have gone through three five-year research evaluations in Aalborg and a couple of research assessment exercises at Sussex, and invariably the exercise of evaluating one's own work and seeing  other people's opinion on it has been very fruitful. However, such an evaluation should be done by a committee composed of distinguished scientists in the relevant field, and not by accountants.

Unfortunately, the document  Skýrslan „Kostnaður, skilvirkni og gæði háskólakennslu“ í heild gives me the impression of having been written by a bunch of accountants rather than by scientists. For instance, I see no mention of quality in research. Research output is measured in number of papers, and no mention is made of the repute of the publication outlet. Probably a journal or conference paper counts for some number X of points, irrespective of where it has appeared, and so does a book. Looking at those figures made me think about the nice little book "How to Lie with Statistics"

Another thing I have to question is the relevance that a report for the period 2003-2005 has in 2007. I can only speak about computer science, since this is the only subject covered in the report about which I have any experience. It is clear to any objective observer that things have changed quite a bit since 2005 in the landscape of computer science in Iceland, and issuing this report now can mislead prospective students and sponsors into believing that the state of play today is reflected by the "findings" in that report. 

Fortunately, we live in the internet age, and we all have access to Google Scholar and all the information that it indexes. I encourage prospective students, their parents and people in research policy not to believe blindly in that report, and to look for the relevant status of the researchers in computer science in Iceland themselves. For computer science at Reykjavik University I have done some of the work for you. Look here, and compare. Then draw your own conclusions. 

I hope to post a proper research evaluation for my department for the period 2003-2007 on this blog by the spring 2008.