05.01.2024
Maria Chiara Pievatolo, The Scale and the Sword: Science, State and Research Evaluation, Bollettino Telematico di Filosofia Politica, 03.01.2024
“[…] discussing how research is evaluated is a waste of time if we do not address the question of who is entitled to do it”.
Introductory note from the author.
“If this were only a domestic issue, the fact that some of the international literature on research assessment in Italy appears misleading to many Italian-speaking researchers would not be so important. Now, however, the ANVUR, the Italian agency for research assessment appointed by the government, is participating in the research assessment reform process initiated by the COARA coalition, in a way that is not only inconsistent, but may put the entire COARA project at serious risk of failure. Therefore, we decided to present a translation of a 2017 article dealing with Andrea Bonaccorsi’s closed-access book La valutazione possibile. Teoria e pratica nel mondo della ricerca. Bologna. Il Mulino, 2015. Andrea Bonaccorsi is a former member of ANVUR’s board of directors, who has attempted to provide one of the broadest theoretical justifications for the Italian research assessment system, which is pervasive, centralized, mostly bibliometric, and under government control. A highly abbreviated English version of Bonaccorsi’s argument, which is also behind a paywall, can be found here.
Anonymous peer review has been, and continues to be, an important part of the process leading to the publication of an article in scholarly journals that are still built on the “affordances” of print technology. Two or more scholars from suitable disciplinary fields, chosen at the discretion of the journal’s editorial board and protected by anonymity, are asked to give an ex ante opinion on the acceptability of a text for publication. What the reviewers reject never sees the light of day, nor do their opinions and any discussions with the authors.
The Italian version of this essay, on the other hand, was born – together with a twin written by the jurist Roberto Caso – as an experiment in open peer review, which at the time was – and still is – rather unusual in Italy, where the state assessment of research requires anonymous peer review in order for a work of scholarship to be considered scientific. Open and post-publication peer review, however, would make it possible to moderate centralized and hierarchical evaluation systems by making the entire discussion public, recognizing the merits of the reviewers, and exposing any conflicts of interest.
To deal with Andrea Bonaccorsi’s justification of state evaluation of research, we will pretend – for the sake of discussion – that the system he theorizes produces a faithful snapshot of the way the scientific community evaluates itself. But even so, it can be shown that his justification leads to a research evaluation system that is practically despotic and theoretically retrograde. The system is despotic because it transforms an informal and historical ethos into a fixed rule of administrative law, which ceases to be an object of choice for the scientific community. And it is retrograde because, by establishing this rule, it blocks evolution in a still image, like the Sleeping Beauty’s castle, which cannot be overcome without further bureaucratic intervention.
In addition to the main argument, there are two ancillary parts: the first deals with the question, proposed by Bonaccorsi, of the empirical verifiability of some of the criticisms made against him; the second is an examination of a sample of quotations used by him to support some important passages. Finally, the conclusion briefly outlines the ideal and critical perspective of open science that inspires this paper.
In this spirit, whenever possible, we have cited legally accessible versions and reviews of the paywalled sources that Bonaccorsi prefers, so that the reader can check our argument without having to overcome further economic barriers.The Italian version of this essay was written in 2017, but we resisted the temptation to add a lot of updated references, even if they would have helped to support our points. We trust that our critique of the centralized and bureaucratic system of research evaluation can survive not only by relying on the literature of six years ago, but also on that of a hundred years ago.
A final warning: the term “state evaluation” is modeled on the term “state capitalism“. Just as state capitalism is an economic system in which the state is involved in business and profit-making economic activity, state science and state evaluation of research suggest that the state, directly or indirectly through its agencies, is involved in defining what is good science and what is not. In Italian we use “di stato” after a noun both in a neutral sense (“esame di stato”: state examination) and in a polemical sense, as in “delitto di stato” (a crime committed by the state itself). Translating “valutazione di stato” as “centralized evaluation” would lose the nuances of the Italian expression, while the adjective “governmental” might suggest a Foucauldian undertone that would betray the spirit of this essay, whose main point is closer to Kant: discussing how research is evaluated is a waste of time if we do not address the question of who is entitled to do it.
We will be inviting some reviewers, but even if you are not invited, your comments are welcome: to take part, read the instructions in the grey box at the bottom of this page. The article to comment is available here (https://commentbfp.sp.unipi.it/maria-chiara-pievatolo-the-scale-and-the-sword-statal-science-and-research-evaluation/)”.