Taking scholarly books into account, part II: a comparison of 19 European countries in evaluation and funding

Abstract

In May 2016, an article published in Scientometrics, titled ‘Taking scholarly books into account: current developments in five European countries’, introduced a comparison of book evaluation schemes implemented within five European countries. The present article expands upon this work by including a broader and more heterogeneous set of countries (19 European countries in total) and adding new variables for comparison. Two complementary classification models were used to point out the commonalities and differences between each country’s evaluation scheme. First, we employed a double-axis classification to highlight the degree of ‘formalization’ for each scheme, second, we classified each country according to the presence or absence of a bibliographic database. Each country’s evaluation scheme possesses its own unique merits and details; however the result of this study was the identification of four main types of book evaluation systems, leading to the following main conclusions. First, countries may be differentiated on the basis of those that use a formalized evaluation system and those that do not. Also, countries that do use a formalized evaluation system either have a supra-institutional database, quality labels for publishers and/or publisher rankings in place to harmonize the evaluations. Countries that do not use a formalized system tend to rely less on quantitative evaluation procedures. Each evaluation type has its advantages and disadvantages; therefore an exchange between countries might help to generate future improvements.

Appendices

Appendix 1: Access interface and text version of the online questionnaire

Appendix 2: Clustering by Joint Correspondence Analysis (JCA) (see e.g. Greenacre 2007 or Camiz and Gomes 2013 for a comparison with multiple correspondence analysis)

The empirical clustering along the variables presented in Table 1 shows that the variables can be reduced to two dimensions quite comprehensively: The first dimension reflects the degree of formalization and use of metrics; the second dimension represents the use of rankings and labels.

Figure 2 shows the map of the JCA. The position of the variables shows that a strong relationship between the existence of a national or institutional database and a formalized evaluation system (i.e., very close to each other in the map), and that the non-existence of a specific book evaluation system lies just opposite to the former. These variables define the x-axis. The y-axis is defined by the existence of a quality label or a publisher ranking on the one hand and by the use of peer review panels on the other hand. The rankings and peer review panels, however, are also spread on the x-axis.

Fig. 2

Two dimensions of evaluation processes in 19 countries using Joint Correspondence Analysis. Note: Formal stands for formalized evaluation procedure, Supra DB stands for supra-institutional database, Inst. DB stands for institutional database, Rank stands for publisher ranking, QualiLab stands for quality label, Panels stands for expert panels, and Book Eval stands for specific procedure for book evaluation (see Table 1). [ES: Spain; FI: Finland; BEFL: Flanders (Belgium); LV: Latvia; DK: Denmark; NO: Norway; SK: Slovakia; LT: Lithuania; ME: Montenegro; CZ: Czech Republic; PL: Poland; IL: Israel; CH: Switzerland; RS: Serbia; PT: Portugal; FR: France; IT: Italy; HR: Croatia; SI: Slovenia]

The countries form four clusters. In the top right quadrant, we find France, Italy, Portugal and Serbia. These countries do not have a specific book evaluation procedure in place. In the bottom right quadrant, Israel, Latvia and Switzerland form a cluster of countries having no formal evaluation procedure for books in place and do not have supra-institutional databases. Books are evaluated using peers in ex-post evaluations. There are small differences between the countries: Switzerland has databases on the institutional level, Israel includes books in evaluations systematically while in Latvia neither is the case. In the bottom left quadrant, the Czech Republic, Denmark, Lithuania, Montenegro, Norway, Poland, Slovenia and Slovakia form a cluster of countries having a formal evaluation system as well as comprehensive databases on a supra-institutional level in place. There is, however, a divide of this cluster into three groups, the first is having a ranking in place (DK, NO, SI), the second does additionally evaluate using peer review panels (SK) while the third has neither a ranking in place nor uses peer review (CZ, LI, ME, PL). Finally, on the top left quadrant, Finland, Flanders (BE) and Spain build a cluster having a formalized evaluation system, a quality label and a publisher ranking in place. Croatia reveals itself as a special case, because it stands for a formal evaluation system and a supra-institutional database. However, there are no other metric instruments used and books are evaluated using expert panels, pulling Croatia into the middle of the x-axis.

The empirical clustering thus reveals different insights than the theoretical clustering represented in Fig. 2. While the axes represent similar dimensions, it groups the countries differently. The emphasis lies more in the differentiation of quantification and peer review, for example. Nevertheless, the results are, not surprisingly, similar. Rotating Fig. 1 by 45° results in a similar solution.