1 Handbook of Operations

International Series in
Operations Research & Management Science
Zilla Sinuany-Stern Editor
Handbook
of Operations
Research and
Management
Science in Higher
Education
International Series in Operations Research
& Management Science
Founding Editor
Frederick S. Hillier, Stanford University, Stanford, CA, USA
Volume 309
Series Editor
Camille C. Price, Department of Computer Science, Stephen F. Austin State
University, Nacogdoches, TX, USA
Associate Editor
Joe Zhu, Foisie Business School, Worcester Polytechnic Institute, Worcester, MA,
USA
More information about this series at http://www.springer.com/series/6161
Zilla Sinuany-Stern
Editor
Handbook of Operations
Research and Management
Science in Higher Education
Editor
Zilla Sinuany-Stern
Department of IEM
Ben-Gurion University of the Negev
Beer-Sheva, Israel
ISSN 0884-8289 ISSN 2214-7934 (electronic)

International Series in Operations Research & Management Science
ISBN 978-3-030-74049-8 ISBN 978-3-030-74051-1 (eBook)
https://doi.org/10.1007/978-3-030-74051-1
© Springer Nature Switzerland AG 2021

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface
This handbook highlights the potential to apply the various operations research
(OR) and management science (MS) models in all functional areas and levels
of higher education (HE). OR is “a unique combination of mathematics, statis-
tics, and computers—to assist in decision making for complex organizational
problems,” as described on the website of the first university in the world to
grant a doctorate in this field—Case Western Reserve University in Cleveland,
Ohio (accessed in June, 2020, https://weatherhead.case.edu/degrees/doctorate/phd/
operations-research/). OR is the original term used upon its inception during World
War II where it was utilized to improve military operations. Its synonym, MS,
points at OR’s managerial orientation in general, as applied to business, industry,
and various types of organizations including HE. Other synonyms for OR include
decision science, systems analysis, operations management, analytics, and more.
We believe that researchers in this field have, in their own HE institution, an imme-
diate framework for applying its methods, demonstrating to their administrators
and students OR’s effectiveness in improving their own department, school, or
institution.
From time to time, discussions revolve around whether HE should be considered
a product (see Chap. 1, Sect. 4.1). Does it behave as a non-profit organization or is
it becoming a for-profit organization? The 16 chapters in this handbook address this
question from various angles.
The chapters of this book are divided into three parts:
Part I: Reviews and Overviews
Part II: New Methodologies
Part III: Applications
v
vi Preface
The Contents
Part I includes 9 chapters (Chaps. 1, 2, 3, 4, 5, 6, 7, 8, and 9) of reviews and

overviews of OR methods and specific functional areas that have been central in
HE since the turn of the millennium.
Chapter 1, by Sinuany-Stern, overviews OR in HE in 40 major OR journals.
In a search in the Web of Science, out of 58 OR methods, the simulation method
appeared most often in articles’ topic, and out of 30 HE functional areas, strategy
appeared most often. Moreover, reviewing OR methods by functional area, the
leading combination in the number of articles was the data envelopment analysis
(DEA) by the efficiency area.
Chapter 2, by Oplatka and Hemsley-Brown, systematically reviews the literature
of HE marketing between 2005 and 2019, pointing to five major themes: the
marketization of HE; marketing communications; branding, image, and reputation;
marketing strategy; and recruitment, alumni, and gift-giving. Marketing has become
a central tool in HE strategy for recruiting students. Almost 44% of the 105 articles
reviewed used quantitative methods.
Chapter 3, by Tiram and Sinuany-Stern, suggests a new classification of HE
simulation methods into two main types: analytic simulation used mainly for
planning purposes, and the more widely used educational simulators and games.
The analytic simulations include discrete-event simulation and Monte Carlo, agent-
based simulation, and system dynamics. The educational simulations include virtual
reality, educational games, simulators, and manikin.
Chapter 4, by Yair, argues that robust HE information systems are crucial for
decision makers at all academic levels and for analytical analysis. He describes the
reasons for the failure of Current Research Information Systems (CRIS) to modern-
ize routine managerial tasks involved in academic administration and suggests ways
to close the gap to modernize academic decision making in universities.
Chapter 5, by Sinuany-Stern, is an overview on forecasting, suggesting a new
classification of HE forecasting methods into two main groups: passive and active
methods. It presents the change in the twenty-first century from passive predictions
via ratio methods and time series analysis to active targeted forecasts via marketing,
causal regression, and data mining—aimed mainly at achieving a targeted planned
enrollment and retaining existing students from dropping out.
Chapter 6, by Pino-Mejias and Lukue-Calvo, presents a benchmarking interactive
tool that includes the most likely OR methods for HE administrators: multivariate
statistics, analytical hierarchy process (AHP), PROMETHEE, and DEA, with an
example given of university ranking. Moreover, for each of the above methods, they
provide examples of HE applications from the literature.
Chapter 7, by Sinuany-Stern and Sherman, reviews the literature on balanced
scorecard (BSC) application in HE strategic planning. In the university context,
BSC translates the institution’s mission statement and strategy into specific mea-
surable goals (e.g., number of students and graduates, number of publications) and
monitors the organization’s performance regarding achievement of these goals.
Preface vii
Chapter 8, by Bentur, Getz, and Shacham, provides a systems analysis of Tech-

nology Transfer Office that is needed in universities to control industry relations,
e.g., research funding and technology commercialization. They present examples
of technology transfer in the fields of artificial intelligence, data science, and smart
robotics, recommending that universities’ achievements regarding the third mission
(technology transfer) should be quantified and ranked on national and international
levels.
Chapter 9, by Sinuany-Stern, presents an optimization simulation scheme for
budget planning in HE. A variety of budgeting procedures, along with cost
simulation, and mathematical models for budget allocation from the literature are
presented, such as linear programming and quadratic programming with intuitive
solutions for most of the models presented for non-quantitative administrators.
Part II of the handbook includes 3 chapters (Chaps. 10, 11, and 12) on specific
methodological advancements in HE, giving OR-oriented readers examples of
theoretical models.
Chapter 10, by Keren, Hadad, and Minchuk, develops a unique game theory
model for allocating a research budget among several academic departments (the
contestants). The prize for this contest is the desired research budget. Moreover, a
new bargaining model is also developed. A numerical example demonstrates the
model and its applicability.
Chapter 11, by Leite, Melicio, and Rosa, deals with an examination timetabling
problem—a famous problem that in HE practice becomes too large to reach an
optimal solution in reasonable computer running time; thus, an accelerated variant
of the threshold acceptance (TA) algorithm called Fast TA is developed here. Its
efficiency is measured in relation to other known algorithms, utilizing the known
Toronto and ITC 2007 benchmark sets for comparison—showing significantly less
running time for Fast TA.
Chapter 12, by Sinuany-Stern and Friedman, develops a robust method to
identify the best rank-scaling efficiency from several efficiency methods, based on
the maximum average correlation of each rank-scale method with all the others. The
method is exemplified using data from a previous article on academic departments
at a given university. The scale-ranking methods used in the example are DEA super
efficiency, DEA cross efficiency, discriminant analysis, and canonical correlation.
Part III of the handbook includes 4 chapters (Chaps. 13, 14, 15, and 16) on
specific case studies, applying OR methods in various functional areas and levels
for HE institutions.
Chapter 13, by Sitaridis, Kitsios, Stefanakakis, and Kamariotou, uses preference
disaggregation analysis, a multi-criteria approach, to evaluate an ICT course in
Greece by students utilizing a course experience questionnaire with 36 items. This
method supports educators in organizing a course more flexibly and facilitates
student communication with teachers and their more active participation in the
educational process.
Chapter 14, by Pietrzak, applies the balanced scorecard (BSC) for strategic plan-
ning at a Polish university, tracking its development over 10 years of major changes
viii Preface
in the Polish HE system while highlighting the difference between businesses and
HE in applying BSC, and the adaptations needed in HE.
Chapter 15, by Davidovitch and Wadmany, reports on the perception of students
and lecturers on using E-learning (mainly the Zoom platform) during the COVID-19
pandemic in Israel. They used questionnaires distributed to a variety of Israeli HE
institutions and used statistical testing and regression analysis. Their main finding
is that less than half of the students and about a third of the lecturers were satisfied
with E-learning.
Chapter 16, by Sinuany-Stern and Hirsh, is a case study devoted to the relative
efficiency of HE in 29 OECD countries based on their 2019 data of selected HE
inputs and outputs. The efficiency of spending public resources on HE at the national
level is of key interest in the context of accountability, especially in developed
countries. The robust method from Chap. 12 is applied. In the study, based on two
inputs and six outputs, the USA ranks as the country that is most efficient.
For each of the 16 chapters, Table 1 lists all author’s last names, the OR
method(s) used, and the functional areas of HE for which each method is used.
In Summary
The content of this handbook is evidence that the third HE revolution is at its
height (Chap. 1, Sect. 4.5), namely that the HE sector has both business and for-
profit characteristics such as marketing (Chap. 2), measuring customer satisfaction
(student course evaluations in Chaps. 13 and 15), strategic planning via BSC
(Chaps. 7 and 14), managing HE research performance (Chap. 4), business relations
with industry (technology transfer, Chap. 8), ranking in HE (Chaps. 6, 12, and
16), internationalization (Chap. 1, Sect. 4.4) and E-learning (Chap. 15), which is
the highlight of the 4.0 industrial revolution, supported by educational web-based
simulators (Chap. 3), and measuring HE efficiency (Chap. 16). Intertwined with the
main classical areas of HE planning are simulation (Chap. 3), forecasting (Chap. 5),
planning and budgeting (Chap. 9), resource allocation (Chap. 10), and examination
timetabling (Chap. 11).
This handbook on OR and MS in HE is an essential tool for planning in all areas
and levels at any time, just as in any other business and industry, and even more so
in times of crisis such as during the COVID-19 epidemic.
Audience
This book is aimed at HE administrators, business administration graduates,

management scientists, operations researchers, analytic and industrial engineers,
applied mathematicians, and computer scientists at all levels who are interested
Preface ix
Table 1 List of chapters with OR/MSa method used and HEa functional areas to which they apply
Chapter Author(s) OR/MS method(s) applieda HE functional area
Part I Reviews and Overviews
1 Sinuany-Stern 58 OR/MS methods 30 HE functional areas
2 Oplatka and Questionnaires and multivariate Marketing
Hemsley-Brown statistics
3 Tiram and Sinuany-Stern Analytic simulation and Planning, training, and
educational simulators labs
4 Yair Research information systems Improving academic
management
5 Sinuany-Stern Passive versus active forecast: Forecasting of costs,
time series, regression, data income, enrollment,
mining faculty, etc.
6 Pino-Mejías and PCA, PROMETHEE, and DEA University ranking,
Luque-Calvo benchmarking
7 Sinuany-Stern and Balanced scorecard Strategic planning
Sherman
8 Bentur, Getz, and Systems analysis Technology transfer
Shacham
9 Sinuany-Stern Linear programming and Planning, budgeting,
quadratic model cost simulation
Part II New Methodologies
10 Keren, Hadad, and Game theory and AHP Funding research
Minchuk
11 Leite, Melício, and Rosa Fast threshold acceptance Examination timetabling
algorithm
12 Sinuany-Stern and Robust rank-scale, correlation, Robust efficiency of
Friedman DEA, multivariate statistics academic departments
Part III Applications
13 Sitaridis, Kitsios, Multi-criteria analysis, Course evaluation and
Stefanakakis, and disaggregation analysis satisfaction of ICT
Kamariotou
14 Pietrzak Balanced score card Strategy in a Polish HEI
15 Davidovitch and Questionnaire, regression, E-learning in HEIs in
Wadmany statistical testing COVID-19 crisis
effectiveness in Israel
16 Sinuany-Stern and Hirsh DEA, multivariate statistics, Efficiency of OECD
robust scale countries in HE
a HE—higher education, HEI—HE institution, OR—operations research, MS—management sci-
ence, ICT—information and communications technology, PCA—principal component analysis,

DEA—data envelopment analysis, PROMETHEE—Preference Ranking Organization Method for
Enrichment Evaluations, AHP—analytic hierarchy process, OECD—Organisation for Economic
Co-operation and Development
x Preface
in contributing to the enhancement of the HE system. Each of these individuals

will find chapters fitting their level, as most are overviews and are not quantitative.
Moreover, the chapters are independent of each other and cover many planning areas
in HE and OR/MS methods.
Beer-Sheva, Israel Zilla Sinuany-Stern

January 2021
Acknowledgments
I express my thanks to the authors of the various chapters of this handbook, to

the reviewers, to the Associate Editor Joe Zhu who encouraged this project, to the
Springer Nature team, and to Eng. Timoti Alexander Shektov my assistant.
xi
Contents
Part I Reviews/Overviews
1 Operations Research and Management Science in Higher
Education: An Overview .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3
Zilla Sinuany-Stern
2 A Systematic and Updated Review of the Literature
on Higher Education Marketing 2005 –2019 . . . . . . . .. . . . . . . . . . . . . . . . . . . . 35
Izhar Oplatka and Jane Hemsley-Brown
3 Overview of Simulation in Higher Education: Methods
and Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 81
Efrat Tiram and Zilla Sinuany-Stern
4 Managing Minds: The Challenges of Current Research
Information Systems for Improving University Performance . . . . . . . . . 117
Gad Yair
5 Forecasting Methods in Higher Education: An Overview . . . . . . . . . . . . . 131
Zilla Sinuany-Stern
6 Survey of Methods for Ranking and Benchmarking Higher
Education Institutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 159
José-Luis Pino-Mejías and Pedro-Luis Luque-Calvo
7 Balanced Scorecard in Strategic Planning of Higher
Education: Review .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 213
Zilla Sinuany-Stern and H. David Sherman
8 System Analysis of Technology Transfer Policies and Models
in Higher Education .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 241
Arnon Bentur, Daphne Getz, and Oshrat Katz Shacham
9 Models for Planning and Budgeting in Higher Education . . . . . . . . . . . . . 263
Zilla Sinuany-Stern
xiii
xiv Contents
Part II New Methodologies

10 Funding Research in Higher Education Institutions: The
Game Theory Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 303
Baruch Keren, Yossi Hadad, and Yizhaq Minchuk
11 A Fast Threshold Acceptance Algorithm for the Examination
Timetabling Problem .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 323
Nuno Leite, Fernando Melício, and Agostinho C. Rosa
12 Robust Efficiency via Average Correlation: The Case
of Academic Departments . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 365
Zilla Sinuany-Stern and Lea Friedman
Part III Applications

13 Course Evaluation Using Preference Disaggregation
Analysis: The Case of an Information Communication
Technology Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 389
Ioannis Sitaridis, Fotis Kitsios, Stavros Stefanakakis,
and Maria Kamariotou
14 The Application of a Balanced Scorecard in Higher
Education Institutions: A Case Study of Wuls . . . . . .. . . . . . . . . . . . . . . . . . . . 419
Michał Pietrzak
15 E-Learning in Times of Crisis: An Incidental or Facilitative
Event? . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 453
Nitza Davidovitch and Rivka Wadmany
16 The Relative Efficiencies of Higher Education in OECD
Countries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 481
Zilla Sinuany-Stern and Arthur Hirsh
Index . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 513
About the Editor and Contributors
About the Editor
Zilla Sinuany-Stern is a Professor Emerita in the Department of Industrial

Engineering and Management (IEM) at Ben-Gurion University (BGU) of the Negev
in Israel. Dr. Sinuany-Stern has served in many administrative roles in HE: as the
Head of IEM at (1000 students) and as a member and chair of various important
committees at the university. In an interim period of 8 years, she was the Provost
of Ariel University (8000 students at that time). On the national level, she was the
President and VP of the Israel Society of OR for 8 years, and for 5 years (until 2017),
she was a member of the Council for Higher Education of Israel, also chairing its
Quality Assurance Committee. At the international level, she was Vice President of
the European Association of OR (EURO) for 4 years. She has been a member of the
International Advisory Board of the Journal of the Operational Research Society
since 1993 (in the UK, the first OR journal ever, opened in 1950). Sporadically,
over the years, she has consulted for various public authorities in Israel and for HE
institutions in the USA such as the Commission for HE in Indiana. She received
her PhD in OR from Case Western Reserve University in 1978 and her BA and MA
from Tel Aviv University in economics and statistics.
Contributors
Arnon Bentur Samuel Neaman Institute, Haifa, Israel

Nitza Davidovitch School of Education, Ariel University, Ariel, Israel
Lea Friedman Department of Industrial Engineering & Management, Ben Gurion
University of the Negev, Beer Sheva, Israel
Daphne Getz Samuel Neaman Institute, Haifa, Israel
xv
xvi About the Editor and Contributors
Yossi Hadad Department of Industrial Engineering and Management, SCE—

Shamoon College of Engineering, Beer Sheva, Israel
Jane Hemsley-Brown Surrey Business School, University of Surrey, Guildford,
UK
Arthur Hirsh Webster Vienna Private University, Vienna, Austria
Maria Kamariotou Department of Applied Informatics, University of Macedonia,
Thessaloniki, Greece
Baruch Keren Department of Industrial Engineering and Management, SCE—
Fotis Kitsios Department of Applied Informatics, University of Macedonia, Thes-
saloniki, Greece
Nuno Leite Instituto Superior de Engenharia de Lisboa, Instituto Politécnico de
Lisboa, Lisboa, Portugal
LARSyS—Laboratory for Robotics and Systems in Engineering and Science,
Universidade de Lisboa, Lisboa, Portugal
Pedro-Luis Luque-Calvo Departamento de Estadística e Investigación Operativa,
Universidad de Sevilla, Seville, Spain
Fernando Melício Instituto Superior de Engenharia de Lisboa, Instituto Politéc-
nico de Lisboa, Lisboa, Portugal
Yizhaq Minchuk Department of Industrial Engineering and Management, SCE—
Izhar Oplatka School of Education, Tel Aviv University, Tel Aviv, Israel
Michał Pietrzak Institute of Economics and Finance, Warsaw University of Life
Sciences, Warsaw, Poland
José-Luis Pino-Mejías Departamento de Estadística e Investigación Operativa,
Universidad de Sevilla, Sevilla, Spain
Agostinho C. Rosa Department of Bioengineering/Instituto Superior Técnico,
Oshrat Katz Shacham Samuel Neaman Institute, Haifa, Israel
H. David Sherman DMSB—Northeastern University, Boston, MA, USA
About the Editor and Contributors xvii
Zilla Sinuany-Stern Department of Industrial Engineering & Management, Ben

Gurion University of the Negev, Beer-Sheva, Israel
Ioannis Sitaridis Department of Applied Informatics, University of Macedonia,
Thessaloniki, Greece
Stavros Stefanakakis Department of Applied Informatics, University of Macedo-
nia, Thessaloniki, Greece
Efrat Tiram Sackler Faculty of Medicine, Tel Aviv University, Ramat Aviv, Tel
Aviv, Israel
Rivka Wadmany The Council for Higher Education, Jerusalem, Israel
Gad Yair Department of Sociology and Anthropology, The Hebrew University of
Jerusalem, Jerusalem, Israel
Part I
Reviews/Overviews
Chapter 1
Operations Research and Management
Science in Higher Education: An
Overview
Zilla Sinuany-Stern
Abstract Operations Research (OR) is an interdisciplinary discipline that evolved

during WWII to solve complex operational military problems such as scarce
resource allocations and logistics. Thereafter, it has been used successfully in the
private and public sectors—including in higher education (HE). The purpose of
this chapter is to give an overview of the use of OR methodologies in HE and
the functional areas in which they are used. First, a search in 40 OR journals
reveals that HE and OR methodology appeared in the title of 0.3% of their articles
during 1950–2019. Moreover, in 17 OR journals, HE did not appear in any article.
Second, looking at the web of science during 2000–2019, we identified 58 main OR
methods relevant to HE. Leading these methods (in terms of the number of articles)
is simulation. Third, we identified 30 functional areas relevant to HE. Leading
these areas is strategy. For specific methods by area, the leading combination is
data envelopment analysis by efficiency. In summary, there is great potential for
academics in OR to utilize OR models to improve their own HE system, especially
during and after crises such as a pandemic.
Keywords Higher education · Operations research · Management science ·

Overview
1.1 Introduction
Operations Research (OR) and Management Science (MS) aid decision-makers

in business, industry, government, and academia, putting emphasis on the strong
relationship of OR with computer science, information systems, and mathematics
(Gass & Fu, 2013).
Z. Sinuany-Stern ()
Department of Industrial Engineering & Management, Ben-Gurion University of the Negev, Beer
Sheva, Israel
e-mail: zilla@bgu.ac.il
© Springer Nature Switzerland AG 2021 3

Z. Sinuany-Stern (ed.), Handbook of Operations Research and Management
Science in Higher Education, International Series in Operations Research
& Management Science 309, https://doi.org/10.1007/978-3-030-74051-1_1
4 Z. Sinuany-Stern
Operations research began during World War II, first in the UK and then in the
USA, in a cooperative effort to solve complex military decisions such as radar
locations, allocation of supplies and ammunition, game theory (war games), and
logistics. Hartcup (2000) wrote about the: “Birth of a new science: operational
research” as one of the outcomes of applying science in World War II.
The OR/MS profession is unknown to many—sometimes even in academia,
as well as in other disciplines. The president of IFORS (International Federation
of Operations Research), representing 40,000 members worldwide, acknowledged
that he finds it hard to explain exactly what “Operational Research” is; moreover,
the history of OR has had many proposals for new names, embracing various
buzzwords—the latest one is “Analytics” (Trick, 2018). The main other terms used
for OR are Operational Research (UK) Operations Research (US), Management Sci-
ence (US in schools of business), Decision Science (NSF), Scientific Management,
Operations Management, Systems Analysis, and Analytics.
Indeed, to solve real-life complex problems that managers often face, the
mathematical models representing the problem are large, with many constraints
under uncertainty, and often have no closed-form solution. Thus, computerized
algorithms are often required, and good computerized management information
systems (MIS) are crucial for the success of OR. OR has been applied in many areas
to date, such as in business, industry, the public sector, healthcare, transportation,
and education.
Education is one of the major components in measuring the Human Development
Index (HDI) of nations. Since 2010, the UN has used HDI to measure a country’s
development. The index comprises three indicators: 1. life expectancy, 2. education
(years of schooling), and 3. per capita income (GNI in PPP). (UN; accessed in
June 2020, http://hdr.undp.org/en/countries). Higher Education (HE) is the highest
level of education. According to the World Bank (2020), this term refers to
all post-secondary education, including colleges, technical training institutes, and
vocational schools. HE is instrumental in fostering growth, reducing poverty,
and boosting shared prosperity. A highly skilled workforce with a solid post-
secondary educational level is a prerequisite for innovation and growth, based on
the assumption that well-educated people are more employable, earn higher wages,
and cope with economic shocks better (World Bank; accessed on Feb. 2020, https://
www.worldbank.org/en/topic/tertiaryeducation).
The transformation of the industrial era into the information age has changed
the basis of competitive advantage from a financial and physical resource-based
economy to one that is intellectual and knowledge-based (Stewart, 1997). This
change has placed universities in a central position to produce economic benefits.
As the springs of knowledge and places that produce knowledgeable beings, along
with the advancement of technology, universities have a vital role in determining
the wealth of a nation, especially today, when HE is one of the major enterprises
producing the most precious product—knowledge.
Already in the 1970s, HE institutions (HEIs) were described as organized
anarchy (Cohen & March, 1986) with weak regulation and inadequate control.
Schierenbeck (2013) claims that HE across the globe is in a productivity crisis
1 Operations Research and Management Science in Higher Education: An Overview 5
that prevents broad access to affordable and high-quality educational services.

He suggests closing the productivity gap in HE by academic managers using
some managerial practices applied by the world’s leading business enterprises.
HEIs that generate knowledge to foster productivity are often unproductive—
“the shoemaker’s children have no shoes.” In academic circles, often the topic of
productivity is widely considered “a taboo subject” (ibid.).
Along with this progress, public demand for accountability in HE is increasing.
OR offers various types of models for improving HE management at all levels
of HEIs from the department level to the national and international levels. The
application of OR in education was pointed at from its initiation by the first OR
journal JORS (originally called OR Quarterly) as an applied research field. The
purpose of this chapter is to give an overview of the use of OR methodologies in
HEIs, and the functional areas in which they are used.
Other books and review papers were devoted to OR in HE from the beginning
(such as Terrey, 1968; Schroeder, 1973; Cheng, 1993; Johnes, 2015); however, their
scope is quite limited compared to the 58 OR/MS methods and 30 functional areas
covered in this review in the web of science (WoS), as listed in Appendices 2 and 3.
Section 1.2 of this chapter reviews the appearance of HE articles in the main 40
OR journals from the WoS. Section 1.3 covers the appearance of 58 OR methods
(clustered into ten groups) in all WoS journals and 30 functional areas applied to
HE (clustered into four groups). Section 1.4 covers several HE issues. Finally, Sect.
1.5 concludes and draws future research directions for OR in HE.
1.2 OR/MS Publications
In academia, sometimes OR courses are given in various departments/schools such

as Mathematics, Business, Economics, Industrial Engineering, Statistics, Computer
Science, Electrical Engineering, Logistics, and Supply Chain. In some HEIs, the
teaching of OR is not taught in a separate OR department but is included in one or
more of the above departments.
The first OR journal was the Journal of the Operational Research Society
(JORS). It was first published in 1950 by the UK OR Society (and was called: A
Quarterly Journal of OR until 1976). In the beginning, only abstracts (extended) of
OR papers were published, collected “from as wide fields as possible,” including
education (Editorial Notes, JORS, 1, 1950). The interdisciplinary nature of OR was
expressed there as follows: “Operational research shares with other branches of
science . . . in knowledge and technique. Studies in social science, . . . , productivity
studies, education, etc., have common features.” This was the justification for the
first OR journal, JORS—the emergence of a new discipline (I am honored to have
been on the international advisory board of JORS since the early 1990s). Today in
the WoS alone, there are over 75 OR/MS journals.
One of the first books that started shaping OR discipline was by Churchman et
al. (1957), along with other major books such as Hillier and Lieberman (2014, 10th
6 Z. Sinuany-Stern
edition), Winston (2004, 4th edition), and Taha (2017, 10th edition) who included
the main OR methodologies such as linear programming, transportation models,
network optimization, scheduling, integer programming, goal programming, non-
linear programming, dynamic programming, metaheuristics, game theory, queuing
theory, decision analysis, inventory theory, Markov decision processes, queuing
theory, forecasting models, and simulation.
In a bibliometric general review of OR, Merigo and Yang (2017) concluded that
the American School of OR leads the most influential analysis of OR and MS in
general, based on parameters such as the number of citations in OR journals from
the WoS. They identify the leading countries in OR/MS as follows: leading is the
American School with the two most influential journals in the field: Management
Science and Operations Research. Moreover, Americans have published most of the
leading articles thus far. They are followed by Continental Europe (EJOR), China
(with several leading researchers), and the British School (JORS).
1.2.1 Reviews on OR in HE
Already in 1968, it was claimed that HE was a complicated business. Terrey (1968),
in his review of HE management tools (for the period 1963–1968), highlighted
four techniques: a. the program planning budgeting system (PPBS), b. systems
analysis, c. CPM/PERT, and d. the Delphi Technique. He stated that budgeting is
both a science and an art (ibid.). Schroeder (1973), in a review of the literature
on MS in university operations, listed four main areas: a. PPBS, b. management
information systems, c. resource allocation models, and d. mathematical models.
He concluded there were four problem areas in HE at that time: student flow
projections, information to be used in decision-making processes, measurement
of outputs, and alternative improved planning methodologies (ibid.). Cheng (1993)
focused on summarizing 15 papers on various OR models applied in HE covering
the following main areas: a. resource allocation, b. budgeting, c. project group
formation, d. scheduling courses and examinations, e. classroom allocation, f.
student registration, g. tuition and fee structures, and h. the Ph.D. submission rate.
This review concluded that while in the 1960s and 1970s OR models applied mainly
LP, in the 1980s, goal programming was mostly used, and most published papers
were financial planning models. However, Cheng (1993) claims that models for
budgeting were implemented more often than resource allocation. He points at the
often-used scheduling problems such as classroom allocation and examination and
course timetabling, ranging from manual procedures to fully automated systems. He
blames the lack of application on the communication gap between OR analysts and
academic administrators (ibid.).
Johnes’ (2015) review is devoted to education in general but also relates to
HE. This review includes over 200 papers covering mainly the following areas:
a. planning, b. resource allocation, c. efficiency and performance measurement, d.
vehicle routing and scheduling, e. the assignment problem, f. student grouping, and
g. examination scheduling and course timetabling.
Bell et al. (2012) edited a book on HE management and OR that covers various
important issues. Some are rather philosophical, such as the various metaphors for
HE management, questioning the use of industrial approaches to quality in HE,
HE management and university culture, and web-based learning. The major OR
method for which two full chapters were devoted is system dynamics. Overall, their
book deals more with soft OR. It fits the less mathematically oriented administrator
(ibid.).
The review in the next section counts the number of HE applications in 40 of the
main OR journals over time, while Sect. 1.3 covers most of the above OR methods
and others (58 OR methods) and more functional areas in HE (30). Most of the
above OR methodologies and areas are included.
1.2.2 HE in OR Journals
Various OR journals have different subareas (“departments”) listed for paper

submission. The Institute for Operations Research and the Management Sciences
(INFORMS) publishes 16 refereed journals, for example, INFORMS’ journal Man-
agement Science has numerous departments for paper submissions: Accounting,
Big Data Analytics, Business Strategy, Decision Analysis, Entrepreneurship and
Innovation, Finance, Healthcare Management, Information Systems, Marketing,
Operations Management, Optimization, Organizations, Revenue Management and
Market Analytics, and Stochastic Models and Simulation. The other main journal
of INFORMS, Operations Research’s submission departments include Decision
Analysis, Environment, Energy and Sustainability, Financial Engineering, Machine
Learning and Data Science, Revenue Management and Market Analytics, Military
and Homeland Security, Operations and Supply Chains, Optimization, OR Practice,
Policy Modeling and Public Sector OR, Simulation, Stochastic Models, and Trans-
portation.
Table 1.1 summarizes the number of articles with HE mentioned in their title
in 40 OR journals from the WoS during the period 1950–2019 in comparison to
the total number of articles. The journals are grouped into five subgroups according
to the range of years when they began publishing. In addition, the same data for
the period 2000–2019 is given. A detailed list of the 40 journals with the same
data for each journal appears in Appendix 1. The largest number of journals were
established before 1980 (16). During this period of 70 years, 352 articles were
published in all 40 journals on HE according to the WoS; namely, in the titles
of these papers, at least one of the following synonyms for HE appeared: higher
education, tertiary education, university, college, or academic. The total number of
articles in the journals is 114,698. Thus, HE articles are a mere 0.3%. The journals
that began before 1980 published more than half of the total number of articles
(71,620) and more than half of the HE application articles (194). In the last 20 years,
8
Table 1.1 Number of OR/MS journals opened over time from a selected 40 OR/MS journalsa in the WoS and number of their articles devoted to HEb over
time
Number of
Number of articles journals with no
published in all Percent of HEb Number of articles HEb articles
Year journal Number of journalsa during articles from published in all HEb articles % during
began journals opened 1950–2019 total journalsa 2000–2019 from total 1980–2019
Total Applied to HEb Total Applied to HE
1950–1979 16 71,620 194 0.27 42,397 101 0.24 5
1980–1989 8 18,218 105 0.58 13,279 66 0.50 2
1990–1999 10 22,328 50 0.22 20,336 41 0.20 4
2000–2009 6 2532 3 0.12 2532 3 0.12 3
2010–2019 0 – – – – – – 0
Total 40 114,698 352 0.31% 78,544 211 0.27% 18
a Source: summary from Appendix 1, a full list of the 40 journals, and the data of each is in Appendix 1—each is from the WoS (assessed during Aug. 2020).
The bold italics figures are the maximal values of each column. The bold figures in the last row indicates the total for each column
b The search HE appearance was done only for article titles. HE synonyms used were: higher education, academic, college, university, tertiary education
Z. Sinuany-Stern
more than half of the total articles in relation to the first 50 years (1950–1999)—
78,544 articles were published. Similarly, the last 20 years had more than half of the
HE applications (211 articles).
As shown in Appendix 1, looking into the number of HE articles in OR journals
during 2000–2019, the Pareto rule of 20/80 applies: 20% of the journals (8 out of
40) contain 82% of the articles published on HE (in column 6 of Appendix 1, the
sum of the articles in the eight leading journals is 173 out of 211 articles during
2000–2019)—each of these eight journals had more than ten articles over the last
20 years with HE articles. In 17 journals, no HE article was found, in 7, there was
only one HE article in each, and in 4 journals, only two articles were found in each.
The leading eight journals with at least 10 HE articles in their titles were: Interfaces
(with 41 articles), European Journal of Operational Research (30 articles), Expert
Systems with Applications (28 articles), Journal of Operational Research Society (20
articles), Management Science (18 articles), Omega (13 articles), Decision Support
Systems (12 articles), and Annals of Operations Research (11 articles).
Indeed, the search performed here was limited to articles where the appearance
of HE and its synonyms is only in the title of the article. Very likely, there are more
HE applications that our search words did not cover. Nevertheless, it is obvious that
OR journals give little attention to HE applications. While our count is a proxy, it
is representative. Looking in Google Scholar, for example, at “health” applications,
using the synonyms: health, hospital, clinic, or medical in the article titles of the
journal Interfaces alone during the years 2000–2019, 289 articles were found—
many more than the number of articles with HE in their title in all the above 40
journals combined (211).
Note that although INFORM dedicates a specific journal for OR education titled:
INFORM Transactions on Education, the journal is focused on the teaching aspects
of OR in HE, not on the application of OR methods in HE; thus, this journal
is not included in the list of the 40 OR journals. Indeed, this journal is devoted
completely to the pedagogy of OR/MS, as claimed. However, it is not focused on
applying OR/MS in HEIs; in searching in this educational journal during 2000–
2019, no article was found with the words “higher education” (or its synonyms)
with any of the following OR methodologies: linear programming, timetabling,
analytics, data mining, information systems, regression, data envelopment analysis,
analytical hierarchical process, balanced scorecard, faculty, planning, budgeting, or
forecasting (for each methodology, its conjugations were also used in the search).
The term HE (or a synonym) appeared only eight times in the context in which
the pedagogical approach was tested or suggested. Only once appeared testing for
university hospital operations and budget (Hicklin et al., 2017)—again, a health
application rather than a university one.
In summary, over the 70 years since the appearance of the first OR journal, only
352 articles have been published in the 40 journals selected with HE in their title
(or its synonyms), which is 0.31% of the total number of articles in these journals.
Most of the journals have published one or no articles on HE in the past 20 years.
The journal with the maximal number of articles with HE mentioned in the title is
Interfaces, which is devoted to OR applications.
10 Z. Sinuany-Stern
The next section extends the search to count HE applications in all WoS journals
(not necessarily OR journals) by OR method and by functional area.
1.3 OR Methods and Functional Areas in HE: Articles in the

WoS
The OR methods solicited for this section are taken from: a. tables of contents
of leading introductory OR textbooks, and b. major general OR journals, as noted
above, which are relevant to HE, as listed in Appendix 2, including 58 OR methods
(e.g., linear programming and integer programming), clustered into ten main groups,
as summarized in the upper part of Table 1.2 (details of each group are in Appendix
2). Moreover, 30 functional areas are listed in Appendix 3, where OR methods are
applied in HE (such as transportation, marketing, and efficiency) that are clustered
into four main groups, as summarized in the second part of Table 1.2. A search was
done in all WoS journals in all its disciplines since OR is a multidisciplinary field
that draws methods from other disciplines such as optimization (from mathematics),
simulation (from industrial engineering), multivariate statistics (from statistics), and
expert systems (from computer science). Moreover, OR methods are applied in
many other disciplines such as engineering, economics, and business. The count
of the number of articles with OR methods and functional areas applied to HE
was done either from the title of each article or from its topic (“topic” includes:
keywords, abstract, and title). The HE search words (synonyms) are at the bottom
of Table 1.1b.
1.3.1 OR Methods in HE: Articles Frequency
Table 1.2 and Appendix 2 display the number of articles counted from the WoS
during 2000–2019 by OR methods. There are five columns as follows:
Column (1) counts for each OR method, its appearance in the article topic in general.
Column (2) counts for each OR method, its appearance in the topic of articles with
HE in their title.
Column (3) counts for each OR method, its appearance in the title of articles with
HE in their topic.
Column (4) counts for each OR method, its appearance in the title of articles with
HE also in their title.
Column (5) is the percentage of column (4) from column (1).
Table 1.2 The number of articles by main OR groups of methods and groups of functional areas in HE during 2000–2019
Number of articles with:
(7) Number of
(6) Specific articles for
method with method with
maximal # of maximal # of
(2) Method in (3) Method in (5) Percentage of articles from the articles in the
(1) Method in topicb & HE in title & HE in (4) Method in column (4) from group for column group for column
Method name topic title topic title & HE in title column (1) (4) (4)
A. Group of OR methods
1. Operations 236,072 1884 1941 435 0.184 Analytics 352
research
2. Decision 150,236 981 1383 318 0.212 AHP 156
analysis
3. 283,038 656 791 219 0.077 DEA 159
Mathematical
programming
4. Discrete 38,722 66 75 19 0.049 Integer program. 18
optimization
5. Stochastic 424,449 219 807 48 0.011 Stochastic 21
6. Forecasting 328,003 519 1644 112 0.034 Forecasting 81
7. Simulation 2,555,896 1852 7369 525 0.021 Simulation 465
8. Multivariate 1,340,733 11,315 1361 198 0.015 Regression 137
statistics
9. Heuristics 277,468 528 829 106 0.038 Heuristics 50
10. Expert 324,151 2601 3553 647 0.200 Information systems 284
1 Operations Research and Management Science in Higher Education: An Overview
systems
Total A 5,958,768 20,621 19,753 2627 0.045 1723
(continued)
11
12
Table 1.2 (continued)

Number of articles with:
(7) Number of
(6) Specific area articles for area
with maximal # with maximal #
(5) Percentage of of articles from of articles in the
(2) Area in topicb (3) Area in title (4) Area in title column (4) from the group for group for column
Area name (1) Area in topic & HE in title & HE in topic & HE in title column (1) column (4) (4)
B. Group of functional areas
1. Transportation 3,753,957 14,852 16,377 3312 0.088 Networks 2582
2. Industry 1,700,795 13,935 12,072 3357 0.197 Industry 2474
3. R&D 239,074 7432 8136 3315 1.387 Entrepreneur 2255
4. Bussiness 7,599,208 52,658 40,604 9807 0.129 Internationalization 4253
Total B 13,293,034 88,877 77,189 19,791 0.149 11,564
a Theseare groups of methods and functional areas. Detailed methods and areas are in Appendices 2 and 3
b Topicis keywords or abstract or title (source: summaries from Appendices 2 and 3)
The bold italics figures are the maximal values of each column. The bold figures in the lust row indicates the total for each column
Z. Sinuany-Stern
1.3.1.1 Groups of OR Methods
The first part of Table 1.2 presents ten groups of OR methods (as detailed in
Appendix 2). Table 1.2 is formed as the sum of the specific methods of each group
taken from Appendix 2. The title of each group is as the first method in the group
(e.g., the group simulation is as the first method in the group). The three leading
groups of methods (with the maximal number of articles in each column) are:
(a) Simulation, which leads in general (2,555,896 articles in Column 1), and for
method in the title and HE in the topic (7369 articles in Column 3).
(b) Multivariate statistics, which leads for method in the topic and HE in the title
(11,315 articles in Column 2).
(c) Expert system, which leads for method in the title and HE in the title (647
articles in Column 4).
Column 5 of Table 1.2 shows that decision analysis leads in percent of the method
and HE in title from the general number of the articles with the method (0.212% of
Column 4 from Column 1).
Columns 6 and 7 indicate the specific method in each group of methods that had
the maximal number of articles with the method and HE in their titles—simulation
specific method in HE leads (with 465 articles with simulation and HE in their
title). We consider Column (4) since the articles in this column are more certain to
be devoted to OR in HE than the other columns because both the OR method and
HE appear in their titles.
1.3.1.2 Specific OR Methods
The previous Sect. 1.3.1.1, deals with groups of OR methods (Table 1.2), whereas
this section deals with specific OR methods from Appendix 2. According to this
appendix, the two leading specific OR methods (with the maximal number of
articles by column) are:
(a) Simulation in general (2,269,462 articles in Column 1), and for method in the
title and HE in the topic (6655 articles in Column 3), and for method and HE in
article titles (465 articles in Column 4); and.
(b) Regression analysis for method in the topic and HE in the title (8088 articles in
Column 2).
Column (5) of Appendix 2 shows that scientific management leads in percent
of the method and HE in title from the general number of the articles with this
method (0.873% of Column 4 from Column 1). However, scientific management is
the least used term for operations research. Moreover, most synonyms of OR are
not used often, rather the specific OR method appears in the title or topic. The new
buzzword, “analytics,” used for OR and in other contexts as computer science, is
the most common of all OR synonyms over the last 20 years. From the specific OR
14 Z. Sinuany-Stern
methods, DEA has the next highest percentage of articles in which HE and method
appear in the title (0.79% of Column 4 from Column 1).
Overall, the eleven leading specific OR methods, out of 58 in HE, in Columns
2, 3, and 4 are: simulation, regression (including logistic), analytics, information
systems, data mining, big data, AHP, forecasting, DEA, and DSS. The Pareto 20/80
rule holds here, as less than 20% of the methods (18.96%) comprise more than 80%
of the articles in Column (2) of Appendix 2 (85.72%). Together, these methods make
up most of the articles (over 72%) in all three columns of HE in Appendix 2.
Column (2) is the most representative since HE in the title means that the articles
are most likely applied to HE, and the method is likely to appear in the topic (title,
keywords, or abstract). Since many HE users (administrators) are not quantitatively
oriented, the functional area where the method is used is more attractive to them.
Indeed, in the next sections dealing with functional areas, there are many more
articles than OR methods.
1.3.2 Functional Areas in HE: Articles Frequency
Table 1.2 and Appendix 3 display the number of articles in the WoS during 2000–
2019 by functional area. There are five columns as follows:
Column (1) shows for each functional area its appearance in the article topic (“topic”
includes title, abstract, and keywords) in general.
Column (2) shows for each functional area its appearance in the topic of articles
with HE in their title.
Column (3) shows for each functional area its appearance in the title of articles with
HE in their topic.
Column (4) shows for each functional area its appearance in the title of articles with
HE also in their title.
Column (5) is the percentage of column (4) from column (1).
1.3.2.1 Groups of Functional Areas
There are 30 functional areas clustered into four groups. The second part of Table
1.2 presents the four groups of functional areas: transportation, industry, R&D,
and business (as detailed in Appendix 3). For example, the group transportation
is composed of eight specific areas, of which networks has the largest number
of articles from all transportation specific areas for each of the five columns (as
indicated in Appendix 3, and Table 1.2). The title of each group is mostly as one
of the areas in the group (e.g., the group transportation is the first method in the
group). The leading group of functional areas (with the maximal number of articles)
in all four columns is business.
In all the columns of Table 1.2, the total number of articles for functional areas
is much larger than the number of articles with OR methods (in Table 1.2, see that
the line of Total B versus the line of Total A is more than double in each column).
Column (5) of Table 1.2 shows that R&D leads in the percentage of the functional
area and HE in the title from the general number of articles with the functional area
in their topic (1.387% of Column 4 from Column 1)—indicating that R&D is more
central in the HE context than other functional areas in general.
Columns (6) and (7) indicate the specific functional area in each group of
methods that had the maximal number of articles with the functional area and HE
in their titles. For the leading group business, internationalization is the specific
functional area in HE that leads (4253 articles with internationalization and HE in
their title).
1.3.2.2 Specific Functional Areas
In Appendix 3, the three leading specific functional areas (with the maximal number
of articles for each column) are:
(a) Strategy for the functional area as the topic and HE in the title (16,261 articles
in Column 2, and 12,151 articles in Column 3).
(b) Internationalization for functional area and HE in the title (4253 articles in
Column 4).
(c) Efficiency in general (2,643,368 articles in Column 1).
Overall, the eight leading functional areas in HE in all three Columns 2, 3,
and 4 (in order of Column 3) are strategy, networks, internationalization, industry,
marketing, efficiency, entrepreneurship, and finance.
Column (5) of Appendix 3 shows that timetabling leads in the percentage of the
functional area and HE in the title from the general number of articles with the
functional area as their topic (11.56% of Column 4 from Column 1)—indicating
that timetabling is more central in the HE context than in general—often applied for
the course or examination timetabling.
1.3.3 Group of OR Methods by Group of Functional Areas
Not all articles with HE and functional areas include an OR method, thus, we
looked at those that mention both: OR method and area. Table 1.3 presents the ten
OR groups of methods (appearing in the title) by four groups of functional areas
(appearing in the topic) that are applied to HE (in the title). The leading OR group
of methods is: mathematical programming by business as the functional area (with
213 articles), where the total number of articles of the leading group of methods
is mathematical programming with 298, and the number of articles of the leading
group of functional areas is business with 606 articles, as highlighted in Table 1.3.
16 Z. Sinuany-Stern
Table 1.3 Number of articles with OR groups of methodsa by groups of functional areasb in HEa
during 2000–2019
Areas of application
Transportation Industry R&D Business Total % from total
OR group of methods
1. Operations research 7 20 2 42 71 6.1
2. Decision analysis 40 29 12 73 154 13.1
3. Mathematical programming 40 30 15 213 298 25.5
4. Discrete optimization 12 0 0 6 18 1.5
5. Stochastic 2 3 1 11 17 1.5
6. Forecasting 12 5 3 20 40 3.4
7. Simulation 52 33 10 86 181 15.5
8. Multivariate statistics 26 15 4 31 76 6.5
9. Heuristics 42 2 0 17 61 5.2
10. Expert systems 103 35 25 117 280 21.7
Total 336 172 72 616 1196 100
% from total 28.1 14.4 6.0 51.5 100
a The bold figures indicate the totals of each row and each column. The bolditalic figure within the
table indicate the maximal value, and its row and column’s totals
b Appears in the topic (assessed from WoS during Sept. 2020)
1.3.4 Specific OR Methods by Specific Functional Areas
Table 1.4 provides a deeper look at 12 selected specific OR methods in HE (out

of 58 in Appendix 2) by eight selected specific functional areas in HE (out of
30 in Appendix 3). Note that in Table 1.4, the method appears in the topic (title,
keywords, or abstract) and HE, and the specific functional area appears in the title
of the articles. First, the group of methods with at least 50 articles in Table 1.3 was
selected. Thus, from 10 groups of methods, only seven were considered (1, 2, 3, 7,
8, 9, and 10). Each of these seven groups has at least 50 articles. From the selected
groups, the methods—in total 12 such specific methods were selected—with the
largest numbers in Column (2) of Appendix 2 (as listed in Table 1.4). Similarly, for
the group of functional areas, the two largest were selected from Table 1.3 (with
more than 200 articles). The selection of the specific areas is based on Appendix 3.
Looking into the 12 leading specific OR methods in Table 1.4, the table is not
as consistent as Table 1.3, since the entry with the maximal number of articles by
method and functional DEA by efficiency (with 170 articles) is not the intersection
of the OR method with the maximal number of articles (regression with 341 articles)
or the functional area with the maximal number of articles (network with 269
articles). Thus, as highlighted in Table 1.4, we point at three leading methods:
DEA by efficiency is the leading combination of specific OR methods and specific
functional areas (with 170 articles). This is consistent with the leading combination
in Table 1.3, where mathematical programming by business leads, as DEA is
a specific method of mathematical programming and efficiency is a sub-area of
Table 1.4 Number of articles in major OR/MS-specific methodsa by specific functional area in HE during 2000–2019
Title
Network Time tabling Marketing Finance Strategy Efficiency Accounting International Total
Topic
Analytics 15 0 3 3 29 5 3 15 73
DSS 5 7 1 2 5 5 0 2 27
AHP/ANP 8 0 1 2 5 6 0 2 24
DEA 14 0 3 3 2 170 1 2 195
Simulation 66 6 11 10 23 20 6 14 156
Regression 64 0 26 51 106 17 23 54 341
Logistic 25 0 9 25 27 1 5 5 97
Heuristics 3 48 0 1 6 0 0 0 58
Genetic Alg. 8 42 1 0 3 2 0 1 57
Big data 16 0 7 11 14 1 3 3 55
Info. Systems 30 0 4 14 12 5 6 7 78
Data mining 20 1 5 5 7 2 0 2 42
Total 274 104 71 127 239 234 47 107 1203
a Note that OR methods appear here in the topic (title, keyword, or abstract) of the articles, and HE and the area appear in the title of the articles, while in Table
1.3, the method appears in the title. The search was done in the WoS (assessed from the Web of Science during Aug. 2020). The bold figures indicate the
maximal value of the columns’ totals and the maximal value of the rows’ totals. The bold italics figures indicate the largest number in the table (not including
the the column and row of totals) and its row and column total
17
18 Z. Sinuany-Stern
business. Moreover, DEA is composed of many linear programming (LP) problems,

where LP is also a specific mathematical programming and optimization method—
see the review by De la Torre et al. (2016) on the literature review of DEA in HE in
the context of efficiency.
1.4 Issues in the HE Sector
This section presents examples of major issues in HE where OR methods can be

applied.
1.4.1 Is HE a Product?
There is considerable debate in the HE literature about whether students are

customers or consumers, or both, in their role as participants in higher education.
Barrett (1996) published a warning on students as consumers, and Baldwin and
James (2000) wrote about the concept of students as informed customers. Some use
course descriptions in universities as “products” (Conway et al., 1994). Although
others also argue that participation in higher education is not strictly a product
and put forward a well-argued case for HE as a quasi-commercial service industry
(Brookes, 2003). Passionate, articulate arguments were also put forward against the
marketization of HE and business terminology, increasingly adopted and accepted
by some educationalists. Those who oppose the emergence and imposition of the
marketization of higher education argue strongly that business values have no place
in higher education, because such values are ethically in conflict with educational
values (Gibbs, 2018). Nevertheless, in 2006, Hemsley-Brown and Oplatka surveyed
63 articles on HE marketing, during 1992–2005, and again, in 2021 Oplatka and
Hemsley-Brown (2021) reviewed 105 such articles, for the period 2005–2019,
indicating that HE marketing has grown greatly. They report that their 2006 review
was cited 1000 times (ibid.). Obviously by now, the use of marketing has become
one of the main tools for recruiting students as part of globalization in HE.
At the turn of the millennium, it became evident that HE is an industry, and an
important one (Goldin & Katz, 1999), with all the parameters of the industry: selling
products (courses, degrees at various levels and in different professions, research,
and knowledge), in large quantities (massification of HE), with import (globaliza-
tion of HE) and export (international students), and marketing (e.g., advertisements
and ranking). HE has a price (tuition), quality assurance (accreditation), research
and development (R&D) contracts with industry, and entrepreneurial initiatives (see
Sect. 1.4.2 below). The size of HE in the industry was estimated by Altbach et al.
(2017) at about 200 million students in 22,000 HE Institutions (HEIs) worldwide.
In many developed countries, over 50% of the 18–24 age group are students. The
commodification of HE is here, and OR/MS methods are applicable, as shown in

the previous sections.
1.4.2 University–Industry Relations
STEM (Science, Technology, Engineering, and Mathematics) graduates/students

who acquired knowledge from HEIs are at the core of entrepreneurial successes
in the industry. Universities and faculty have strong relations with industry R&D, in
the form of knowledge transfer, patents, incubators, and industrial parks (Etzkowitz,
2001). Etzkowitz and others developed the “triple Helix interaction” of:
1. University, which creates basic knowledge
2. Industry, which produces commercial goods
3. Government, which regulates the markets, and provides some public infrastruc-
ture, and support.
Etzkowitz and Zhou (2018) present various models of cooperation between these
three sectors—including evolving first, two out of three cooperation’s, and then
cooperation of all three. They point at the two famous successful examples: Silicon
Valley in California and the Boston area in the USA. Furthermore, often, coopera-
tion of two of the three elements creates intermediary institutions, such as “science
park” (ibid.). The European Union’s programs, such as, Horizon 2020, is another
example that combines university-industry scientific cooperation across countries,
see: https://ec.europa.eu/programmes/horizon2020/sites/horizon2020/files/H2020_
inBrief_EN_FinalBAT.pdf
Tole and Czarnitzki (2010) were concerned that—commercializing science via
academic spinoffs—may cause a “brain drain” from academia.
To estimate the use of OR methods in the context of industry, Table 1.2 shows the
grouping of ten OR groups of methods and four groups of functional areas in HE.
The industry group is composed of five specific areas, as shown in Appendix 3. The
total number of articles of the industry group applied to HE is 12,072 (in column 3
of Table 1.2); this high number points at the centrality of the industry area in HE, in
the last 20 years. However, as shown in Table 1.3, out of these 12,072 articles, the
total number of articles with OR method in their title is only 172—a low number
(only 1.425%).
Similarly, the R&D group of areas, as shown in Appendix 3, is composed of six
specific areas: transfer of knowledge, research and development, patents, industrial
parks, incubators, and entrepreneurship. The total number of articles of the R&D
group in HE is 8136 (in column 3 of Table 1.2), this high number points at the
centrality of R&D in HE, over the last 20 years. However, as shown in Table 1.3,
the total number of articles with OR methods in the R&D group is only 72 articles
out of the above, 8136 articles—a very low number (only 0.885%). Thus, there is a
great potential for applying OR methods to the above six areas of R&D and industry.
20 Z. Sinuany-Stern
1.4.3 Student Debt
Student debt is a major issue in the USA, to be resolved at the national and state
levels. Many students are not able to repay their loans; one of the main factors
is the decline in the income premium for graduates compared to non-graduates
[see, for example, the documentary film Ivory Tower [https://en.wikipedia.org/wiki/
Ivory_Tower_(2014_film) directed by Rossi, assessed on Sept. 2020]. Over 40
million borrowers in the US owe more than $1.5 trillion in loans. Krishnan and
Wang (2019) show that student debt is negatively related to the propensity to start
a firm, which affects the economy negatively; thus, the impact of student debt has
additional ramifications. Although there are doubts, about the power of planning to
resolve such issues (Thelin, 2018), we believe that without immediate and long-run
planning, the problem may intensify.
1.4.4 Internationalization in HE
De Wit and Altbach (2020) review the development of internationalization in HE

during the past 50 years—from a minor activity to a central strategically important
issue in HE, worldwide. Internationalization fosters the global knowledge economy.
Research intensive universities play a central role in internationalization—the
exchange, import, and export of researchers, students, and knowledge. In ranking
universities, internationalization is one of the central factors. Many universities
even opened branches abroad, as a strategy. Online programs and courses such
as MOOCs (massive open online courses) are available worldwide via advanced
ICT (information and communication technology), which widen the boundaries of
internationalization, and reduces its costs—the concept of internationalization at
home is growing.
Interestingly, Altbach and de Wit (2018) questioned the continuation of interna-
tionalism in HE, due to issues of academic freedom, nationalist–populist arguments,
ethics, the role of English, branch campuses, and anti-immigration (e.g., Brexit).
Unfortunately, these predictions of the drastic reduction in internationalization
materialized, completely from another reason—the COVID-19 pandemic, as they
admit later (De Wit & Altbach, 2020).
As shown in Appendix 3 internationalization functional area in HE had 4253
articles during 2000–2019—the maximal number of articles in column (4). But
in Table 1.4, out of 4253 articles, internationalization had only 106 papers with
applications of OR methods (about half of them used regression model). Thus, there
is a great potential for using OR methods for internationalization in HE.
1.4.5 EduTech and the Fourth Industrial Revolution
HE follows the development of technology, economy, political, and social processes.

More specifically, the materialization of these processes is expressed in what
we may call EduTech (educational technology), its expression is in: E-learning,
MOOCs, ICT, educational simulators, simulations, multimedia, PowerPoint, com-
puter educational games (e.g., business game), gamification, virtual library, virtual
laboratory, web-based educational technologies, big data, analytics, etc. The term
EduTech has been used mainly in the industry that produces technological educa-
tional tools. We use it in the context of these various tools and methods for HE in all
academic disciplines and in the administration of HE (see also, Mosteanu, 2020).
We predict that more and more HEIs will have to establish EduTech units to support
faculty and students, in the use of all these technological methods, just like computer
services today—some already exist, but not enough (especially for social sciences
and humanities users).
Every industrial revolution (IR) has required more and more skilled manpower.
Schwab and Davis (2018) discuss the fourth IR (also termed Industry 4.0) and its
implication on the economy and society. (See Table 1.5, which summarizes the
four IR eras in simple general terms, employing the main emerging technologies,
along with new academic disciplines, in each IR era.) Gleason (2018) addresses the
effect of the fourth IR on various aspects of HE: liberal arts, integration of STEM
innovations, changing role of libraries in the digital age, global trends in youth
Table 1.5 Evolvement of the four Industrial Revolution (IR) and HE

Eraa Main emerging technologies Main new academic discipline evolved
Pre IR Raw materials, man & animal Humanities, Sciences, Civil
(pre-1770) power, wheels, tools Engineering, Medicine, Arts
First IR Machines, steam, waterpower, Mechanical Engineering,
(1770–1880) trains Social Sciences
Second IR Electricity, radio, telephones, Electrical Engineering.,
(1880–1940) production lines, advanced Industrialb Engineering,
materials Managementb , Material
Engineering
Third IR Electronics, television, Nuclear Engineering,
(1940–1995) computers, information. Aeronautical Engineering,
Systems, spaceships, artificial Electronic Engineering,
intelligence OR/MSb , Computer Science
Fourth IR Smart robotics, biotechnology, Robotics (Smart Sys.), Big
(1995+) cyber technology, cellular Data Sustainable
phones, the internet, ICTc , big Environment, Analyticsb ,
data EduTechc
a The exact years can never be accurate due to the overlap between eras; moreover, there are some
variations among countries—the leading countries are North America and Europe
b Often OR/MS is part of Management or Industrial Engineering, even called Analytics
c EduTech is educational technology, the use of technology in teaching and learning
22 Z. Sinuany-Stern
mobility, development of lifelong learning programs, E-learning, and working from

home (also teaching and research).
Already, in 1997, Slaughter and Leslie (1997) wrote about academic capital-
ism and the entrepreneurial university—highlighting such issues as globalization,
financial-driven academic policies, technology transfer strategies, and intellectual
property (see Sect. 1.4.2 above). They claim that: “As the industrial revolution at
the end of the nineteenth century created the wealth that provided the base for
postsecondary education and attendant professionalization, so the globalization of
the political economy at the end of the twentieth century is destabilizing patterns of
university professional work developed over the past hundred years. Globalization
is creating new structures, incentives, and rewards for some aspects of academic
careers and is simultaneously instituting constraints and disincentives for other
careers.”
Etzkowitz (2001) wrote about the second academic revolution and the rise
of entrepreneurial science, in which began: the entrepreneurial involvement of
universities in industry, incubators, research parks, university spinouts, intellectual
property, licensing strategies, and entrepreneurial faculty (see also Etzkowitz &
Zhou, 2018). The third and fourth IR require workers with HE, which is the
justification for the massification of HE, since the 1950s. This is the period when OR
developed as an academic discipline—for improving civil organizations in business
and industry and in public organizations, such as HE.
1.4.6 The Pandemic and EduTech
The years 2020–2021 will be remembered because of the COVID-19 pandemic,

which shook up mankind’s major systems: health, economy, education, trans-
portation, culture, etc. HE was affected greatly, as campuses were closed to most
frontal-teaching activities, and online teaching took over. In the beginning, HEIs
created teaching guides and conducted online workshops to equip staff and students
in the use of various online learning platforms, such as Zoom, Skype, Moodle, and
Google Drive (Crawford et al., 2020). Teachers were trained online by information
technology teams, who also provided support as needed. As shown in Table 1.6,
in countries that reported campuses closed by the end of March 2020 (17 out of
20 countries), only three countries (including the USA) reported that only some
campuses (not all) were closed. Only eight countries reported that all HEIs moved
to online teaching, while the other 11 reported that some HEIs moved to online
teaching, but not all (ibid.). While the long-run ramifications of the pandemic on
HE delivery and organization are not yet clear, HE is currently in a new era of
ICT dominance and at a faster pace than other systems. It appears that blended
learning—defined as the combination of traditional face-to-face education and
technology-mediated instruction (Porter et al., 2014) may be the future trend in HE.
One of the potential negative impacts of the coronavirus on HE is on the mental
Table 1.6 Campus closures and the move to online teaching during COVID-19 pandemic in
selected countriesa by the end of March 2020
Type of Economy Countries Campus closure Move to online
Developed Australia, Germany, Italy, Ireland, All All
United Kingdom
United States of America Some Some
Developing Brazil, Singapore Some Some
China, Egypt, Hong Kong All All
Chile, India, Indonesia, Jordan, All Some
Nigeria, South Korea, South Africa,
United Arab Emirates
a Based on Crawford et al. (2020)
health of students and academic staff, which may require special attention on the
part of the administration (Sahu, 2020).
1.4.7 Planning in HE
Marshall (2018) wrote about using technology to catalyze change in university

learning and teaching. Almog and Almog (2020) point at many wrongs in the
university model and try to portray what will come in its place—new organization
of the learning space and more: E-learning, MOOCs, badges, vendors, gaming in
HE learning, networking, etc.
It seems that the COVID-19 pandemic is the crisis that is pushing all the above
forecasts, of various researchers (Almog & Almog, 2020; Altbach & de Wit, 2018;
Etzkowitz, 2001; Gleason, 2018; Marshall, 2018; Schwab & Davis, 2018; Slaughter
& Leslie, 1997), into a variety of combinations of new directions, as shown in the
limited survey, presented in Table 1.6. For example, internationalization declined,
in the physical sense (traveling between countries), but ICT (such as Zoom) allowed
more E-learning, which increases internationalization in the virtual sense. Teachers
who refused to use E-learning in the past are forced to do it now—something
that could happen only in a crisis. Working conditions have changed completely—
flextime has changed to working from home, which may affect the future amount
of office space required. In conclusion, planning is urgent—more than ever—in the
short term and in the long term. OR models were born during crisis (World War II)
and are urgently needed in the COVID-19 pandemic crisis—thereafter in general
(i.e., unemployment, health) and in HE.
It seems that digital literacy and ICT will be crucial for the success of all
participants in HE: lecturers, students, and administrators. For their survival, more
and more HEIs will probably agree to give credit for external courses in the
academic supermarkets that will be supplied by external providers. Special bodies
will evaluate the quality rating of these academic products, and the reliability of
24 Z. Sinuany-Stern
student evaluation systems, grades, and the number of credits (Almog & Almog,
2020). The universities need to price partial tuition for those who purchase courses
elsewhere. It is not clear whether the overall tuition for a degree will drop in
price. Students may prefer to attend universities that give them credit for previous
courses and attractively reduced tuition fees. The universities, for their part, may
gain a reduced dropout rate—as they will accept students with some academic
background from online courses. Moreover, universities may offer (sell) online or
MOOCs (Wong et al., 2019) in the virtual academic supermarket to gain visibility.
To motivate teachers to develop good products, incentives can be developed, like e-
book royalty payments (per student). In a successful situation, all may gain: reduce
tuition, dropout rate will be reduced, and student satisfaction will increase. See
further on blended learning in Porter et al. (2014).
1.5 Summary and Conclusions
This chapter provides an overview of the use of OR methods and functional areas
in HE. A search in 40 OR journals revealed that HE (or its synonyms) appears
in the title of only 0.3% of their articles, from 1950–2019; this is an astonishing
low rate, compared to other fields (e.g., health). Moreover, in 17 of the journals,
no article was found with HE in its title. In a web of science search, out of 58
OR methods listed in our study, the 11 leading methods (in number of articles)
are: simulation, regression (including logistics), analytics, information systems,
data mining, big data, AHP, forecasting, DEA, and DSS. The Pareto 20/80 rule
holds here—as less than 20% of the methods comprise about 80% of the articles.
Similarly, overall, from 30 functional areas, we identified eight leading areas in
HE: strategy, internationalization, efficiency, network, industry, entrepreneurship,
marketing, and finance. Looking at methods by functional area, DEA by efficiency
is the leader.
In summary, there is great potential for academics in OR to utilize OR models
to improve their own HE institutions in all functional areas. The main problems of
applying OR methods in HE (in other systems too) is the lack of unified information
systems at all levels of HE. Getting the data takes up most of the time and cost of
the application of OR methods. Indeed, one of the leading methods in our analysis
of OR methods in HE is information systems.
OR was born at a time of crisis, like the COVID-19 pandemic today, calling OR
specialists to take the lead once again in using the widened arsenal of OR methods
to save the HE system. HE had been in trouble during the last decade, even in the
leading country in HE—the US (see Merigo & Yang, 2017)—addressing problems
such as student debt in the US and EduTech. As Thelin (2018) claims, American
HE needs a substantive “New Deal” in purposes and policies.
For future research, various combinations of OR methods have great potential.
Mingers and Gill (1997) foster combining management science methodologies.
Along these lines, Sinuany-Stern et al. (1994) used DEA with some statistical
methods to measure the efficiency of academic departments at a university. Cooper

(1999) suggested to combine DEA with commonly used statistical methods—new
and old—to further enhance the scope of OR. One of the leading OR methods is
data mining, which combines statistical models, such as clustering, regression, and
decision-making—see reviews on educational data mining by Aldowah et al. (2019)
and by Romero and Ventura (2020).
Another direction, for future research, is to extend the search method of articles
on OR in HE and their areas of application (in Appendix 2 and 3), such as 1.
Additional OR methods (e.g., optimal control and fuzzy logic/set) 2. Additional
functional areas (e.g., enrollment and faculty/lecturer); 2. Extending the search
from the WoS to Google Scholar or other sources; 3. To perform a study among
HEIs’ administrators on the specific OR methods they actually use and their area of
application within HE.
Appendix 1: The number of articles published in 40 OR journals with HE in their title during
26
1980–2019
Number of articles published in Number of articles published in

1980–2019a 2000–2019a
Year journal
Journal title opened Total number HEb number HE % Total number HEb number HE %
Journal of the Operational Research Societya 1950 8449 58 0.686 3827 20 0.522
Journal of Optimization Theory and Applications 1967 5873 0 0 3346 0 0
Management Science 1954 5817 23 0.395 3306 18 0.544
Operations Research 1952 3842 6 0.156 1933 2 0.103
Transportation Science 1969 1594 0 0 943 0 0
International Journal of Systems Science 1970 6459 1 0.015 3362 0 0
INFOR 1963 973 19 1.952 459 9 1.960
Mathematical Programming 1971 3571 0 0 1964 0 0
Networks 1971 1969 2 0.101 1119 1 0.089
Omega 1973 2949 17 0.576 1603 13 0.810
Computers and Operations Research 1974 5496 15 0.272 4172 5 0.119
Mathematics of Operations Research 1976 2112 0 0 1062 0 0
European Journal of Operational Research 1977 18,412 49 0.266 11,916 30 0.251
RAIRO–Operations Research 1977 689 0 0 689 0 0
Transportation Research Part B–Methodological 1979 2754 1 0.036 2035 0 0
OR Spectrum 1979 661 3 0.453 661 3 0.453
Optimal Control Applications and Methods 1980 1394 1 0.071 850 0 0
Interfaces 1983 3712 76 2.047 1513 41 2.709
Annals of Operations Research 1984 4571 11 0.240 3783 11 0.290
Applied Stochastic Models in Business and Industry 1985 1122 1 0.089 1078 1 0.092
Decision Support Systems 1985 3073 14 0.455 2525 12 0.475
Z. Sinuany-Stern
(continued)
Optimization 1985 1970 0 0 1685 0 0
Queueing Systems 1986 1324 0 0 1008 0 0
Asia-Pacific Journal of Operational Research 1987 1052 2 0.190 837 1 0.119
Expert Systems with Applications 1990 12, 552 37 0.294 11, 844 28 0.236
Discrete Event Dynamic Systems Theory and Applications 1991 494 0 0 410 0 0
Journal of Global Optimization 1991 2684 0 0 2357 0 0
Computational Optimization and Applications 1992 1597 2 0.125 1476 2 0.135
TOP 1993 544 0 0 544 0 0
International Transactions in Operational Research 1994 711 2 0.281 711 2 0.281
INFORMS Journal on Computing 1996 947 2 0.211 916 2 0.218
Journal of the Operations Research Society of Japan 1996 849 1 0.117 289 1 0.346
Mathematical Methods of Operations Research 1996 1169 0 0 1008 0 0
Journal of Scheduling 1998 781 6 0.768 781 6 0.768
Fuzzy Optimization and Decision-Making 2002 313 0 0 313 0 0
SORT 2003 215 1 0.465 215 1 0.465
Discrete Optimization 2004 498 0 0 498 0 0
Central European Journal of Operations Research 2006 576 1 0.173 576 1 0.173
European Journal of Industrial Engineering 2008 401 1 0.249 401 1 0.249
Pacific Journal of Optimization 2008 529 0 0 529 0 0
Total 114, 698 352 0.306 78, 544 211 0.268
a formerlyQuarterly Journal of Operational Research
b Assessedfrom the WoS on July 2020; HE synonyms used were: higher education, academic, college, university, and tertiary education. The bold italics figures
are the maximal values of each column. The bold figures in the last row indicates the total of each column
27
28 Z. Sinuany-Stern
Appendix 2: The number of articles published in WoS

Journals with HE and OR methoda 2000–2019
Percent of
Method in Method in column
topic and title and Method (4) from
Method in HE in title HE in and HE in column
Method topic (1) (2) topic (3) title (4) (1)
1. Operations research
Operations 8013 81 94 20 0.250
research/Operational
research
Management 1865 57 51 9 0.483
science
Scientific 1146 79 14 10 0.873
management
Operations 4448 56 93 16 0.360
management
Analytic/Analytics 188, 877 1355 1538 352 0.186
Quantitative 15, 134 178 48 10 0.066
model/methods
System 16, 589 78 103 18 0.109
analysis/Systems
analysis
2. Decision analysis
Decision 16, 969 58 94 13 0.077
science/Decision
analysis/Decision
theory
Decision 67, 361 315 630 103 0.153
support/DSS
Multi-criteria 9631 62 58 9 0.093
decision
Analytic hierarchy 31, 930 404 428 156 0.489
process/AHP/ANP
PROMETHEE 1431 8 13 5 0.349
Decision tree 22, 914 134 160 32 0.140
3. Mathematical programming
Mathematical 10, 604 16 28 1 0.009
program-
ming/Optimization
Linear 66, 016 75 100 13 0.020
programming/LP
Chance constraint 576 0 0 0 0.000
(continued)
(continued)
Percent of
Data envelopment 20, 128 318 351 159 0.790
analysis/DEA
Goal 2684 19 34 8 0.298
Programming/Goal
Program
Soft constraint/Soft 1312 56 4 3 0.229
constraints
Nonlinear 16, 951 23 41 4 0.024
programming/NLP
Quadratic 115, 781 65 122 4 0.003
Separable 25, 466 12 29 0 0.000
Game theory 23, 520 72 82 27 0.115
4. Discrete Optimization
Discrete 2464 5 3 1 0.041
optimization
Integer 15, 477 53 47 18 0.116
programming
Mix integer/MIP 20, 781 8 25 0 0.000
5. Stochastic
Stochastic 243, 606 119 464 21 0.009
Markov 115, 921 52 220 14 0.012
Queue/queuing 41, 925 41 89 11 0.026
Dynamic 22, 997 7 34 2 0.009
programming
6. Forecasting
Forecasting 150, 520 328 1146 81 0.054
Time series 161, 508 178 473 30 0.019
Moving average 14, 266 8 18 0 0.000
Exponential 1709 5 7 1 0.059
smoothing
7. Simulation
Simulation 2,269,462 1632 6655 465 0.020
Industrial 329 2 1 0 0.000
dynamic/dynamics
Monte Carlo 225, 155 101 336 18 0.008
Discrete events 13, 894 20 80 4 0.029
Agent based/ABS 47, 056 97 297 38 0.081
(continued)
30 Z. Sinuany-Stern
(continued)
Percent of
8. Multivariate statistics
Multivariate 4049 12 7 1 0.025
statistics
Regression 916, 392 8088 983 137 0.015
Logistics 300, 691 2972 240 37 0.012
Principle 79, 897 105 60 9 0
component/PCA
Canonical 5491 50 20 6 0.109
correlation
Discriminant 34, 213 88 51 8 0.023
analysis
9. Heuristics
Heuristics 111, 764 299 371 50 0.045
Greedy 22, 354 14 35 5 0.022
Metaheuristics 5877 28 20 3 0.051
Tabu Search 8586 31 32 3 0.035
Simulated 19, 441 31 58 6 0.031
annealing
Genetic Algorithms 109, 446 125 313 39 0.036
10. Expert systems
Expert 15, 517 76 129 18 0.116
system/Expert
systems
Machine learning 112, 381 207 528 37 0.033
Data science 3528 45 98 15 0.425
Data mining 66, 293 606 761 4 0.006
Big data 47, 119 481 901 283 0.601
Information 73, 957 1177 1105 284 0.384
systems/
Information
management
Computational 5356 9 31 6 0.112
intelligence
Total 5, 958, 768 20, 621 19, 753 2627 0.044
a Assessed from the WoS on August 2020; HE synonyms used were: higher education, academic,
college, university, and tertiary education. The bold italics figures are the maximal values of each
column. The bold figures in the last row indicates the total of each column
Appendix 3: The number of articles published in WoS

journals with HE and functional areaa during 2000–2019
Percent of
Area in Area in column
topic & title & HE Area & (4) from
Functional Area in HE in title in topic HE in title column
area topic (1) (2) (3) (4) (1)
1. Transportation
Transportation 144, 001 474 587 98 0.068
Transshipment 1426 1 1 0 0
Assignment 139, 140 1343 666 124 0.089
Project manage- 24, 381 327 450 97 0.398
ment/CPM/Gant/
PERT
Network/Networks 2, 395, 625 8865 11, 883 2582 0.108
Location 781, 639 2159 1312 147 0.019
Scheduling 266, 136 1443 1120 78 0.029
Timetabling 1609 240 358 186 11.560
2. Industry
Industry 641, 524 6500 6169 2474 0.386
Logistics 41, 095 315 603 112 0.273
Maintenance 361, 693 1143 1075 146 0.040
Reliability 489, 641 3110 2753 300 0.061
Inventory 166, 842 2867 1472 325 0.195
3. R&D
Transfer of Knowl- 27, 078 851 1073 387 1.429
edge/Knowledge
manage-
ment/Management
of knowledge
R&D/Research and 91, 236 1567 1107 324 0.355
development
Patent/Patents 63, 285 1080 1087 267 0.422
Industrial 2157 12 20 3 0.139
park/Industrial
parks
Incubator/Incubators 6717 274 209 79 1.176
Entrepreneurship/ 48, 601 3648 4640 2255 4.640
Entrepreneurial
4. Bussiness/management
Marketing 626, 982 6226 5345 1339 0.214
Finance/Financial 318, 978 5048 3505 1146 0.359
Strategy 1, 880, 330 16,261 12,151 1261 0.067
Balanced 6976 189 160 70 1.003
scorecard/BSC
(continued)
32 Z. Sinuany-Stern
(continued)
Percent of
Area in Area in column
topic & title & HE Area & (4) from
Functional Area in HE in title in topic HE in title column
area topic (1) (2) (3) (4) (1)
Efficiency/Efficient 2,643,368 5862 4644 804 0.030
Accounting 1, 196, 791 6642 2388 655 0.055
Investment 206, 858 1698 1057 216 0.104
Revenue 2087 6 31 4 0.192
management
Policy 2619 11 6 2 0.076
modeling/policy
model/Policy
models
Renovation 8184 176 116 57 0.696
International/ 706, 035 10, 539 11, 201 4253 0.602
Internationalization
Total 13, 293, 034 88, 877 77, 189 19, 791 0
a Assessed from the WoS in August 2020. HE synonyms used were: higher education, academic,
college, university, and tertiary education. The bold italics figures are the maximal values of each
column. The bold figures in the last row indicates the total of each column
References
Aldowah, H., Al-Samarraie, H., & Fauzy, W.-M. (2019). Educational data mining and learning
analytics for 21st century higher education: A review and synthesis. Telematics and Informatics,
37, 13–49.
Almog, T., & Almog, O. (2020). Academia—All the lies: What went wrong in the University Model
and What will come in its Place. Haifa University Publishing.
Altbach, P., & de Wit, H. (2018). Are we facing a fundamental challenge to higher education
internationalization? International Higher Education, 2(93), 2–4.
Altbach, P. G., Reisberg, L., & de Wit, H. (2017). Responding to massification: Differentiation in
Post-secondary education worldwide. Sense Publishers.
Baldwin, G., & James, R. (2000). The market in Australian higher education and the concept of
student as informed consumer. Journal of Higher Education Policy and Management, 22(2),
139–148.
Barrett, L. R. (1996). On students as customers—Some warnings from America. Higher Education
Review, 28(3), 70–71.
Bell, G., Warwick, J., & Galbraith, P. (2012). Higher education management and operational
research: Demonstrating new practices and metaphor. In M. Peters (Ed.), Educational Futures
Rethinking Theory and Practice (Vol. 54). Sense Publication.
Brookes, M. (2003). Higher education: Marketing in a quasi-commercial service industry. Interna-
tional Journal of Non-profit and Voluntary Sector Marketing, 8(2), 134–142.
Cheng, T. C. E. (1993). Operations research and higher education administration. Journal of
Educational Administration, 31(1), 77–91.
Churchman, C. W., Ackoff, R. L., & Arnoff, E. L. (1957). Introduction to Operations Research.
Wiley.
Cohen, M., & March, J. (1986). Leadership and Ambiguity, the American College President (2nd
ed.). Harvard Business School Press.
Conway, T., Mackay, S., & Yorke, D. (1994). Strategic planning in higher education: who are the
customers? International. Journal of Educational Management, 8(6), 29–36.
Cooper, W. W. (1999). Operational research/Management Science: Where it’s been. Where it
should be going. Journal of the Operational Research Society, 50, 3–11.
Crawford, J., Butler-Henderson, K., Rudolph, J., Malkawi, B., Glowatz, M., Burton, R., Magni, P.
A., & Lam, S. (2020). COVID-19: 20 countries higher education intra-period digital pedagogy
responses. Journal of Applied Learning & Teaching, 3(1), 1–20.
De la Torre, E. M., Sagarra, M., & Agasisti, T. (2016). Assessing organizations’ efficiency adopting
complementary perspectives: An empirical analysis through data envelopment analysis and
multidimensional scaling, with an application to higher education. In S.-N. Hwang, H.-S.
Lee, & J. Zhu (Eds.), Handbook of operations analytics using data envelopment analysis
(International Series in Operations Research & Management Science) (pp. 145–166). Chief
Ed. Zhu J.
De Wit, H., & Altbach, P. (2020). Internationalization in higher education: Global trends and
recommendations for its future. Policy Reviews in Higher Education, 5(1), 28–46. https://
doi.org/10.1080/23322969.2020.1820898
Editorial notes. (1950). Journal of Operational Research Society, 1(1), 1–2.
Etzkowitz, H. (2001). The second academic revolution and the rise of entrepreneurial science.
IEEE Technology and Society Magazine, 20(2), 18–29.
Etzkowitz, H., & Zhou, C. (2018). The Triple Helix: University-Industry-Government Innovation
and Entrepreneurship (2nd ed.). Routledge.
Gass, S. I., & Fu, M. C. (Eds.). (2013). Encyclopedia of operations research and management
science. Springer.
Gibbs, P. (2018). Higher education marketing—Does inducing anxiety facilitate critical thinking
or more consumerism? Journal of Marketing for Higher Education, 28(1), 1–11.
Gleason, N. W. (2018). Higher education in the era of the fourth industrial revolution. Palgrave
Macmillam. https://doi.org/10.1007/978-981-13-0194-0_1
Goldin, C., & Katz, L. F. (1999). The shaping of higher education: The formative years in the
United States, 1890 to 1940. Journal of Economic Perspectives, 13(1), 37–62.
Hartcup, G. (2000). Birth of a new Science: Operational research. In G. Hartcup & B. Lovell (Eds.),
The effect of Science on the Second World War (pp. 100–121). Palgrave Macmillan.
Hemsley-Brown, J., & Oplatka, I. (2006). Universities in a competitive global marketplace: A
systematic review of the literature on higher education marketing. International Journal of
Public Sector Management, 19(4), 316–338.
Hicklin, K., Ivy, J. S., & Vila-Parrish, A. R. (2017). A prescription for budget Woes at Gracious
University Hospital. INFORMS Transactions on Education, 17(3), 107–109.
Hillier, F. S., & Lieberman, G. J. (2014). Introduction to operations research (9th ed.). McGraw-
Hill.
Horizon. (2020). Retrieved August 2020, from https://ec.europa.eu/programmes/horizon2020/
sites/.horizon2020/files/H2020_inBrief_EN_FinalBAT.pdf
Johnes, J. (2015). Operational research in education. European Journal of Operational Research,
Operational Research in education, 243(3), 683–696.
Krishnan, K., & Wang, P. (2019). The cost of financing education: Can student debt Hinder
Entrepreneurship? Management Science, 65(10) https://doi.org/10.1287/mnsc.2017.2995
Marshall, S. J. (2018). Shaping the University of the Future: Using Technology to Catalyse Change
in University Learning and Teaching. Springer.
Merigo, J. M., & Yang, J.-B. A. (2017). Bibliometric Analysis of Operations Research and
Management Science. Omega: The International Journal of Management Science, 73(c), 37–
48.
Mingers, J., & Gill, A. (Eds.). (1997). Multi Methodology: The theory and practice of combining
management science methodologies. Wiley.
Mosteanu, N. R. (2020). Using internet and EduTech become a primary need rather than a luxury—
The reality: A new skilled educational system—Digital university campus. International
Journal of Engineering Technologies, 4(6), 1–9.
34 Z. Sinuany-Stern
Oplatka, I., & Hemsley-Brown, J. (2021). A systematic and updated review of the literature on
higher education marketing 2005–2019. In Z. Sinuany-Stern (Ed.), Handbook of Operations
Research and Management Science in Higher Education. Springer.
Porter, W. W., Graham, C. R., Spring, K. A., & Welch, K. R. (2014). Blended learning in higher
education: Institutional adoption and implementation. Computer and Education, 75, 185–195.
https://doi.org/10.1016/j.compedu.2014.02.011
Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated sur-
vey. WIREs Data Mining and Knowledge Discovery, 10(3). https://doi.org/10.1002/widm.1355
Sahu, P. (2020). Closure of Universities Due to Coronavirus Disease 2019 (COVID-19): Impact
on education and mental health of students and academic staff. Cureus, 12(4), e7541. https://
doi.org/10.7759/cureus.7541
Schierenbeck, C. (2013). Fixing Higher Education: A Business Manager’s Take on How to Boost
Productivity in Higher Education. Springer.
Schroeder, R. G. (1973). A survey of management science in university operations. Management
Science, 19(8), 841–971.
Schwab, K., & Davis, N. (2018). Shaping the future of the fourth industrial revolution: A Guide to
Building a Better World. World Economic Forum, Currency.
Sinuany-Stern, Z., Mehrez, A., & Barboy, A. (1994). Academic departments’ efficiency via DEA.
Computers & Operations Research, 21(5), 543–556.
Slaughter, S., & Leslie, L. L. (1997). Academic Capitalism: Politics, Policies, and the
Entrepreneurial University. Johns Hopkins University Press.
Stewart, T. A., (1997) Intellectual Capital: The New Wealth of Organizations. Doubleday.
Taha, H. A. (2017). Operations research: An introduction (10th ed.). Pearson.
Terrey, I. N. (1968). Program budgeting and other newer management tools in higher education:
A description and annotated bibliography. Center for Development of Community College
Education. University of Washington Paper no. 6, Seattle, Washington.
Thelin, J. R. (2018). Missions impossible: Prospects and planning in American Higher Education.
Society, 55, 556–559.
Tole, A. A., & Czarnitzki, D. (2010). Commercializing Science: Is there a University “Brain Drain”
from Academic Entrepreneurship? Management Science, 56(9), 1599–1614.
Trick, M. (2018). Getting the world out of Operational Research. IFORS News, 12(4), 1.
UN. (2020). Retrieved June 2020, from http://hdr.undp.org/en/countries
Winston, W. L. (2004). Operations Research: Applications and Algorithms (4th ed.). Thomson
Learning.
Wong, J., Baars, M., Davis, D., Van Der Zee, T., Houben, G.-J., & Paas, F. (2019). Supporting
Self-regulated learning in online learning environments and MOOCs: A systematic review.
International Journal of Human-Computer Interaction, 35(4), 356–373.
World Bank. (2020). Retrieved February 2020, from https://www.worldbank.org/en/topic/
tertiaryeducation
Chapter 2
A Systematic and Updated Review
of the Literature on Higher Education
Marketing 2005–2019
Izhar Oplatka and Jane Hemsley-Brown
Abstract The purpose of this review is to identify key research themes in the field
of higher education (HE) supply-side marketing through a systematic search of
journal article databases of papers published between 2005 and 2019, to report
on current issues and themes, and ascertain research gaps in the literature for
exploitation in future research. Based on an analysis of 105 papers from the field
of HE marketing, five major themes characterizing the research on HE marketing
are presented in the paper: the marketization of HE; marketing communications;
branding, image, and reputation; marketing strategy; and recruitment, alumni and
gift-giving. Some thoughts about the nature of the knowledgebase in this field and
recommended topics for research conclude the paper. Note, 46 papers were based
on quantitative methodologies (constitutes 43.8% of the reviewed papers).
Keywords Educational Marketing · Branding · Higher Education · International

Marketing · Advertising · Institutional Image
2.1 Introduction
In 2006, the current authors published a systematic review of journal papers in

the area of HE marketing (Hemsley-Brown & Oplatka, 2006) covering 1992–
2004, and since publication, this article has been cited in almost a thousand works
(949 citations on (Google Scholar, 2019)). Since 2004, however, research on HE
marketing has increased dramatically due to a paradigm shift toward the increasing
marketization of HE worldwide, which inevitably generates considerable research
I. Oplatka ()
School of Education, Tel Aviv University, Tel Aviv, Israel
e-mail: oplatka@tauex.tau.ac.il
J. Hemsley-Brown
Surrey Business School, University of Surrey, Guildford, UK

36 I. Oplatka and J. Hemsley-Brown
in all fields related to HE markets marketing, marketization, and choice (Hemsley-

Brown & Oplatka, 2015a); this is also clear from the findings of the current review.
Authors have provided a range of rationales for research in this field, includ-
ing (see Hemsley-Brown & Oplatka, 2015a): increasing globalization (Maringe,
2010); internationalization and the need for international marketing (Chen, 2008);
increasing international student mobility (Wilkins & Huisman, 2011); expansion
in student demand (Leslie, 2003); increasing access to information (Veloutsou
et al., 2005); changes in fees and funding (Abu Bakar & Abdu Talib, 2013)
changing demographics (Bornholt et al., 2004) and concerns about the fundamental
transformation of education itself (Gibbs, 2007a)
Therefore, it is unsurprising that together with increasing competition nationally
and internationally for home and overseas students and reduced public funding HEIs
are now all too aware of the need to be more sophisticated and professional in their
marketing (Favaloro, 2015; Hemsley-Brown & Oplatka, 2006). The objectives of
this chapter, therefore, are to return to the literature to conduct an update of the 2006
(Hemsley-Brown & Oplatka, 2006) review; to document the search process; identify
relevant papers in line with the previous search parameters; draw up revised research
themes, document the findings; and make recommendations for future research in
the field.
2.2 Key Issues from the 2006 Literature Review
In our 2006 paper, we commented that higher education marketing tended to rely
on marketing approaches in use in the business sector in the absence of specific HEI
marketing techniques and methods (Hemsley-Brown & Oplatka, 2006), although
concerns have often been raised about the relevance of ideas imported from the
for profit sector (Gibbs, 2001; Gibbs, 2007a). Before 2006 it was clear that books
and manuals available for marketers in HE were based on business marketing
models—many focusing on a “how-to” approach. In particular, a leading text in
higher education marketing was Kotler and Fox (1985) Strategic Marketing for
Educational Institutions, re-printed in 1995. The research agenda in 2019 has
widened, and the need for “how-to-do” books has seen little expansion, with the
exception of two HE marketing specific texts: Marketing Higher Education: Theory
& Practice; and Marketing Further & Higher Education: an educator’s guide to
promoting courses, departments and institutions (Gibbs & Knapp, 2012; Maringe
& Gibbs, 2008).
In the 2006 paper, the authors highlighted the tendency of authors to also
rely on business marketing texts to provide definitions of the concepts covered
in research articles, particularly where current or existing theory is utilized for
empirical research. The authors commented that in particular texts by Kotler (e.g.,
Kotler & Armstrong, 2003; Kotler & Fox, 1985) were frequently cited, and several
articles were named in this regard (Ivy, 2001; Nguyen & LeBlanc, 2001; Klassen,
2 A Systematic and Updated Review of the Literature on Higher Education. . . 37
2002; Maringe & Foskett, 2002; Binsardi & Ekwulugo, 2003; cited by Hemsley-
Brown & Oplatka, 2006).
Findings from our 2006 review also revealed that there was considerable
debate in journal articles about students as customers and whether students were
consumers, customers, or both in their role as participants in higher education.
Authors also reported opposition to the notion of a student as a consumer (Baldwin
& James, 2000; Barrett, 1996) and to the description of courses in universities as
“products” (Conway et al., 1994), although authors also argued that participation
in higher education is not strictly a product and put forward a well-argued case
for HE as a service (e.g., Brookes, 2003; Mazzarol, 1998; Nicholls et al., 1995).
Passionate, articulate arguments were also put forward against the marketization of
HE and the adoption of business terminology (Hemsley-Brown, 2011), increasingly
adopted and accepted by some educationalists. Those who oppose the emergence
and imposition of the market in higher education argued strongly that business
values have no place in higher education because such values are morally and
ethically in conflict with educational values (Gibbs, 2002; Gibbs, 2007b).
The following sections in this chapter focus on an update of the 2006 review and
present the thematic findings of our systematic literature review of HE marketing
for the period 2005 (the end date used for the 2006 review) to 2018. The purpose of
this review is to identify key research themes in the field of higher education (HE)
supply-side marketing through a systematic search of journal article databases of
papers published between 2005 and 2019; to report on current issues and ascertain
research gaps in the literature for exploitation in future research. The following
section sets out the methodology for the review and is followed by the findings, with
subsections based on the themes which emerged from the analysis: the marketization
of HE; marketing communications; branding, image and reputation; marketing
strategy including relationship marketing; and recruitment, alumni and gift-giving.
2.3 Methodology
A previous review paper on higher education marketing by the current authors

applied a systematic review approach suggested by Booth (2001) and focused
on presenting the “best evidence” (Hemsley-Brown & Oplatka, 2006). The main
principles of this approach have been adopted for this review by searching named
databases, systematically recording the hits, and downloading papers relevant for
the review based on previously identified search criteria. The current searches were
based on updating our previous systematic review (with papers published between
1992 & 2004); therefore, the current review is based on searching within the period
2005 to 2018; the search terms matched to the themes which were reported in
the original review and extended where new topics emerged following the re-
categorization of selected papers (e.g., the inclusion of “social media,” “marketing
strategy and “alumni”).
For the updated literature review, the authors of the current research conducted
systematic, documented, and extensive searches of academic databases, namely
British Education Index; ERIC; Emerald Full Text; EBSCO (Business Source Com-
plete; & PSYCINFO and Psych Articles). Specific internet searches using Google
Scholar were carried out the follow up secondary references, to access some aca-
demic articles in the full text more directly and to identify further articles related and
relevant to the focus of the review. Searches were initially very open with no limits
(with the exception of dates (2005-2018) and “scholarly” articles. However, due to
the high number of hits (93,996 hits for EBSCO business source complete search
using “marketing” and “university” or “higher education”), and the large number of
irrelevant articles included, the search terms were de-limited as follows “marketing”
+ “university” or “higher education,” not teaching/learning/instruction—the latter
phrase was used to exclude the high number of papers on pedagogy. In addition, the
search parameters included specific databases (see above), “English language,” and
either “abstract” or “subject terms” (requiring separate searches). The purpose of
the approach to searching and identifying articles was first, to ensure that as many
articles as possible within the search parameters were identified, second to focus on
the study’s objectives, and third, to provide a search approach that excluded a large
number of articles which were outside the parameters.
The review parameters filtered “hits” to focus on articles published in the
English language but from anywhere in the world; scholarly journal papers were
prioritized; opinion pieces and conference papers were excluded either during the
searching or after the initial sort process; searches used both abstract and subject
terms for searching. Following pre-reading or visual scanning of all papers, and
some deselection, 165 papers (empirical and conceptual) from 2609 hits, were
downloaded for further scrutiny. Reference manager version 12 software (with links
to original sources) was used to capture and track the papers for further assessment.
After reading each paper to determine the focus and objectives, 60 papers were
excluded from the original selection of 165. Reasons for deselection were based on
lack of relevance to the study; research sample was a single institution—frequently
from a single discipline; papers were short opinion pieces; or papers related to
sectors other than higher education. This resulted in a final total of 105 papers for
the review.
In our 2006 paper, the authors used “thematic analysis” and summarized the
findings based on various relevant themes (Tranfield et al., 2003) which emerged
from analysis of the selected papers. For the earlier paper the authors curated a small
selection of papers, resulting in a final selection of 15 papers, which were considered
the “best evidence” for each of the identified themes (based on objectives, design
of the study, sample size, methodology, and rigor) and restricted the final selection
to empirical papers. For the current review, the authors approached the task in a
more traditional review style, and therefore did not curate the selection to only
“best evidence,” or to empirical papers only, but included all papers meeting
the objectives and defined parameters. The authors conducted searches using the
original thematic categorization from the 2006 paper and later added sub-headings
or changed headings where appropriate to extend or to divide the thematic sections.
These sections (original themes are marked with *) are marketization, *marketing
communications (with the addition of advertising and social media marketing),
*image and reputation (with the addition of branding), application of *marketing
models, and *strategic marketing were replaced by marketing strategy, which in the
updated review includes *segmentation, targeting and *positioning, and *relation-
ship marketing; *widening participation is now excluded, due to lack of relevant
papers (related to HE marketing) since 2005; a new section on marketization is
included in this review, and a small number of papers on recruitment and alumni are
also new, based only on articles which relate to HE marketing (rather than to student
choice which is reviewed elsewhere (Hemsley-Brown & Oplatka, 2015b). Table 2.1
provides a summary of topics with relevant sources and shows the number of papers
included in this review.
The presentation of the findings section is structured as follows: first papers on
marketing communications are analyzed, with subsections on advertising and a new
topic, social media marketing. This is followed by a section covering marketing
strategy, which includes subsections on segmentation, targeting and positioning,
and relationship marketing. A substantial section follows on branding, which was
not included in the previous review, with a subsection on image and reputation—
which provides an update on research articles since our previous paper. A substantial
section on marketization in higher education follows, and is a new topic that has
emerged since 2005; finally, a short section on recruitment marketing and alumni,
including gift-giving completes the review section of this review. A discussion and
conclusion follow which highlight the themes which emerge from the review and
suggest possible ways forward for researchers in the HE marketing field.
2.4 Findings
The inductive analysis of the reviewed papers yielded five major themes—
marketization of HE, marketing communications, branding, marketing strategy,
and relationship marketing, with sub-themes (see summary in Table 2.1). See
Appendix for a list of all the articles included in the study, listed under headings by
theme.
2.4.1 The Marketization of Higher Education
The searches, after applying the strict selection criteria, resulted in 17 papers on
the marketization of higher education being selected for the review: 14 conceptual
papers and three empirical papers (i.e., Bowl, 2018; Cardoso et al., 2011; Symes
& Drew, 2017) (see Appendix 1). Marketization as a topic of research in higher
education marketing emerged from this study as second in frequency after higher
education branding. Marketization is defined as follows:
Table 2.1 Sources for higher education marketing literature review shown by subject focus
40
Main subject focus Number of papers Authors

Marketization 17 Gibbs, 2007; Brown, 2008; Mause, 2009; Molesworth et al., 2009; Newman & Jahdi, 2009; Miller & Brian, 2010;
Weymans & Wim, 2010; Cardoso et al., 2011; Hemsley-Brown, 2011; Lowrie & Hemsley-Brown, 2011; Susanti &
Dewi, 2011; Craven & Anne, 2012; Natale & Doran, 2012; Favaloro, 2015; Symes & Drew, 2017; Bowl, 2018; Gibbs,
2018.
Marketing 9 Adams & Eveland, 2007; Bonnema & Van der Waldt, 2008; Edmiston-Strasser, 2009; Schüller & Chalupský, 2011;
communications Popović, 2015; Vuori, 2015; Gribanova, 2016; Royo-Vela, 2016; Pizarro Milian & Davidson, 2018.
• Advertising 3 Clayton et al., 2012; Papadimitriou & Ramírez, 2015; Jan & Ammari, 2016.
• Social media marketing 8 Zailskaite-Jakste & Kuvykaite, 2012; Palmer, 2013; Bélanger et al., 2014; Linvill et al., 2015; Brendzel-Skowera &
Lukasik, 2016; Brech et al., 2017; Clark et al., 2017; Peruta & Shields, 2017.
Branding 23 Baker et al., 2005; Judson et al., 2006; Temple, 2006; Chapleo, 2007; Hemsley-Brown & Goonawardana, 2007; Judson
et al., 2009; Chapleo, 2011; Furey et al., 2014; Aula & Tienari, 2015; de Heer & Tandoh-Offin, 2015; Delmestri et al.,
2015; Holmberg & Stannegård, 2015; Kosmützky & Krücken, 2015; Sataøen, 2015; Chapleo, 2015a; Chapleo, 2015b;
Fay & Zavattaro, 2016; Rooksby & Collins, 2016; Lee et al., 2017; Rose et al., 2017; Rutter et al., 2017; Winter &
Thompson-Whiteside, 2017; Lomer et al., 2018.
• Image and reputation 13 Wolverton, 2006; Brown & Mazzarol, 2009; Angulo-Ruiz & Pergelova, 2013; Finch et al., 2013; Steiner et al., 2013;
Idris & Whitfield, 2014; Tolbert, 2014; Drori et al., 2016; Plungpongpan et al., 2016; Chen, 2016; Saleem et al., 2017;
Huisman & Mampaey, 2018; Lee & Chen, 2018.
Marketing strategy 5 Naidoo & Wu, 2011; Schofield et al., 2013; Alhakimi & Qasem, 2014; Nkala et al., 2014; Mogaji, 2016.
• Market orientation 6 Pavičić et al., 2009; Camelia & Marius, 2013; Ross et al., 2013; Tayar & Jack, 2013; Alnawas & Phillips, 2014; Baber
& Upadhyay, 2015.
• Segmentation, targeting, 4 Ghosh et al., 2008; Mazzarol & Soutar, 2008; Durkin et al., 2014; McAlexander et al., 2016
and positioning
• Marketing mix 3 Enache, 2011; Moogan, 2011; Artur, 2015.
• Relationship marketing 5 Kim et al., 2010; de Macedo Bergamo et al., 2012; Purgailis & Zaksa, 2012; Iskhakova et al., 2017; Schlesinger et al.,
2017.
Recruitment, alumni and 9 Frølich et al., 2009; Pippert et al., 2013; Hulme et al., 2014; Stephenson & Bell, 2014; Alnawas & Phillips, 2015;
gift-giving Asaad et al., 2015; Stephenson & Yerger, 2015; Onk & Joseph, 2017; Pedro et al., 2018.
Total 105
Source: The authors
I. Oplatka and J. Hemsley-Brown
“Marketization in higher education refers to the adoption of free market practices in running
universities. These include the business practices of cutting production costs, abandoning
courses and programs not in demand, offering more popular programs and facilities and
advertising to increase brand image, sales and the profit margins” (Hemsley-Brown, 2011)
p118
The process of marketization in HE that began in the 1990s in many Anglo-

American countries continues to receive attention in the second decade of the
twenty-first century. Two papers succinctly clarified the challenges of the phe-
nomenon in the context of higher education. Lowrie and Hemsley-Brown (2011) set
out the differences between marketization and marketing and highlight the increase
in the complexity of HE; the expansion of a consumerist dialog; the increasing
focus on university rankings, and the importance of image, identity, and brand.
Similarly, Miller and Brian (2010) analyze definitions of the commodification of
HE and conclude that the commodification of this sector is already well established
but outcomes based on values, principles, and liberalism are on the decline in favor
of credentials, skills, and financial gain.
Three papers focus on HEIs’ responses to marketization, examining how uni-
versities in New Zealand and the UK balance the need to maintain a successful
global brand (e.g., elitism) with national expectations (e.g., equality and fair access)
(Bowl, 2018); and whether the political change in public and private universities
and polytechnics in Portugal is transforming students into consumers (Cardoso
et al., 2011). Similarly, Favaloro (2015) examined growth in student numbers
and marketing investment patterns in Australian universities. They concluded that
marketization brought about increased attention to performance, aggressive and
intrusive announcements, and intensive competition between non-prestige univer-
sities and non-university providers.
As far as the outcomes of HE marketization are concerned, four conceptual
papers (i.e., Brown, 2008; Craven & Anne, 2012; Newman & Jahdi, 2009; Susanti &
Dewi, 2011) explored the implications, benefits, and limitations of the marketization
of HE in general (Brown, 2008); marketization in respect to a particular social and
educational aspect, such as social justice and unfairness of markets for students from
lower-socio-economic groups (Craven & Anne, 2012); the impact of marketing on
staff (Newman & Jahdi, 2009); and new regulations in Indonesia (Susanti & Dewi,
2011) who claim that marketization contributes to polarization particularly in terms
of social class, poses barriers to access to HE for disadvantaged groups of students
and fails to address educational inequity.
Finally, in this section, eight papers put forward a critical argument and concep-
tual debate in terms of a marketized HE system and the increasing marketization of
HE—predominantly raising a raft of concerns regarding the operation of a free mar-
ket in HE. Scholars set out an argument against advertising as part of the marketing
of HE claiming that it intends to convince rather than inform prospective students
(Gibbs, 2007b). Gibbs (2018) further argues against consumerism, claiming that it
changes learners into customers who simply aim to gain a degree as if it was a mere
commodity and transforms tutors into service providers. Authors also argue that
by engaging with market discourse which is a neoliberalist construction (Symes &
Drew, 2017), universities are participating in the selling of HE experience as an

economic asset. Hemsley-Brown (2011) examined the potential damage the market
can inflict on HE and raises concerns that the core values of HE are in conflict
with the market: free markets are known to exacerbate inequalities, she argues,
and intervention is needed by governments to reduce this inevitable unfairness;
Mause (2009) highlights the lack of progress by economists in identifying and
measuring the harmful and beneficial effects of market signaling in the HE context;
and Natale and Doran (2012) claim that academic freedom is considered less
important than accountability and trust less important than utility in the market-like
environment. Papers further warn against a decline in creativity and professional
judgment following the marketization of HE (Weymans & Wim, 2010) and the
shift in higher education towards a commodity that can be bought, rather than as
a transformational experience (Molesworth et al., 2009).
2.4.2 Marketing Communications
The topic of marketing communications includes, according to Gribanova (2016),

advertising, public relations (PR), direct marketing communications; personal
selling; and sales promotion, both in an online and offline environment. Nine papers
focused on marketing communication in their attempt to expose the problems of
different types of communication, organizational factors, and outcomes in a higher
education context (see Appendix 2). Authors from Serbia probed the types of
marketing communications used by 131 HEIs (Popović, 2015) and found that they
lacked integration. This problem could be related to institutional leadership, since
Edmiston-Strasser (2009) identified the link between integration and leadership in
research with a sample of 42 US universities. Similarly, (Schüller & Chalupský,
2011) explored types of problems for university marketing communications and
examined how these communications are organized. Based on interviews with 20
marketing managers from the Czech Republic, they identified management structure
problems in the universities, including faculty relationships, and suggested that
faculties should have more responsibility for marketing communications within
their faculties.
Further papers extended the concept of marketing communication and explored
the connection with the institutional image. Thus, based on large-scale survey of
students, researchers aimed to determine whether marketing messages need to be
different for each market segment, and examined the content of symbolic images
in promotional materials in different types of HE institutions (Bonnema & Van der
Waldt, 2008). Research seeking to identify how universities can develop to improve
their image by using social media tools found that advertising, word-of-mouth,
content, and information about HEIs are the main influences on decisions in Spanish
universities (Royo-Vela, 2016). While researchers studying public universities in
Canada found that they tend to emphasize research achievements and rankings
(Pizarro Milian & Davidson, 2018).
Finally, three papers explored marketing communications in the virtual world.

Accordingly, Gribanova (2016) wrote a theoretical paper about developments in
mobile marketing communications technology tools and the benefits to universities
in engaging with these methods to communicate with students and prospective
students. From a different viewpoint, research (Vuori, 2015) explored how 68
HEIs in Finland convey market differentiation on webpages targeted at prospective
students and found that the websites emphasized the relevance of programs for
employability and career prospects in addition to an international experience and
context. Similarly, websites were used to divide institutions into three groups:
accredited residential, regionally accredited online, and non-accredited online in a
study of 150 universities in the USA (Adams & Eveland, 2007). Two further aspects
of marketing communications justify further discussion, based on advertising and
social media marketing.
Despite the apparent popularity of marketing communications as a topic for
research, there is an overlap with social media marketing and website postings and
website content.
2.4.2.1 Advertising
Advertising is a paid form of mass communication by an advertiser or company to

specific readers, viewers, or the wider audience to achieve the specific goals (Kotler
& Armstrong, 2003). Two papers explored marketing messages in the advertising
campaigns of institutional communications and found striking similarities across
institutions. In the first paper, the authors reveal five themes that 115 American
HEIs use in messaging and advertising specific to institution communication:
campus characteristics, academics, co-curricular activities, prestige building, and
mission/purpose (Clayton et al., 2012). These authors argue that institutions are not
creating unique messages; there is a great deal of similarity among them. The second
paper reveals that the most salient content of advertisement campaigns in five US
cities show similarities, despite the wide variation in types of institution. Similarities
in the content of communication campaigns are based on students; logos; global
scope, open campus, and careers (Papadimitriou & Ramírez, 2015).
In contrast, based on a survey with 350 prospective Malaysian students, Jan and
Ammari (2016) examined the impact of online advertising on prospective students’
decision marking and found that while online advertising is important in the student
decision-making process, intrusive or intensive advertisements and campaigns have
a negative impact on students’ positive attitude and selection of a university.
The topic of advertising specifically remains relatively limited as a research
focus, similar to the conclusions provided in the 2006 paper. We might speculate
that the expansion of social media marketing is likely to have contributed somewhat
to the lack of focus in research on a more traditional approach such as advertising
and that advertising research is combined with other methods such as websites and
social media messages.
2.4.2.2 Social Media Marketing
Social media marketing (SMM) utilizes the Internet and in particular social net-
working websites for communication and marketing purposes (Brech et al., 2017).
Social media marketing has become a popular subject for research in HE marketing
since the publication of our 2006 paper. This is a form of Internet marketing
that utilizes social networking websites as a marketing tool for communication
(Brech et al., 2017). Eight papers focused on social media marketing from multiple
theoretical stances (Bélanger et al., 2014; Brech et al., 2017; Brendzel-Skowera
& Lukasik, 2016; Clarke, 2007; Linvill et al., 2015; Palmer, 2013; Peruta &
Shields, 2017; Zailskaite-Jakste & Kuvykaite, 2012); four of the papers focused
on social media in general. First, authors develop a theoretical model to explain
engagement with social media marketing and the outcome factors (Brech et al.,
2017); second, the research identifies how universal the use of social media sites
is for Polish universities (Brendzel-Skowera & Lukasik, 2016); a third, examined
whether social media engagement impacts on relationship quality in respect of
American universities and students (Clark et al., 2017); and finally researchers
constructed a model of communication in social media and tested the model in
14 Lithuanian HEIs (Zailskaite-Jakste & Kuvykaite, 2012). In Lithuania, Facebook
is the most common social media site where universities have a presence. This
study found that universities with a strong reputation tend to have more Facebook
fans; however, a large fan-base negatively impacts student engagement. The authors
note that communication between business organizations and consumers, consumer-
to-consumer communication, and communication between consumer and business
organizations (feedback) are the three key aspects of communication on social
media in Lithuania and can be successfully applied in the university sector.
Four further papers focus on particular social media platforms such as Twitter and
Facebook. Using content analysis, Peruta and Shields (2017) examined Facebook
posts from 66 top HE institutions in the US and found differences between posts
based on institution type. Based on a study of 106 prestigious Canadian universities
(Bélanger et al., 2014), varying levels of activity on Twitter were recorded, with a
high level of sharing by retweeting (37%). Twitter was found to be the most-used
platform for conversations between users, whereas Facebook was used more for
institution-led postings. In a study of six Australian universities, Palmer (2013) high
levels of retweeting were also found, which indicated complex interactions between
Twitter users. Finally, in contrast, in a study of 52 American universities, researchers
indicated that Pinterest is not being used effectively by institutions. They claim that
using Pinterest to share images of news and past events is not in line with the best
qualities and features of Pinterest (Linvill et al., 2015).
The increase in research about social media marketing is not surprising given the
exponential growth in the use of these platforms since our original literature review
paper was published in 2006. However, the nature of the papers is still somewhat
exploratory and concentrates on examining aspects of the uses, popularity, and levels
of activity in social media usage by universities and colleges.
2.4.3 Branding, Image, and Reputation
A brand is a symbol, a name, or a distinctive design that identifies a product or

a company; and distinguishes the seller’s offering based on value (Temple, 2006).
Branding enables an organization to develop a sustainable, distinctive identity in the
market to achieve competitive advantage (Kotler & Andreasen, 1996). Branding as a
research topic results in one of the largest sections for this review (see Table 2.1); 23
papers focused on branding in HEIs and examined the characteristics of universities’
branding, factors, and consequences; and a further 13 papers conducted research
on a related topic: (brand) image and (brand) reputation which featured in our
earlier review (Hemsley-Brown & Oplatka, 2006) (see Appendix 3). Branding was
a topic which we recommended in our 2006 paper as a key area for future research.
Categories of research include visual images and icons; challenges of applying
theories; organizational issues; private sector branding; branding metrics; drivers
and strategies for branding; and the benefits and consequences of HE branding.
First, a study of the worldwide branding of universities (Delmestri et al., 2015)
examined the images, icons, and logos employed on the Internet to promote HE
institutions from 20 countries and five continents. Data were collected from 821
university home pages, and analysis revealed differences between the west and other
parts of the globe: more abstract visual types of icons as a means of expression
are dominant in the west, but there is a greater diversity of icons, logos and logo
types in other parts of the world. Other authors examined icons, logos, trademarks,
and brand activities, such as brand equity, brand positioning, and brand identity by
universities, to establish what these activities reveal about university branding in the
USA (Rooksby & Collins, 2016).
Second, researchers have also pursued a greater understanding of university
branding and the challenges of applying theories of branding to universities. Chap-
leo (2015a, p.8 ) explored the emergence, development, and growth of the branding
of higher education in the UK and concluded that there are clear opportunities
to examine branding in the HE context. He claims that although “corporate-,
experiential- and internal-branding” from the business sector “have a degree of
applicability” for HE “further empirical work is needed.”
More recently, research has also examined the branding of UK higher education
since the “Britain is GREAT” campaign of 1999 (Lomer et al., 2018) to 2014 using
a Bourdieusian approach. The authors concluded that the British higher education
brand relies on the reputations of its highest quality elite institutions and follows
the brand positioning of the nation as a whole—for example, in terms of Britain’s
objective to achieve international competitiveness.
Third, researchers (Aula & Tienari, 2015) found that university branding is
predominantly an organizational purpose and strategy linked to identities and (non-)
identification of internal and external players in a socio-cultural and societal context;
and others argue there are clichéd brand values associated with the branding of HEIs
in Norway and Sweden, with tendencies towards both conformity and differentiation
(Sataøen, 2015). Temple (2006) concludes that branding in universities is best
described and reputation management ensuring the university has a strong vision,
while others (Winter & Thompson-Whiteside, 2017) assert that HEIs are not writing
copy for any specific target audience—they claim that references to location tend to
mix messages of safety on the one hand, and excitement on the other.
Fourth, as far as private education is concerned, researchers who studied univer-
sity brands in the private sector found that private and major university respondents
were positive about all forms of communications and found them to be more
effective than those from more minor universities, who believed that the brochure
was the most effective (Judson et al., 2006). However, (Judson et al., 2009) indicated
that 343 university administrators from private HEIs reported greater brand clarity
in internal brand communications than respondents from public HEIs.
Fifth, measurement has also been the subject of branding research. The work of
Chapleo (2007) sought to establish whether branding activity in British universities
should be measured and how it should be measured, including the importance of
applying metrics as part of university branding strategy (Chapleo, 2011);
Next, in addition, papers are devoted to exploring the factors affecting HE
branding in different countries across the globe. Researcher explored the drivers for
branding and attitudes towards university brands: the importance of brand promise
and brand positioning (Furey et al., 2014); key brand personality positioning accord-
ing to a further article (Rutter et al., 2017) is based on sophistication, competence,
sincerity, ruggedness, and excitement. Further articles focus on the challenges
of university branding and the key features that contribute to making university
branding different from commercial branding; and how and when universities adopt
a branding and re-branding strategy (Baker et al., 2005; Chapleo, 2015b; Fay &
Zavattaro, 2016).
Four further research articles on branding strategies found that HEIs need to
develop their brand by constructing clear and effective brand architecture, and main-
taining coherent and harmonious links within the brand architecture” (Hemsley-
Brown & Goonawardana, 2007 p.947); according to more recent research, the
university heritage factor also has a greater influence on parents than on prospective
students (Rose et al., 2017). Among the barriers to branding are the language skills
of international students (Lee et al., 2017) and negative opinions towards the concept
of the brand in HEIs due to its commercial connotations (Chapleo, 2007).
Finally, four papers explored the consequences of HE branding in terms of
benefits, promises, and mission statements. The researchers indicate that the benefits
of branding are differentiating the institution; greater loyalty from staff and students;
credibility to charge high tuition fees; and greater loyalty from all stakeholders (de
Heer & Tandoh-Offin, 2015); and that mission statements enable universities to
claim a market position in comparison with competitors (Kosmützky & Krücken,
2015). Holmberg and Stannegård (2015) further found that the culture of the
university explored in the study is marketized managerialism expressed by students
who adopted the same idea and language in discussions about their future.
2.4.3.1 Image and Reputation
Related areas of study to higher education branding are image and reputation that
are also included in the original 2006 literature review. Brand image is a corporate
asset, used to distinguish the organization from the others as well as contributing to
sustainability and value creation (Lee & Chen, 2018). Conversely, an organization
with a high reputation implies that its achievements considerably exceed that of most
of its competitors in one or more particular features. Reputation is the sum of the
attitudes and evaluations of stakeholders, students, funding bodies, and government
(Steiner et al., 2013).
In the current review, thirteen (13) papers focus on image and reputation, but
also on identity, seven of which approach the issues from the demand-side, i.e.,
the students’ side, and the factors affecting their retention. Four of these papers
use student satisfaction and retention as variables in their study of institutional
image. First, for example, Brown and Mazzarol (2009) report that perceptions of the
institution’s image have an impact on both perceived value and student satisfaction.
Student retention and dropout are also related to perceptions of the institutional
image according to later research (Angulo-Ruiz & Pergelova, 2013); institutional
image influences both student performance and commitment, which in turn has an
impact on students’ decisions to dropout.
Chen (2016) explored customer satisfaction in the higher education sector and
concluded that positive brand image and student satisfaction lead to positive word-
of-mouth, and sharing satisfying experiences is related to loyalty to the institution.
In a study of similar themes, (Saleem et al., 2017) tested the effect of service
quality with a sample of 747 students from Pakistani. The findings indicate that
the quality of the service is positively related to student satisfaction and university
reputation has a moderating effect.
Three further demand-side papers focused on institutional identity. (Tolbert,
2014) examined the extent of the identity of 59 American universities in promotional
materials and focuses on its faith (Christian) dimension. Two further papers included
other stakeholders in the research design, focusing on university identity in the
USA and how this is conveyed to the target audience in the marketing materials
(Wolverton, 2006). The second of these identity papers examines whether the
corporate visual identity influences a viewer’s perception of a university and its
lecturing staff (Idris & Whitfield, 2014); researchers found that perceptions of
a university [created for the study] were more positive when images of a male
lecturer were shown, and reputation and trustworthiness were more positive when a
traditional logo was used.
Of the 13 papers which focus on image and reputation, six were concerned with
the supply side, i.e., on universities’ websites, managers, documents, and executives.
Dominant themes were visual or brand image and identity. Thus, (Drori et al., 2016)
explored universities’ identities and narratives through the visual images, logos, and
icons presented on their websites and Lee & Chen et al. (2018) aimed to identify
image asset management factors using a Delphi technique, and (Plungpongpan et
al., 2016) examined the impact of university (corporate) social responsibility on the
brand image of a university.
Three papers merit highlighting in this area of study and focus more on
reputation in addition to identity and image. First, (Finch et al., 2013) focused
on the attributes and the implications of reputation for three categories of higher
education institutions. Second, based on documents from 29 British universities,
(Huisman & Mampaey, 2018) asked how universities want students, stakeholders,
and the outside work to view them and how similar and distinctive UK universities’
images are. Interestingly, despite competition in the UK HE sector, there were few
images that deviated from the mainstream, although there were some examples of
distinctiveness. Finally, (Steiner et al., 2013) proposed a multidimensional model
that identifies internal and external factors influencing university identity and
reputation. Accordingly, universities’ identities and reputation strategies comprise
four dimensions: organizational identity, symbolic identity, image, and reputation.
2.4.4 Marketing Strategy
Marketing strategy is the perception and identification of potential markets and the
design of specific tactical approaches for reaching those audiences (Schofield et al.,
2013). The topic of marketing strategy in HEIs is the topic of interest for 23 papers
in broad terms, covering the topics of utilization and delivery of strategy (5 papers);
market orientation (5); segmentation, targeting and positioning (4); marketing mix
(4); Recruitment strategy; and relationship marketing (5) (see Appendix 4). Notably,
marketing strategy is perception and identification of the potential markets and the
design of specific tactical approaches for reaching those audiences (Schofield et al.,
2013).
Five marketing strategy papers examined the level of utilization and effectiveness
of marketing strategies in universities and colleges (e.g., (Mogaji, 2016) and found
that marketing planning and resource analysis are utilized and perceived to be highly
effective (Alhakimi & Qasem, 2014); secondly, implementing a successful market
strategy is based on a marketing manager’s commitment to the strategy and to their
role, especially in a highly competitive market (Naidoo & Wu, 2011). Schofield
and co-authors (Schofield et al., 2013) found that newer universities in the UK
are more influenced by a government policy agenda than older, more traditional
institutions, who continue to rely on their reputations while others (Nkala et al.,
2014) examined marketing strategies to deal with strong competition for staff and
students; they revealed that expectations of service quality and delivery are high
among Zimbabwean students, but service experience is low.
2.4.4.1 Market Orientation
Market orientation refers to an organizational culture that prioritizes superior market

performances in an organization by creating, and exploiting behaviors that focus on
and lead to superior market value for buyers and continuous superior performance
for the business (Narver & Slater, 1990). This topic has been explored conceptually
or empirically in six papers. Thus, while (Camelia & Marius, 2013) reviewed
the literature and discussed the importance of market orientation in HEIs, other
researchers explored the level of market orientation in 87 Croatian universities
(Pavičić et al., 2009); how market orientation strategies influence international
student recruitment performance in Australia (Ross, Grace, Shao, & Mitchell
Ross, 2013); and the market orientation and economic performance relationship of
universities in India (Baber & Upadhyay, 2015). In contrast, (Alnawas & Phillips,
2014) examined the constructs which constitute the concept of Prospective Student
Orientation (PSO) and identified three second-order formative constructs to measure
the concept of PSO. Insights from these studies indicate that private universities that
face low competition perform better than private universities which operate in highly
competitive environments.
Authors (Tayar & Jack, 2013) also conclude that universities are naturally
risk averse, in a study investigating the marketing behaviors and perspectives
of Australian university managers making decisions about entering international
markets. Universities were motivated by simultaneously pursuing increasing income
and raising reputation (Tayar & Jack, 2013). Others found that international student
recruitment departments of Australian HE institutions largely adopt a customer
orientation approach rather than a competitor or inter-functional focus (Ross
et al., 2013). In contrast, Croatian universities demonstrate an underdeveloped
market orientation despite recognition of the significance of developing stakeholder
relationships (Pavičić et al., 2009).
2.4.4.2 Segmentation, Targeting, and Positioning (STP)
Market-oriented HEIs usually engage in market segmentation, develop a marketing

mix approach and pay attention to student recruitment. Segmentation is a process
of dividing a market into identifiable groups based on their needs, characteristics,
or behavior for steering or targeting the marketing effort (Kotler & Armstrong,
2003). Four papers analyze the STP process: one paper explores the use of
market segmentation to establish whether this approach improves the utilization of
resources (Ghosh et al., 2008); another explores university marketing managers’
awareness and perceived importance of market segmentation (Durkin et al., 2014);
McAlexander et al. (2016) demonstrate the importance of using a segmentation
and targeting approach for identifying and engaging with alumni to maximize
alumni giving; finally, authors also focus on the market positioning of Australian
universities (Mazzarol & Soutar, 2008). STP authors recommend improved use of
resources and the adoption of targeted marketing strategies and highlight the lack of
awareness and failure to capitalize on the use of online communications to target and
communicate with more mature prospective students. Authors also note, however,
that it is increasingly difficult for universities to maintain quality and growth in a
context where reduction in spending is also a priority (Mazzarol & Soutar, 2008).
2.4.4.3 The Marketing Mix
Three (3) authors explored the application of the marketing mix and the marketing
communications mix as approaches to the marketing of HEIs. The 4Ps have a special
place in the marketing mix, and each of them affects the buyers in one or several
phases of consumption. The marketing mix refers to the “4Ps” which are defined as
Product; Price; Place, and Promotion. Authors have also added a 5th P: People; and
by also adding two additional Ps: “Process,” “Physical evidence,” 7Ps model is an
adapted model of the marketing mix for services marketing (Enache, 2011).
First, the application of the 7Ps framework is used to gain a better understanding
of the Romanian higher education market (Enache, 2011). In contrast, Artur (2015)
focused specifically on strategic marketing communications activities and the role
and importance of the Internet specifically, in the marketing activity of higher
education institutions in Poland; and (Moogan, 2011) similarly sought to identify
the key marketing communications mix activities needed to meet students’ needs
when applying to university to inform university marketing managers. While a study
of the 7Ps model is somewhat limited in scope, the conclusions from the studies on
marketing communications mix strategies unsurprisingly highlight the importance
of online communications. In defense of these conclusions, perhaps—the papers are
more than five years old, and marketing communications has moved on apace since
2011—and even since 2015.
2.4.4.4 Relationship Marketing Strategy
A major concept in the marketing of services and educational organizations refers to

relationship marketing, a strategy where the management of interactions, relation-
ships, and networks are fundamental for interaction with buyers (Grönroos, 1997).
Yet, only five papers focused specifically on this concept, none from the UK or the
USA, and the studies largely examined the antecedents to institutional loyalty. Thus,
a study from Brazil tested the association between relationship quality and loyalty to
an institution and found that the strongest contributing variable to student customer
loyalty is the perceived quality of the institution (de Macedo Bergamo et al., 2012).
Purgailis and Zaksa (2012) examined the relationship between service quality and
student loyalty to a university in Latvia; Schlesinger et al. (2017) examined the
role of brand image, trust, satisfaction, and shared values as a direct and indirect
explanation of alumni loyalty in Spain; one conceptual paper further analyses the
relationship between alumni loyalty and relationship marketing (Iskhakova et al.,
2017).
Finally, in Korea, (Kim et al., 2010) research found that students’ strong identi-
fication with their sports program is positively related to their strong identification
with the university as a whole and leads to a sense of belongingness & support for
the university.
In sum, the topic of relationship marketing goes beyond loyalty to an institution,
however, and offers potential for research on trust, commitment, and long-term
support for institutions; in addition to studies on partnerships with international
institutions, use of technology for managing relationships and data analytics, and
risk, salience, and emotional bonds in relationships. A related topic of research in
this respect is alumni management which is the final topic of this review.
2.4.5 Recruitment, Alumni, and Gift-Giving
Finally, papers were identified which focus on marketing for recruitment purposes,
management of alumni, and the related topic of gift-giving (see Appendix 5).
Alumni or previous students are the best ambassadors, advocates, and volunteers
who can also be mentors, help to build enrolments, actively recommend the
institution, and be enthusiastic fundraisers (Pedro et al., 2018). This is allied to
gift-giving and donations, which refer to the monetary contributions made to an
institution by its alumni (Stephenson & Bell, 2014).
The topic of student choice is considered substantial, and a review of literature
has been published elsewhere (Hemsley-Brown & Oplatka, 2015b), therefore
recruitment, the process of identifying, targeting, attracting, and managing the
enrolment of students to the institution (Mortimer, 1997), is only included when
there is a marketing, not choice focus.
Five papers focused on student recruitment strategies of HEIs: how these
strategies are linked to a university’s market position (Frølich et al., 2009); and
what emerging universities can learn from the strategies employed by established
universities regarding student recruitment (Onk & Joseph, 2017). Two papers focus
on international recruitment agents, asking how agents justify their value and why
universities are employing agents (Hulme et al., 2014), including their export market
orientation (Asaad et al., 2015), while another explored the way ethnic minorities
and racial groups are portrayed in university recruitment materials (Pippert et al.,
2013). It is somewhat surprising that there are so few papers on international
recruitment strategy, given the high level of international recruitment activity in
universities around the world. We acknowledge that with different search criteria,
perhaps more papers on this topic might emerge.
Papers on alumni (two papers) and gift-giving (two papers) are new topics for
the review compared with our last review. One paper focuses on elements of an
effective strategy for alumni management, and the second focuses on factors that
encourage alumni to develop a sustained relationship with the institution. First,
Alnawas and Phillips (2015) examine and test the concept of alumni orientation
(AO) in UK universities and identify nine multi-item components to measure AO.
They offer a model of AO and emphasize the importance of database management,

social media management, and events management in successful alumni relations.
Second, a propensity to form a strong alumni relationship is based on a positive
attitude to making a commitment to an alumni relationship, good communication,
and satisfaction with both the social and academic environment while students
are studying at the institution (Pedro et al., 2018). Both papers on gift-giving are
based in the USA. The focus of both gift-giving papers is the motivation for gift-
giving, and identifying the reasons alumni make donations (Stephenson & Bell,
2014; Stephenson & Yerger, 2015). Both papers use a very large sample for the
study and conclude that motivation for alumni donating is explained by using social
identity theory (Stephenson & Bell, 2014); and satisfaction with university resources
is associated with the positive brand association and brand image, which increases
the likelihood of gift-giving (Stephenson & Yerger, 2015).
The next section examines the differences between the 2006 review and the
current review; and identifies HE marketing topics—potential topics, which are
missing from the current searches based on examining and comparing topics that
are currently the focus of research on marketing outside the HE sector.
2.5 Discussion
Looking back at our paper from 2006 (Hemsley-Brown & Oplatka, 2006) and
comparing the current review with the review from 14 years ago, a number of
striking differences between our first review of the literature and the current review
arise. First, not unexpectedly, the number of hits during searches has increased
to the extent that we had to be more specific in our search terms (largely to
exclude research on pedagogy in HE). We also searched separate topics based
on headings from the previous review and new topics which emerged from these
searches. The earlier review identified 63 papers (empirical and theoretical), and the
current review: 105 (empirical and theoretical); we also excluded “opinion” papers
in both reviews, and excluded some papers based on single-institution samples due
to generalizability issues. Above all, the current review emphasizes our conclusion
that HE marketization is here to stay and extends to other areas due to increased
competition among HEIs and the continuing need for image- and brand-building.
Second, new topics and sub-topics emerged from this review, compared with
the previous review; and one topic was no longer the focus of study in marketing
HE (widening participation). New topics are social media marketing, branding,
strategic marketing, marketization, alumni and gift-giving, and the original themes
are expanded—particularly all aspects of marketing communications, which has a
new focus on online communications. Marketization, branding, and social media
marketing are the central focus of papers in HE marketing during the period
2005–2019. However, in this respect, we have a number of observations and
recommendations:
(1) Marketization research, which is predominantly “critical marketing” on the

topic, reveals that most papers are conceptual rather than empirical. This field could
expand into more empirical research, conceptual modeling for empirical research,
and a stream of robust empirical evidence for some of the concerns and claims
associated with this phenomenon, which is now a global issue.
(2) Although some authors debate the weaknesses and deficiencies of the
adoption of a market concept in HE, there are limited conceptual articles arguing
in support of the effective application of market concepts and marketization of
HE. Conceptual papers tended to take a critical stance against the market in HE,
while empirical papers tend to accept the irrevocability of the free market without
challenge as a rationale for their research. This can generate, among other things, a
sense of uncritical acceptance with current marketization in HE in terms of empirical
research.
(3) There is also potential to examine specific elements or dimensions of
marketization in more focused, empirical studies. In doing so, researchers may
shed more light on the complexity of market concepts in HE and their inherent
contradictions regarding education experience and management approaches.
Third, research in branding has also expanded considerably during this period
and there is potential to extend research in this field further by considering the
study of more focused elements or dimensions of branding currently and previously
the focus of research in marketing in other sectors. For example, brand attachment
(Grisaffe & Nguyen, 2011; Park, 2016); brand proposition (Lee et al., 2017); brand
trust (Chaudhuri & Holbrook, 2001; Lombart & Louis, 2010); brand commitment
(Chouthoy & Kazi, 2016; Shuv-Ami, 2012); and brand personality (Lombart &
Louis, 2010). A recent edited book covers strategic brand management in higher
education, and includes sections on brand strategy, planning, and measurement—
chapters on measurement, covering scale development and evaluation of scales in
HE branding, also offer a valuable resource and starting point for further research
in the field of HE branding (Nguyen et al., 2019). In contrast, the broader study
of marketing strategy is relatively fragmented, consisting of several sub-fields of
research, each of them encompassing a small amount of empirical work (e.g., the
marketing mix, market orientation, segmentation research). This reflects, by and
large, the initial stages of intellectual maturity that characterize the study of HE
marketing and may change in the future as more younger scholars join the field and
extend its areas of interest.
Notably, when looking at the type of the papers, the knowledge about HE
marketization has been produced mainly by conceptual papers. In contrast, the
other four streams of research have employed both quantitative and qualitative
research designs. Thus, 46 empirical works have used the former design. More
accurately, 10 out of 20 papers about marketing communication, advertising, and
social media, 14 out of 36 papers about branding, image and reputation, 15 out of
23 papers about marketing strategy, market orientation, segmentation and marketing
mix/relationships marketing, and 7 out of 9 papers about recruitment, alumni and
gift-giving are quantitative.
Fourth, the topic of social media marketing is still in its infancy in some respects,
and more empirical studies are required to improve understanding of social media
marketing and wider marketing communications methods, using more ambitious
research approaches. To date, papers tend to be exploratory, examining media use,
popularity, and levels of activity. Although research exploring usage, engagement,
and types of engagement offer valuable insights, there are further topics currently of
interest in the wider marketing literature that still need further study. These topics
relate to sharing, co-creation of content, and experience. For example, co-creation
of content (Fujita et al., 2019); media message sharing (Ordenes et al., 2019); user-
generated content (Roma & Aloini, 2019); social network mechanisms (Levitan,
2018); online brand communities (Munjal et al., 2019); consumer empowerment
(Bachouche & Sabri, 2019); user-sharing (Chen et al., 2018); and how these
contribute to value for users.
Finally, our review is a second attempt to chart the progress of the current
research in the study of marketing in higher education. Future reviewers of this field
will probably have much more work to do, as the field is only in its incipient phases
of scholarly development. A future review might be needed to examine the literature
in only one of the key fields we identified due to the anticipated rapid expansion of
published literature in the broader field of HE marketing.
Appendix 1 Summary of articles on marketization for the literature review
Authors, date Sample Method Summary Results

The Marketization of Higher Education
Bowl, 2018 4 universities (2 in Qualitative: How do universities balance the need to Differentiation is fundamental for the
New Zealand, and Document content maintain a successful global brand (e.g., communication strategies of universities.
2 in the UK) analysis elitism) with national expectations (e.g., Distinctiveness is essential in a marketing
equality and fair access). strategy, but also implies exclusiveness.
Commitment to social inequality is muted in
marketing communications while
performance is highlighted.
Brown, 2008 N/A Conceptual paper What are the implications, benefits, and Competition is beneficial in terms of
limitations of the marketization of higher responsiveness, innovation, and efficiency.
education? Marketization has limitations, particularly
regarding course quality and the provision of
sufficient information for customers.
Marketization contributes to polarization,
particularly in terms of social class.
Cardoso et al., N/A Qualitative: Content The study aims to determine whether the Press advertisements sought to stimulate
2011 analysis of newspaper political change in public and private demand low to high levels of aggressive
advertisements universities and polytechnics in Portugal is tactics. Many institutions, particularly
transforming students into consumers. private colleges, used more aggressive and
2 A Systematic and Updated Review of the Literature on Higher Education. . .
intrusive announcements.
Craven & N/A Conceptual The discussion focuses on social justice, Facilitating access to higher education for
Anne, 2012 particularly the unfairness of markets for disadvantaged groups is not sufficient;
students from lower socio-economic institutions must also focus on student
groups in the UK. retention and progression for these students.
55
56
Favaloro, 2015 N/A Conceptual Examines growth in student numbers and Prestige universities are much less affected
marketing investment patterns in by a highly competitive environment than
Australian universities. other universities; non-prestige universities
also face competition from non-university
providers.
Gibbs, 2007 N/A Conceptual Sets out an argument against advertising as Argues that advertising intends to persuade
a part of the marketing of higher education rather than inform, and is intrusive and
on the basis that it goes against critical manipulative, which is against the purpose
thinking and should be avoided on moral of education, which is for the common good.
grounds.
Gibbs, 2018 N/A Conceptual Challenges the marketization of higher Consumerism changes learners into
education and sets out the damage and customers, tutors into service providers, and
dissatisfaction it has wrought on education higher education has become competitive
itself. rather than collaborative; therefore trust in
higher education has now been undermined,
and dissatisfaction is the result.
Hemsley- N/A Conceptual Examines the potential damage the market A combined market and government
Brown, can inflict on higher education and raises interventionist approach is necessary for the
2011 concerns that the core values of higher benefit of those who are doubtful or unable
education are in conflict with the market to participate in HE—particularly where
there are financial barriers.
Lowrie & N/A Conceptual Focuses on the concept of marketization, Discusses the emerging themes and
Hemsley- makes a comparison with marketing and characteristics of marketization including an
Brown, explains the differences. increase in the complexity of HE; the
2011 expansion of a consumerist dialog; a focus
on rankings; and the importance of image,
identity, and brand.
Mause, 2009 N/A Conceptual The author discusses the problems which The author highlights the lack of progress by
arise through excessive marketing economists in identifying and measuring the
signaling in a highly competitive higher harmful and beneficial effects of market
education market. signaling in the HE context.
Miller & N/A Conceptual The author analyses definitions of the The author concludes that the commodification of
Brian, 2010 commodification of higher education and higher education is already well established, but
identifies what universities are selling. outcomes based on values, principles, and
liberalism are on the decline in favor of
credentials, skills, and financial gain.
Molesworth et N/A Conceptual The purpose of the paper is to explore The marketization of higher education has led to a
al., 2009 Fromm’s argument that students in decline in intellectual complexity of content and a
higher education no longer seek to “be greater focus on the workplace and employability
learners,” but rather aim simply to “get a issues.
degree.”
Natale & N/A Conceptual Higher education was once a public Academic freedom is considered less important
Doran, 2012 good, but is now a commodity. than accountability; trust less important than
utility; performance and self-interest are more
important than introspection.
Newman & N/A Conceptual The reasons behind the shift to The authors argue there is now an increasing
Jahdi, 2009 marketization and a focus on marketing focus on the 4Psa of higher education marketing
and the impact on staff and universities. and a decline of those in HE who challenge the
marketization of education.
Susanti & N/A Conceptual Explores the impact of the marketization Criticizes regulations for failing to address
Dewi, 2011 of higher education in Indonesia in the educational inequity and the impact of
context of new regulations. privatization and marketization academic values
and the purpose of higher education.
Symes & 50 advertisements Qualitative: Provides a critical analysis of Advertisements reveal careerist, vocational
Drew, 2017 (Australia) Content analysis advertisements for universities in a images dan messages portraying a university as a
of advertisements. climate of high competition inspired by higher level technical college rather than a
neo-liberalism. university.
Weymans & N/A Conceptual The authors discuss a possible alternative The authors argue that a market-orientated HE
Wim, 2010 to market orientation and traditional culture stifles creativity, professional debate, and
universities by examining key professional judgment.
dimensions of a democratic culture.
Source: The authors
57
a The 4Ps of the marketing mix: product, price, promotion, and place.
58
Appendix 2 Summary of articles on marketing communications, advertising, and social media

marketing for the literature review

Marketing communications, advertising and social media marketing
Marketing Communications
Adams & 150 university Qualitative: Content Websites were used to divide institutions Accredited residential institutions were not
Eveland, 2007 Web sites analysis of web pages into three groups: accredited residential, utilizing advantages in order to differentiate
regionally accredited online, and their institutions from online accredited and
non-accredited online. Promotional non-accredited institutions.
imagery, marketing messages, and
marketing themes were analyzed for
similarities and differences.
Bonnema & 716 prospective Quantitative survey To determine whether marketing messages Key factors identified are employability,
Van der Waldt, students need to be different for each market course content, student experience, sport
2008 segment and identify the information needs opportunities, media and social sources.
of segments of prospective students. Marketing should target individual
sub-groups of students.
Edmiston- 42 US Qualitative and Examined whether integrated marketing Institutional leadership was shown to be the
Strasser, universities; 9 quantitative: Mixed communications is applied in universities most important determinant of whether
2009 interviews methods survey and in the USA, and the process of IMC integrated marketing communications (IMC)
interviews (integrated marketing communications). was in place in universities. IMC also needs
to work alongside institutional branding and
reinforce the branding equity of the
institution.
Gribanova, N/A Conceptual paper An exploration of developments in The authors recommend that universities adopt
2016 mobile marketing communications new mobile marketing communications tools both
technology tools and the benefits to for information dissemination and engaging with
universities in engaging with these target students.
methods to communicate with students
and prospective students.
Pizarro Milian 43 home pages Qualitative and An examination of the content of Public universities emphasize research
& Davidson, (Canada) quantitative: symbolic images in promotional achievements and rankings, while colleges tended
2018 Mixed methods materials: a comparison of different to focus on short phrases and catchy tag lines.
content analysis types of HE institutions.
Popović, 2015 131 HEIs (Serbia) Qualitative: The study focuses on which types of The findings indicate that marketing
Content analysis marketing communications are used by communications are not integrated; the
of web pages HEIs, how effective they are and whether researchers recommend more exploitation of
they are integrated. interactive communications.
Royo-Vela, 121 student Qualitative and The study seeks to identify how The findings show that: advertising,
2016 respondents quantitative: universities can develop to improve their word-of-mouth, content, and information about
(Spain) image by using social media tools HEIs are the main influences on decisions.
Schüller & 20 marketing Survey and What are the problems for university The research identified management structure
Chalupský, managers (Czech interviews marketing communications, and how is problems in the universities/faculty relationship
2011 Republic) marketing communications organized? and suggests that faculties should have more
responsibility for marketing communications.
Vuori, 2015 68 homepages of Qualitative: Explores how HEIs in Finland convey Websites emphasized the relevance of programs
universities content analysis market differentiation on webpages for employability and career prospects in addition
(Finland) of web pages targeted at prospective students. to an international experience and context.
Advertising
Clayton et al., 115 American Qualitative: This research explores the message The advertising model identified five themes the
2012 higher education Content analysis strategies of institutional institutions use in messaging: campus
institutions of advertising communications by US universities’ characteristics, academics, co-curricular activities,
material schools of sports. prestige building, and mission/purpose.
(continued)
59
(continued)
60
Jan & 350 Malaysian Quantitative Examined the impact of online Intrusive advertisements on university websites
Ammari, 2016 prospective survey advertising on prospective students. have a negative impact on student’s selection of a
students university.
Papadimitriou 5 campaigns Qualitative: Explores the main marketing messages in The most salient content of advertisements were:
& Ramírez, Content analysis university advertising campaigns in five students; logos; global scope, open campus, and
2015 of advertising cities worldwide. careers and the similarity across cities was
materials notable.
Social media marketing
Bélanger et 106 Canadian Qualitative: Studies the social media of Canadian Utilizing a campus newsfeed strategy is the most
al., 2014 university social analysis of data universities to examine institutional frequently used strategy by Canadian universities
media sites from social media branding strategies for recruitment of (50%). More elite schools are highlighting the
sites home and overseas students. results of research by faculty, which attracts
students who aspire to join a prestigious
university.
Brech et al., 159 university Quantitative: Drawing on theories of self-presentation Universities with a strong reputation tend to have
2017 social media analysis of social and community engagement, the authors more Facebook fans; however, a large fan-base
websites media data develop a theoretical model to explain negatively impacts on engagement.
engagement with social media marketing
and the outcome factors.
Brendzel- 16 Polish Quantitative: To identify how universal the use of Facebook is the most common sites for
Skowera & universities analysis of social social media sites is for Polish universities in Poland to have a presence. Most
Lukasik, 2016 media data universities. interaction is with students and potential students
and attempts to encourage two-way
communication.
Clark et al., 24 respondents Quantitative: The research examines whether social The results show a positive relationship between
2017 (USA) survey media engagement impacts on the extent to which students follow a university on
relationship quality in relation to social media and their perception of establishing a
universities and students. high-quality relationship with their university.
Using several multi-media platforms to follow a
university leads to higher perceived relationship
quality.
Linvill et al., 52 university Qualitative: The study used content analysis to Findings indicate that Pinterest is not being used
2015 Pinterest accounts Content analysis develop a typology of university pins effectively by institutions. Using Pinterest to share
(USA) of social media based on information sharing, images of news and past events is not in line with
accounts community, group experience and the best qualities and features of Pinterest.
promotion.
Palmer, 2013 6 universities’ Quantitative: An examination in the use of Twitter by Varying levels of activity on Twitter were
Twitter sites analysis of social Australian universities. recorded, and a high level of retweeting (sharing)
(Australia) media accounts relies on the number of followers.
Peruta & Facebook posts 66 Qualitative: Differences in postings by frequency and The type of institution is a factor in Facebook
Shields, 2017 universities Content analysis type level of engagement based on the posting strategies and type of engagement. Photos
(USA) of Facebook posts type of institution. are less frequently posted by liberal arts colleges.
Zailskaite- 14 HEIs Qualitative: Case The aim of the study was to build a Findings show that three aspects of
Jakste & (Lithuania) study and content model of communication in social media communication in social media are key:
Kuvykaite, analysis and to test the model in public higher communication between business organizations
2012 education institutions in Lithuania. and consumers; consumer-to-consumer
communication; communication between
consumer and business organizations (feedback).
61
62
Appendix 3 Summary of articles on branding, image, and reputation for the literature review

Branding, image, and reputation
Branding
Aula & Convenience Qualitative: Content Analyses data from one new university University branding is predominantly an
Tienari, 2015 sample of 3 HE analysis of created in a merger argue that branding is organizational purpose linked to identities
institutions stakeholder texts and characterized by different interests among and (non-)identification of internal and
involved in a in-depth interviews players, with different means to influence external players in a socio-cultural and
merger brand development. societal context. It goes beyond marketing
and marketing communications.
Baker et al., 490 respondents Qualitative: focus Research focuses on understanding the People construct their own beliefs about
2005 groups with faculty drivers for attitudes towards university University corporate partnerships, and this is
and staff brands. not necessarily related to reality. The
respondents held both positive and negative
views about the benefits of university
corporate partnerships.
Chapleo, 2007 Judgment sample Qualitative interviews To explore current knowledge and opinion Reputation is viewed as distinctly different
of 15 respondents with executives of of issues and perceptions of potential and from brand, which has commercial
UK universities actual barriers to branding by academics in connotations. Brand is a less favorable term
UK universities. in an HE context due to commercial
associations. However, CEs perceived their
own institution as having a distinctive brand.
Chapleo, 2011 In-depth Qualitative interviews Seeks to establish whether branding Universities, on the whole, have not found
Interviews with with marketing activity in universities should be measured commercial approaches to branding helpful
20 managers managers of UK and how it should be measured. or relevant to the complex contexts of a
universities university. Newer universities were more
able to use the commercial definition of
branding than older universities, where the
concept of reputation was a closer match
with their activities.
Chapleo, A snowball Qualitative To gain a clearer understanding of An overall conceptual model of branding for
2015a sample of 15 interviews with university branding and the challenges of universities remains elusive since commercial
managers marketing applying theories of branding to branding models are not a good fit. Corporate
managers of UK universities. branding and experiential branding have some
universities applicability to universities.
Chapleo, 55 respondents Qualitative: Explore the challenges of university Universities are considered too complex to apply
2015b from 32 UK interviews with branding and the key features that a commercial branding model, particularly for
universities managers from contribute to making university branding stating a concise brand proposition. The complex
UK universities different from commercial branding. nature of higher education and the multiple
stakeholders means that conventional branding
models are limited.
de Heer & 2 case study Qualitative and The study examines the benefits of The benefits of branding are: differentiating the
Tandoh-Offin, universities in quantitative: university branding and whether institution; greater loyalty from staff and students;
2015 Ghana; 50-minute (interviews and enhanced performance and increased credibility to charge high tuition fees; and greater
interviews secondary data recruitment of students and faculty result loyalty from all stakeholders.
analysis) from a successful branding strategy.
Delmestri et 821 university Qualitative: The study examines universities’ Findings show that more abstract visual icons are
al., 2015 home pages Content analysis branding and visual self-representation, expression has dominance in the west, but there is
of university icons in an international context. The focus is a greater diversity of logos and logo types in other
and visual images on the images, icons, and logos parts of the world.
employed on the Internet to promote the
institutions.
Fay & 109 HEIs (USA) Quantitative: Examines whether changes in tenure and Private universities are early adopters of branding
Zavattaro, secondary data increased teaching loads have affected strategies, but they are not followed by their
2016 analysis dyadic institutional branding initiatives. Focuses public university counterparts. Universities tend
event history on how and when universities adopt a to re-brand as their performance and income
analysis branding and re-branding strategy. increase.
Furey et al., 35 interviews with Qualitative: To explore the nature of different brand There is evidence of the application of branding,
2014 university Inductive content promises in different types of universities and brand promise. Key themes provide the basis
managers; 85 analysis and in the UK. of distinctions between universities: environment,
websites interviews experiences, status, heritage and global
63
positioning
64
(continued)
Hemsley- 30 respondents Qualitative and The study aims to examine key factors in The findings indicate that universities need to
Brown & and 2 managers quantitative: the development of a articulate and develop their brand by constructing
Goonawar- applicant data university and business school brand in clear and effective brand architecture and
dana, analysis; in-depth an international context and focuses on maintaining coherent and harmonious links within
2007 interviews the development of the brand the brand architecture.
architecture.
Holmberg & 32 interviews; and Qualitative and Explores the benefits of university The culture of the university school was identified
Stannegård, secondary data quantitative branding as a way of supporting students’ as marketized managerialism, which was reflected
2015 sources interviews; self-branding and brand identity. in the students who adopted the same ideas and
content analysis language in discussions about their future.
from documents
Judson et al., 157 respondents Quantitative Investigates the internal communication Private and major university respondents were
2006 survey of the university brand using a sample positive about all forms of communications and
from private, major and mid-major found them to be more effective than those from
universities. less major universities, who believed that the
brochure was the most effective.
Judson et al., 343 university Quantitative The study focused on how well Respondents from private HEIs reported greater
2009 administrators survey university administrators understood brand clarity in internal brand communications
their university brand and whether they than respondents from public HEIs.
implemented their knowledge of the
brand as part of their role.
Kosmützky & 110 university Qualitative: Do university mission statements Results show that mission statements enable
Krücken, 2015 mission Content analysis differentiate the institution from its universities to claim a market position in
statements and discourse competitors? comparison with competitors that highlights both
analysis of the similarities with other universities and the
institution texts differences.
Lee et al., Historical Quantitative: The purpose of the study was to identify International students who are able to integrate
2017 statistics on Statistical the key factors associated with brand with their peers through a high level of language
international secondary data satisfaction from a student perspective. skills (e.g., English) are more likely to be satisfied
students in Korea analysis with their school and study program.
Lomer et al., N/A Conceptual Explores the emergence, development, The development of the branding of higher
2018 and growth of the branding of higher education in Britain was a commodification of HE
education in the UK since the “Britain is and stressed the elitism and high quality of British
great” campaign of 1999. education, targeted at international students.
Rooksby & 10265 university Quantitative and To explore trademarks and brand Universities continue to use trademarks as a way
Collins, 2016 trademarks (USA) qualitative: activities by universities to establish what of positioning their institutions in the market and
analysis of these activities reveal about university to support market value and market signaling.
multiple datasets branding.
Rose et al., 127 students; 240 Quantitative The role of university heritage in Developed a measure for brand heritage; the
2017 respondents; surveys marketing communications of heritage factor had a greater influence on parents
universities. than on prospective students.
Rutter et al., 10 university Qualitative: The study seeks to explore how brand The study identifies key brand personality
2017 prospectuses Content analysis personality is communicated in the positioning based on sophistication, competence,
(UK) of prospectuses marketing communications of sincerity, ruggedness and excitement.
universities.
Sataøen, 2015 39 and 36 HEIs Qualitative and The research studies how different core Findings suggest there are clichéd brand values
(Norway and quantitative values are used in the branding of associated with the branding of HEIs of the two
Sweden) content analysis universities in two countries. countries and tendencies towards both conformity
of prospectuses and differentiation.
Temple, 2006 N/A Conceptual Examines the meaning of branding and Concludes that branding in universities is best
presents a case using examples to described and reputation management and
analyze the current so-called branding ensuring the university has a strong vision.
activities of universities.
Winter & 15 HEI managers Qualitative: The study explores how universities of HEIs are not writing copy for any specific target
Thompson- (UK); 105 content analysis different types exploit an HEI’s location audience; references to location tend to mix
Whiteside, prospectuses in their marketing. messages of both safety and excitement.
2017 (UK)
(continued)
65
(continued)
66
Image and reputation

Angulo-Ruiz Random sample Quantitative: To develop and test a student retention Early personal and institutional characteristics
& Pergelova, of 396 students survey model including system and institutional (e.g., higher education institutional image;
2013 dropout as outcome variables and students’ goal commitment), institutional
examines differences in factors that experience and integration of students to the
affect them. Models the image of the academic environment influence the level of
institution as an influence on institutional student performance and institutional
commitment and dropout intention. commitment, which further influence dropout
decisions.
Brown & 373 student Quantitative: The study focuses on customer Institutional image positively predicted perceived
Mazzarol, respondents survey (Partial satisfaction as a driver for loyalty in value and student satisfaction
2009 Least Squares) higher education settings in Australian
universities.
Chen, 2016 115 former HE Quantitative: The research explores brand image, Brand image is significantly related to loyalty.
students in Taiwan survey loyalty, and positive word-of-mouth: University students’ brand image, satisfaction,
sharing satisfying experiences and and loyalty positively influence the sharing of
recommendations to others. satisfying experiences and making
recommendations to others.
Drori et al., 826 university Qualitative: The research seeks to study universities’ Four institutional narratives are identified:
2016 home pages Content analysis identities and narratives through the guild-like classic, professional scientific, localized
of internet front visual images, logos, and icons presented national pride, and universal organization
page icons on their websites. narrative.
Finch et al., 30 manager Mixed methods: The research focuses on the attributes Findings indicate that each category of higher
2013 respondents; 164 interviews and and the implications of reputation for education institution displays distinctly different
manager quantitative three categories of higher education reputational attributes which facilitate
respondents survey institutions. competitiveness in the market.
Huisman & Documents and Qualitative and The study focuses on asking how Despite competition in the UK HE sector, there
Mampaey, websites from 29 quantitative: universities want students, stakeholders, were few images that deviated from the
2018 UK universities content analysis and the outside work to view them: how mainstream, although there were some examples
similar and how distinctive are UK of distinctiveness.
universities’ images?
Idris & 888 international Quantitative: The study focused on whether the The study found that perceptions of a university
Whitfield, respondents survey corporate visual identity influences a [created for the study] were more positive when
2014 viewer’s perception of a university and images of a male lecturer were shown, and
its lecturing staff. reputation and trustworthiness were more positive
when a traditional logo was used.
Lee & Chen, 5 discussants and Qualitative: The study aimed to identify image asset The study found that in terms of asset
2018 15 respondents Delphi management factors using a Delphi management, stakeholder image and student
methodology technique. image were a top priority for managers in HEIs.
Plungpongpan 6 parents and 6 Qualitative: Examines the impact of university University corporate social responsibility (USR)
et al., 2016 students from 3 interviews corporate social responsibility (USR) on is not a factor in the choice of university and is
universities the brand image of a university. not very visible to prospective students, but it is
(Thailand) viewed positively and needs more effective
marketing.
Saleem et al., 747 students Quantitative: To test the effect of service quality on Service quality positively relates to student
2017 (Pakistan) survey student satisfaction and the moderating satisfaction and is strengthened by the culture;
effect of university culture, price, and price and reputation negatively impact this
reputation. relationship.
Steiner et al., N/A Conceptual The theoretical paper develops a Universities’ identities and reputation strategies
2013 multidimensional model identifying comprise four dimensions: organizational identity,
internal and external factors influencing symbolic identity, image, and reputation.
university identity and reputation.
Tolbert, 2014 59 institutions Qualitative: The research focuses on to what extent Findings show that focusing on faith in marketing
(USA) Content analysis the branding of university focuses on its materials was low: 14%-20% on home pages and
of marketing faith (Christian) dimension. 18%–19% on admissions materials.
materials
Wolverton, 3 business schools Qualitative: The study focuses on university identity Institutional identity was used to identify
2006 (USA) Descriptive case and how this is conveyed to the target institutional strengths, which in turn were used for
research audience in the marketing materials. targeting marketing based on the institutional
identity.
Source the authors
67
68
Appendix 4 Summary of articles on marketing strategy, market orientation, segmentation, targeting

and positioning, marketing mix and relationship marketing for the literature review

Marketing strategy, market orientation, segmentation, targeting and positioning, marketing mix and relationship marketing
Marketing strategy
Alhakimi & 21 managers from Qualitative and This study examined the level of utilization Positive and significant relationships were
Qasem, 2014 universities in Quantitative: survey and effectiveness of marketing strategies in found between usage and perceived
Yemen and descriptive private universities recruiting effectiveness of marketing strategies.
analysis undergraduate students. Marketing planning and resource analysis
were utilized and perceived to be highly
effective; a market segmentation strategy
was least utilized.
Mogaji, 2016 134 university Qualitative: Content The purpose of the paper is to scrutinize During the last stages of recruitment, most
websites (UK) analysis of webpages the marketing strategies adopted by UK universities substantially revised their
universities on their websites at the end of websites to focus on student satisfaction,
the recruitment process. employability and availability of
accommodation
Naidoo & Wu, 570 marketing Quantitative: survey The study explores the implementation of Implementing a successful market strategy is
2011 managers (SEM) internal marketing strategies in based on a marketing manager’s
universities. commitment to the strategy and to their role,
especially in a highly competitive market.
Nkala et al., 80 academics Qualitative: Examines the service quality strategies Findings suggest that expectations of service
2014 (Zimbabwe) interviews using universities in Zimbabwe are using the quality and delivery are high among
content analysis challenge competition in higher education. Zimbabwean students, but service
experience is low.
Schofield et 19 institutions (10 Qualitative: inductive The research study examines the marketing Newer universities are more influenced by a
al., 2013 HE; 9 FE) (UK) thematic analysis strategies of different types of higher government policy agenda than older, more
education institutions. traditional institutions that continue to rely
on their reputation.
Market orientation
Alnawas & 21 in-depth Qualitative and To conceptualize, and empirically The measurement instrument for PSO, (36 items)
Phillips, 2014 interviews; 13 quantitative: examine the constructs which constitute was valid and reliable using 6 multi-item
pilot study interviews; and the concept of Prospective Student components: information generation; information
respondents survey Orientation (PSO). The study identified use; inter-functional coordination; intra-functional
three second-order formative constructs coordination; managing recruitment / promotional
to measure the concept of PSO. activity; and managing prospective.
Baber & 119 respondents Quantitative: The study examines the relationships Private university environments with low
Upadhyay, from private survey between market orientation and the competition perform better than private
2015 universities in economic performance of universities in universities which operate in highly competitive
India Indian. environments.
Camelia & N/A Conceptual paper Reviews the literature and discusses the One of the most important benefits of a market
Marius, 2013 importance of market orientation in orientation strategy is the increased performance
higher education institutions. of the institution. Other improvements include
increased enrolments and retention of students.
Pavičić et al., 87 university staff Quantitative How market oriented are Croatian Croatian universities demonstrate an
2009 (Croatia) survey universities, and how far do they gather underdeveloped market orientation despite
information appropriate for different recognition of the significance of developing
stakeholders? stakeholder relationships.
Ross et al., 159 international Quantitative: The study examines market orientation The findings show that ISR departments of HE
2013 recruitment survey strategies (by individual components) to institutions largely adopt a customer orientation
practitioners establish how they influence international approach rather than a competitor or
(Australia) student recruitment (ISR) performance. inter-functional focus.
Tayar & Jack, 4 university case Qualitative: data The researchers examine investigate the The case universities were motivated by income
2013 studies (Australia) analysis behavior and perspectives of university and a quest for a prestigious reputation; they were
managers making decisions about risk averse and influenced by past mistakes.
entering an international market.
(continued)
69
70
(continued)
Segmentation Targeting and Positioining (STP)
Durkin et al., 8 marketing Qualitative: data Explores university marketing managers’ Highlights the lack of awareness and failure to
2014 managers; 10 analysis and awareness and perceived importance of capitalize on the use of online communications to
websites content analysis market segmentation, in particular target and communication with more mature
segments of more mature students and prospective students.
the use of the Internet for marketing
communications.
Ghosh et al., Secondary data Quantitative: Exploratory research to establish whether Cluster analysis of enrolment data resulted in five
2008) 16,000 students secondary data using segmentation analyses offers distinct segments based on ten segmentation
analysis improved use of resources and adoption variables.
of a targeted marketing strategy.
Mazzarol & 258 HE Quantitative: data Explores the market positioning behavior The findings indicate that it is difficult for
Soutar, 2008 institutions analysis (cluster of Australian universities to examine universities to maintain quality and growth with a
(Australia) analysis) whether the strategy impacted their simultaneous cost-cutting strategy, especially in a
competitiveness in the market. context of high competition.
McAlexander 4834 alumni (2 Quantitative: The study examines the factors that The findings demonstrate the importance of using
et al., 2016 US universities) survey impact philanthropic giving by university segmentation and targeting approach for
alumni. identifying and engaging with alumni to
maximize alumni giving.
Marketing Mix (4Ps: price, product, promotion, and place)
Artur, 2015 100 school Qualitative: online Explores and seeks to gain an The Internet is a crucial element of the marketing
marketing survey understanding of the role and importance activity of universities in Poland; however, the
managers of the Internet in the marketing activity utilization of the Internet by universities has not
of higher education institutions in kept up-to-date with the rapid developments in
Poland. Internet technology.
Enache, 2011 All universities in Quantitative: The 7Psa framework is applied to gain a Applying the 7Psa model from services marketing
Romania Secondary data better understanding of the Romanian provides a useful framework for understanding,
analysis of higher education market. analysing, and managing the market forces that
enrolled students affect the higher education market.
Moogan, 2011 318 applicants Qualitative and The study was designed to identify the The study found that while marketing
4 managers Quantitative: key marketing communications activities communications, particularly program
survey needed to meet students’ needs when information, was readily available for students,
questionnaire; applying to university to inform more online resources would be welcome by
in-depth university marketing managers. undergraduate applicants.
interviews
Relationship Marketing
de Macedo 352 Brazilian Quantitative: The study examines and tests the The strongest contributing variable to student
Bergamo et students survey; multiple association between relationship quality customer loyalty is the perceived quality of the
al., 2012 linear regression and loyalty to an institution. institution. Trust and commitment also positively
influence student customer loyalty.
Iskhakova et N/A Conceptual The theory and perspectives used in The results reveal that alumni research is
al., 2017 studies of alumni loyalty and the conducted within a number of research areas
constructs, measurement and scales used including services marketing, education science;
in the studies. relationship marketing; management, charitable
giving and microeconomics.
Kim et al., 302 respondents Quantitative: Examines students’ identification with Students’ strong identification with their sports
2010 from 3 Korean survey (structural their university to measure belongingness program is positively related to their strong
universities equation and satisfaction. identification with the university as a whole and
modeling) leads to a sense of belongingness & support for
the university.
Purgailis & 2010 university Quantitative: Examined the relationship between Student-perceived quality comprises academic
Zaksa, 2012 students (Latvia) survey (structural service quality and student loyalty to the staff, study content, readiness for labor market,
equation HEI (University of Latvia). and acquired skills, which positively influence
modeling). student loyalty to the HEI.
Schlesinger et 1000 university Quantitative: The study examines the role of brand The findings show that university image plays a
al., 2017 alumni (Spain) survey (structural image, trust, satisfaction, and shared key role, and four variables determine student
equation values as a direct and indirect loyalty: graduate satisfaction, trust, shared values,
modeling) explanation of alumni loyalty. and image.
Source: The authors.
a The 7Ps are the 7 elements of the marketing mix (price, promotion, product, place (4Ps) plus: people, process, physical evidence (3Ps).
71
72
Appendix 5 Summary of articles on recruitment, alumni, and gift-giving for the literature review

Recruitment, alumni and gift-giving (excluding choice and consumer behavior)
Asaad et al., 63 international Quantitative: survey Aims to investigate export market Concludes that four higher education
2015 managers in UK (Partial least squares orientation export-specific variables are important
universities analysis) (EMO) in the context of British drivers of EMO in universities. Also
universities when recruiting international confirms the direct effects of EMO on
students university export performance and (indirect
effects) mediated through international
reputation.
Frølich et al., University Qualitative and Explores the recruitment strategies of HEIs The findings indicate that all universities in
2009 admissions data quantitative: and examines how these strategies are the study focused on the traditional market
(Norway); Secondary data on linked to their market position. where they faced competition from other
recruitment HEIs in Norway; 10 institutions. Institutions tended not to
managers interviews capitalize on their uniqueness in the market.
Hulme et al., N/A Conceptual paper The study focuses on international The authors reveal contradictions in the
2014 recruitment agents and questions, how stated aims of international recruitment via
agents justify their value, why universities agents and the focus is on widening
are employing agents, and how potential participation and offering opportunities; in
students view agents. reality, they reveal a market- and
profit-driven system.
Onk & Joseph, Top 10 world Qualitative and The study focuses on what emerging Priorities for successful international
2017 economies quantitative: universities can learn from the strategies recruitment are: identifying sources of
Secondary data employed by established universities funding, setting up an international office,
analysis regarding student recruitment. recruitment team, and recruitment process.
Pippert et al., 10,000 images, Qualitative: Content The study focuses on the way ethnic Images were presented showing
2013 165 institutions analysis minorities and racial groups are portrayed over-representation of African-American
(USA) (photographic in university recruitment materials. students: these students were
images) over-represented in institutions’ materials.
Alumni and gift-giving
Alnawas & 22 alumni from 6 Qualitative and To conceptualize and examine the Confirms 48 item measurement instrument for AO
Phillips, 2015 UK universities; quantitative: concept of Alumni Orientation (AO) is valid and reliable using nine multi-item
10 pilot in-depth using a discovery-oriented approach. A components: Case for Support; Alumni Database
respondents interviews; pilot self-administrated survey to validate the Management; Social Media Management;
study to develop a six identified constructs that form AO. Financing Event Management; Financing
measurement Publication Management; Promoting Best
scale. Practices of Event Management; Promoting Best
Practices of Publication Management;
Intra-functional Coordination; and
Inter-functional Coordination.
Pedro et al., 2544 alumni Quantitative: To explore the satisfaction, image, and The findings indicate that communication and
2018 (Portugal) survey (structural commitment of university alumni. satisfaction with the social and academic
equation environment at their university contribute to a
modeling) positive alumni commitment relationship.
Stephenson & 2852 alumni Quantitative: The study focuses on university brand Findings indicate that motivation for alumni
Bell, 2014) (USA) survey (negative identification and the willingness to donating is explained by using social identity
binomial make alumni donations. theory.
regression)
Stephenson & 2763 alumni Quantitative: The study examines the relationship Findings show that alumni satisfaction with
Yerger, 2015 (USA) survey (multiple between alumni satisfaction, brand student affairs and satisfaction with campus
regression) identification, university brand image, resources positively affect brand identification and
promotion of the institution and choosing brand image.
to donate.
Source: The authors
73
References
Abu Bakar, A. R., & Abdu Talib, A. N. (2013). A Case Study of an Internationalization Process
of a Private Higher Education Institution in Malaysia. Gadjah Mada International Journal of
Business, 15(3), 211–230.
Adams, J., & Eveland, V. (2007). Marketing Online Degree Programs: How Do Traditional-
Residential Programs Compete? Journal of Marketing for Higher Education, 17(1), 67–90.
Alhakimi, W., & Qasem, A. A. (2014). Toward an understanding of Marketing Strategies in Higher
Education Institutions. Euro Asia Journal of Management, 24(1/2), 23–35.
Alnawas, I., & Phillips, C. (2014). Prospective student orientation in higher education: Develop-
ment of the construct. Journal of Customer Behaviour, 13(2), 135–163.
Alnawas, I., & Phillips, C. (2015). Alumni Orientation: Development of the Construct. Journal of
Nonprofit & Public Sector Marketing, 27(2), 183–215.
Angulo-Ruiz, L. F., & Pergelova, A. (2013). The Student Retention Puzzle Revisited: The Role of
Institutional Image. Journal of Nonprofit & Public Sector Marketing, 25(4), 334–353.
Artur, K. (2015). The Internet as an element of Marketing Strategies of Polish Universities During
Recent Years. Economic Processes Management, 4, 80–91.
Asaad, Y., Melewar, T. C., & Cohen, G. (2015). Export market orientation behavior of universities:
the British scenario. Journal of Marketing for Higher Education, 25(1), 127–154.
Aula, H. M., & Tienari, J. (2015). The University Branding Game. International Studies of
Management & Organization, 45(2), 164–179.
Baber, R., & Upadhyay, Y. (2015). Examining the Role of Competition Intensity as Moderator on
Market Orientation and Performance Relationship in Private Universities. South Asian Journal
of Management, 22(1), 97–113.
Bachouche, H., & Sabri, O. (2019). Empowerment in marketing: synthesis, critical review, and
agenda for future research. AMS Review, 1, 1–20.
Baker, S. M., Faircloth, J. B., & Simental, V. (2005). Perceptions of University—Corporate
partnership influences on a brand. Journal of Marketing Theory & Practice, 13(2), 32–46.
Baldwin, G., & James, R. (2000). The Market in Australian Higher Education and the Concept of
Student as Informed Consumer. Journal of Higher Education Policy and Management, 22(2),
139–148.
Barrett, L. R. (1996). On Students as customers—some warning from America. Higher Education
Review, 28(3), 70–71.
Bélanger, C. H., Bali, S., & Longden, B. (2014). How Canadian universities use social media to
brand themselves. Tertiary Education & Management, 20(1), 14–29.
Binsardi, A., & Ekwulugo, F. (2003). International Marketing of British Education: research on
the students’ perception and the UK market penetration. Marketing Intelligence & Planning,
21(5), 318–327.
Bonnema, J., & Van der Waldt, D. L. R. (2008). Information and Source Preferences of a Student
Market in Higher Education. International Journal of Educational Management, 22(4), 314–
327.
Booth, A. (2001). Cochrane or cock-eyed? How should we conduct systematic reviews of
qualitative research? Paper presentation the Qualitative Evidence-based Practice Conference.
In “Taking a Critical Stance” (Ed.), ( University of Coventry.
Bornholt, L., Gientzotis, J., & Cooney, G. (2004). Understanding Choice Behaviours: Pathways
from School to University with Changing Aspirations and Opportunities. Social Psychology of
Education, 7(2), 211–228.
Bowl, M. (2018). Differentiation, distinction and equality—or diversity? The language of the
marketised university: an England, New Zealand comparison. Studies in Higher Education,
43(4), 671–688.
Brech, F. M., Messer, U., Vander Schee, B. A., Rauschnabel, P. A., & Ivens, B. S. (2017). Engaging
Fans and the Community in Social Media: Interaction with Institutions of Higher Education on
Facebook. Journal of Marketing for Higher Education, 27(1), 112–130.
Brendzel-Skowera, K., & Lukasik, K. (2016). Polish Universities in Social Media. Valahian
Journal of Economic Studies, 7(4), 29–36.
Brookes, M. (2003). Higher Education: marketing in a quasi-commercial service industry.
International Journal of Non-Profit and Voluntary Sector Marketing, 8(2), 1465–1520.
Brown, R. (2008). Higher education and the market. Perspectives: Policy & Practice in Higher
Education, 12(3), 78–83.
Brown, R. M., & Mazzarol, T. W. (2009). The importance of institutional image to student
satisfaction and loyalty within higher education. Higher Education, 58(1), 81–95.
Camelia, G., & Marius, P. (2013). Incorporating Market Orientation in Higher Education Institu-
tions. Annals of the University of Oradea, Economic Science Series, 22(1), 1743–1752.
Cardoso, S., Carvalho, T., & Santiago, R. A. (2011). From students to consumers: reflections on the
marketisation of Portuguese higher education. European Journal of Education, 46(2), 271–284.
Chapleo, C. (2007). Barriers to brand building in UK universities? International Journal of
Nonprofit & Voluntary Sector Marketing, 12(1), 23–32.
Chapleo, C. (2011). Exploring rationales for branding a university: Should we be seeking to
measure branding in UK universities? Journal of Brand Management, 18(6), 411–422.
Chapleo, C. (2015a). An exploration of branding approaches in UK universities. International
Journal of Nonprofit & Voluntary Sector Marketing, 20(1), 1–11.
Chapleo, C. (2015b). Brands in Higher Education. International Studies of Management &
Organization, 45(2), 150–163.
Chaudhuri, A., & Holbrook, M. B. (2001). The chain of effects from brand trust and brand affect
to brand performance: the role of brand loyalty. Journal of Marketing, 65(2), 81–93.
Chen, L. H. (2008). Internationalization or International Marketing? Two Frameworks for Under-
standing International Students’ Choice of Canadian Universities. Journal of Marketing for
Higher Education, 18(1), 1–33.
Chen, C. T. (2016). The Investigation on Brand Image of University Education and Students’ Word-
of-Mouth Behavior. Higher Education Studies, 6(4), 23–33.
Chen, T., Drennan, J., Andrews, L., & Hollebeek, L. D. (2018). User experience sharing. European
Journal of Marketing, 52(5/6), 1154–1184.
Chouthoy, S., & Kazi, R. (2016). En route to a Theory—Building Consumer Brand Commitment
through CSR Reputation. Global Business & Management Research, 8(3), 67–82.
Clark, M., Fine, M. B., & Scheuer, C. L. (2017). Relationship Quality in Higher Education
Marketing: The Role of Social Media Engagement. Journal of Marketing for Higher Education,
27(1), 40–58.
Clarke, M. (2007). The Impact of Higher Education Rankings on Student Access, Choice, and
Opportunity. Higher Education in Europe, 32(1), 59–70.
Clayton, M. J., Cavanagh, K. V., & Hettche, M. (2012). Institutional Branding: A Content Analysis
of Public Service Announcements from American Universities. Journal of Marketing for
Conway, T., Mackay, S., & Yorke, D. (1994). Strategic Planning in Higher Education: who are the
customers? The International Journal of Educational Management, 8(6), 29–36.
Craven, A., & Anne, C. (2012). Social justice and higher education. Perspectives: Policy &
Practice in Higher Education, 16(1), 23–28.
Delmestri, G., Oberg, A., & Drori, G. S. (2015). The Unbearable Lightness of University Branding.
International Studies of Management & Organization, 45(2), 121–136.
Drori, G. S., Delmestri, G., & Oberg, A. (2016). The Iconography of Universities as Institutional
Narratives. Higher Education: The International Journal of Higher Education Research, 71(2),
163–180.
Durkin, M., Filbey, L., & McCartan-Quinn, D. (2014). Marketing to the mature learner: exploring
the role of web communications. Service Industries Journal, 34(1), 56–70.
Edmiston-Strasser, D. M. (2009). An Examination of Integrated Marketing Communication in U.S.
Public Institutions of Higher Education. Journal of Marketing for Higher Education, 19(2),
142–165.
Enache, I. C. (2011). Marketing Higher Education Using the 7Ps Framework. Bulletin of the
Transilvania University of Brasov. Series V: Economic Sciences, 4(1), 23–30.
Favaloro, C. (2015). Marketing in the Australian Higher Education Sector. Journal of Higher
Education Policy and Management, 37(5), 490–506.
Fay, D. L., & Zavattaro, S. M. (2016). Branding and Isomorphism: The Case of Higher Education.
Public Administration Review, 76(5), 805–815.
Finch, D., McDonald, S., & Staple, J. (2013). Reputational interdependence: an examination of
category reputation in higher education. Journal of Marketing for Higher Education, 23(1),
34–61.
Frølich, N., Brandt, S., Høvdhaugen, E., & Aamodt, P. O. (2009). Coping by copying? Higher
education institutions’ student recruitment strategies. Tertiary Education & Management,
15(3), 227–240.
Fujita, M., Harrigan, P., & Soutar, G. N. (2019). The strategic co-creation of content and student
experiences in social media. Qualitative Market Research: An International Journal, 22(1),
50–69.
Furey, S., Springer, P., & Parsons, C. (2014). Positioning University as a Brand: Distinctions
between the Brand Promise of Russell Group, 1994 Group, University Alliance, and Million+
Universities. Journal of Marketing for Higher Education, 24(1), 99–121.
Ghosh, A. K., Javalgi, R., & Whipple, T. W. (2008). Service Strategies for Higher Educational
Institutions Based on Student Segmentation. Journal of Marketing for Higher Education, 17(2),
238–255.
Gibbs, P. (2001). Higher Education as a Market: a problem or a solution? Studies in Higher
Education, 26(1), 85–94.
Gibbs, P. (2002). From the Invisible Hand to the Invisible Hand-Shake: marketing higher education.
Research in Post Compulsory Education, 7(3), 325–338.
Gibbs, P. (2007a). Does advertising pervert higher education? is there a case for resistence? Journal
of Marketing for Higher Education, 17(1), 3–11.
Gibbs, P. (2007b). Does Advertising Pervert Higher Education? Is There a Case for Resistance?
Journal of Marketing for Higher Education, 17(1), 3–11.
Gibbs, P. (2018). Higher education marketing—does inducing anxiety facilitate critical thinking or
more consumerism? Journal of Marketing for Higher Education, 28(1), 1–11.
Gibbs, P., & Knapp, M. (2012). Marketing higher and further education: An educator’s guide to
promoting courses, departments and institutions. Routledge.
Google Scholar. (2019). Citations for “University in a competitive global marketplace: a sys-
tematic review of the literature on higher education marketing”. https://scholar.google.co.uk/
citations?user=JaFl1IoAAAAJ&hl=en [On-line].
Gribanova, A. (2016). Mobile Communication Technologies as an Integrated Marketing Com-
muncations Instrument in Promoting Higher Education Institutions. Journal of Business
Management, 11, 114–125.
Grisaffe, D. B., & Nguyen, H. P. (2011). Antecedents of emotional attachment to brands. Journal
of Business Research, 64(10), 1052–1059.
Grönroos, C. (1997). From marketing mix to relationship marketing: towards a paradigm shift in
marketing. Management Decision, 35(4), 322–329.
de Heer, F., & Tandoh-Offin, P. (2015). Exploring the Benefits of Branding Universities: A
Developing Country Perspective. IUP Journal of Brand Management, 12(4), 58–71.
Hemsley-Brown, J. (2011). Market heal thyself: the challenges of a free market in higher education.
Journal of Marketing for Higher Education, 21(2), 115–132.
Hemsley-Brown, J., & Goonawardana, S. (2007). Brand Harmonization in the International Higher
Education Market. Journal of Business Research, 60(9), 942–948.
Hemsley-Brown, J., & Oplatka, I. (2006). Universities in a competitive global marketplace: a
Hemsley-Brown, J., & Oplatka, I. (2015a). Higher Education Consumer Choice. (e-book: http://
www.palgrave.com/gp/book/9781137497185 ed.) London: Palgrave.
Hemsley-Brown, J., & Oplatka, I. (2015b). University choice: what do we know, what don’t we
know and what do we still need to find out? International Journal of Educational Management,
29(3), 254–274.
Holmberg, I., & Stannegård, L. (2015). Students’ Self-Branding in a Swedish Business School.
International Studies of Management & Organization, 45(2), 180–192.
Huisman, J., & Mampaey, J. (2018). Use your imagination: what UK universities want you to think
of them. Oxford Review of Education, 44(4), 425–440.
Hulme, M., Thomson, A., Hulme, R., & Doughty, G. (2014). Trading places: The role of agents in
international student recruitment from Africa. Journal of Further and Higher Education, 38(5),
674–689.
Idris, M. Z., & Whitfield, T. W. A. (2014). Swayed by the Logo and Name: Does University
Branding Work? Journal of Marketing for Higher Education, 24(1), 41–58.
Iskhakova, L., Hoffmann, S., & Hilbert, A. (2017). Alumni Loyalty: Systematic Literature Review.
Journal of Nonprofit & Public Sector Marketing, 29(3), 274–316.
Ivy, J. (2001). Higher Education institution image: a correspondence analysis approach. The
International Journal of Educational Management, 15(6), 276–282.
Jan, M. T., & Ammari, D. (2016). Advertising Online by Educational Institutions and Students’
Reaction: A Study of Malaysian Universities. Journal of Marketing for Higher Education,
26(2), 168–180.
Judson, K. M., Gorchels, L., & Aurand, T. W. (2006). Building a University Brand from within: A
Comparison of Coaches’ Perspectives of Internal Branding. Journal of Marketing for Higher
Education, 16(1), 97–114.
Judson, K., Aurand, T., Gorchels, L., & Gordon, G. (2009). Building a University Brand from
Within: University Administrators’ Perspectives of Internal Branding. Services Marketing
Quarterly, 30(1), 54–68.
Kim, T., Chang, K., & Jae Ko, Y. (2010). Determinants of organisational identification and
supportive intentions. Journal of Marketing Management, 26(5-6), 413–427.
Klassen, M. (2002). Relationship Marketing on the Internet: the case of top-and lower-ranked
universities and colleges. Journal of Retailing and Consumer Services, 9, 81–85.
Kosmützky, A., & Krücken, G. (2015). Sameness and Difference. International Studies of
Management & Organization, 45(2), 137–149.
Kotler, P., & Andreasen, A. R. (1996). Strategic marketing for nonprofit organizations. Prentice
Hall.
Kotler, P., & Armstrong, G. (2003). Principles of Marketing. Prentice Hall.
Kotler, P., & Fox, K. F. A. (1985). Strategic Marketing for Educational Institutions. Prentice-Hall.
Lee, C. K., & Chen, H. C. (2018). Configuring School Image Assets of Colleges in Taiwan.
Universal Journal of Educational Research, 6(1), 195–200.
Lee, M., Kang, S., & Hua Jin, A. (2017). Antecedents of Education Brand Satisfaction: A
Conceptual Framework and Research Propositions. Journal of Marketing Thought, 4(1), 45–51.
Leslie, D. (2003). Using success to measure quality in British higher education: which subjects
attract the best-qualified students? Journal of the Royal Statistical Society: Series A (Statistics
in Society), 166(3), 329–347.
Levitan, L. C. (2018). Social Constraint and Self-Doubt: Mechanisms of Social Network influence
on Resistance to Persuasion. Political Psychology, 39(4), 957–975.
Linvill, D. L., Rowlett, J. T., & Kolind, M. M. (2015). Academic Pinstitution: Higher Education’s
Use of Pinterest for Relationship Marketing. Journal of Relationship Marketing, 14(4), 287–
300.
Lombart, C., & Louis, D. (2010). Impact of brand personality on three major relational con-
sequences (trust, attachment, and commitment to the brand). Journal of Product & Brand
Management, 19(2), 114–130.
Lomer, S., Papatsiba, V., & Naidoo, R. (2018). Constructing a national higher education brand
for the UK: positional competition and promised capitals. Studies in Higher Education, 43(1),
134–153.
Lowrie, A., & Hemsley-Brown, J. (2011). This thing called marketisation. Journal of Marketing
Management, 27(11-12), 1081–1086.
de Macedo Bergamo, F. V., Giuliani, A. C., de Camargo, S. H. C. R. V., Zambaldi, F., & Ponchio,
M. C. (2012). Student loyalty based on relationship quality: an analysis on higher education
institutions. Brazilian Business Review (English Edition), 9(2), 26–46.
Maringe, F. (2010). The Meanings of Globalization and Internationalisation in HE: findings from
a world survey. In F. Maringe & N. H. Foskett (Eds.), Globalization and Internationalization
of Higher Education (pp. 17–34). Continuum.
Maringe, F., & Foskett, N. H. (2002). Marketing university education: the South African
experience. Higher Education Review, 34(3), 35–51.
Maringe, F., & Gibbs, P. (2008). Marketing higher education: Theory and practice. McGraw-Hill
Education (UK).
Mause, K. (2009). Too Much Competition in Higher Education? Some Conceptual Remarks on the
Excessive-Signaling Hypothesis. American Journal of Economics & Sociology, 68(5), 1107–
1133.
Mazzarol, T. (1998). Critical success factors for international education marketing. International
Journal of Educational Management, 12(4), 163–175.
Mazzarol, T. W., & Soutar, G. N. (2008). Strategy matters: strategic positioning and performance in
the education services sector. International Journal of Nonprofit & Voluntary Sector Marketing,
13(2), 141–151.
McAlexander, J. H., Koenig, H. F., & DuFault, B. (2016). Millennials and Boomers: increasing
alumni affinity and intent to give by target market segmentation. International Journal of
Nonprofit & Voluntary Sector Marketing, 21(2), 82–95.
Miller, B., & Brian, M. (2010). Skills for sale: what is being commodified in higher education?
Journal of Further & Higher Education, 34(2), 199–206.
Mogaji, E. (2016). Marketing Strategies of United Kingdom Universities during Clearing and
Adjustment. International Journal of Educational Management, 30(4), 493–504.
Molesworth, M., Nixon, E., & Scullion, R. (2009). Having, being and higher education: the
marketisation of the university and the transformation of the student into consumer. Teaching
in Higher Education, 14(3), 277–287.
Moogan, Y. J. (2011). Can a Higher Education Institution’s Marketing Strategy Improve the
Student-Institution Match? International Journal of Educational Management, 25(6), 570–589.
Mortimer, K. (1997). Recruiting overseas undergraduate students: are their information require-
ments being satisfied? Higher Education Quarterly, 51(3), 225–238.
Munjal, P., Mishra, M. S., & Shanker, R. (2019). The Drivers and Outcomes of Customer
Engagement in Brand Communities: Review and Future Research. Journal of Management
Research (09725814), 19(1), 56–76.
Naidoo, V., & Wu, T. (2011). Marketing strategy implementation in higher education: A mixed
approach for model development and testing. Journal of Marketing Management, 27(11-12),
1117–1141.
Narver, J. C., & Slater, S. F. (1990). The effect of a market orientation on business profitability.
Journal of Marketing, 54, 20–35.
Natale, S., & Doran, C. (2012). Marketization of Education: An Ethical Dilemma. Journal of
Business Ethics, 105(2), 187–196.
Newman, S., & Jahdi, K. (2009). Marketisation of education: marketing, rhetoric and reality.
Journal of Further & Higher Education, 33(1), 1–11.
Nguyen, N., & LeBlanc, G. (2001). Image and reputation of higher education institutions in
students retention decisions. The International Journal of Educational Management,15(6),
303–311.
Nguyen, B., Melewar, T. C., & Hemsley-Brown, J. (2019). Strategic Brand Management in Higher
Education. Routledge.
Nicholls, J., Harris, J., Morgan, E., Clarke, K., & Sims, D. (1995). Marketing Higher Education:
the MBA experience. The International Journal of Educational Management, 9(2), 31–38.
Nkala, D., Mugwati, M., Mudzurandende, F., Mazhindu, K., & Mhere, F. (2014). Marketing:
A New Strategy for State Universities in Zimbabwe. Journal of Marketing Development &
Competitiveness, 8(2), 14–25.
Onk, V. B., & Joseph, M. (2017). International Student Recruitment Techniques: A Preliminary
Analysis. Journal of Academic Administration in Higher Education, 13(1), 25–34.
Ordenes, F. V., Grewal, D., Ludwig, S., Ruyter, K. D., Mahr, D., & Wetzels, M. (2019). Cutting
through Content Clutter: How Speech and Image Acts Drive Consumer Sharing of Social Media
Brand Messages. Journal of Consumer Research, 45(5), 988–1012.
Palmer, S. (2013). Characterisation of the use of Twitter by Australian Universities. Journal of
Higher Education Policy & Management, 35(4), 333–344.
Papadimitriou, A., & Ramírez, G. B. (2015). Exploring Advertising in Higher Education: An
Empirical Analysis in North America, Europe, and Japan. Tertiary Education and Management,
21(2), 99–110.
Park, C. W. (2016). Brand Attachment: Theory and Practice. Advances in Consumer Research, 44,
3–6.
Pavičić, J., Alfirević, N., & Mihanović, Z. (2009). Market orientation in managing relationships
with multiple constituencies of Croatian higher education. Higher Education, 57(2), 191–207.
Pedro, I. M., Pereira, L. N., & Carrasqueira, H. l. B. (2018). Determinants for the Commitment
Relationship Maintenance between the Alumni and the Alma Mater. Journal of Marketing for
Peruta, A., & Shields, A. B. (2017). Social Media in Higher Education: Understanding How
Colleges and Universities Use Facebook. Journal of Marketing for Higher Education, 27(1),
131–143.
Pippert, T. D., Essenburg, L. J., & Matchett, E. J. (2013). We’ve got minorities, yes we do:
visual representations of racial and ethnic diversity in college recruitment materials. Journal
Pizarro Milian, R., & Davidson, C. (2018). Symbolic resources and marketing strategies in Ontario
higher education: a comparative analysis. Journal of Further and Higher Education, 42(2),
143–157.
Plungpongpan, J., Tiangsoongnern, L., & Speece, M. (2016). University Social Responsibility
and Brand Image of Private Universities in Bangkok. International Journal of Educational
Management, 30(4), 571–591.
Popović, A. (2015). Marketing Communications of Higher Education Institutions in the Republic
of Serbia. Marketing (0354-3471), 46(3), 166–178.
Purgailis, M., & Zaksa, K. n. (2012). The Impact of Perceived Service Quality on Student Loyalty
in Higher Education Institutions. Journal of Business Management, 6, 138–152.
Roma, P., & Aloini, D. (2019). How does brand-related user-generated content differ across social
media? Evidence reloaded. Journal of Business Research, 96, 322–339.
Rooksby, J. H., & Collins, C. S. (2016). Trademark Trends and Brand Activity in Higher Education.
Review of Higher Education, 40(1), 33–61.
Rose, M., Rose, G. M., & Merchant, A. (2017). Is Old Gold? How Heritage “Sells” The University
to Prospective Students: The Impact of a Measure of Brand Heritage On Attitudes toward the
University. Journal of Advertising Research, 57(3), 335–351.
Ross, M., Grace, D., Shao, W., & Mitchell Ross, D. G. a. W. S. (2013). Come on higher ed . . .
get with the programme! A study of market orientation in international student recruitment.
Educational Review, 65(2), 219–240.
Royo-Vela, M. (2016). Effects of Inbound Marketing Communications on HEIs’ Brand Equity: The
Mediating Role of the Student’s Decision-Making Process. An Exploratory Research. Journal
Rutter, R., Lettice, F., & Nadeau, J. (2017). Brand Personality in Higher Education: Anthropomor-
phized University Marketing Communications. Journal of Marketing for Higher Education,
27(1), 19–39.
Saleem, S. S., Moosa, K., Imam, A., & Khan, R. A. (2017). Service Quality and Student
Satisfaction: The Moderating Role of University Culture, Reputation and Price in Education
Sector of Pakistan. Iranian Journal of Management Studies, 10(1), 237–258.
Sataøen, H. L. (2015). Higher education as object for corporate and nation branding: between
equality and flagships. Journal of Higher Education Policy & Management, 37(6), 702–717.
Schlesinger, W., Cervera, A., & Pérez-Cabañero, C. (2017). Sticking with your university: the
importance of satisfaction, trust, image, and shared values. Studies in Higher Education, 42(12),
2178–2194.
Schofield, C., Cotton, D., Gresty, K., Kneale, P., & Winter, J. (2013). Higher Education Provision
in a Crowded Marketplace. Journal of Higher Education Policy and Management, 35(2), 193–
205.
Schüller, D., & Chalupský, V. (2011). Internal Marketing Communication of Higher Education
Institutions. Economics & Management, 16, 1316–1322.
Shuv-Ami, A. (2012). Brand commitment: A new four-dimensional (4 Es) conceptualisation and
scale. Journal of Customer Behaviour, 11(3), 281–305.
Steiner, L., Sundström, A. C., & Sammalisto, K. (2013). An analytical model for university identity
and reputation strategy work. Higher Education, 65(4), 401–415.
Stephenson, A. L., & Bell, N. (2014). Motivation for alumni donations: a social identity perspective
on the role of branding in higher education. International Journal of Nonprofit & Voluntary
Sector Marketing, 19(3), 176–186.
Stephenson, A. L., & Yerger, D. B. (2015). The Role of Satisfaction in Alumni Perceptions and
Supportive Behaviors. Services Marketing Quarterly, 36(4), 299–316.
Susanti, D., & Dewi, S. (2011). Privatisation and marketisation of higher education in Indonesia:
the challenge for equal access and academic values. Higher Education, 61(2), 209–218.
Symes, C., & Drew, C. (2017). Education on the Rails: A Textual Ethnography of University
Advertising in Mobile Contexts. Critical Studies in Education, 58(2), 205–223.
Tayar, M., & Jack, R. (2013). Prestige-oriented market entry strategy: the case of Australian
universities. Journal of Higher Education Policy & Management, 35(2), 153–166.
Temple, P. (2006). Branding Higher Education: Illusion or Reality? Perspectives: Policy and
Practice in Higher Education, 10(1), 15–19.
Tolbert, D. (2014). An Exploration of the Use of Branding to Shape Institutional Image in
the Marketing Activities of Faith-Based Higher Education Institutions. Christian Higher
Education, 13(4), 233–249.
Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-
based management by means of systematic review. British Journal of Management, 14, 207–
222.
Veloutsou, C., Paton, A., & Lewis, J. (2005). Consultation and reliability of information sources
pertaining to university selection: Some questions answered? International Journal of Educa-
tional Management, 19(4), 279–291.
Vuori, J. (2015). Excellent Prospects for Beautiful Minds: Marketing International Education.
International Journal of Educational Management, 29(5), 582–595.
Weymans, W., & Wim, W. (2010). Democracy, knowledge and critique: rethinking European
universities beyond tradition and the market. London Review of Education, 8(2), 117–126.
Wilkins, S., & Huisman, J. (2011). International student destination choice: the influence of home
campus experience on the decision to consider branch campuses. Journal of Marketing for
Winter, E., & Thompson-Whiteside, H. (2017). Location, Location, Location: Does Place Provide
the Opportunity for Differentiation for Universities? Journal of Marketing for Higher Educa-
tion, 27(2), 233–250.
Wolverton, M. (2006). Three Georgias in Atlanta: Lessons from Business Schools about Finding
Your Identity. International Journal of Educational Management, 20(7), 507–519.
Zailskaite-Jakste, L., & Kuvykaite, R. (2012). Implementation of Communication in Social Media
by Promoting Studies at Higher Education Institutions. Engineering Economics, 23(2), 174–
188.
Chapter 3
Overview of Simulation in Higher
Education: Methods and Applications
Efrat Tiram and Zilla Sinuany-Stern
Abstract Simulation is one of the most widely used methodologies in many areas
such as industry, army, leisure activities, and education. This chapter is devoted to
simulation in higher education (HE) and its use for administrative and educational
purposes. Many types of simulation models have been used in HE for various
purposes. Literature reviews have been published on specific types of simulations in
HE but, to the best of our knowledge, there has not been a comprehensive overview
of all types of simulation models in HE. We classify simulation models in HE into
two types: (a) analytic simulations models such as discrete event simulation and
system dynamics, and (b) educational simulation models such as, virtual reality,
games, and simulators for training. This overview presents examples for each
method, which gives some idea of the specific areas of HE they were used for.
Moreover, taxonomy tables summarize the number of articles published over the last
20 years for each simulation method in HE in the Web of Science, along with the
area of applications they are used for in HE. The number of articles on educational
simulations is larger by far than the number of articles on analytic simulations. This
overview is useful for developers of HE simulations and for administrators in HE
institutions.
Keywords Higher education · Simulation · System dynamics · Discrete-event

simulation · Agent-based simulation · Educational simulation · Simulator ·
Educational game · Virtual learning
E. Tiram ()
Sackler Faculty of Medicine, Tel Aviv University, Ramat Aviv, Tel Aviv, Israel
Z. Sinuany-Stern
Department of Industrial Engineering and Management, Ben Gurion University of the Negev,
Beer Sheva, Israel

82 E. Tiram and Z. Sinuany-Stern
3.1 Introduction
Simulation is a model which imitates a system or a process behavior to learn

about the system over time (Yang, 2014). Simulation is used in many areas such as
industry, business, military, engineering, city planning, medicine, leisure activities,
and education. Banks (2009) defines “modeling and simulation” as an academic
discipline with its own research methodology. Ideally, simulation has several steps,
the first of which is building a model that approximates a real-world event. Such a
model, which is usually computerized, often includes its own dynamics and involves
repetitions and data collection. This step is followed by the analysis of the data and
the drawing of conclusions (often with visualization). Ideally, the final steps are
verification and validation of the model. This chapter is devoted to simulation in
higher education (HE) and its use for administrative and educational purposes.
Simulation is a multidisciplinary method that advances technology. In 1999,
the NSF officially declared simulation as the third branch of science (ibid., p.
3). There are many general simulation journals (e.g., Journal of Simulation), in
addition to those focused on specific disciplines (e.g., Simulation in Healthcare,
Simulation, and International Journal of Simulation Modeling) or on a specific
sub methodology (e.g., Simulation and Gaming). Moreover, several disciplines
(e.g., operations research, computer science, and industrial engineering) include
simulation as a subdiscipline, some even offering a required course in simulation
in their academic program either on how to build or apply a simulation model.
Others just use simulation models for educational training (e.g., simulators and
games). Almost all disciplines (and often subdisciplines) in HE have their own
types of simulation models, including the following: games in business studies
(Rosa et al., 2017), virtual reality in surgeon training (Papanikolaou et al., 2019),
simulation-based learning using manikins in nursing education (Cant & Cooper,
2017), simulation in engineering education (Deshpande & Huang, 2011), and even
dynamic simulation games for university management (Barlas & Diker, 2000).
Simulation is one of the most popular methods in HE. During the last 20 years,
simulation appeared in the topic (article’s title or abstract or keywords) of over
1.8 million articles in Web of Science (WoS)—of them, about 100,000 articles
were devoted to HE. Simulations can be digital or mathematical or both, physical
versus virtual, visual versus interactive, static versus dynamic, web-based versus
local simulator, digital game vs. role-playing game. Simulation models are usually
computerized; the development of most simulation models was in tandem with the
development of the computer and other technological platforms. During the last
10 years, smart phone simulation applications have become popular along with edu-
cational gaming, internet-based simulations, and, more recently, the cloud platform
for simulation with big data (see the forthcoming special issue of the Journal of
Simulation devoted to “Modeling and Simulation in the Cloud Computing Era”).
Simulation is merely a proxy for a real system; it cannot capture the full capacity of
the real system due to the latter’s complexity and the lack of information available
about the real system. Thus, verification and validation are important. Sometimes
the term simulation is used loosely in various contexts that are hard to define.
3 Overview of Simulation in Higher Education: Methods and Applications 83
There are review articles on specific simulation methods in HE, as shown in

Table 3.2, but no overview of all types of simulation models in HE is found in
the scientific literature, to the best of our knowledge. The purpose of this chapter
is to remedy this lacuna. The purpose of our study is to classify and present the
various types of simulation methods as related to HE and to describe their areas of
application and their frequency in the literature.
The closest to our classification is Banks (2009), with the introductory chapter
to the book by Sokolowski and Banks (2009) that tries to present an overview on
“Principles of Modeling and Simulation: A Multidisciplinary Approach” (ibid.).
The article by Banks (2009) is on simulation models in general, not necessarily
on HE, and is directed at graduate students, who have an analytic background,
to be trained as builders of models and simulation. In contrast, our chapter is
directed at HE management and researchers who do not have an analytic orientation.
Banks (2009) classifies modeling and simulation into two types: (1) Stand-alone
simulations. These are grouped into five categories: training, decision support,
understanding, education and learning, and entertainment. (2) Integrated simulation.
The criterion for this classification is “whether or not the simulation program runs
independently from the system it represents” (ibid.).
The approach to classification in our overview is somewhat different. We first
classify the various simulation methods and their areas of applications in HE;
second, we give examples of articles on these methods in HE; and third, we count the
frequency of appearance of the various simulation methods in the scientific literature
in the context of HE.
We identify, in this overview, two main types of simulation models in HE:
1. Analytic simulations are subdivided into several methods: (a) System dynamics,
(b) Discrete-event simulation and Monte Carlo, (c) Agent-based simulation, (d)
Deterministic simulation.
2. Educational simulations are subdivided into several methods: (a) Virtual, (b)
Games, (c) Simulators, (d) Box trainer, (e) Manikins.
Analytic simulations models (Yang, 2014) consist of mathematical equations or a
series of relations between various components, be they deterministic vs. stochastic,
discrete vs. continuous, or static vs. dynamic. An analytic simulation model in HE
usually allows management to plan their affairs, to conduct virtual experiments,
or to test policies and system behavior under various scenarios—answering “what
if” questions regarding enrollments, income, costs, etc. (e.g., Zaini et al., 2017).
Evidently, educational simulations in HE are much more popular than analytic
simulation models and tools (Vlachopoulos & Makri, 2017).
This is not a comprehensive review of all simulation publications in HE, since
there are too many such articles. Thus, this review provides a wide range of
simulation methods applied in HE, classified by type, with examples that give some
idea of the specific areas of HE they are used for and the level of HE hierarchy
to which they are applied. Articles of literature review for specific simulation
methods or areas of application are reported wherever possible. The examples are
taken mainly from journal papers, mostly from the last two decades (from Google
scholar). Moreover, taxonomy tables summarize the number of articles published

for each simulation method in HE in WoS and the areas of applications they are
used for in HE.
This overview is useful for administrators in HE institutions, including heads of
departments, deans, consultants, and presidents, to apprise them of the growing use
of simulations in HE and the advantages and disadvantages of their potential use in
HE.
Section 3.2 of this chapter is devoted to analytic simulation methods in HE. Fol-
lowing that, Sect. 3.3 covers the main educational simulation methods. Section 3.4
provides the number of articles on simulation methods in each of these two groups
and their areas of application in HE during 2000–2019. Finally, Sect. 3.5 concludes
the chapter.
3.2 Analytic Simulations for Managerial Planning in HE
Analytic simulation models consist of mathematical equations or series of logical

relations between various components, be they deterministic vs. stochastic, static
vs. dynamic. Analytic simulation models in HE allow management to plan their
affairs, to conduct virtual experiments, or to test policies and system behavior under
various scenarios—answering “what if” questions regarding enrollments, income,
costs, etc. In the Encyclopedia of Business Analytics and Optimization, Yang (2014)
categorizes analytic simulation models into three types: system dynamics, discreet
event simulation, and agent-based simulation or model. Brailsford et al. (2019) also
used the same three simulation models in the context of operations research. These
three models are digital models with commercial software—for example, AnyLogic
supports all three models (ibid.) and simulation in R (Hallgren, 2013). We also
added here subcategory Monte Carlo simulations which often appear with stochastic
simulations as discrete-event simulation or agent-based simulation. Historically, the
term simulation had been used loosely in the context of HE planning; thus, a fourth
category can be devoted to those various mathematical deterministic static analytic
simulations. In this section, the analytic simulation methods are presented:
(a) System dynamic (SD)
(b) Discrete-event simulation (DES) and Monte Carlo
(c) Agent-based simulation or model (ABS or ABM)
(d) Deterministic and other Analytic Simulation Models
As summarized in Appendix 2, in HE, all these simulation methodologies were
used by management, mainly for planning in all the levels of HE: department,
college, university, national, and international. There are other ways of classifying
these analytic simulations such as, stochastic (Monte Carlo) versus deterministic
simulation and discrete time versus continuous simulation. However, the classifi-
cation we used here is the most common. In addition to the type of model, our
taxonomy includes the area of application in HE, the level of the organization where
the model was used, and the country of application (wherever relevant). We tried to
cover a large variety of countries. In HE, all these three simulation methods were
used by management mainly for planning efficiency and quality evaluation, strategy,
forecasting, and resource allocation, as detailed in Appendix 2, which is organized
in chronological order of the articles. The four analytic simulations are described in
the following four subsections.
3.2.1 System Dynamic (SD)
SD was developed originally for industrial applications—termed originally Indus-

trial Dynamics by its initiator, Forrester (1961). SD is based on a set of differential
equations describing the relevant system as based on stock, flow, and feedback
in each subsystem and the relationships among them. This is a deterministic
continuous-time system, where rather than individual entities, a continuous flow
of entities is moving in the subsystems. SD model consists of a set of differential
equations which represent non-linear systems. An example of a computer package is
STELLA (Richmond, 2004). STELLA is short for Systems Thinking, Experimental
Learning Laboratory with Animation [also marketed as iThink since 1986]. The
advantage of SD is that it is a non-linear model, which describes the interrela-
tionships among various subsystems within a large system which are captured
dynamically over time.
Already in Thompson, 1970, Thompson suggested the use of Industrial Dynamic
simulation, where the model conceptualizes the HE institution as a system of
interacting flows of students, faculty, capital assets, information, and money—the
flows are continuous over time. By the late 1980s, White (1987) reported that
Simulation is one of the most reported OR methodologies used in HE—third
in frequency, with 24 out of 146 articles. However, White states that “many of
the large-scale simulation models for financial planning have fallen into disuse.”
Nevertheless, 15 years later, Bell et al. (2012) highlighted SD as the most useful
hardcore OR methodology used in HE. Thus, in their book, they devoted two
chapters to SD approaches (chapter 11 by Galbraith, 2012; and chapter 8 by Voyer
et al., 2012) while none to other types of simulation. Kennedy (2008) performed
a taxonomy of SD educational policy. Later, Kennedy (2011) reviewed forty SD
papers on HE (two more in K-12 education) and produced a taxonomy of SD models
of educational pedagogic issues that were published during the period 2000–2010.
His taxonomy was performed along two dimensions: five “hierarchical levels,” and
seven “areas of concern” in HE. In the hierarchical HE level dimension, most of the
papers were devoted to university wide management issues. In the largest group
in the “areas of concern” the papers were devoted to planning, resourcing, and
budgeting. We identify in a WoS search that 2432 SD articles are applied to HE;
namely, HE and SD appear in their topic (title or abstract or keywords), of which
for 99 articles HE and SD appeared the articles’ title. Here are some examples of
applications of SD in HE, as summarized in Appendix 2 in chronological order.
Voyer et al. (2012) used the SD approach to improve an advising system for
Business School undergraduates in the USA. Based on a survey of graduating
seniors, advising services received low marks. With the school team, four key
variables were identified for the main four subgroups: Advisors (total workload,
advising workload, time spent with students, etc.); students (wait time, queue
length, student satisfaction, etc.); faculty (faculty involvement, the complexity of
curriculum, and guidance requirements; other (time frame, automated advising,
and budget). The relationships among the key variables were verified to construct
SD flowcharts which considered possible external disturbances outside of the
Business School. Various policies were tested, and policy recommendations were
presented to the management to improve the student advising system: increasing
their satisfaction via using automation, simplifying the curriculum, and involving
faculty in the advising process in critical periods.
Brailsford and De Silva (2015) used an SD model to plan the better provision
of state-funded dental care and the future university intake of dental students in Sri
Lanka. The model had two components—the supply and demand for dental services
on the national level. The supply component considered the career progression of
dentists from recruitment and training at the university through various career paths
through retirement via SD. The demand component calculated various scenarios for
the national future dental care, based on the potential future economic development
of Sri Lanka.
Kara (2015) used a simple SD nonlinear stochastic model for the provision of
HE service in Turkey on the national level during a global economic crisis. Two
main assumptions were used: the quality performance was rising stochastically but
modestly, and the increase in the technology level leads to improved employment
perception and higher levels of employment. The demand and supply nonlinear
time-dependent equations are estimated (via regression) as a function of HE price,
customers’ income, service quality, the degree of using technology in HE and
expected job placement. The author presents the SD differential equations and the
fluctuations of the main two variables over time as a function of the explanatory
variables and the budget allocation to HE.
Fateh Rad et al. (2015) investigated the relationship between university and
industry in Iran as two major infrastructures of national innovation systems in
leading scientific and industrial settings. The relationship between the two systems
is captured by SD nonlinear differential equations, based on a joint investment
between industry and university, in three levels of communication from the lower to
the higher levels.
Strauss and Borenstein (2015) developed their SD model for long-term planning
of undergraduate education in Brazil at an aggregate level. As the purpose of the
government was to increase undergraduate enrollment, it met with partial success.
The scenario analysis considered government regulations/policies, demand, and the
balance between the public and private sector.
Dandagi et al. (2016) used a questionnaire with 207 responses to establish causal
relationships between factors for strategically governing a technical university in
India. Factor analysis was used to generate latent variables, which were utilized
to build the structural equations of SD model, for strategic management of the

university.
Al Hallak et al. (2017) used SD for student enrollment in the Syrian private
HE sector, which examined the dynamic interactions of student flows, staff ratios,
and investments in facilities. This was a decision support system that might help to
increase enrollments. Various scenarios were examined.
Faham et al. (2017) analyzed the key factors and obstacles related to education
for sustainable development in HE. They created a model using SD to develop
education for sustainable development in HE, with the emphasis on sustainability
competencies of students. The approach of SD helped to recognize the interven-
tional procedures in fostering competencies of students in the university. The model
described the dynamic interactions among the students, scientists, content, policy,
pedagogy, community, and government. The model was applied to the University of
Tehran, Iran.
Ghaffarzadegan et al. (2017) used SD to model the education workforce pipeline
on the national level. They introduced two mechanisms that act on the education
enterprise, causing the number of educated people to increase dramatically with
relatively short-term changes in the job market. Their illustrative dynamic model
showed that the system was susceptible to small changes, and they introduced self-
driving growth engines that were adequate to over-incentivize degree attainment.
Moreover, they showed that the mechanisms magnified the effects of short-term
recessions or technological changes and created long-term waves of mismatch
between workforce and jobs. The implication of the theory was that degree inflation
magnified pressures on those with lower degrees, and increased under-employment,
job market mismatch, and inefficiency.
Zaini et al. (2017) reported on a real-life study in a USA university. In response to
financial problems, the administration reduced undergraduate tuition, which resulted
in an increase in student enrollment. The faculty, who argued that the quality of
education had been declining, resisted the expansion. The growth in the number
of students also affected the use of the university’s infrastructure. The authors
developed a simple SD model of university expansion with emphasis on the areas
of planning, resourcing, and budgeting.
Kara (2018) was concerned with the mediocre, low productivity, and low-
performance levels of many universities in developing countries. He suggested an
SD model, where university performance was expressed by teaching quality, and
research productivity levels were related to three resources: human capital, social
capital, and physical capital. He used structural equations expressing the dynamic
behavior of the system over time. The model measured the improvement in quality
and productivity of the university as a function of subsidy and reinvestment. The
simulation experiments obviated the need for the university to undergo a long and
expensive trial and error process.
As shown in Table 3.1, in our search of WoS during 2000–2019, SD, in
general, appeared in the topic (title, abstract, or keywords) of 37,763 articles; the
combination system dynamic and HE appeared in the topic of 2432 articles, and
system dynamic and HE appeared in the title of 99 articles.
Table 3.1 Number of articles on simulation methods in general and in higher educationa
Number of articles during 2000–2019 Percentage
2. Method 3. Method
1. Method in and HE in and HE in Column (2) Column (3)
Method Name topic topic title from (1) from (1)
1. Simulation 1,834,848 99,026 4223 5.397 0.230
A. Analytic simulations
2. System 37,763 2432 99 6.440 0.262
dynamic (SD)
3. Monte Carlo 225,568 8851 176 3.924 0.078
4. Discrete-event 69,259 5426 129 7.834 0.186
(DES)
5.a Agent-based 48,772 3502 171 7.180 0.351
(ABS)
5.b Deterministic 407 34 0 8.354 0
and others
A. Total of 381,769 20,245 575 5.303 0.151
Analytic Sim.
B. Education simulations
6. Virtual 261,559 30,390 2650 11.619 1.013
7. Simulator 102,142 9583 449 9.382 0.440
8. Game 188,519 14,878 1447 7.892 0.768
9. Box trainer 316 126 3 39.873 0.949
10. Manikin 3516 963 41 27.389 1.166
Total of 556,052 55,940 4590 10.060 0.825
Education Sim.
Total A + B 937,821 76,185 5165 8.12 0.551
a Assessed from the Web of Science, Oct. 2020. See Appendix 1 on synonyms for HE and each
simulation method. Topic includes title, abstract, or keywords. Bold lines are the totals of the
groups of simulation methods
3.2.2 Discrete-Event-Simulation (DES) and Monte Carlo
Discrete-event simulation (DES) first emerged in the late 1950s and steadily grew in
popularity to become the most frequently used of the classical Operational Research
techniques across a range of industries and users. DES models look at the operation
of a system as a discrete sequence of events in time, typical of a manufacturing
process. Each event occurs at a particular instant in time and marks a change
of state in the system. This contrasts with continuous simulation, in which the
simulation continuously tracks the system dynamics over time. Conventional DES
constructs are entities, activities, and queues; these constructs are linked to form a
complex process in which entities flow. Probability functions are also used to model
stochastic processes. DES, alternatively termed Activity-Based Simulation or Monte
Carlo Simulation (Banks et al., 2005), is the main analytic simulation model used
in practice since the 1960s. In DES, the incoming entities stand in queues, and they
receive service activities or resources in a station. In some systems, human service
becomes part of the activities and resources. DES allows for the randomness of
the service time of the service provider within a given probability distribution and
randomness of the inter-arrival time between entities all within given probability
distributions. The following are some examples of the application of DES in HE.
Schellekens et al. (2010) developed a process-focused, demand-driven opera-
tional model for delivering educational programs to support the development of
individual students and to provide a flexible alternative for the traditional supply-
driven and class-based organization of educational programs. DES was developed
and applied to the higher education of professionals in the Netherlands.
Oltean et al. (2017) dealt with facility planning and renovation using DES. The
purpose of DES was the viability of the building and maintaining process in a
generic public university organization. The events were driven by internal, external,
or mixed causes generated randomly. Some of the uncontrollable events occurring
in the process may drive the discrete evolution to disturbance rejection states, where
specific recovery strategies are applied. If recovery is successful, disturbances may
become sources of innovation.
As shown in Table 3.1, in our search of WoS during 2000–2019, discrete-event,
in general, appeared in the topic (title, abstract, or keywords) of 69,259 articles; the
combination discrete-event and HE appeared in the topic of 5426 articles; discrete-
event and HE appeared in the titles 129 times.
Since most models of DES and ABS are stochastic, using randomness, where
the randomness often materializes via sampling like Monte Carlo gambling. Often
Monte Carlo appears together with DES and sometimes also with ABS. Thus, it
appears in the literature more than DES and ABS combined. Referring to Table 3.1,
in our search of WoS during 2000–2019, Monte Carlo, in general, appeared in the
topic (title, abstract, or keywords) of 225,568 articles; the combination Monte Carlo
and HE appeared in the topic of 8851 articles; Monte Carlo and HE appeared in the
titles of 176 articles. However, the count in the title of the articles shows that Monte
Carlo is even less than ABS. Monte Carlo simulation without DES or ABS stands
for various other analytic stochastic simulation methods.
3.2.3 Agent-Based Simulation (ABS)
Agent-based Simulation (ABS) is a relatively new paradigm that simulates the

simultaneous operations and interactions of multiple autonomous agents (determine
their own actions) to predict complex processes. The agents can be individuals,
groups, or institutions. Agents may behave differently from each other. ABS
assesses agents’ effects on the system as a whole—it adds behavior (in terms of
decision rules) to micro-simulation. Agents’ behavior can be adaptive in response
to other agents or environmental changes or other agents’ behavior; feedback and
memory are crucial in ABS. Monte Carlo is utilized to generate randomness. ABS
has elements of game theory, complex systems, and multi-agent systems (Yang,
2014). Siebers et al. (2010), who reviewed DES in general (not necessarily in the
context of HE), claimed that DES is “dead” while ABS, which replaces it, will stay
for a longer period. They state that in the early 1990s, ABS “promised to offer
something novel, interesting, and potentially highly applicable to OR. However,
there is relatively little evidence that ABS is much used in the OR community,
there being few publications relating to its use in OR and OR-related simulation
journals.” Indeed, ABS is a more general model than DES, and thus it is more
complicated, requiring more data. This makes it more difficult to use, especially
in the case of HE administrators. Moreover, not every situation requires ABS, and
a basic course in simulation is not likely to include ABS—it is dependent on the
instructor and the software chosen for the course. Nevertheless, OR specialists have
been using ABS without using this term since the late 1980s. See, for example, Stern
and Sinuany-Stern (1989), Sinuany-Stern and Stern (1993), and Sinuany-Stern et al.
(1997), where the terms used were behavioral based in DES, within the framework
of operations research.
In the literature review of ABS by Gu and Blackmore (2015), they see ABS as
a bottom-up model where the macro-level behavior is a result of the micro-level
agents and their interactions. In their systematic review of the applications of ABS
in HE by area of application, they present thirty-five articles during the period 1997–
2013. During the years 1997–2005, the number of articles published was one or
none every year. The areas with the maximal number of articles were academic
activities (with twelve articles) and teaching and learning (ten articles). Three more
areas with four articles each were: student performance, application and enrollment,
and university collaboration. They also found ABS applications in the ranking of
universities.
Molders et al. (2011) suggested modeling scientists as agents. Base on this
idea, Gu et al. (2015) built a hypothetical ABS, where the productivity of the
agents (scientists) is measured under several environmental assumptions. Three
types of academic agents (scientists) were categorized by their strategy: Careerists
who strive for high IF journals, Orthodox who strive for topical fitting factor, and
Mass who strive for high acceptance rate journals. The papers are characterized
by impact factor, acceptance rate, and topical fitting factor. The simulation has six
sub models: (1) Create journal, (2) Create academics, (3) Academics move around,
(4) Academics submit papers, (5) Journals publish papers, (6) Retire and recruit
academics. The number of journals and total number of acceptable papers stayed the
same in all scenarios. This was a competitive environment where the composition
of each group of academics changed over the scenarios. The results showed that the
Orthodox type has the most consistent achievements.
Tan et al. (2019) built a discrete-timed stochastic simulation model based on
parallel systems to explore the relation between the number of submissions and the
overall standard of academic journals within a similar discipline under peer review.
The model simulates the submission, review, and acceptance behaviors of academic
journals in a distributed manner. The results point to the possibility that the standard
of academic journals deteriorates due to excessive submissions. Over twelve articles
are cited on this issue using ABS, including a review of the literature on the use of
ABS in scientific publication and the peer-review system by Kovanis et al. (2016).
As shown in Table 3.1, in our search of WoS during 2000–2019, agent-based,

in general, appeared in the topic (title, abstract, or keywords) of 48,772 articles;
the combination agent-based and HE appeared in the topic of 3502 articles; agent-
based and HE appeared in the titles of 171 articles.
3.2.4 Deterministic and Other Analytic Simulation Models
Already in 1967, Weathersby applied a linear regression simulation model to predict

costs of academic departments in the University of California, Berkeley. Cost
simulation is defined as a collection of mathematical expressions that attempt to
interrelate analytically; these are the costs and input parameters of HE institutions.
Hopkins (1971) called such mathematical models deterministic simulation models.
Some of them are large scale for university planning. Hopkins raised questions
about their large expense and accuracy. Often the models are not well understood
by HE administrators, and they require major data collection. Sinuany-Stern (1983)
illustrated the general structure of a cost simulation model for university planning
(Fig. 2 and Fig. 4; p. 212, 214, ibid.) utilizing a set of equations where some led to
others.
Larson et al. (2014) claim that there were too many PhD graduates, i.e., too
few academic job openings. They used a demographic model, utilizing a basic
reproductive number R0 —the mean number of new PhDs that a typical tenure-
track faculty member will advise during his career. The model calculated that in
the then current situation, R0 = 1 was sufficient to secure tenure track positions
for all PhD graduates, but in a situation of regional population growth, a higher
value was needed R0 > 1. Applying the model to MIT data in 2011, the R0 average
in engineering was 7.8, which was too high. Some fields had a reproductive value
R0 < 1, which indicated a severe shortage of PhDs in the future (e.g., mechanical
engineering). However, another field had R0 = 19 (environmental engineering). The
authors suggested that the intake of PhD students by disciplines could be improved
by also considering the industry demand for PhDs in areas such as STEM and by
raising the level of master’s degree graduates, which may replace the access demand
for PhDs by industry.
Siniksaran and Satman (2020) designed a simulation software, “World Universi-
ties Ranking Simulator” (WURS), for the three popular world university rankings:
THE, QS, and ARWU. The purpose of the simulator was to enable the institution to
detect the parameters they need to improve in order to raise their scores in these three
rankings. The data needed was outlined, and the transformations and manipulations
performed by the simulator were detailed.
3.2.5 Combining Analytic Simulation with OR and Other

Methods
Simulation is one of the most popular methods in HE and in operations research in

general. Yang (2014) stated about analytic simulation methods (i.e., SD, DES, and
ABS): “In OR context, simulation is often used as part of business optimization to
understand the impact of changes” and is used sometimes to improve the system.
As a future direction, Brailsford et al. (2019), in their general review of simulation,
suggested using a hybrid of some of the above-listed simulation methods with each
other and with other OR methodologies, such as optimization. They pointed at
AnyLogic software as the most used simulation software, which accommodates the
three types of analytic digital simulations: SD, DES, and ABS.
Below are some HE applications which combine simulation with other OR
methods.
Barlas and Diker (2000) developed an interactive dynamic simulation model
into a university management game, UNIGMAE, based on SD. They generated
numerous performance measures and demonstrated the systemic nature of univer-
sity management. This game enabled stakeholders to understand those individual
decisions, in isolation, yield counterintuitive results when not coordinated with
related decisions. Sinuany-Stern (1978) combined deterministic simulation for cost
prediction with optimization for budget planning in a multi-campus community
college.
Hussein and El-Nasr (2013) used SD simulation to understand the relationships
among various design factors of an HE quality model. The TQM (total quality
management) approach was used, where eleven main factors affecting quality were
identified: building, facilities, courses, marketing, counseling, library and informa-
tion centers, student services, legalism and morals, scientific evaluation, staff level,
and management. The institutional budget can be allocated proportionally to the
weights given to each factor by management. However, an optimization model
utilizing a genetic algorithm was suggested to achieve an optimal budget allocation
of the SD model to maximize the quality of the HE institution. NetLogo software
was used (https://ccl.northwestern.edu/netlogo/).
3.3 Educational Simulation in Higher Education (HE)
There are additional special types of simulations, educational simulations, specific

to pedagogy in HE, which are used much more often than the analytic simulations
mentioned in Sect. 3.2. These educational simulations include: serious games, video
games, computerized and non-computerized simulators, physical simulators, virtual
worlds, and massive multi-player online games—most are digital, internet bases,
and computerized—and all are used for teaching and training in HE (Bradley, 2006).
We classify educational simulations into five main methods:
1. Virtual
2. Simulator
3. Game
4. Box trainer
5. Manikin
“Simulation and gaming” sometimes is used as a general term for educational
simulation. Simulation in the pedagogy of HE is spreading with the evolution
of computer technologies and internet-based communications. These pedagogical
tools and computer-based simulations, and serious games enrich the learning and
practical training in many disciplines.
Since there are too many articles on educational simulation (4590 articles vs. 575
for analytic simulation, as shown in Table 3.1), Table 3.2 and our overview contain
mainly review articles in this section, while covering these five methods. Sometimes
the terminology of the five methods is fuzzy; thus we divided Sect. 3.3 into two main
parts: Sect. 3.3.1 is devoted to the five educational simulation methods, while Sect.
3.3.2 is classified by academic disciplines.
3.3.1 Educational Simulation by Method
3.3.1.1 Virtual Learning
Virtual simulation enables 3-dimensional environments and objects to look real

when transmitted, which creates for the student an immersive and engaging learning
experience. A virtual Lab allows the student to perform experiments virtually, saving
materials, reducing cost, increasing safety, and enhancing accessibility, especially in
STEM disciplines. Virtual simulations are often used for training in various medical
fields such as nursing, surgery, radiotherapy, midwife training, radiation therapy,
and pharmacy education. The main advantage of using virtual simulations is the
ability of the trainees to repeat the procedure tasks several times. For instance, in the
field of surgery, the possibility to repeat a certain surgical procedure enhances the
muscle memory ability by internalizing effective feedback within its virtual platform
(Papanikolaou et al., 2019). For example, the Moog Simodont dental trainer is
Virtual reality (VR) based platform which was introduced within the pre-clinical
studies and was examined in a study within an operative density restoration course
for undergraduate students (see Murbay et al., 2020). Manual and digital methods
were included to evaluate the students’ performance. It was found that the students
who were exposed to the Moog Simodont were evaluated significantly higher with
performance and preparation (manually and digitally). These findings support and
strengthen the claim that this virtual reality simulator is essential to improve dentist
student’s skills and contributes to such kind of undergraduate restoration course
(ibid.).
Table 3.2 Review of articles on educational simulation in higher education

No. of
HE area Years Type of Articles
Author (year) reviewed covered simulation reviewed Type of reviewa
Faria et al. Business 1970–2009 Business games 304 Allb
(2009)
McGaghie et Simulation- 2003–2009 All medical 64 selected Compared to 3
al. (2010) based medical simulations articles reviews from
Edu. (SBME) 1969–2003
Cant and Nurse education 1999–2009 Manikin 12 NSR
Cooper (2010)
Deshpande Engineering 1960–2006 Computer 70 SR
and Huang Edu. game
(2011) simulation
Kennedy Pedagogic 1972–2008 System 40 SR
(2011) dynamic
Cooper and Midwifery Edu. 2000–2010 Actor, manikin 24 SR
Fox-Young and virtual
(2012)
Smithson et al. Pharmacy 2000–2013 Standardized 27 SR
(2015) patient
Warren et al. Nurse Edu. 2007–2014 Manikin 10 SR
(2016)
Doolen et al. Nurse graduate 2009–2015 High-fidelity 7 of 34 Umbrella SR
(2016) Edu. simulation reviews
Sun et al. Airway Until 2016 Virtual, comp. 17 SR
(2017) management Software or meta-analysis
Edu. manikin
Adib- Nurse research 1975–2015 Manikin 16 SR
Hajbaghery Edu.
and Sharifi
(2017)
Vlachopoulos Across academic 2010–2016 Game 123 SR
and Makri fields
(2017)
Cant and Nurse Edu. 2010–2015 Manikin 25reviews Umbrella SR
Cooper (2017)
Rosa et al. Business 1970–2016 Business game 157 SR
(2017)
Frank et al. Surgery 1999–2016 Simulator 57 SR
(2018) arthroscopic meta-analysis
Horsley et al. Nursing research 2010–2016 Nursing 48 SR
(2018) interprofessional simulation
Kane (2018) Radiation 2000–2018 Virtual 40 NR
therapy Edu.
Papanikolaou Surgeons Until 2018 Box trainer 50 NR
et al. (2019) comparative
Foronda et al. Nursing 1996–2018 Virtual 80 SR
(2020) education simulation
a SR—Systematic Review. NR—Narrative Review, NSR—Narrative Systematic Review
b The authors reviewed all articles of one journal “Simulation and Gaming” since its inception in
1970
As shown in Table 3.1, in our search of WoS during 2000–2019, virtual in

general appeared in the topic (title, abstract, or keywords) of 261,559 articles; the
combination virtual and HE appeared in the topic of 30,390 articles; virtual and HE
appeared in the titles of 2650 articles.
3.3.1.2 Simulator
A simulator is a device that allows the person to train in an environment like the real
one, such as a driving simulator (e.g., car or airplane) for training or practice driving
under various conditions. Simulators are also used in medical training. For example,
Frank et al. (2018) reviewed 57 articles on the use of arthroscopic simulators for
training surgeons to perform joint surgery. In the first level, the improvement in
diagnostic arthroscopic operation was noticeable within the simulator itself. In the
next stage, a correlation was observed between basic diagnostic arthroscopy and
standard of practice. Finally, limited findings indicate that when basic diagnostic
arthroscopy procedure is applied in patients, as a continuation of simulator training,
it is much more improved and successful (ibid.). The advantages of the simulator
include increased safety for patients, a reduction in the time needed for “hands-on”
learning by surgeons on the real system (patients in this case), and improvement in
diagnostic capabilities.
As shown in Table 3.1, in our search of WoS during 2000–2019, simulator, in
combination simulator and HE appeared in the topic of 9583 articles, and simulator
and HE appeared in the titles of 449 articles.
3.3.1.3 Games
A simulation game presents students with real-life scenarios relevant to their

discipline. Usually, it is competitive, involves several players or groups of players,
and is computerized. The game simulation provides the players (students) with rules,
strategies, and chances of events to happen, all of which develop students’ skills.
Juan and Loch (2017) claimed that innovative, serious games and simulations in
the pedagogy of HE are spreading with the evolution of computer technologies and
internet-based communications. They stated that the use of games and simulation
in HE is still in its early stage. Nevertheless, it is just a question of time until such
games will gain widespread acceptance. They reviewed five articles that appear in
a collection devoted to games and simulations in HE (ibid.). One of these articles
is by Vlachopoulos and Makri (2017), who lump together the terms “game” and
“simulation,” listing the following types of games/simulations in HE: educational
games, digital game-based learning, applied games, interactive exercises, and next-
generation video and simulation games (games modeled after natural or man-made
systems or phenomena, in which players must achieve pre-specified goals). The
delivery of games and simulations can be via computer, web browser, consoles,
mobile phones, and other mobile gaming devices. These games include serious
games, as well as game-based learning in various academic disciplines. The learning
goals of the game/simulation can include knowledge acquisition and understanding,
increased motivation, engagement, and skill acquisition. The technique can be
described as single/multi player, linear/nonlinear, collaborative, competitive, per-
suasive, synchronous, or immersive. They performed a systematic review of games
and simulations in HE, reviewing 123 articles between 2010 and 2016. When the
breakdown of the articles was by discipline, business/management led with 21 of
the 123 articles; next was health sciences (16 articles), and then computer science
(11 articles). Interestingly, 25 of the articles were devoted to Meta-Analysis and
Systematic Reviews (ibid.). The results indicated that games and simulations were
effective and had a positive impact on cognitive and behavioral learning goals. They
concluded that the use of gamified mobile apps and virtual learning (e.g., a student
game to practice a business negotiation) increased student engagement, retention,
and academic achievements (ibid.).
As shown in Table 3.1, in our search of WoS during 2000–2019, game, in
combination game and HE appeared in the topic of 14,878 articles; the combination
game and HE appeared in the titles of 1447 articles.
3.3.1.4 Box Trainer
Box trainer is a specific tool designed for laparoscopic surgery that replaces
invasive surgeries, thus minimizing cuts of body tissues. Consequently, it is almost
impossible for surgeons to have a 3-dimensional view of the operating area. This
means more training is required for surgery students. The box trainer is a specific
tool with three-dimensional graphics and the use of motion sensors for realistic
movements (motion control) to simulate the real situation. Papanikolaou et al.
(2019) published a narrative comparative review focused on box trainers and virtual
reality simulators. Their review went up to 2018 and covered about fifty articles.
Surgical training with box trainers conferred a significant benefit in terms of
surgical skills development, increased patient safety, and overall cost reduction even
though the use of virtual reality simulators was significantly more expensive. They
concluded that simulation training allowed trainees to learn from their mistakes, to
repeat surgical tasks multiple times, to establish muscle memory, and to enhance
skill competency with the aid of informative feedback.
An example of laparoscopy box trainer type for training basic laparoscopy
skills is a “Portable Ergo-Lap Simulator” (which was designed by Dong
juan Xiao, within the industrial design engineering in Delft University of
Technology, the Netherlands www.tudelft.nl/en/ide/research/research-labs/applied-
labs/laparoscopy-box-trainer/). The main aim of this simulator is to help medical
staff to improve their proficiency under ergonomic conditions, and its tasks are
based on scientific research regarding laparoscopy necessary and relevant skills in
this field, as well as feedback of sixty surgical participants after they carried out
tasks on the prototype.
As shown in Table 3.1, in our search of WoS during 2000–2019, box trainer,
in general, appeared in the topic (title, abstract, or keywords) of 316 articles; the
combination box trainer and HE appeared in the topic of 126 articles; game and HE
3.3.1.5 Manikin
Manikin is an example of a physical simulator used in nursing education. The use

of manikins is associated with reduced costs, increased efficiency in performing
certain tasks, and above all, with safety. For example, Lapkin and Levett-Jones
(2011) performed a cost-utility analysis to compare medium vs. high-fidelity human
patient simulation manikins in nursing education. They used multi-attribute utility
function and calculated the costs associated with each of the two alternatives. The
conclusion was that medium-fidelity manikins are much more cost-effective than
high-fidelity manikin with the ratio of 1:5.
Another study by Curran et al. (2015) examined a different approach to the
differences between low and high-fidelity manikin simulators and was conducted
within the neonatal resuscitation program (NRP). This program aimed to improve
the skills of physicians and other health care staff regarding newborn resuscitation.
Medical students participated in the study and were divided into two groups (low
and high-fidelity manikin), and the following variables were examined: integrated
skills, satisfaction, confidence, and teamwork. The findings raised that satisfaction
and confidence were significantly higher in the high-fidelity manikin simulator
group comparing to the low-fidelity manikin (yet no significant difference was found
regarding integrated skills as well as teamwork).
As shown in Table 3.1, in our search of WoS during 2000–2019, manikin, in
general, appeared in the topic (title, abstract, or keywords) of 3516 articles; the
combination manikin and HE appeared in the topic of 963 articles; manikin and HE
3.3.2 Educational Simulation by Discipline
In this subsection, we cover some disciplines that use educational simulation. We

concentrate here mainly on review articles of educational simulations.
3.3.2.1 Health Disciplines
Bradley (2006) wrote about the history of simulation in medical education and how
simulation-based training became a standard for medical education. Simulation-
based learning affords opportunities to address multiple domains of learning and
performance. Unlike traditional clinical education, simulation-based learning pro-
vides learners exposure to events that are rare in the clinical setting too and allows
learners to assume leadership roles in emergencies. In the health sciences context,
simulation means a suitably analogous situation, using digital, virtual, simulator,
or other physical apparatus for student/personnel training in many academic health
disciplines. McGaghie et al. (2010) reviewed 64 selected articles on simulation-
based medical education (SBME) from 2003–2009 and three review articles from
1969–2003. They discuss twelve features and best practices of SBME that teach the
need to know to best utilize SBME, concluding that SBME is a complex service
intervention that needs to be planned and practiced with attention to organizational
contexts (ibid.). Below are some examples of several health disciplines.
Nursing Education: The main simulation tools in nursing education are those
that represent patients, from manikins to virtual Lab. The major concern is the
fidelity of this simulation, as Nehring and Lashley (2010) claim in the book they
edited, containing twenty chapters by various authors, covering various aspects
of high-fidelity in nursing education such as research, setting a patient simulation
program, and creating an interdisciplinary simulation center. As shown in Table 3.2
there are many literature reviews on nursing simulations (Cant & Cooper, 2010;
Cooper & Fox-Young, 2012; Warren et al. 2016; Horsley et al., 2018; and Foronda
et al., 2020). Cant and Cooper (2010) reviewed twelve articles on simulation in
nursing education (1999–2009), concluding that further exploration is needed to
determine the effect of team size on learning and to develop a universal method
of outcome measurement. Warren et al. (2016) reviewed ten articles (2007–2014)
on the effectiveness of simulation-based education on satisfaction and learning
outcomes in nurse practitioner programs, concluding that high fidelity simulation
increases students’ knowledge and confidence. Adib-Hajbaghery and Sharifi (2017)
reviewed sixteen articles (1975–2015) about the effect of simulation training on the
development of nurses’ and nursing students’ critical thinking, concluding that more
studies with careful designs are needed to produce more credible evidence on the
effectiveness of simulation on critical thinking.
Review of reviews (umbrella review) was performed by Doolen et al. (2016), who
reviewed thirty-four review articles on high—fidelity simulation in undergraduate
nursing education. They found that only seven articles met some inclusion criteria.
They conclude that faculty do not receive enough support in using the simulation.
Furthermore, there are great differences among the reviews and the studies, which
limit the findings (ibid.). Cant and Cooper (2017) used also applied an umbrella
review of twenty-five systematic reviews (2010–2015), covering 700 studies on
simulation-based learning in undergraduate nurse education. They concluded that,
although most reflect strong satisfaction with simulation education, there are still
some gaps in comparable design of the reviews.
Eighty articles on virtual simulations (published during 1996–2018) were
reviewed by Foronda et al. (2020) systematically via meta-analysis. They found
that most of the articles (86%) pointed at the effectiveness of virtual simulation in
nursing education. They conclude that there is a need to improve the research design
(ibid.). Horsley et al. (2018) reviewed 48 articles (published during 2010–2016) on
interprofessional health care teams in nursing education. Their findings show that
collaborative health care in nurse education increased safety and quality of patients’
care and satisfaction and reduced health care cost.
In HE, there is a crucial advantage for using virtual reality simulations to
achieve improvement in practical skills. For example, to examine the process of
medication administration, a study was performed to evaluate the effectiveness
of Pharmacology Inter-Leaved Learning Virtual Reality (PILL-VR) simulation in
nursing students versus traditional methods (see Dubovi et al., 2017). The students
were divided into two groups: one learned with regular lecture-based curriculum,
while the other used the platform of PILL-VR simulation platform. A higher
significant achievement in knowledge learning (conceptual as well as procedural)
were obtained in learning activities with the assistance of the PILL-VR simulation
(ibid.).
Airway Management Education: Airway management is often life-saving
procedure for patients in acute situations. Sun et al. (2017) reviewed seventeen
articles (until 2016) on simulation in airway management education, using meta-
analysis to compare simulation-based training for airway management (virtual,
computer software, or manikin) versus non-simulation-based training methods.
They recommended combining the two methods.
In the field of nursing, methods for training in difficult airway management
are not common in study courses. A Virtual Reality (VR) airway intervention is
presented and examined within a pilot study to teach difficult airway management
(Samosorn et al., 2020). According to the findings, the VR airway Lab was
rated high by students and faculty regarding different indices such as improving
knowledge in the field of airway management.
Midwifery Education: Like the situation in nursing, patient simulation practices
can be defined as the substitution of a bona fide patient encounter with artificial
models or manikins, virtual reality, or live actors enacting a scenario that replicates
substantial aspects of the real experience in a controlled, safe environment for
midwifery.
Cooper and Fox-Young (2012) reviewed twenty-four articles (2000–2010) on
simulation-based learning in midwife education, utilizing various simulation meth-
ods/tools (e.g., actors, manikin, virtual), concluding that simulated learning of
midwifery skills is beneficial, though the clinical practice is still needed. Articles
with evidence from obstetrics, neonatology, technical and non-technical skills
(teamwork) were included.
A pilot evaluation was performed at the Queensland University by Downer et al.
(2020). The use of 3D midwifery visualization resources (3DMVR) by midwifery
students was evaluated. The traditional methods (such as books, lectures, and
clinical skills in Lab sessions) were available to the midwifery students in addition
to the 3D tool. The aim was to evaluate if the 3D tool enhances the students’
conceptual understanding of the processes related to birth. According to the results,
all midwifery students expressed that the 3D experience was beneficial. The
participating students were satisfied with the tool and indicated that the experience
allowed an improved and deep understanding of the anatomical and physiological
aspects of the process.
Pharmacy Education: Smithson et al. (2015) reviewed twenty-seven articles on

standardized patients in pharmacy education, concluding that gaps in knowledge
around transferability, scalability, and cost benefit of this technique still exist, and
there is a need for pharmacy educators to address these gaps to justify this resource-
intensive teaching method. Virtual reality technology provides an immersive and
interactive learning environment for pharmacy students as they provide service to
patients. Coyne et al. (2019) provided an updated overview of virtual reality in
pharmacy education.
Cheesman et al. reported on their experiment on the effective use of Virtual Labo-
ratory Practical Class (VLPC). They took two groups of second-year pharmacology
students. One group had access to the VLPC before their Lab session, and the second
group had access to the VLPC after their Lab session. The results showed that the
first group performed better—with a lower mean completion time compared to the
second group. Moreover, the first group reported an increase in their confidence
in successfully completing the live practical Lab experiment. They conclude that
attending VLPC before the live practical Lab is more effective than attending VLPC
after the live practical Lab.
Radiation Therapy Education: Kane (2018) wrote a systematic narrative
review of about forty articles on simulation-based education, mainly in virtual
environment radiation therapy (VERT) education during 2000–2018. He concludes
that: “The evidence suggests that future inquiry involving VERT should explore
different ways in which VERT can be used to contribute to the skillset required by
the radiation therapist of tomorrow” (ibid.).
3.3.2.2 Simulations in Business Management
Vlachopoulos and Makri (2017), in their review of simulation and gaming in

HE, in general, reported that out of 123 articles they reviewed, business manage-
ment/marketing had the largest number of articles (21); the rest of the articles were
devoted to various other disciplines.
Faria et al. (2009) wrote a review of the history of business gaming covering
one specific journal, Simulation and Gaming, where 304 articles were devoted to
some aspects of business games out of 1115 published in that journal during 1970–
2009, representing 27.3% of Simulation and Gaming total publications. In HE,
business games started in a university course in the USA (University of Washington,
1957). Since 1962, business games development stages were tied to computer
development: first on mainframes and then growing in complexity until 1984, when
PC-based games were developed. Since 1998 internet-based business games were
introduced and used in HE. The five main topics that appeared in the literature of
business games over the years were: experience gained, strategy aspects of the game,
decision-making experience gained, learning outcomes provided, and teamwork
experience provided by business games. Since the turn of the millennium, games
focusing on specific areas were more common. Examples include games related to
marketing, accounting, stock market and finance, and human resource management.
Some of the games allowed the use of analytical tools such as forecasting.
For example, Rosa et al. (2017) performed a systematic review of 157 articles
on business games during 1970–2016 to analyze the relationship between games
and the development of creative potential in business games. They found that
business games were not used specifically to promote creativity. Nevertheless,
some other related themes were reviewed, such as cognitive and behavioral aspects
(ibid.). Business games are often digital and involve several participants in a
competitive situation. Sometimes the term gamification was used. Another example
is Avramenko (2012) which referred to enhancing students’ employability through
business simulation.
Computerized Simulation Games for Software IT Project Management: Lui
et al. (2015) designed an improved simulation game to train a group of students to
take the role of project managers, where the objective was to complete a software
design project on time, while adhering to the requirements of the stakeholders, and
staying within a limited budget. The players were able to monitor the progress and
the effects of their decisions. This design attempted to improve on various elements
in other such games existing at the time.
3.3.2.3 STEM Disciplines
For STEM (science, technology, engineering, and mathematics), there are various
types of simulations, specific for each discipline and dealing with education and/or
research, and sometimes as developers of new simulations. The most frequent use
of educational simulations for STEM education is for virtual laboratories.
Alkhaldi et al. (2016) reviewed eleven articles on the implementations of virtual
and remote laboratories in various STEM disciplines. Among the Labs in this review
Labs were those in physics, biology, chemistry, computer science, information
technology, robotics, and mechanical engineering. The advantages of such Labs
were summarized as follows: accessibility and flexibility from anywhere at any time,
the ability to fit individual pace and schedule, the retrial of experiments without
wasting resources, and safe environment. These simulation Labs were often online
or cloud based. It was suggested that integrating physical Labs with virtual Labs in
creative way would enrich learning.
Deshpande and Huang (2011) studied the state of the art of computer game sim-
ulation, where active multi-sensory experiential learning methodologies were used
in engineering education. Their review covered the period 1960–2006 and included
seventy articles. They reviewed simulation games applications in engineering edu-
cation in the following eighteen engineering areas (and subareas): civil engineering,
electrical engineering (including digital signal processing and power electronics),
computer engineering (digital logic, security, protocols, software engineering,
information systems, and artificial intelligence), chemical engineering, mechanical
engineering (including Engineering, graphics, mechanics, thermodynamics and
system dynamics), industrial engineering (including enterprise resource planning,
production planning and control and supply chain management) and environmental
engineering. In their table of simulation games, they refer to about seventy games
in various areas, in addition to engineering, such as business, medical, health,
architecture, and physics. They conclude that proper application of simulation
games in engineering education would maximize the student’s transferability of
academic knowledge to industry.
Another example, in the field of developmental biology, a neurosphere simulator
became into use during the limitations of the COVID-19 pandemic via online,
because of movement from “face to face” Lab classes into an online teaching era
(Zupanc et al., 2021). This simulator was developed based on cellular automata
models of neurosphere growth and is freely available for students on the web (http://
neurosphere.cos.northeastern.edu/). The simulator allows students to experiment
with different components of the biological processes which were expressed in the
computerized model during this Lab class exercise online, and to download data for
further specific analysis, as well (ibid.).
City Planning education: Minnery and Searle (2013) presented an innovative
way of using a toy computer game of a city (SimCity 4) in student assignments to
develop urban and strategic plans.
In summary, the use of simulation and games in the pedagogy of HE has been
growing over the years. For the most part, students were satisfied, though faculty
support was sometimes insufficient. Cost effectiveness was usually justified, but
not always reported. The literature review and umbrella systematic review articles
in Table 3.2 reflect the great variability of the studies’ purposes and designs,
thus comparability is not always evident. The health educational simulations
reported most often in the literature were manikins and virtual simulations. STEM
educational simulations were mostly for virtual Labs, while business educational
simulations were mainly games.
3.4 Counting Articles About Simulation Methods and Their

Area of Application in HE
In Sects. 3.2 and 3.3, only some examples of articles from the literature on
simulation in HE are presented due to the huge number of such articles, running
between 4223 and 99,026 (based on line # 1 of Table 3.1). To get an idea about
the frequency of publications on the various simulation methods and the areas of
application, this section provides the relevant enumeration. Section 3.4.1 counts the
number of articles on each simulation method in general versus their count in HE
(higher education). Section 3.4.2 counts the number of articles on each application
area in general versus their count in HE. Section 3.4.3 presents the number of
articles on each simulation method by area of application as applied to HE. In
all three tables of this section, the count of articles was performed in the Web of
Science (WoS) assessed October 2020. In Table 3.1 and Table 3.3, column 1 counts
Table 3.3 Number of articles on areas of applications and simulation in general and in HEa
Areas of
applications Number of articles Percentage
2. Area and 3. Area and Column 2 Column 3
1. Area in topic HE in topic HE in title from 1 from 1
1. Transportation 2,687,092 182,392 8431 6.788 0.314
2. Industry 1,535,172 172,676 6993 11.248 0.455
3. Knowledge 224,253 34,688 3886 15.468 1.733
transfer
4. Marketing 7,730,644 767,185 43,741 9.924 0.566
5. Projection 274,406 13,867 351 5.053 0.128
6. Enrollment 697,521 353,146 81,184 50.629 11.639
7. Faculty 217,093 123,689 9714 56.975 4.475
8. Teaching 1,803,689 450,020 49,850 24.950 2.764
Total 15,169,870 2,097,663 204,150 13.828 1.346
a Assessed from the Web of Science on Oct. 2020. See Appendix 1, on synonyms for “HE” and
“areas” of application
the number of articles with methods/area in their topic. Topic means that the search
word (or one of its synonyms, see Appendix 1 for the synonyms for each search
word) appears either in the title, abstract, or keywords of the article. Column 2
counts the number of articles with HE and method/area in their topic. Column 3
counts the number of articles with HE and method/area in their title. Obviously,
for each method, column 2 articles are included in column 1 articles, and column
3 articles are included in column 2 articles. Thus, the count of column 2 is always
less than the count in column 1, and the count of column 3 is less than the count of
column 2.
3.4.1 Number of Articles on Simulation Methods
As shown in Table 3.1, the simulation methods were divided into two groups:
analytical simulations methods listed in Sect. 3.2 and educational simulation
methods listed in Sect. 3.3, used mainly for pedagogic purposes. Educational
simulations articles out-number the analytic simulation articles, since the former are
intended for a wider audience, i.e., those disciplines which do not require as much
mathematical proficiency as disciplines where analytic simulation is used. Even the
authors of educational simulations come from a variety of disciplines, such as health
professions and education, in addition to the STEM disciplines.
Obviously, out of ten simulation methods listed in Table 3.1, the method with the
highest number of articles in all three columns is the generic method simulation.
Moreover, simulation appears more than all the other nine simulation methods
combined. For example, in column 1 in general, there were 1,834,848 articles on
simulation methods, versus a total of 381,769 for analytic simulation and 556,052
for educational simulation. Obviously, there are overlaps. For example, in some
articles for method 4, discrete-event, the word simulation is often added, thus such
article will be counted twice in the total of analytic simulation. Thus, in the last row
of Table 3.1, the total count is given without the first row of method 1, simulation.
In column 2, where the search words HE and the specific simulation method
appear in the topic of the articles, the total number of analytic articles is 20,245,
while the number of education articles is 55,940. However, in column 3, where the
search is for titles that contain the words HE and a specific method, the total number
of analytical simulation articles is 575, while the number of educational simulation
articles is 4590. This proves that educational simulations became very important
during the last two decades. About 5% of the analytical simulation articles in WoS
are in HE, where the search words are in the topic (percent of column 2 total from
column 1), but 10% of the educational simulation articles in WoS are in HE. When
the search words are only in the articles’ titles (column 3 as a percentage of column
1) then the corresponding percent is 0.151% for analytical simulation and 0.825%
for educational simulation.
Of the four analytical simulation methods in Table 3.1, Monte Carlo is the
method with the largest number of articles in the first three columns (176 articles
in column 3). There are overlaps between Mote Carlo and the other two stochastic
analytic methods: discrete-event simulation and agent-based simulation. Neverthe-
less, each of these stochastic simulation methods alone has more articles than system
dynamics. Note that the method with the smallest number of articles is deterministic
in all three columns of Table 3.1.
From the five educational simulations, the virtual simulation method is leading
with the maximal number of articles in the first three columns of Table 3.1—with
261,559 articles, of which 30,390 articles include HE in their topic, of which 2650
articles include HE and virtual method in their title. The method game has the
second maximal number of articles in all three columns, and the method simulator
is the third.
In summary, educational simulations have many more articles than analytical
simulation methods. From educational simulations, the leading methods in terms of
the number of articles in WoS are as follows in order of frequency: virtual, game,
and simulator. The leading methods of analytical simulation are Monte Carlo,
discrete-event, and agent-based simulation.
3.4.2 Number of Articles on Application Areas
Table 3.3 presents eight application areas where HE can be applied. The leading
application areas, according to the number of articles in general (column 1), were
marketing, transportation, and teaching. For HE and area of application in the
topic of the articles (column 2), the order of leading application areas is marketing,
teaching, and enrollment. For HE and area appearing in the title of the articles
(column 3), the order of leading areas of applications is enrollment, teaching, and
marketing. Note that Table 3.3 does not refer to simulation at all.
In summary, the leading application areas in HE are enrollment, teaching, and

marketing. Overall, the number of articles in areas of applications as presented in
Table 3.3, which come under the rubric of area in topic in column 1 (about 15 million
articles), is much larger than the number of articles by methods in all three columns
of Table 3.1 (1.8 million articles).
3.4.3 Number of Articles on Simulation Methods

by Application Areas in Higher Education
Table 3.4 presents the number of articles with HE and simulation method in an
article title by area of application in their topic. The assumption behind this is
that most administrators and researchers in HE are likely to use HE and area of
application in their title to attract a wider range of readers, and thus they tend to
extend the method term to the keywords or abstract.
Table 3.4 Number of articles with HEa and area of applicationb by simulation method
Areab :
1 2 3 4 5 6 7 8
Methods TRNSb INDb KTb MKTGb PROJb ENRLb FACb TCHb Total
1. Simulation 540 217 47 977 36 1900 96 1804 5617
2. System 13 19 6 50 10 15 6 21 140
dynamic
3. Monte Carlo 19 46 1 83 5 14 13 10 191
4. Discrete- event 17 9 1 61 0 50 11 40 189
5. Agent- based 25 7 2 40 1 13 2 14 104
Sum of methods 74 81 10 234 16 92 32 85 624
2–5
6. Virtual 219 74 42 455 4 1036 66 1003 2899
7. Simulator 48 12 4 85 2 306 10 350 817
8. Game 59 60 32 216 3 770 29 390 1559
9. Box trainer 0 0 0 0 0 7 0 5 12
10. Manikin 0 1 0 4 0 86 2 71 164
Sum of methods 326 147 78 760 9 2205 107 1819 5451
6–10
Total 940 445 135 1971 61 4197 235 3708 11,692
Total w/o 400 228 88 994 25 2297 139 1904 6075
simulation
a Seesynonyms used in Appendix 1. HE (higher education) and application area in article title and
method in article topic
b Abbreviations of areas of applications are: TRNS —transportation, IND—industry KT—
knowledge transfer, MKTG—marketing, PROJ—projection, ENRL—enrollment, FAC—faculty,

TCH—teaching
The figures in Table 3.4 strengthen the results of Sects. 3.4.1 and 3.4.2 (as
reflected in Tables 3.1 and 3.3), although the counting method changed. Again, the
leading simulation methods, with the maximal number of articles, are the generic
term simulation, and then come the three leading educational simulations: virtual,
game, and simulator (last column of Table 3.4). Again, the leading application areas,
in terms of number of articles, are enrollment, teaching, and marketing (last row of
Table 3.4). Within each of the three leading methods: virtual, game, and simulator,
the maximal number by area follow the same order holds as the total: enrollment
teaching and marketing.
For example, focusing on the generic simulation method by application areas
in HE (row 1 in Table 3.4), the use of simulation for enrollment has the maximal
number of articles (1900 articles, in row 1 column 6). The area of application
with the second largest number of articles by area is simulation for teaching (1804
articles, in row 1, column 8). The area of application with the third largest of articles
is simulation for marketing (977 articles, in row 1 column 4), the same above
order of applications areas (enrollment, teaching, and marketing) applies for the
number of articles of each of the main three simulation methods: virtual, game and
simulator—in lines 6, 8, 7, respectively.
Similarly, looking at the last column of Table 3.4 (total) the simulation methods
with the maximal number of papers and their orders are virtual, game, and simulator
(2899, 1559, and 817, respectively). The same order applied to the three leading
application areas: enrollment, teaching, and marketing (rows 6, 8, 7, respectively).
In the following is a comparison between the analytic simulation methods and
the educational simulation methods:
(a) Analytic simulation—four methods: Monte-Carlo, discrete-event simulation,
system dynamics, and agent-based simulation, with 191, 189, 140, 104 articles,
respectively, and 624 articles in total compose only 10.27% of the articles.
(b) Educational simulations—five methods: virtual, games, simulator, manikin,
and box trainer, with 2899, 1559, 817, 164, 12 articles, respectively, and 5471
articles in total compose 89.73% of the articles (out of 6075).
Obviously, the educational simulation methods comprise most of the articles in
Table 3.4, almost 90%, in the WoS during the period 2000–2020.
Moreover, each of these simulation methods: virtual, game, and simulator; their
areas with the maximal number of articles are the same: enrollment, teaching, and
marketing (as shown in Table 3.4, in the intersect of these three methods and three
areas). These nine entries (of virtual, game, and simulator methods by enrollment,
teaching, and marketing areas) compose 4611 (76%) of the articles in Table 3.4, not
including the generic method simulation. Namely, 12.5% of the entries of Table 3.4
cover 76% of the articles—thus, the Preto rule basically holds true.
As shown in Appendix 1, marketing (MKTG) area of application includes
the following synonyms: finance, financial, strategy, . . . , policy models, . . . ,
planning, budgeting, cost, costing, managerial, management, administration, and
administrative. Section 3.2 shows that analytical simulations are used mainly in
marketing areas, as listed here. Indeed, we see in Table 3.4 that for each of the four
analytic simulation methods, including the sum of the analytic simulation methods
(line of “Sum of methods 2–5” in Table 3.4), marketing has the highest number
of articles, while the educational simulation methods achieve their maximum in
enrollment and teaching, and marketing is in the third place.
In summary, as seen in Table 3.4, the educational simulation methods have by
far more articles than analytical simulation methods. The educational simulation
methods with the largest number of articles are virtual, game, and simulator. The
areas of applications with the maximal number of articles are enrollment, teaching,
and marketing. However, the analytical simulation methods have the maximal
number of articles in the marketing area of application as expected, while enrollment
and teaching areas of applications are in the second and third places.
Simulation is one of the leading methodology/tool in HE, besides its use in Industry,
military, leisure, etc. The number of articles in WoS during the recent years on
simulation, in general, is over 1.8 million articles; of which simulation in HE
composes about 10%. This chapter provides an overview of simulation in HE: first
by classifying the simulation methods in HE by type, second by pointing at areas
of application where simulation is used in HE; third by verifying the most used
simulation methods and their areas of application in HE. We suggest classifying the
simulation methods in HE into two main types of simulation models: a. analytical
simulation methods and b. educational simulation methods—each is divided into
sub-methods.
In summary, we found that educational simulations have many more articles than
analytical simulations (in WoS during 2000–2019). The leading simulation methods
(in terms of number of articles) are (1) Virtual (2) Game and (3) Simulator—all
three methods are from the educational simulations group.
The advantages of educational simulation are that they are characterized by
safety (e.g., in medicine) and accessibility (virtual Lab); are more economic with
respect to training, with less disruption of training than exists in real situations
(health system simulations, or business game); have more flexibility in training
under various conditions, and enable flexibility in training scheduling.
We consider here eight areas of application of simulation in HE: (1) transporta-
tion (2) Industry (3) Knowledge transfer (4) Marketing (5) Projection (6) Enrollment
(7) Faculty (8) Teaching (see synonyms used for each area in Appendix 1). The
leading areas of application in HE in terms of number of articles are (1) enrollment,
(2) teaching, and (3) Marketing. Looking at the ten simulation methods by eight
areas of application in HE (Table 3.4), the same three simulation methods (virtual,
game, and simulator) are leading within each area of application and mostly in
the same above three leading areas of applications (i.e., enrollment, teaching, and
marketing). However, the analytical simulation has its maximal number of articles
in marketing—enrollment and teaching are the next in the number of articles.
The advantages of analytic simulation are manifold. They do the following:

improve the ability to choose from alternatives by easily testing various options,
policies, and scenarios without disrupting the real system; diagnose problems in
complex situations; accelerate real life processes in time via simulation runs,
predicting future development and “what if” behavior of the system; and enable
visualizing the system operation.
The main disadvantage of simulation methods, in general, is that simulation
is an imitation of the real system. Other disadvantages include the need to train
the users and teachers to use the specific simulation model with its technicalities,
assumptions, options, applications, and interpretation of the results. Verification
and validation of the simulation sometimes are neglected, and sometimes, effec-
tiveness/cost value may be questionable.
This overview is useful for developers of HE simulations and for administrators
at all levels of HE institutions, including heads of departments, deans, consultants,
and councils for HE at the state level, so they can become aware of the growing use
of simulations in HE and their potential use.
For future research, enlarging the synonyms and avoiding double counting will
improve the accuracy of the articles’ counting results. There is a lack of literature
reviews on three of the analytical simulation methods: discrete-event simulation,
agent-based simulation in HE, and deterministic and other simulations in HE.
Finally, there is a need to review the economic viability of the various educational
simulation tools (digital and virtual) for HE. The COVID-19 pandemic may increase
the utilization of the various simulation tools. It would be interesting to study
the change in the use of educational simulation, mainly online and web-based
simulations, a year and more after the pandemic will be over. Moreover, it remains
to be seen what forms of virtual classroom and the virtual university will evolve (see
Almog & Almog, 2020).
Appendix 1: Synonyms for: (a) Higher Education (b)

Simulation Method (c) Areas of Application
a. Higher Education:
Higher education: academic, college, university, tertiary education, medical education, nurse,
nurses, nursing, pharmacy, undergraduate education, undergraduate, airway management
education, medical, engineering, business education, internship, higher professional education
b. Simulation Methods:
1. Simulation: simulating, simulate, simulation-based
2. Industrial dynamic, industrial dynamics, system dynamic, system dynamics,
system-dynamic, dynamic simulation
3. Monte Carlo
(continued)
4. Discrete-event: discrete-event, DES

5. Agent-based: agent-based, ABS, ABM
6. Simulator: simulators
7. Game: games
8. Box trainer: box training.
c. Areas of Applications:
1. Transportation: transshipment, assignment, project management, CPM, Gant, PERT,
network, networks, location, scheduling, timetabling
2. Industry: logistics, maintenance, reliability, inventory
3. Knowledge transfer: knowledge management, management of knowledge, R&D research
and development, patent, patents, industrial park, industrial parks, incubator, incubators,
entrepreneurship, entrepreneurial
4. Marketing: finance, financial, strategy, strategic, balanced scorecard, BSC, efficiency,
efficient, accounting, investment, revenue management, policy modeling, policy model, policy
models, renovation, international, internationalization, planning, budgeting, cost, costing,
managerial, management, administration, administrative
5. Projection: projecting, forecasting, forecast
6. Enrollment: student/students
7. Faculty: staff, manpower
8. Teaching: teach, practice, practicing, train, training, pedagogy, pedagogic, practitioner
Appendix 2: Taxonomy of Simulation Models Applied

to Higher Education
Type of Levels HE Country of

Authors (year) simulationa HE area hierarchy application
Weathersby Regression Costs of University USA, UC
(1967) departments by Berkeley
department
Baisuck and DES Enrollment University USA
Wallace projection and regional
(1970)
Hopkins Deterministic. University University USA
(1971) simulation planning
Sinuany- Simulation Budgeting Community USA (multi
Stern and and planning college (CC) campus CC)
(1978) optimization
Kennedy SD Policy issues All levels Review
(2008) 1975–2005
Schellekens DES Program University The
et al. (2010) organization Netherlands
Kennedy SD Policy and All levels 40 Review
(2011) pedagogy articles 1972–2008
(continued)

Galbraith SD Various Various Methodology
(2012)
Voyer et al. SD Performance Business USA
(2012) sch.
Hussein and SD and Budget Institutional Methodology
El-Nasr genetic allocation NetLogo
(2013) algorithm and quality
Larson et al. R0, Supply and National USA
(2014) reproduction demand of
number Eng. PhDs
Brailsford SD Supply and National Sri Lanka
and De Silva demand of
(2015) dentists
Gu et al. ABS/ABM Academic All HE Review
(2015) activities levels 35 during
and teaching articles 1997–2013
Gu et al. ABS Researchers University Australia
(2015) prototype in publications by
NetLogo strategy department
by
researcher
Kara (2015) Stochastic Stud. Supply National Turkey
SD and demand
and quality
Fateh Rad et SD Strategic— University Iran
al. (2015) industry Regional
relations
Little (2015) DES and Quality University General
ABS improve
Strauss and SD Long run National HE Brazil
Borenstein Undergrad. planning, pub-
(2015) enrollment strategy lic/private
sector
Dandagi et SD and Strategy University India
al. (2016) Structural Question-
equations naire factor
anal.
Oltean et al. DES— Building Public Romania
(2017) facility viability university
planning
Al Hallak et SD—facility Enrollment Regional Syria
al. (2017) & faculty Mgmt. private
plan sector
Faham et al. SD Strategy University Iran
(2017)
(continued)

Ghaffarzadegan SD— Policy Edu. Regional Methodology
et al. (2017) Student job
quantity vs. mismatch
quality
Zaini et al. SD Strategy University USA
(2017)
Kara (2018) SD— Quality and University Turkey
Structural productivity
equation
Tan et al. Discrete- Quality of International China
(2019) timed Peer review
model, process
Parallel
systems
Siniksaran Simulation Universities Universities Review
and Satman ranking
(2020)
a SD—System Dynamics, DES—Discrete-event Simulation, ABS—Agent-Based Simulation also
ABM (Agent-Based Model). b indicating number of reviewed papers in a review paper
References
Adib-Hajbaghery, M., & Sharifi, N. (2017). Effect of simulation training on the development of
nurses and nursing students’ critical thinking: A systematic literature review. Nurse Education
Today, 50, 17–24.
Al Hallak, L., Ayoubi, R. M., Moscardini, A., & Loutfi, M. (2017). A system dynamic model of
student enrollment at the private higher education sector in Syria. Studies in Higher Education,
44(4), 663–682.
Alkhaldi, T., Pranata, I., & Athauda, R. I. (2016). A review of contemporary virtual and remote
laboratory implementations: Observations and findings. Journal of Computers in Education,
3(3), 329–351.
Almog, T., & Almog, O. (2020). Academia—All the lies: What went wrong in the university model
and what will come in its place. Tamar & Oz Almog Publishing.
Avramenko, A. (2012). Enhancing students’ employability through business simulation. Education
+ Training, 54(5), 355–367.
Baisuck, A., & Wallace, W. A. (1970). A computer simulation approach to enrollment projection
in higher education. Socio-Economic Planning Sciences, 4(3), 365–381.
Banks, C. M. (2009). What is Modeling and simulation? In J. A. Sokolowski & C. M. Banks (Eds.),
Principles of Modeling and simulation: A multidisciplinary approach. Wiley, Chapter 1.
Banks, J., Carson, J. S., Nelson, B. L., & Nicol, D. (2005). Discrete-event system simulation (4th
ed.). Prentice-Hall.
Barlas, Y., & Diker, V. (2000). A dynamic simulation game (UNIGAME) for strategic university
management. Simulation and Gaming, 31(3), 331–358.
Bell, G., Warwick, J., & Galbraith, P. (2012). Higher education management and operational
research: Demonstrating new practices and metaphors (Eds). In M. A. Peters (Ed.), Educational
futures rethinking theory and practice (Vol. 54). Sense Publishers.
Bradley, P. (2006). The history of simulation in medical education and possible future directions.
Medical Education, 40(3), 254–262.
Brailsford, S., & De Silva, D. (2015). How many dentists does Sri Lanka need? Modelling to
inform policy decisions. Journal of the Operational Research Society, 66(9), 1566–1577.
Brailsford, S. C., Eldabi, T., Kunc, M., Mustafee, N., & Osorio, A. F. (2019). Hybrid simulation
modelling in operational research: A state-of-the-art review. European Journal of Operational
Research, 278(3), 721–737.
Cant, R. P., & Cooper, S. J. (2010). Simulation-based learning in nurse education: Systematic
review. Journal of Advanced Nursing, 66(1), 3–15.
Cant, R. P., & Cooper, S. J. (2017). Use of simulation-based learning in undergraduate nurse
education: An umbrella systematic review. Nurse Education Today, 49, 63–71.
Cheesman, M. J., Chen, S., Manchadi, M.-L., Jacob, T., Minchin, R. F., & Tregloan, P. A.
(2014). Implementation of a virtual laboratory practical class (VLPC) module in pharmacology
education. Pharmacognosy Communications, 4(1), 2–10.
Cooper, S., & Fox-Young, S. (2012). Simulation based learning in midwifery education: A
systematic review. Women and Birth, 25(2), 64–78.
Coyne, L., Merritt, T. A., Parmentier, B. L., Sharpton, R. A., & Takemoto, J. K. (2019). The
past, present, and future of virtual reality in pharmacy education. American Journal of
Pharmaceutical Education, 83(3), https://doi.org/10.5688/ajpe7456
Curran, V., Fleet, L., White, S., Bessell, C., Deshpandey, A., Drover, A., Hayward, M., & Valcour,
J. (2015). A randomized controlled study of manikin simulator fidelity on neonatal resuscitation
program learning outcomes. Advances in Health Sciences Education, 20(1), 205–218.
Dandagi, S., Bhushi, U., Bagodi, V., & Sinha, D. (2016). Strategic management of technical
university: Structural equation modelling approach. Journal of Modelling in Management,
11(1), 75–90.
Deshpande, A. A., & Huang, S. H. (2011). Simulation games in engineering education: A state-of-
the-art. Computer Applications in Engineering Education, 19(3), 399–410.
Doolen, J., Mariani, B., Atz, T., Horsley, T. L., O’ Rourke, J., McAfee, K., & Cross, C. L. (2016).
High-fidelity simulation in undergraduate nursing education: A review of simulation reviews.
Clinical Simulation in Nursing, 12(7), 290–302.
Downer, T., Gray, M., & Andersen, P. (2020). Three-dimensional technology: Evaluating the use
of visualization in midwifery education. Clinical Simulation in Nursing, 39, 27–32. https://
doi.org/10.1016/j.ecns.2019.10.008
Dubovi, I., Levy, S. T., & Dagan, E. (2017). Now I know how! The learning process of medication
administration among nursing students with non-immersive desktop virtual reality simulation.
Computers and Education, 113(1), 16–27.
Faham, E., Rezvanfar, A., Mohammadi, S. H. M., & Nohooji, M. R. (2017). Using system
dynamics to develop education for sustainable development in higher education with the
emphasis on the sustainability competencies of students. Technological Forecasting and Social
Change, 123, 307–326.
Faria, A. J., Hutchinson, D., Wellington, W. J., & Gold, S. (2009). Developments in business
gaming: A review of the past 40 years. Simulation and Gaming, 40(4), 464–487.
Fateh Rad, M., Sayed Esfahani, M. M., & Jalilvand, M. R. (2015). An effective collaboration model
between industry and university based on the theory of self-organization: A system dynamics
model. Journal of Science and Technology Policy Management, 6(1), 2–24.
Foronda, C. L., Fernandez-Burgos, M., Nadeau, C., Kelley, C. N., & Henry, M. N. (2020). Virtual
simulation in nursing education: A systematic review spanning 1996 to 2018. Simulation in
Healthcare: The Journal of the Society for Simulation in Healthcare, 15(1), 46–54.
Forrester, J. W. (1961). Industrial dynamics. Productivity Press.
Frank, R. M., Wang, K. C., Davey, A., Cotter, E. J., Cole, B. J., Romeo, A. A., Bush-Joseph, C. A.,
Bach, B. R., & Verma, N. N. (2018). Utility of modern arthroscopic simulator training models:
A meta-analysis and updated systematic review. Arthroscopy: The Journal of Arthroscopic and
Related Surgery, 34(5), 1650–1677.
Galbraith, P. (2012). Making a bed to lie in: System dynamics behind university management
stress. In G. Bell, J. Warwick, & P. Galbraith (Eds.), Higher education management and
operational research. Educational futures (rethinking theory and practice) (Vol. 54, pp. 179–
208). Sense Publishers, Brill Sense. chapter 11.
Ghaffarzadegan, N., Xue, Y., & Larson, R. C. (2017). Work-education mismatch: An endogenous
theory of professionalization. European Journal of Operational Research, 261(3), 1085–1097.
Gu, X., Blackmore, K., Kornforth, D., & Nesbitt, K. (2015). Modelling academics as agents: An
implementation of an agent-based strategic publication model. Journal of Artificial Societies
and Social Simulation, 18(2). https://doi.org/10.18564/jasss.2725
Gu, X., & Blackmore, K. L. (2015). A systematic review of agent-based modelling and simulation
applications in the higher education domain. Higher Education Research and Development,
34(5), 883–898.
Hallgren, K. A. (2013). Conducting simulation studies in the R programming environment. Tutor
Quant Methods Psychology, 9(2), 43–60.
Hopkins, D. S. P. (1971). On the use of large-scale simulation models for university planning.
Review of Educational Research, 41(3), 467–478.
Horsley, T. L., O’Rourke, J., Mariani, B., Doolen, J., & Pariseault, C. (2018). An integrative review
of Interprofessional simulation in nursing education. Clinical Simulation in Nursing, 22, 5–12.
Hussein, S. E., & El-Nasr, M. A. (2013). Resources allocation in higher education based on system
dynamics and genetic algorithms. International Journal of Computer Applications, 77(10), 40–
48.
Juan, A. A., & Loch, B., Daradoumis T., & Ventura, S. (2017) Games and simulation in higher
education. International Journal of Educational Technology in Higher Education, 14, 37.
https://doi.org/10.1186/s41239-017-0075-9
Kane, P. (2018). Simulation-based education: A narrative review of the use of VERT in radiation
therapy education. Journal of Medical Radiation Sciences, 65(2), 131–136.
Kara, A. (2015). Simulations of technology-induced and crisis-led stochastic and chaotic fluctua-
tions in higher education processes: A model and a case study for performance and expected
employment. Educational Science: Theory and Practice, 15(2), 303–312.
Kara, A. (2018). Escaping mediocre-quality, low-productivity, low-performance traps at uni-
versities in developing countries: A human capital-based structural equation model with
system-dynamics simulations. Educational Science: Theory and Practice, 18(3), 541–559.
https://doi.org/10.12738/estp.2018.3.0255
Kennedy, M. (2008). A taxonomy of system dynamics models of educational policy issues.
Retrieved September, 2009, from https://proceedings.systemdynamics.org/2008/proceed/
papers/KENNE182.pdf
Kennedy, M. (2011) A taxonomy of system dynamics models of educational pedagogic
issues. Retrieved May, 2019, from https://www.systemdynamics.org/assets/conferences/2011/
proceed/papers/P1455.pdf
Kovanis, M., Porcher, R., Ravaud, P., & Trinquart, L. (2016). Complex systems approach to
scientific publication and peer-review system: Development of an agent-based model calibrated
with empirical journal data. Scientometrics, 106, 695–715.
Lapkin, S., & Levett-Jones, T. (2011). A cost–utility analysis of medium vs. high-fidelity human
patient simulation manikins in nursing education. Journal of Clinical Nursing, 20(23–24),
3543–3552.
Larson, R. C., Ghaffarzadegan, N., & Xue, Y. (2014). Too many PhD graduates or too few academic
job openings: The basic reproductive number R0 in academia. Systems Research Behavioral
Science, 31(6), 745–750.
Little, D. (2015). Guiding and modelling quality improvement in higher education institutions.
Quality in Higher Education, 21(3), 312–327. https://doi.org/10.1080/13538322.2015.1111008
Lui, R. W. C., Lee, P. T. Y., & Ng, V. T. Y. (2015). Design and evaluation of PMS: A computerized
simulation game for software Project Management. The Computer Games Journal, 4(1–2),
101–121.
McGaghie, W. C., Issenberg, S. B., Petrusa, E. R., & Scalese, R. J. (2010). A critical review of
simulation-based medical education research: 2003–2009. Medical Education, 44(1), 50–63.
Minnery, J., & Searle, G. (2013). Toying with the City? Using the computer game SimCity™4.
Planning Practice & Research, 29(1), 41–55.
Molders, M., Fink, R. D., & Weyer, J. (2011). Modeling scientists as agents. How scientists cope
with the challenges of the new public management of science. Journal of Artificial Societies
and Social Simulation, 14(4), 6.
Murbay, S., Neelakantan, P., Chang, J. W. W., & Yeung, S. (2020). Evaluation of the introduction
of a dental virtual simulator on the performance of undergraduate dental students in the pre-
clinical operative dentistry course. European Journal of Dental Education, 24(1), 5–16.
Nehring, W. M., & Lashley, F. R. (2010). High Fidelity patient simulation in nursing education.
Jones and Bartlett Publishers.
Oltean, V. E., Borangiu, T., & Drăgoicea, M. (2017). A discrete event model of viability building
in a public university organization. Service Science, 9(4), 263–352.
Papanikolaou, I. G., Haidopoulos, D., Paschopoulos, M., Chatzipapas, I., Loutradis, D., & Vlahos,
N. F. (2019). Changing the way we train surgeons in the 21th century: A narrative comparative
review focused on box trainers and virtual reality simulators. European Journal of Obstetrics
and Gynecology and Reproductive Biology, 235(13–18).
Richmond, B. (2004). An introduction to systems thinking with iThink paperback. isee systems.
Rosa, M., González, M., Araújo, A. C. C., & Santiago, G. (2017, August 21–25). Business
game and its relationship with creativity: A systematic literature review. 21st International
Conference on Engineering Design, ICED17. The University of British Columbia, 409–418.
Samosorn, A. B., Gilbert, G. E., Bauman, E. B., Khine, J., & McGonigle, D. (2020). Teaching
airway insertion skills to nursing faculty and students using virtual reality: A pilot study.
Clinical Simulation in Nursing, 39(1), 18–26.
Schellekens, A., Paas, F., Verbraeck, A., & van Merrienboer, J. J. G. (2010). Designing a flexible
approach for higher professional education by means of simulation modelling. Journal of
Operational Research Society, 61(2), 202–210.
Siebers, P. O., Macal, C. M., Garnett, J., Buxton, D., & Pidd, M. (2010). Discrete-event simulation
is dead, long live agent-based simulation! Journal of Simulation, 4(3), 204–210.
Siniksaran, E., & Satman, M. H. (2020). WURS: A simulation software for university rankings—
Software review. Scientometrics, 122, 701–717. https://doi-org.ezproxy.bgu.ac.il/10.1007/
s11192-019-03269-8
Sinuany-Stern, Z. (1978). Financial planning models for hierarchical higher education systems:
Application of simulation and multi goal network optimization. Ph.D. Thesis, Case Western
Reserve University.
Sinuany-Stern, Z. (1983). Cost simulation models for university budgeting. Computer Environment
and Urban Systems, 8(4), 211–216.
Sinuany-Stern, Z., & Stern, E. (1993). The role of traffic factors and route choice behavior in
simulating urban evacuation. Socio-Economic Planning, 27(2), 97–108.
Sinuany-Stern, Z., Stern, E., Sfaradi, Z., & Holm, E. (1997). Effect of commuters behavior
on congested network performance: A micro simulation approach. European Journal of
Operations Research, 96(3), 455–470.
Smithson, J., Bellingan, M., Glass, B., & Mills, J. (2015). Standardized patients in pharmacy
education: An integrative literature review. Currents in Pharmacy Teaching and Learning, 7(6),
851–863.
Sokolowski, J. A., & Banks, C. M. (Eds.). (2009). Principles of Modeling and simulation: A
multidisciplinary approach. Wiley.
Stern, E., & Sinuany-Stern, E. (1989). A Behavioral-based simulation model for urban evacuation.
Regional Science Association, 66(1), 87–103.
Strauss, L. M., & Borenstein, D. (2015). A system dynamics model for long-term planning of the
undergraduate education in Brazil. Higher Education, 69(3), 375–397.
Sun, Y., Pan, C., Li, T., & Gan, T. J. (2017). Airway management education: Simulation based
training versus non-simulation based training-a systematic review and meta-analyses. BMC
Anesthesiology, 17(1), 17.
Tan, Z., Cai, N., Zhou, J., & Zhang, S.-G. (2019). On performance of peer review for academic
journals: Analysis based on distributed parallel system. IEEE Access, 7, 19024–19032. https://
doi.org/10.1109/access.2019.2896978
Thompson, R. K. (1970). Higher education administration: An operating system study utilizing
dynamic simulation model. In A. N. Schreiber (Ed.), Corporate simulation models. Seattle
University of Washington Printing.
Vlachopoulos, D., & Makri, A. (2017). The effect of games and simulations on higher education:
A systematic literature review. International Journal of Educational Technology in Higher
Education, 14, 22. https://doi.org/10.1186/s41239-017-0062-1
Voyer, J., Brown, S. B., Gage, N., Kovalenko, D., & Williams, T. (2012). A system dynamics
approach to improving an advising system for business school undergraduates. In G. Bell,
J. Warwick, & P. Galbraith (Eds.), Higher education management and operational research.
Educational futures (rethinking theory and practice) (Vol. 54, pp. 125–152). Sense Publishers,
Brill Sense. chapter 8.
Warren, J. N., Luctkar-Flude, M., Godfrey, C., & Lukewich, J. (2016). A systematic review of
the effectiveness of simulation-based education on satisfaction and learning outcomes in nurse
practitioner programs. Nurse Education Today, 46, 99–108.
Weathersby, G. (1967). The development and applications of a university cost simulation model.
University of California, Berkeley, Office of Analytical Studies.
White, G. (1987). The implementation of management science in higher education administration.
OMEGA International Journal of Management Science, 15(4), 283–290.
Yang, L. (2014). Evolution of simulation paradigms in OR. In Wang J. Encyclopedia of business
analytics and optimization. https://doi.org/10.4018/978-1-4666-5202-6.ch083
Zaini, R. M., Pavlov, O. V., Saeed, K., Radzicki, M. J., Hoffman, A. H., & Tichenor, K. R. (2017).
Let’s talk change in a university: A simple model for addressing a complex agenda. Systems
Research and Behavioral Science, 34(3), 250–266.
Zupanc, G. K. H., Lehotzky, D., & Tripp, I. P. (2021). The Neurosphere simulator: An educational
online tool for modeling neural stem cell behavior and tissue growth. Developmental Biology,
469(1), 80–85.
Chapter 4
Managing Minds: The Challenges
of Current Research Information Systems
for Improving University Performance
Gad Yair
“Data and analytics help inform decision-making. Rendering

decisions without accurate, robust, and topical data is the same
as flying blind, relying on intuition, “gut feel,” and
unreproducible methods.”
(Peter Maglione, CEO, Academic Analytics)
Abstract This chapter describes the failure of Current Research Information

Systems (CRIS) to modernize routine managerial tasks involved in academic
administration. It argues that most universities maintain traditional decision-making
procedures in regards to hiring, promotions, preparation of annual reports, and
submission of portfolios for accreditation and assessment. I argue that developers of
CRIS programs failed to appreciate the tasks of academic management. At the same
time, management teams seem reluctant to change traditional academic practices.
I discuss reasons for this disconnect and suggest ways to close the gap so as to
modernize academic decision-making in universities.
Keywords Current Research Information Systems (CRIS) · Higher education ·

Academic management
4.1 Introduction
Two decades into the twenty-first century, many universities still administer central
academic procedures without accurate and robust research data. Top and middle-
level administrators carry out organizational tasks like the hiring of new faculty,
conducting promotion procedures, submitting annual reports, and to a lesser extent
G. Yair ()
Department of Sociology & Anthropology, The Hebrew University of Jerusalem, Jerusalem, Israel
e-mail: gad.yair@mail.huji.ac.il

118 G. Yair
preparing for accreditation and assessment exercises through manual procedures.

Rarely do they rely on new digital tools that appeared on the academic scene during
the past decade. Indeed, Current Research Information Systems, or CRIS programs,
had promised to solve those problems while modernizing the abovementioned tasks.
However, they have not. Both parties share the responsibility for failing to
modernize academic management by implementing CRIS programs. The problem
results from problematic marketing strategies by major companies; it also results
from an origins story in library science. However, the disconnection also results
from faculty resistance and from top administrations’ misspecification of academic
needs.1 But as a result, resources are wasted, and academic units are left unaccount-
able.
The current chapter seeks to connect the two spheres by providing examples for
possible specifications and their academic rationales. It urges university leaders to
abandon current “medieval” practices by adopting digital tools. It calls on CRIS
developers to better cater to the routine needs of academic management. With joint
efforts, both sides can help universities to close the digital divide that separates many
of them from modern organizational realities.
A caveat: This essay benefits from a decade-long experience as the person
who developed CoolCite, one of the first CRIS programs on the market. I am
also a science sociologist, with keen observations of Israeli science and Israeli
academia (Yair, 2019a, 2019b). I gained comparative perspectives on the problems
of integrating CRIS programs into academic decision-making processes through
visits to American and European universities as well. Surprisingly or not, I found
most universities share similar disjuncture between the medieval realities of their
academic management and leading technological advances in the field of knowledge
management. I hope my call below helps to close this gap.
4.2 Context: Accountability and Transparency of Research

Outputs
As the global race for scientific innovation speeds up, administrators are challenged
to provide stakeholders with data about research performance and impact. They
face demands to lower costs while increasing productivity and impact. Multiple
stakeholders pressure universities to modernize while becoming accountable and
transparent. Government allocations for research require more precise data about
productivity ever. Private donors pend endowments on data concerning academic
excellence. Corporations hinge resources for research by demanding transparent
1 As a developer of a CRIS program I had the opportunity to follow Israeli University presidents and
rectors drag the process for more than a decade. First attempts for national implementation took
place in 2008. Only in 2017 has PURE by Elsevier won a bid. We are at the verge of 2020—and
we are yet to see implementation (checked with the bidding agency, Machba, November 2019).
4 Managing Minds: The Challenges of Current Research Information Systems. . . 119
data about scientific outputs. As a result, boards of governors entrust university

leaders with responsibility for maximizing scientific innovation within growing
budgetary constraints. To answer those diverse stakeholders, administrators need
robust evidence about inputs (e.g., funding, positions, FTEs) and outputs (e.g.,
students, patents, publications, prizes, citations). They need information about
successful units as well as about bottlenecks and areas of malfunction.
With growing public pressures for accountability and transparency, top university
administrators are therefore responsible for two interrelated tasks: (A) Improving
the conditions that allow scientists to maximize their research potential; (B)
Employing Current Research Information Systems (CRIS) for responsible and
transparent decision-making procedures at different levels under their jurisdiction.
They are assisted in attaining the first task by abundant research concerning
management practices. Indeed, decision-makers have a knowledge-base from which
to extract organizational strategies for increasing scientific productivity and impact.
For example, studies have pointed to organizational levers like funding sources,
monitoring practices, or performance-based incentive programs. However, leading
university administrators still regard the second area as terra incognita. Up until
recently, indeed, they had little data for supporting evidence-based managerial
practices or the digital tools to support routine, labor-intensive administrative
procedures. Furthermore, they often lack familiarity with new technologies for
academic management or the skills to use them. As a result, some university leaders
still oversee their universities while relying on outdated, oftentimes arbitrary, and
possibly biased procedures for decision-making. Some just keep using the practices
that Peter Maglione criticized, namely “flying blind, relying on intuition, “gut feel,”
and unreproducible methods.”
In response to those growing needs, entrepreneurs have developed tools for the
management of academic information that supposedly serves university administra-
tions. Giants like Thomson Reuters, Elsevier, Clarivate Analytics, and ProQuest
envisioned offering administrators with digital tools for research management.
Smaller innovative companies introduced alternative metrics for assessing indi-
vidual or unit productivity and impact (Altmetrics). Indeed, CRIS programs like
Esploro, Scival, Pure, and Vivo promised to bring universities up to speed into
the digital revolution.2 However, they missed central clients in University admin-
istration, since developers provided librarians, rather than administrators, with data
and analytics for assessing science. True, some programs provide aggregated data
arranged in multiple organizational levels; some standardized supply metrics for
assessing journals and publications; and several enrich views on data by adding
“alternative metrics” that broaden the scope of scientific assessments. However,
most CRIS programs have remained detached from routine academic realities.
2 These are but prominent examples. The field contains many other platforms. Readers are
advised to see a taxonomy of those programs and their features here (extracted Decem-
ber 21, 2019): https://en.wikipedia.org/wiki/Comparison_of_research_networking_tools_and_
research_profiling_systems.
120 G. Yair
Developers have offered sophisticated tools with horns and whistles, but they have
mostly answered for needs of librarians and research managers. They thus failed
serving universities in the main tasks that they conduct on an annual basis.
The current mismatch between advanced digital tools and routine university
practices results from the fact that most companies in the field arrived from scientific
publishing and library management. As a result, they designed CRIS programs with
the intent to satisfy the needs of university librarians, not administrators in the front
where science is managed. One often observes this biased orientation in marketing
efforts: CRIS programs mainly target libraries and librarians. In some cases, they
also target research administrators. Consequently, those advanced digital products
have largely remained unused during central, labor-intensive academic procedures
that task university administrations on an annual basis. Indeed, scrutiny of main
academic procedures suggests that department chairs, deans, provosts, rectors, and
university presidents have remained aside from the digital revolution. Those are
truly smart people; their technologies, however, are a bit more advanced than pencil
and paper.3
The present chapter seeks to mend the disconnection between current research
information systems and routine academic operations in universities. It first reviews
the development of the burgeoning field of current research information systems
and explains why their publishing and library orientations contributed to the dis-
connection. Second, the chapter shows where CRIS developers and administrators
need to meet to finally connect the digital revolution to traditional, almost medieval
administrative practices. To do so, it focuses on major annual, labor-intensive
organizational procedures. It demonstrates how administrators would benefit from
appropriately specified CRIS programs that would target routine organizational
needs. Specifically, it dwells on hiring practices, promotion procedures, preparation
of annual reports, and the submission of assessment and accreditation portfolios.
Most universities currently carry those routine organizational practices through
the same procedures that they used three decades ago, when Bill Gates described
the digital revolution and the paperless office (B. Gates, 1995). It is time for
department chairs, deans, rectors, and universities to jump on the digital bandwagon.
With adapted CRIS programs, they would be in a better position to support
scientific innovation, productivity, and excellence. Furthermore, by redesigning
current research information systems, developers can boost their integration with
the digital campus, advancing beyond purveyors of research information systems.
Their products do have the potential to animate and modernize the university of the
twenty-first century.4 But they need to better address administrative needs.
3 Some administrators boast that they moved to Dropbox sharing CV documents rather than using
printed papers. This is hardly a technological advance.

4 The organizational challenges I describe and my suggestion for software development are set
from a bird’s eye. Each should be carefully studied with possible adaptations to local institutional
variations.
4.3 Libraries Are Not Universities: The Problematic

Evolution of CRIS Programs
CRIS programs are the applied side of the field nowadays called “The Science
of Science” (Fortunato et al., 2018; Eugene Garfield, 2009; Price, 1963). This
century-long research tradition was inaugurated by Alfred Lotka, who was the first
to model the rarity of scientific excellence (Lotka, 1929; Yair et al., 2017). The
revolution in this field—and its commercialization—gained pace with the studies
of Eugene Garfield, who founded the area of bibliometrics.5 Garfield’s studies
expanded the understanding that distributions of publication productivity, paper
citation counts, and journal impact factors6 follow power-law or log-normal patterns
(Eugene Garfield, 2009; Eugene Garfield, 2006; Eugene Garfield, 1970; E Garfield,
1972). His leadership gained followers in library science and library management.
However, since his development efforts targeted librarians, he inadvertently created
a limited trajectory for later technological developments in academic management.
The field of information science and scientometrics made tremendous technical
advances with growing data availability (e.g., Web of Science, Google Scholar).7
It was further buttressed by the incorporation of simple metrics for assessments of
productivity and impact, exemplified by the H-Index8 and its many adaptations and
revisions (Hirsch, 2005; Mistele et al., 2019; Praus, 2019). In the last two decades,
scholars have also introduced sophisticated research methods, coupled with network
models and big data, for evaluating and assessing careers, programs, universities,
and entire fields of knowledge (Sinatra et al., 2016; A. J. Gates et al., 2019). Social
network platforms like Facebook and Twitter expanded available information about
the impact and reach of research. They allowed innovators to come up with new
tools for harvesting that information for assessing impact (e.g., PlumX).9
Those technical innovations led in some cases to multi-billion businesses. Small
enterprising individuals who came up with ideas for extracting information and
rearranging it for decision-making ended up being integrated into huge publishing
5 Bibliometrics and scientometrics are fields of study that focus on scientific productivity and
impact—which constitute the two main measures for ranking individuals, journals, and institutions
in terms of success and prestige.
6 Measures of impact factor (IF) denote the number of expected citations for publications in Journal
X in the following year or two. The higher the IF, the more citations garnered by papers. This
measure serves as a proxy for journal quality and its prestige. It is also a proxy for the centrality of
journals in research fields.
7 Some administrators use Publish or Perish, a software that collects publication data from
Google Scholar. Many fail to appreciate problems in Google Scholar. See the rich website of
P&P developer, Anne-Wil Harzing in: https://harzing.com/resources/publish-or-perish (extracted
December 21, 2019).
8 The H-Index measures productivity (number of publications) and impact (number of citations)
with a single number that makes it highly popular and effective.

9 PlumX measures number of mentions in social network platforms like Facebook, Twitter and
Mendeley. See: https://plumanalytics.com/learn/about-metrics/ (extracted December 21, 2019).

122 G. Yair
or library outlets. For example, Eugene Garfield created Current Contents, the
Science Citation Index (SCI), and the Journal Citation Reports, a family of tools
that were gradually incorporated into Thomson Reuter’s products. Those tools were
later bought by Clarivate Analytics, nowadays offering a large suite of solutions
to researchers and institutes (e.g., Endnote, Publons, InCite).10 The library of
Cornell University developed VIVO for integrating internal information.11 This
local CRIS system now serves 140 institutes across more than 25 countries.
Similarly, Aleph, originally developed at The Hebrew University of Jerusalem to
improve its library collection, morphed into Ex-Libris, which was later bought by
ProQuest—now offering its own CRIS program named Esploro.12 Similarly, a small
Danish company named Atira A/S developed Pure, one of the first full-fledged CRIS
programs—only to be bought by Elsevier, another giant of scientific publishing.13
ORCID, another digital platform, created a single ID code for each scholar,
thereby more accurately archiving scholars’ publications.14 However, as most other
CRIS programs—this important innovation suffers from the underidentification of
publications, mainly in the humanities and the social sciences. Such successful
innovative platforms—and the fact that many companies compete for hegemony in
this market—suggest that information about scientific performance has become an
important asset in the competition for scientific excellence. However, most of those
tools have remained on the limited track of library science. It is time to acknowledge,
then: the administration of academic activities requires a very different outlook on
the information. It, therefore, requires expanding the capacities of current research
information systems.
In what follows, I show how developers need to think of those capacities so as
to help department chairs, deans, rectors, and presidents perform their routine, data-
rich academic tasks.15 By developing additional technical solutions for those needs,
CRIS programs promise to lower operational costs and workload while improving
transparency and efficiency. Their revised capacities promise that researchers and
universities would be able to make better use of time and resources for the main
task they are entrusted with: Advancing science for health, knowledge, and the
improvement of the human condition.
Both sides need to adapt to current practices. CRIS developers need to ask
themselves the following questions: Why have we failed to enter into routine
academic practices of university administration? What do we currently lack in
10 https://clarivate.com/webofsciencegroup/ (extracted December 21, 2019).

11 https://duraspace.org/vivo/ (extracted December 21, 2019).
12 (extracted December 21, 2019).
13 https://www.elsevier.com/solutions/pure (extracted December 21, 2019).
14 https://orcid.org/ (extracted December 21, 2019).
15 Marketization of CRIS program faces another organizational problem. Universities provide
units—departments, research centers, faculty units—with autonomy. Those units create their
own websites. They are at times responsible for integrating digital platforms through their own
discretion and with their internal budgets. Market strategies should allow flexible specifications of
service.
functionality? What revisions do we need to make? On their part, university

administrators need to ask the parallel question—which CRIS programs best answer
for our mainstream academic needs? How can we make sure that we run our
university with the tools that would boost our performance and ranking in the global
race for scientific excellence?16
4.4 Academic Managerial Routines in Need of Reform
In the following four sections, I review major institutional operational requirements.

Those operations demand immense organizational efforts and coordination. How-
ever, to date, most universities keep handling those routine tasks through manual
procedures. I focus on what I deem to be major areas of intervention for CRIS
programs: Hiring procedures, promotion procedures, preparation of annual reports,
and the submission of accreditation and assessment portfolios.
4.4.1 Recruitment of New Faculty
Hiring of new faculty stands at the heart of university decision-making. It involves

in this complex but routine operation selection committees, department chairs, and
deans. Furthermore, in many universities, new nominations require the approval
of rectors and presidents. In some countries, they also necessitate state approval.
During this tedious and tasking annual process, hundreds of applicants flood
universities with CVs, research statements, and recommendation letters. Facing this
avalanche of papers and PDFs, faculty have to select the best candidate out of
the pool. They mostly do so through a burdensome manual process, mostly non-
transparent and at times with little accountability regarding standards of evaluation.
Either way, by the end of that annual process, universities replace between 3% to 5%
of their faculty (as similar numbers of faculty members retire). Evidently, systematic
miscarriage of hiring responsibilities means that over the long haul, the university
and departments may lose budgets and esteem and deteriorate in institutional
rankings. Therefore, department chairs and deans need to make sure that their hiring
procedures are optimal. Rectors and presidents need to be accountable for making
sure that their university hires the best candidates.
In facing this challenging annual task, decision-makers often grapple with
important questions. How good were we in selecting candidates? How accurate were
16 In appraising “who needs to change,” CRIS developers need to appreciate that universities are
heavily regulated by ordinances and collective agreements. Academic decision-making procedures

are resistant. It is therefore the task of information giants to adapt their products to the realities and
regulations of academic life.
124 G. Yair
our nominations across years? Are there units that keep electing candidates of lesser
quality? Were our rival departments or institutes able to recruit faculty with better
long-term performance? Those are important questions. However, current organi-
zational practices—often paper-based—do not allow university governing boards
to oversee hiring practices truly. Furthermore, with hundreds of applicants across
years, departments in competitive markets currently lack systematic information
about their capacity to attract and maintain the best talent. They may have anecdotal
stories, but they are basically clueless.
In answering this challenge, CRIS programs should play critical roles in the
management and assessment of hiring procedures. This is especially true in
areas like physics and the life sciences, where the training of young scientists
continues for five or more years of post-doc. In such fields, candidates have already
published papers and accumulated performance data that allows making decisions
on proximate evidence. In fields that attract candidates immediately after graduate
school (e.g., economics), it is post hoc comparisons that become critical.
Advanced CRIS programs should help departments assess candidates during the
hiring process. They should also help deans and provosts by providing answers for
post hoc assessments. Indeed, with appropriate data models, CRIS programs should
allow departments to create annual groups of candidates for managing the selection
process. They should help search committees in weighing their candidates by
using objective and transparent evidence. For example, they should support faculty
assessments of candidates by allowing them to quantify recommendation letters.
They should also allow faculty to score assessments of applicants’ academic merit
(by evaluating contributions in papers and job talks). CRIS programs should provide
transparent weighted scores that rationalize the process of hiring. They should thus
guarantee rectors and presidents that lower-level decisions are backed by evidence.
Furthermore, to allow post hoc evaluations, CRIS programs should continually
harvest applicants’ publication productivity and their impact. This would allow
departments and higher level administrators to assess hiring accuracy.
4.4.2 Promotion Procedures
Academic career ladders motivate faculty to maintain productivity across the years.
They celebrate excellence and reward effort and impact. To maximize the human
potential of their faculty, some universities created four career stages, others three.
Large universities handle hundreds of promotions per year; deans often oversee
dozens. Either way, each promotion creates abundant workload and data. Each
requires a committee, up to seven recommendation letters, coupled with summation
reports by the committee and department chairs. Each promotion also requires
approval by higher level committees. Such procedures often last a year to conclude,
sometimes longer, oftentimes due to bottlenecks, workload, and forgetfulness. Since
procedures are secretive and often employ opaque standards, many people mistrust
the system of promotions. Lack of clear criteria often means that non-academic
justifications may contaminate the process and taint career ladders’ meritocratic
rationale. Indeed, political and social biases at times position underachieving
faculty in higher professorial positions than more prolific professors, raising public
concerns over transparency and fairness regarding university promotions.17
Those problems are exacerbated by highly diverse disciplinary cultures (Becher
& Trowler, 2001). Decision-makers often grapple with comparisons between
physicists—who publish in large groups and with immense publication counts—and
humanists, who mostly opt to publish books as single authors (Elisabeth S. Clemens
et al., 1995). The problem of disciplinary differences at times appears even within
faculty units. Anthropology, for example, opts for publication standards along the
tradition of the humanities, whereas psychologists take to the norms of the life
sciences. But even within psychology—committees are challenged in comparing
neuroscientists with social psychologists. Alas, those disciplinary differences allow
faculty to make ad-hoc arbitrary decisions about promotions. Department chairs
often inflate promotion files with the rhetoric of excellence with little supporting
evidence. Higher level committees rarely consult historical data, even in aggregated
forms—thereby increasing doubts about the fairness and equal conditions. Alas,
few universities manage those burdensome procedures with robust evidence about
the timeliness, fairness, and meritocracy of promotions.
CRIS programs can play a significant role in correcting for those persistent
organizational challenges. Essentially, promotion files can be likened to submitted
papers for journals. For example, systems like Editorial Manager by Aries manage
the entire workflow from the submission of papers up to publication. It accepts
submitted manuscripts, helps editors elect reviewers, and collects revision decisions
with follow-up procedures until a final decision—all in one digital environment.
CRIS programs should adopt similar features for the handling of promotion
procedures. They should allow candidates for promotion to upload their CV,
academic statement, and sample papers. The program would help committee or
department chairs in selecting appropriate recommenders while verifying that they
are independent and in a good position to provide references. Similar to other
document tracking systems, it should deliver all files, manage reminders, and
provide analytics about the timing of all elements during the procedure (e.g., time
from initiation to end, reviewers’ timeliness, and committee deliberation efficiency).
Furthermore, CRIS programs like VIVO or PURE should use their strengths
in providing promotion committees with automatic yet quantitative assessment
of CVs—e.g., number of publications, citations, H-Index, journal impact factor
figures, with all metrics standardized for the field. They should present collaboration
networks and information about grants awarded. They should integrate information
about teaching and student evaluations—thereby providing a rounded, multiple
assessments of excellence (Boyer, 1990). They should also provide clear evidence
17 In 2012 I conducted an Israeli faculty survey with more than 1400 academics replying. 47% said
the promotion criteria are opaque; 33% stated that promotion criteria are not applied equally; 34%
replied that promotion procedures are unfair. These are troubling statistics.
126 G. Yair
about the evolution of the career from prior promotion stages (delta figures). Uni-
versities should be able to upload historical files to help committees contextualize
their decisions. With appropriate data design, those upended CRIS programs would,
in time, create comparative benchmarks and allow all concerned to have a more
efficient and trustable process of maintaining faculty motivation.
4.4.3 Preparing Annual Reports
Universities are often required to provide annual performance reports. Governing

boards, state comptrollers, funding agencies, and other overseeing agencies request
universities to provide detailed information about teaching, research, and public
service. In some universities, academic units are also expected to supply annual
reports about their performance to the provost. And some states require individual
faculty members to submit their own annual reports. In facing those requirements,
universities often engage in labor-intensive efforts of data collection and editing of
information. Deans and department chairs spend administrative hours on reformat-
ting reports arriving from different sources. Similarly, individual faculty members
spend dear time in organizing their CV, writing their annual report while specifying
objectives for the following year. It is clear that those activities derail universities
and faculty from their main tasks, namely engaging in research and teaching.
CRIS programs should make most of this organizational hassle go away. By inte-
grating different institutional databases (e.g., teaching assignments, grants, research
outputs), they should provide automatic reports across various areas of faculty
research, teaching, and service. They should supply universities with templates
for reports starting from the individual faculty level, through departmental reports,
up to aggregated reports at the university level. Indeed, comprehensive CRIS
programs would eventually digitize the entire campus, allowing administrators to
produce quick and updated reports with little human intervention. Furthermore,
current technologies allow creating PowerPoint presentations and Word or PDF
documents with pre-populated elements that simply update with new data. Such
elements include text that describes changes across time. For example, using ever-
updating data, such automatic reports can describe improvement, stability, or point
to mounting challenges. Such report templates should allow administrators to
enter free text for further explanation. But rather than having administrative and
professional employees toiling over long and repetitive annual reports, the entire
process should be computerized. Click and submit.
This is a good example of where CRIS developers need to move out of their
comfort zone in library databases. The challenge they face is integrating campus life
across different systems. As much as that may be challenging, campus life in most
universities is already in digital format—but scattered. Courses are computerized.
Faculty members are assigned to specific courses, with data about course type
and level (e.g., undergraduate), number of students, possibly with course grades
and grade distribution. Similarly, universities maintain information about research
grants in digital format. They record which faculty members hold grants, the sum
of grants, the source, the length of time, etc. CRIS programs are but steps away
from integrating those currently operating databases. There are protocols for data
transfer; there are local computer administrators that can share responsibility for
making data availability.
Having such integrated capacities would critically ease the production of annual
reports. However, they are also likely to generate new capacities for academic
management. Advanced integrative CRIS programs would actually allow university
administrators to ask new questions while creating new targets for improvement.
For example, integration between grant and publication databases (as Academic
Analytics partly enjoys today) would allow universities to ask new questions and
create new efforts at reform. Here are some examples from the economics of science
(Diamond, 1996; Sent, 1999; Stephan, 2012): “How many papers do we publish
per $100k?” “Are there disciplinary differences in such efficiency quotients?” “Can
we set benchmarks for improving the economic efficiency of our university?”
Similarly, integration between library databases (e.g., ProQuest databases) with
teaching databases would possibly transform discussions about the scholarship of
teaching (Shulman, 2004). For example, administrators would be able to ask: “How
updated are our syllabi?” “Which new items can substitute outdated ones?” “How
many item overlaps do students encounter across our program?” Such questions
truly get to the essence of university life, possibly reinvigorating current academic
discussions about research and teaching. They would encourage faculty members to
stay at the front of their discipline; they would thus guarantee that students are led
towards those frontiers too.
4.4.4 Preparing Accreditation and Assessment Portfolios
Some disciplines require schools or departments to undergo external accreditation

procedures (e.g., business administration). Some governments require universities
or specific departments to engage in occasional internal or external assessments.
Those requirements are costly and time-consuming. They necessitate department
chairs to collect data from various sources. They require communication across
administrative units and coordination with external regulations. If there is one task
that administrators truly wish to be simplified, this is it. I know because I served as
a chair when my department underwent national assessment; I know because I later
assisted five more departments in other disciplines to submit their reports; I also
know because I served in accreditation committees for the past two decades.
Familiarity with academic management platforms assures me that CRIS pro-
grams have a lot to offer on that front. They have faculty-level information on
publications, citations, and the h-index. They probably hold information about
the educational background of department members, possibly with the global
rankings of graduating institutes and the majors or disciplines faculty members
heralded from. They supply information about collaboration networks and about the
128 G. Yair
internationalization of those networks. Some also provide Altmetrics in the form

of shares across multiple academic and social networking platforms. Some tools
integrate funding data with productivity and impact measures, allowing to present
assessors with the smart evaluation of performance. In places where department
chairs already enjoy such comprehensive systems, the complex and tedious tasks
involved in the preparation of comprehensive and smart reports for accreditation
and assessment committees are significantly eased.
As suggested in the previous section, the integration of CRIS systems with other
campus databases would immensely improve available specifications while saving
departments and universities significant resources. With improved specifications,
the preparation of reports would require much less time. Data integration across
campus would also minimize cross-unit bottlenecks in the coordination and transfer
of required documents. Obviously, integrative CRIS programs would save costs.
More crucially, however, they would help to generate new insights about institu-
tional strengths and challenges.
For example, accreditation agencies often require a clear description of programs
of study. How many professors are involved in teaching? How many are full-
time faculty members, how many are employed as adjuncts? What are their
academic degrees? How equal are teaching assignments across the program? What
is the distribution of students’ grades? Are there significant trends across the
past five years in those indicators? With smart reporting capacities, answers to
those questions should be easy. Similarly, integration with employment data would
allow department chairs to provide evidence concerning faculty turnaround while
specifying clear horizons concerning future retirements and the hiring of young or
senior scholars. As stated above, most of those reporting tasks can be automated.
CRIS programs should be able to generate graphs, spreadsheets, and PowerPoint
presentations with a click of a button; they should also allow institutes to specify
and generate their own reports without reliance on re-programming by external
developers.
If CRIS programs do not provide those features yet—they should. If developers
have not yet dwelled on cross-campus integration—they should. And if universities
have not put forth such requirements—well, they obviously should.
4.5 Conclusions
Thirty years ago, I served as a research director in one of the units at the
Municipality of Jerusalem. At the time, the administration adopted a new telephony
system—one that could dial back, memorize numbers, and forward calls to other
employees. During the demonstration, the general director of the municipality
declared that “The problem is that the phones are now smarter than the employ-
ees.” My observations of assimilation and implementation of CRIS programs in
universities at times merit similar conclusions. During the past decade, I observed
strong rejections of CRIS programs by universities—largely because they deliver
on the promise of accountability and transparency. In many universities, faculty

and leadership join hands in objecting to those programs, fearing that transparency
would eventuate in greater public pressures. Furthermore, the targeting of marketing
efforts towards librarians and research directors—rather than to presidents, provosts,
rectors, deans, and department chairs—meant that the latter were able to maintain
institutional inertia. One should reckon, though, that librarians have a minor role
in institutional decision-making. Similarly, research directors are mostly occupied
with questions of funding and grant management. Though those two offices are
crucial and merit attention, they are not positioned at the central “command line”
of most university organizations. As a result of this misalignment, most decision-
making processes in higher education today have remained unaffected by innovative
and integrative CRIS programs. In many cases, implementation is localized—being
integrated into one or few units, allowing inertia to continue. As Peter Maglione was
quoted above, the result is that in many universities and research centers, decision-
making is still undertaken by “flying blind, relying on intuition, “gut feel,” and
unreproducible methods.”
This, I surmise, is the main challenge for both sides. Administrators are
responsible for changing institutional practices so as to base their decision-making
on valid, reliable, transparent, and updated data. Developers of CRIS programs
should better align their marketing and technical specifications with concrete routine
institutional operations. Adopting technological innovations is important beyond
market signaling; they should be followed by consulting universities on how to
renovate their organizational decision-making routines. One needs to better care
for the connection between administration and technology. It is, indeed, a challenge
that the public demands university leaders to rise to.
References
Becher, T., & Trowler, P. R. (2001). Academic tribes and territories: Intellectual enquiry and the
cultures of disciplines. Open University Press/SRHE.
Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. Carnegie Foundation
for the Advancement of Teaching.
Clemens, E. S., Powell, W. W., McIlwaine, K., & Okamoto, D. (1995). Careers in print: Books,
journals, and scholarly reputations. American Journal of Sociology, 101(2), 433–494.
Diamond, A. M. (1996). The economics of science. Knowledge and Policy, 9(2–3), 6–49. https://
doi.org/10.1007/BF02696298
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen,
A. M., et al. (2018). Science of science. Science, 359(6379). https://doi.org/10.1126/
science.aao0185
Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.
https://doi.org/10.1126/science.178.4060.471
Garfield, E. (1970). Citation indexing for studying science. Science, 227, 669–671.
Garfield, E. (2006). The history and meaning of the journal impact factor. The Journal of the
American Medical Association, 295(1), 90–93. https://doi.org/10.1001/jama.295.1.90
130 G. Yair
Garfield, E. (2009). From the science of science to Scientometrics visualizing the history of
science with HistCite software. Journal of Informetrics, 3(3), 173–179. https://doi.org/10.1016/
j.joi.2009.03.009
Gates, A. J., Ke, Q., Varol, O., & Barabási, A.-L. (2019). Nature’s reach: Narrow work has broad
impact. Nature, 575(7781), 32–34. https://doi.org/10.1038/d41586-019-03308-7
Gates, B. (1995). The road ahead. Longman.
Hirsch, J. E. (2005). An index to quantify an Individual’s scientific research output. Proceedings
of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.
Lotka, A. (1929). The frequency distribution of scientific productivity. Journal of Washington
Academy of Sciences, 16, 317–323.
Mistele, T., Price, T., & Hossenfelder, S. (2019). Predicting authors’ citation counts and H-indices
with a neural network. Scientometrics, 120(1), 87–104. https://doi.org/10.1007/s11192-019-
03110-2
Praus, P. (2019). High-ranked citations percentage as an indicator of publications quality. Sciento-
metrics, 120(1), 319–329. https://doi.org/10.1007/s11192-019-03128-6
Price, d. S. D. J. (1963). Little science, big science. Columbia University Press.
Sent, E.-M. (1999). Economics of science: Survey and suggestions. Journal of Economic Method-
ology, 6(1), 95–124. https://doi.org/10.1080/13501789900000005
Shulman, L. S. (2004). Visions of the possible: Models for campus support for the scholarship of
teaching. In W. E. Becker & M. L. Andrews (Eds.), Scholarship of teaching and learning in
higher education: Contributions of research universities (pp. 9–24). Indiana University Press.
Sinatra, R., Wang, D., Deville, P., Song, C., & Barabási, A.-L. (2016). Quantifying the evolution
of individual scientific impact. Science, 354(6312). https://doi.org/10.1126/science.aaf5239
Stephan, P. (2012). How economics shapes science. Harvard University Press.
Yair, G. (2019b). Culture Counts More Than Money: Israeli Critiques of German Science. Social
Studies of Science, September. https://doi.org/10.1177/0306312719878788
Yair, G. (2019a). Hierarchy versus symmetry in German and Israeli science (pp. 1–32). American
Journal of Cultural Sociology. https://doi.org/10.1057/s41290-019-00069-8
Yair, G., Gueta, N., & Davidovitch, N. (2017). The law of limited excellence: Publication
productivity of Israel prize laureates in the life and exact sciences. Scientometrics, 113(1),
299–311. https://doi.org/10.1007/s11192-017-2465-0
Chapter 5
Forecasting Methods in Higher
Education: An Overview
Zilla Sinuany-Stern
Abstract Forecasting is the first, crucial stage of planning in any organization,

and in higher education (HE) in particular. Student enrollment projections are
particularly important, since they affect institutions’ income, the number of faculty
needed, facility requirements, budgets, etc. There are overviews of forecasting and
classifications in general and for particular methods and applications. However, to
the best of our knowledge, the last overview of forecasting in HE was published in
1997. Since then, two major approaches sipped from business to HE and became
dominant in HE forecasting: data mining and questionnaires for marketing. The
purpose of this chapter is to provide an updated overview of forecasting methods
used in HE and their main areas of application. We cover a large array of forecasting
methods and areas of HE application, we classify them, and point at examples from
the literature, rather than providing an exhaustive annotated review, since there are
too many publications in the literature on forecasting in HE. Counting the number of
articles published in the Web of Science during the last 20 years, we find that, out of
six main forecasting methods identified and classified, four methods are used most
often in HE: regression, simulation, data mining (including its sub-methods), and
questionnaires. Furthermore, four areas of application for forecasting are used most
often in HE: enrollment, marketing, teaching, and performance. The two relatively
new forecasting methods used in HE, during the last 20 years, are data mining
and questionnaires. These two, relatively new forecasting methods, educational
data mining and questionnaires (for marketing), are classified in this chapter as
active forecasting methods in HE, as they provide the administrator with control
over the forecast by pointing (directly or indirectly) at actions which can achieve
a better-targeted forecast. While the old methods, time series, and ratio methods,
are classified as passive methods with no control. Though regression and simulation
forecasting methods are often active, they can sometimes be passive.
Z. Sinuany-Stern ()
Beer Sheva, Israel

132 Z. Sinuany-Stern
Keywords Forecasting · Higher education · Prediction · Planning · Active

forecast · Data mining · Marketing · Regression · Simulation · Time series
5.1 Introduction
In general, forecasting activity levels is the first stage of planning in any organization
or business. In higher education (HE) in particular, many activities are determined
by student enrollments at various levels (e.g., department, university, state, and
international) and in various categories (e.g., by year of study, state and out-of-state
students, and by gender). Enrollment influences planning and budgeting, and more
specifically income from tuition and from the state, amounts of various types of
manpower needed, space needs, various costs, etc. Each of these parameters needs
to be forecasted too, based on the enrollment projections.
In the general literature on forecasting Makridakis et al. (1983, 1998), in
their classic book on forecasting methods and applications, provided rigorous
mathematical formulations of many forecasting methods. They classified forecast-
ing methods into four main categories: smoothing and decomposition, regression
and econometrics, Box-Jenkins (ARIMA), and qualitative methods (similarly also
Hyndman and Athanasopoulos, 2018). Armstrong and Green (2018) classified fore-
casting methods simply into two main groups: judgmental methods and quantitative
methods (detailed classification in Sect. 5.3.2).
In the context of HE, Schmid and Shanley (1952) wrote about techniques for
forecasting university enrollment in 1952, mentioning only two methods—ratio and
cohort-survival—which were tested empirically on the University of Washington.
They concluded that the ratio method is simpler but is not as accurate as of the
cohort-survival method.
Brinkman and McIntyre (1997), in their comprehensive review on forecasting in
HE, listed forecasting methods used for predicting enrollment and revenues in HE.
They considered a wider range of forecasting methods than Schmid and Shanley
(1952), but not as wide as Makridakis et al. (1983), dividing them into three main
categories: quantitative methods, qualitative methods, and combined approaches.
Their literature review was based on applications of the forecasting methods above
in HE, covering articles during 1980–1995 (ibid.). Almost all the above-listed
methods were taken from Makridakis et al. (1983) except ratio and cohort-survival
which are specific to HE.
The purpose of this chapter is to provide an overview of the main forecasting
methods in HE and the main areas of application. Examples are given, rather than
an exhaustive annotated review, since the literature on forecasting in HE is very
large, with thousands of articles over the last 20 years alone.
The contribution of this chapter lies first in two relatively new forecasting
approaches, which became leading in forecasting in HE, not covered by Brinkman
and McIntyre (1997): data mining for retention of students (as reviewed by Romero
& Ventura, 2020) and questionnaires mainly for marketing (Oplatka & Hemsley-
5 Forecasting Methods in Higher Education: An Overview 133
Brown, 2021). Secondly, we suggest a new approach where the various forecasting
methods are classified into two main groups: active forecasting methods versus
passive forecasting methods. Thirdly, by counting the number of articles according
to forecasting method and areas of application in HE, along with examples from the
literature, we get an overview of the applicability of various forecasting methods for
HE planning and control.
The paper is organized as follows. Section 5.2 presents the forecasting methods,
with examples showing their areas of application in HE, as summarized by
Appendix 1. Section 5.3 presents types of classifications for forecasting methods in
HE and various issues in HE forecasting. Section 5.4 provides a count of the number
of papers published on each forecasting method in HE, from the total number of
forecasting papers in general for each method, and for each area of application.
Section 5.5 concludes the chapter.
5.2 Review of Forecasting Methods: Passive Versus Active

Forecasting
In presenting the main forecasting methods, we divide them into six main groups,
considering old methods (Brinkman & McIntyre, 1997; Makridakis et al., 1998) and
newer methods of data mining (Romero & Ventura, 2007, 2020) and questionnaires
for marketing (Oplatka & Hemsley-Brown, 2021); which were used more and more,
during the last two decades, for forecasting in HE, as follows:
1. Time Series, including moving averages, exponential smoothing, fuzzy time
series, and autoregressive methods (ARIMA).
2. Ratio forecasting methods, including the cohort-survival method and Markov
analysis.
3. Regression analysis, which is very popular in forecasting, and is also included
in data mining as a subsystem.
4. Simulation, including system dynamics and other simulation approaches, which
are popular in forecasting and planning in HE and other fields.
5. Data Mining (DM), which is often used for predicting student retention, even for
individual students. Some DM subsystems, in addition to regression, are decision
trees, random forests, neural networks, genetic programming, and grey models.
6. Questionnaires and surveys are often used for marketing.
The first two methods can be classified as passive forecasting methods, while
the last four methods can be classified as active forecasting methods (see more
explanation in Sect. 5.3).
The following is an explanation of each of the six forecasting methods above,
with their sub-methods. For each of the forecasting methods listed, some examples
are given from the literature in the next subsection. Detailed mathematical formula-
tion is avoided to appeal to a wider audience of HE administrators.
5.2.1 Time Series
In time series, the forecast, Ft + 1 , of Xt + 1 , is a function of time series past values,

X1 , . . . , Xt . There are many time series methods; the three main methods are
discussed here (see Chen et al., 2019). Here are the four main sub-methods of time
series:
(a) Moving average techniques of order N: the forecast is calculated t as an average
of the actual observations over N prior periods Ft +1 = j =t −N+1 Xt /N.
As time progresses, each new forecast drops the observation from the oldest
period and replaces it with the actual observation in the most recent period.
For example, a moving average of order 10 considers the average of the last 10
periods. Pamungkas and Rofiqoh (2015) used a moving average of order 3 to
forecast the number of students in a university (Batam University in Indonesia)
during 2016–2020. As shown in Appendix 1, Lavilles and Arcilla (2012) also
used a moving average.
(b) Exponential smoothing forecast Ft + 1 is a combination of the last forecast, Ft ,
and the last observation, Xt , so that Ft + 1 = aXt + (1 − a)Ft , where 0 < a < 1
is the weight given to the last observation. For example, Anggrainingsih et al.
(2015) used exponential smoothing to predict the number of website visitors
for a university in Indonesia. As shown in Appendix 1, others have also used
exponential smoothing in HE, for example, Ge et al. (2013), Lavilles and Arcilla
(2012), and Tang and Yin (2012).
(c) Fuzzy time series is an interval prediction for time series rather than a point.
For example, Aladag et al. (2010) used a high-order fuzzy time series method
in which fuzzy relationships are defined by feed-forward neural networks, and
an adaptive expectation model is used for adjusting the forecasted values. The
method is applied to enrollments at the University of Alabama. Ge et al. (2013)
suggested integrating fuzzy time series with exponential smoothing. They also
applied their method on enrollment data from the University of Alabama and
showed that the method has a better performance compared to the other five
forecasting methods, based on the mean-squared error. As shown in Appendix
1, Hwang et al. (1998) also used fuzzy time series for HE.
(d) ARIMA—Auto Regressive Integrated Moving Average methods are based
on regression of the forecasted observation with its past observations (auto
regression) Ft + 1 = F(X1 , . . . , Xt , E1 , . . . , Et ),where Et is the error of forecast
Ft . For example, Chen (2008) used ARIMA to forecast student enrollment in
Oklahoma State University. Enrollment data was given by term (spring and fall)
and by type of student entering (freshman, transfer, readmission, and continuer).
Sometimes, the set of first-order differences is used to obtain a linear functional
relationship (when there is nonlinear behavior of the data).
5.2.2 Ratio Methods
Ratio methods use to be the main methods for enrollment projection, due to its
simplicity and intuitive meaning. For example, historical data are used to calculate
a time series of ratios between the total population of some relevant age group and
the number of students in that age group (Sinuany-Stern, 1980). Another example
is from Agboola and Adeyemi (2013), who projected university enrollment in
Nigerian universities assuming 3.5% annual growth and projected the need for
academic staff based on past faculty/student ratios by discipline. As shown in
Appendix 1, others have used ratio models: Schmid and Shanley (1952), Sinuany-
Stern (1984), and Allen (2013). Here are the two main additional sub-methods of
the ratio method:
(a) Cohort-survival method is another widely used enrollment projection method-
ology, also referred to as the “grade progression ratio method.” The approach
develops retention ratios, which are associated with successive levels of aca-
demic attainment (grades). Given the total enrollment at the lowest grade
for each of several years, cohort of students in the lowest grade is “aged”
through the subsequent grades; the result being, in the simplest case, the
distribution of enrollment by grade for any future period. For example, if
60% of the graduating secondary school students in a specific state historically
enroll in state HEIs (HE Institution), then predicting the number of “in-state”
enrollments in the next year is simply Ft + 1 = 0.6Zt where Zt is the number of
“in-state” secondary students graduating, and Ft + 1 is the projected number of
“in-state” students entering state HEIs.
Another example is from Burkett (1985), who used a cohort-survival model
to forecast freshman enrollment at the University of Mississippi during 1985–
1989, based on historical data of grade 7–12 secondary school students, high
school graduates in the state of Mississippi, the University of Mississippi
freshman enrollment, and the number of out-of-state freshmen. The historical
survival ratios were calculated based on the historical data to forecast freshman
enrollment. As detailed in Appendix 1, others who used cohort-survival meth-
ods in the context of HE include Schmid and Shanley (1952), Murtaugh et al.
(1999), Min et al. (2011), Trusheim and Rylee (2011), Tsevi (2018), and Wade
(2019).
(b) Markov analysis has the advantage of being relatively straightforward to set
up and does not require extensive use of computers to find output parameters
like steady-state probabilities. A Markov system is based on a system with K
possible states and transition probabilities for elements moving from one state
to another during one period, where Pij is the probability of moving from state
i to state j.
Bessent and Bessent (1980) used Markov analysis on the progression of
doctoral students, from year to year, through their degree program, based on
past years’ average percentages. Kwak et al. (1986) extend the approach to
a trimester-based institution by modeling each trimester separately. They cap-
tured graduation rates and the number of enrolled students relatively accurately
for the department studied. Later, Shah and Burke (1999) presented a Markov
analysis of undergraduate students as they progress towards their degree. Their
model takes into consideration the course of study and the gender and age of the
student when starting the degree program. They were interested in predicting
the average time to degree completion. As shown in Appendix 1, others also
used Markov analysis, including Hwang et al. (1998), Nicholls (2009), Allen
(2013), Witteveen and Attewell (2017), and Sass et al. (2018).
5.2.3 Regression Analysis
A regression model is a causal model that detects relationships between the variable
to be forecasted (dependent variable) and another given explanatory variable.
Multiple regression deals with multiple explanatory variables. Nonlinear regression
and econometric models (several simultaneous equations where the “dependent”
variable of one equation may become an explanatory variable in another equation)
are hardly used in HE forecasting—they are used more by macro-economists.
Regression analysis is widely used by researchers in HE, but mostly not for
forecasting purposes—often regression is used for pedagogical research. Regression
is also used as sub-method in data mining (see Sect. 5.2.5). Regression modeling is
one of the most popular forecasting methods in HE and in general (see Sect. 5.4.1).
Here are the two main regression models used in HE:
(a) Simple regression model mostly assumes a linear relationship of the forecasted
variable (e.g., enrollment) with another explanatory variable, for example,
Ft + 1 = a + b(t + 1) where the time t + 1 is the explanatory variable and
Ft + 1 is the forecast in period t + 1.
(b) Multiple regression in HE, forecasting is often linear: the forecasted variable,
Ft + 1 , has several explanatory variables, Ft +1 = a + tj =1 bj Xj For example,
Sinuany-Stern (1984) used a student questionnaire to verify explanatory vari-
ables for projecting the numbers of students in each of the three campuses of a
large community college in Cuyahoga Community College (Ohio), utilizing a
multiple linear regression model for the change in enrollments from 1 year to the
next 2 years. The predictors of the regression model were: the unemployment
rate in the county, the dollar amount allocated to financial aid, the opening of
a new facility on the campus, and the number of high school graduates from
the campus’s draw area. The predictors were based on the market research
questionnaire (although at that time, the administrators of the college refused
to use the term marketing!). In a second stage, the past breakdown to sub-
categories, as needed for the state budget funding formula, was predicted via
the ratio model; the ratio was predicted by utilizing the average past breakdown
percentages, using moving average.
As shown in Appendix 1, others also used multiple regression analysis in
HE, including Baisuck and Wallace (1970), Sinuany-Stern (1980), Sinuany-Stern
(1984), Sinuany-Stern and Yelin (1993), Murtaugh et al. (1999), Amber et al.
(2015), Sweeney et al. (2016), Chou (2019), and Wade (2019).
5.2.4 Simulation Models
Simulation is one of the most popular methods used in forecasting (as shown in Sect.
5.4.1); it is a model which imitates a system or a process behavior to learn about
the system over time, sometimes to improve it. It allows one to conduct virtual
experiments or to test behavior under various scenarios. Simulation models can
consist of mathematical equations or series of relations between various components
and are mostly computerized. The simulation is a proxy of the real system—it
cannot capture the full capacity of the real system due to its complexity and the lack
of information available about it. In the context of HE, simulation is used mostly for
planning and forecasting—pedagogical simulators and games are not considered
here at all. As shown in Appendix 2 (items 15–18), the main three simulation sub-
methods are:
(a) System dynamics (SD) is a deterministic continuous model utilizing differential
equations to describing the interaction among the subsystems.
(b) Discrete event simulation (DES) is a stochastic model advancing time by events
or by discrete-time interval follow-up of the system state and its stochastic
movement over time.
(c) Agent-based simulation (ABS) is an extension of DES for several
processes/agents with various behaviors which interact with each other.
In 1970, Baisuck and Wallace were already using a simulation approach for
enrollment projection in various types of HEIs: private universities, public uni-
versities, community colleges, and statewide planning activity. The model takes
sub-groups of students and their movements within the HEI over time, using a
regression model which estimates the changes in each group as a function of
some other planning variables. Xiao and Chankong (2017) used SD simulation for
predicting the supply and demand of medical education talents in the Jiangsu region
of China. They predict the number of doctors needed to maintain the average levels
of 34 OECD countries by the year 2024. The forecast allows HE authorities to make
adjustments to meet the additional demand for Medical Doctors needed. They claim
that the SD model is applicable to other regions in China (ibid.). SD encompasses
the complexity of the system, while other methods focus on parts of the problem.
As shown in Appendix 1, Saltzman and Roeder (2012) also used simulation in HE.
5.2.5 Data Mining (DM)
Since the late 1990s, DM has been used more and more as a forecasting method in
HE (see Sect. 5.4.1). DM combines computer science, statistics, and analytics. DM
uses various old and new forecasting and classification algorithms, amongst which
are linear modeling, regression, correlation, decision trees, random forests, den-
sity estimation, probabilistic graphical models, neural networks, nearest-neighbor,
variance analysis, text mining, principal components analysis (PCA), time mining,
association rule, hierarchical clustering, density-based methods, and others which
are being added over time. DM automatically tests a variety of methods on a big
database and selects the method which fits best. There are several DM toolkits,
such as SPSS Clementine (https://softadvice.informer.com/Spss_Clementine_12.0_
Free_Download.html assessed on Nov. 2020) that specialize in DM. Main DM steps
are data preparation, visualization, prediction (e.g., regression, neural network, time
series) or classification/clustering (e.g., random forest, PCA), and selecting the best
model. Data source in HE can be from institutional database, web-based (e.g., from
online course), and questionnaire.
During the last 20 years, DM has been used very often, mainly to improve
teaching practices for both the learning process and the learners (to improve
student retention), or to identify what type of courses or other managerial actions
may decrease students’ chances of dropping out of a course or the college, thus
increasing internal enrollments (by retention), rather than passively trying to predict
enrollments as dictated by internal or external forces to be revealed via traditional
forecasting methods. As shown by educational data mining (EDM), based on the
HEI database of student achievement by course performance over the semester, the
retention of each student in a course, or in a given semester can be predicted—as
big data analysis improves over the years—see literature review on Educational Data
Mining (EDM) by Romero and Ventura (2007), covering 304 articles up to 2005.
They pointed that data mining techniques can be classified as follows: statistics
and visualization; clustering, classification, and outlier detection; association rule
mining and pattern mining; and text mining. The Journal of Educational Data
Mining started in 2009—its first article was a review of the literature on EDM by
Baker and Yacef (2009)—noting the increased emphasis of EDM on prediction, they
covered 45 most cited articles published during 1995–2009. In the context of EDM,
Siemens (2013) introduced the term learning analytics as data analytics in education
“ . . . to increase the scope of data captured so that the complexity of the learning
process can be more accurately reflected in analysis.” He points at the issue of
privacy and data ownership, and ethics in EDM in general and learning analytics
in particular (ibid.). Additional literature reviews have been published on EDM:
Al-Razgan et al. (2014), who covered over 300 EDM articles published during
2006–2013; and Shahiri et al. (2015), who reviewed 39 articles, about predicting
student performance using DM techniques during 2002–2015. Recently, Romero
and Ventura (2020) extended their past review to include a wide range of new
terms used during the last ten years in EDM and learning analytics: academic
analytics, institutional analytics, teaching analytics, data-driven education, data-

driven decision-making in education, Big Data in education, and educational data
science. Hellas et al. (2018) reviewed 357 articles on EDM, highlighting the use of
DM for predicting the performance of students in HE. There are more reviews of
EDM. Here are few examples on the use of DM in HE in Appendix 1 by Jantan et
al. (2010), Miguéis et al. (2018), and Sweeney et al. (2016).
Here are some details on several DM sub-methods:
(a) Decision tree is a structure that aids decision-makers in presenting options with
their consequences and probabilities. In data mining, there are classification
trees for discrete sets of values and regression trees with continuous variables.
Hamsa et al. (2016) used a decision tree in a prediction model of student
academic performance.
(b) Random forests are an ensemble learning method for classification. They
utilize regression and other tasks that operate by constructing a multitude
of decision trees at training time and outputting the class that is the mode
of the classes (classification) or mean prediction (regression) of the individ-
ual trees. Hasan et al. (2020) used analytics and DM techniques to predict
student performance in HEIs using video learning (dichotomy: success = 1,
failure = 0). The sample included 772 students who registered for e-commerce
courses on video. Data was retrieved from the student information system,
learning management system, and mobile applications. DM was used to explore
eight models and to reduce the number of parameters, transformations were
used, including PCA. The results showed that the random forest accurately
predicted successful students at the end of the class. Sweeney et al. (2016) used
factorization machines, a random forest, and personalized linear regression to
predict students’ grades in the next semester and improve student performance
and retention in the USA. Moon et al. (2018) used time series analysis and a
random forest to forecast university buildings’ electrical loads in Korea.
(c) Neural networks are nonlinear statistical data modeling or decision-making
tools. They can be used to model complex relationships between inputs and
outputs or to find patterns in data. For example, Aladag et al. (2010) used neural
networks. As described above, the method was applied to enrollments at the
University of Alabama.
(d) Genetic programming is a heuristic technique to search for an optimal program
among the space of all programs. For example, Amber et al. (2015) used genetic
programming and multiple linear regression models to forecast electricity
consumption in a UK university. The genetic programming model performed
better than the multiple regression model.
(e) Grey model combines the theoretical structure of the forecasting model with
data to complete the model, for example, combining linear regression with a
neural network. Tang and Yin (2012) used grey predictions and exponential
smoothing for education expenditure and school enrollment, using univariate
time series data from the Chinese National Center for Education Statistics.
Based on several methods for fitting the forecast, such as mean-squared error,
they concluded that the two grey models they used outperformed exponential
smoothing. They concluded that their results provide a basis for further research
in model-building for the short-term prediction of student enrollment and
education expenditure. Along these lines, Ge and Xie (2015) applied a grey
forecasting model based on improved residual correction to university education
cost prediction in China.
5.2.6 Questionnaires and Surveys
Questionnaires and surveys are used as an aid to forecasting with one of the
above-mentioned methods, often with regression or DM. The questionnaire is
the set of questions distributed to a certain group; in HE, it might be students,
faculty, staff, etc. A survey is not necessarily a questionnaire; it can be data from
internal or external sources, often combined with some analysis. In this category,
qualitative methods are imbedded such as experts, Delphi (a specific procedure
for questionnaire with feedback). Questionnaires were very popular in HE and the
social sciences in general, in which researchers make use of questionnaires, not
necessarily for forecasting but to understand people’s behavior or the success of
various teaching methods, etc.
In the past, administrators tended to use surveys—such as collecting comparable
data from other HEIs—rather than questionnaires. For example, Edwards performed
a survey in 1932, collecting data on college enrollment from 96 US Colleges over
the period 1890–1930, during times of economic depression, to verify whether
there was evidence of an enrollment increase because of the depression. Sinuany-
Stern (1976) used questionnaire results to verify reasons for the 20% growth in
student enrollments at a community college in Ohio. Based on the results of the
questionnaires, a multiple regression model was utilized for predicting enrollments
for that college. Banerjee and Igbaria (1993) used questionnaires to plan computer
capacity in 170 US universities.
Since the mid-1980s, marketing became more and more acceptable in HE.
Questionnaires and surveys are often used for HE marketing, as shown in the
literature review of marketing in HE by Hemsley-Brown and Oplatka (2006).
Marketing has become a reality in HE over the past 25 years, despite the ethical
arguments surrounding marketization—as presented in the latest review of the
literature by Oplatka and Hemsley-Brown (2021). The questionnaire/survey may
reveal what motivates students to enroll (or alumnae to donate), pointing towards
actions and policies that management can take to increase enrollment, such as
advertisements and public relations in various media.
In the context of forecasting enrollments, the marketing approach is designed to
allow control at the enrollment level by taking actions such as advertising to increase
enrollments, or to achieve enrollment targets set by administrators.
5.3 Types of Classifications of Forecasting Methods in HE
In the context of enrollment forecasting, both DM methods and questionnaires for

market research represent active forecasting methods, where the main purpose of
the administrators is to maximize enrollments by performing some actions based
on these methods, rather than providing a passive forecast, as was the case until
the 1990s. While DM mostly utilizes internal databases from the institution, market
research often creates new information based on questionnaires or surveys. Barthwal
(2000) also used the terms active and passive forecasts (pp. 112–113), but more in
the sense of an unchanged environment in the passive forecast versus the dynamic
environment in the case of an active forecast.
We introduce the terms active and passive forecasting methods in a slightly
different sense. We describe forecasting as active when it provides a control for
the administrator by pointing at some actions for influencing future events, while a
passive forecast assumes no such control. The two leading forecasting approaches,
DM and marketing are active forecasting methods, pointing the administrators at
actions to be taken to achieve a targeted forecast. Regression analysis and simulation
are not always active forecasting methods. However, time series and ratio methods
are passive forecasting methods.
5.3.1 New Classification: Active Versus Passive Forecasting

Methods
As explained above, we suggest to classify the six main groups of forecasting

methods in HE (Sect. 5.2) into two groups:
(a) Passive forecasting methods: (1) Time series (2) Ratio methods
(b) Active or passive forecasting methods: (3) Regression (4) Simulation
(c) Active forecasting methods: (5) Data mining (6) Questionnaires and quantita-
tive
Other characteristics of each of the six main forecasting methods are as fol-
lows.
1. Time Series: Some sub-methods are deterministic (moving average and exponen-
tial smoothing); some sub-methods are stochastic/probabilistic (ARIMA). Time
series, basically, are not causal methods.
2. Ratio Methods: Cohort-survival is a deterministic method, while Markov analysis
is a stochastic model. Ratio methods are not considered causal, although a causal
relationship is imbedded in the choice of groups.
3. Regression Methods: Are stochastic but not always controlled variables.
4. Simulation: System dynamics is a deterministic method, while discrete event

simulations and agent-based simulations are usually probabilistic. Causality is
often assumed, though not always.
5. Data Mining Methods: Some are deterministic, and some are stochastic, some
are casual, and some are not; some are predicting a continuous level (e.g.,
enrollments), and some are predicting classification-success or failure.
6. Questionnaires and Surveys: Used, often, in conjunction with some of the above
methods.
5.3.2 Other Classifications of Forecasting Methods
There are other classifications of forecasting methods, in general, not specific to HE.
Here are two examples:
Makridakis et al. (1983) full classification is the most general:
(a) Smoothing and decomposition: Average methods, and exponential smoothing,
decomposition, ratio-to-moving average, census decomposition
(b) Regression and Econometric: simple regression, multiple regression, economet-
ric models
(c) Box-Jenkins ARIMA: Box-Jenkins, multivariate time series
(d) Qualitative methods: predicting cycle, qualitative, and technological method
Another example for classification method is by Armstrong and Green (2018),
who foster conservatism as a golden rule in forecasting and simplicity, suggest
another breakdown for two types of forecasting methods:
(a) Judgmental methods, including prediction markets, multiplicative decomposi-
tion, intention surveys, expectation surveys, expert surveys, simulation interac-
tion, structured analogies, experimentation, and expert systems.
(b) Quantitative methods, including extrapolation, rule-based methods, judgmental
bootstrapping, segmentation, simple regression, and knowledge models.
Combining forecasts from several methods is recommended, as it has a high
potential to reduce forecast error (ibid.).
The breakdown of forecasting methods of Armstrong and Green (2018) and
its terminology and sources better fit market research (Market/marketing was
mentioned 42 times in the article, including in 13 marketing references). Indeed,
marketing (marketization) became accepted in HE during the past 30 years (Oplatka
& Hemsley-Brown, 2021).
The third classification method Brinkman and McIntyre (1997) classified fore-
casting methods in HE, divided into three main categories as follows:
(a) Quantitative methods: (1) curve-fitting techniques: trend analyses, enrollment
ratio, moving average, autoregressive integrated moving average (ARIMA),
exponential smoothing, seasonal models, regressing on time. (2) Causal
explanatory methods: multiple regression, Markov, ratio, and flow models

(e.g., cohort-survival)
(b) Qualitative methods: Experts and Delphi
(c) Combined approaches
Their literature review was based on applications of the forecasting methods
above in HE, covering articles from 1980 to 1995 (ibid.). Almost all the above-
listed methods were based on classical textbooks such as Makridakis et al. (1983)
and Hyndman and Athanasopoulos (2018). Brinkman and McIntyre (1997) is not
as detailed as the classical textbooks on forecasting, added methods specific to
HE; namely, Ratio method, cohort-survival, and Markov system. We added another
two forecasting approaches: DM (educational DM–EDM and learning analytic) and
questionnaires/surveys (for marketing HE). Moreover, our classification is based on
active and passive forecasting.
5.3.3 Managerial and Planning Aspects of Forecasting
Linking enrollment predictions and tuition income models is common, for example,
by Trusheim and Rylee (2011) for the University of Delaware. The enrollment
matrix is constructed from past enrollment data by semester over several years
by type of entering students (freshman, transfer, readmitted, and continuer). The
model is based on a cohort-survival method. The predicting model consists of four
steps: (1) collect historical data on enrollment by full-time and part-time ratios; (2)
develop the retention percentage for continuing students; (3) obtain new freshman,
transfer, and readmitted student targets for future semesters; and (4) run the numbers
through the model to generate predictions. The prediction is based on average
historical percentages for each group, where the head of the university can dictate a
total target. The tuition model was integrated, where the enrollment breakdown of
full-time/part-time ratio by resident/non-resident ratio was used for calculating the
desired income from tuition based on the full-time tuition versus part-time tuition,
and resident tuition versus the higher non-resident tuition. The desired ratios are
transmitted to the admission office to consider in their admission process. This
model is a good example of how the policy of HEIs can affect enrollment and the
income from tuition.
The level of detail in the forecast varies from one application to another, and
the horizon of the forecast needed also varies among applications. If there is high
demand for a specific HEI, the institution does not need to forecast enrollment but
to determine a target number for its enrollment according to other parameters which
need to be predicted (e.g., income, faculty, space, etc.). Yet, fluctuations may in fact
occur in actual enrollments.
Breakdowns of student enrollments by discipline, by gender, and by other
demographic parameters to plan the resources needed for each group are essential.
The forecast can be from the top down, where first the total number of students is
calculated by a specific method, and then the total is broken down into its com-
ponents in a second stage by making some assumptions about percentages (based
on past breakdowns) within the total enrollment forecasted. Another approach is a
forecast from the bottom up, by using various methods to forecast the enrollment of
each group under various assumptions and then summing up those groups’ forecasts
to obtain the total student enrollments.
Hoenack and Weiler (1979) present a university (Minnesota) total enrollment
forecasting model in a situation where enrollment may be less than the university
capacity and may decline due to a reduction in the size of the traditional college-
going population and the labor market situation. In their specific situation, the state
allocation to a public university was based on enrollment forecasts for the university;
thus, predicting lower than actual enrollment caused a reduced state allocation to
the university that year. The comprehensive model was composed of ten sequential
equations, which were simulated sequentially within each year and over the years.
Each equation predicts a specific subgroup at time t as a function of other relevant
sub-groups and demographic (e.g., cohort) and economic parameters (some lagged),
such as tuition, unemployment rate, salaries of college and non-college graduates.
Forecasting accuracy is an important issue. The main method for evaluating
forecast accuracy is mean-squared error of their past observations versus their
forecast—the smaller the better (Makridakis et al., 1998). Another less frequently
used method is mean absolute error (ibid.). Combining forecasts is often done to
achieve greater accuracy (Armstrong & Green, 2018).
There is not a widely accepted best forecasting method in general, and specif-
ically in HE for forecasting university enrollment or income, or other parameters
needed for HE planning. Furthermore, many of the existing methods require the
knowledge and use of advanced statistical/mathematical techniques. Many HEIs
do not have administrative staff with the expertise to utilize advanced statistical
methods. The simple methods are often less accurate than the more advanced
methods, but not necessarily. Many HEIs hire consultants for this purpose (such
as the author of this chapter). It is important that the consultant has proficiency in a
wide range of forecasting methods and access to forecasting software.
5.4 Counting Number of Articles on Forecasting in HE

by Method and by Area
Since there are thousands of articles on forecasting methods in HE, it was too much
to review each; thus Appendix 1 presents only examples of articles to present the
various forecasting methods and what they were used for. However, the number of
articles published on forecasting methods in general, versus HE was counted here,
as taken from the Web of Science (WoS) during 2000–2020, as shown in Appendix
2. In total, there are over 5000 articles on forecasting in all 30 methods listed in
Appendix 2. Similarly, looking at the 11 areas of application in Appendix 3, where

forecasting is used in HE, a total of about 14,500 articles were found.
5.4.1 Counting Number of Articles on Each Forecasting

Method in HE
Column-1 of Appendix 2 presents, for each of 30 listed forecasting methods

(and their synonyms), the number of articles for which the method appeared in
their topic (i.e., in the title or abstract or keywords of the article) in general
(not necessarily in HE). Since some forecasting methods are used, also, for other
purposes besides forecasting, we require that in the search engine for HE, in addition
to specific forecasting methods, the word forecast (or some synonym) will appear,
too. The total sum of articles in column-1 is almost 6 million articles. The four
leading forecasting methods in general, with the maximal numbers of articles, are
simulation (with its sub-methods 15–18) with over 2.6 million articles; data mining
(with its sub-methods, 19–29) with over 1.68 million articles, regression with over
0.887 million articles, and questionnaires with over 0.447 million articles.
Column-2 of Appendix 2 presents, for each of 30 forecasting methods, the
number of articles where the word “forecast” and the word “higher education” and
the method (or their synonyms) appear in the topic of the article. The forecasting
methods with the highest numbers of articles are as follows.
(a) Regression, with over 26,000 articles
(b) Simulation or its sub-methods (15–18), with about 22,000 articles
(c) Data mining or its sub-methods (19–29), with about 20,000 articles
(d) Questionnaires with 11,000 articles
Column-3 counts the number of articles where the method is in the topic, and
HE and forecast or their synonyms appear in the title of the article. The sum of
the articles over the 30 forecasting methods is 5219, obviously, less than columns
1 and 3. In this case, again the same four methods forecasting methods are leading
but in different order: questionnaires have the largest number of publications, with
2849 articles; regression is second, with 969 articles; and data mining (with its sub-
methods: 19–29) is third, with 843 articles; simulation (with its sub-methods 15–18)
is the fourth, with 407 articles. For HE, it is more reasonable to take column 2
counting as representing, since HE applications tend not to highlight the specific
type of the methodology in the title of the article but rather in the abstract or
keywords.
In summary, the same four methods in the first three columns, in different
orders, are leading in number of articles: regression, simulation, data mining, and
questionnaires.
5.4.2 Counting Number of Articles in HE, and Forecasting,

by Main Areas of Application
Appendix 3 counts in WoS the numbers of articles in each of the 11 listed application
areas. The leading areas in column-1 are marketing, performance, transportation,
and teaching.
The leading areas in column-2 of Appendix 3, where HE forecasting and the area
of application appear in articles’ topics, are as follows:
1. Marketing, with about 90,000 articles
2. Teaching, with about 52,600 articles
3. Enrollment, with about 46,000 articles
4. Performance, with about 38,700 articles
In column-3 of Appendix 3, where HE and forecasting appear in articles’ titles,
and the area appears in the topic, there are the same four leading areas of application
as in column-2, but in a different order: enrollment with 3484 articles, marketing
with 3149 articles, teaching with 2379 articles and performance 1712 articles.
In Appendix 3, the total number of articles with area in HE in their topic versus
title (columns 2 vs. 3) as a percent from the total number of articles with area not
necessarily in HE (column 1) is 1.69% versus 0.077; this is similar to the parallel
percent of total number of articles with HE and forecasting method in their topic
versus title, 1.416% versus 0.087% (Appendix 2 last two columns in relation to
columns 2 and 3).
Obviously, for Appendices 2 and 3, the relation between the first three columns
is the same: column-3 articles are included in column-2 articles, and column-2
articles are included in column-1 articles. Regarding the exact number of articles,
where each method/area was used in HE for forecasting, column-2 numbers can
be considered as a proxy for the upper bound number of the articles (for each
forecasting method and area of application); while column-3 can be considered
as proxy for their lower bound. Proxies in each column mean that there might be
double counting of the same article (e.g., once in data mining, and another time in
regression, as a sub-method in data mining) or missing synonyms for some methods.
This chapter reviews many forecasting methods used in HE, and their main areas of
application, with examples drawn from the literature (in Appendix 1). By counting
the number of articles published in the Web of Science over the last 20 years, we
found that four forecasting methods were used most often in forecasting in HE:
regression, simulation, DM (including its sub-methods), and questionnaires (for
marketing purposes). Furthermore, four areas of application were used most often
for forecasting in HE: enrollment, marketing, teaching, and performance.
The new upcoming forecasting method during the past 20 years is DM and its
sub-methods: both old methods (e.g., regression) and new methods (e.g., genetic
algorithms). DM selects the best from several methods applied to the data. Often,
Educational DM (EDM and another variation—learning analytics) is, often, used as
a tool to increase student retention from one semester to the next. Moreover, DM can
handle big data to predict if a single student will drop out, and the reason, so that
the administration/lecturer is able to take actions to retain the student before s/he
drops out. EDM and learning analytic combine many aspects of forecasting applied
specifically to HE.
In addition to the classical use of questionnaires by theoretical researchers in edu-
cation to study various phenomena in HE, administrators are using questionnaires
more often in the past 20 years, mainly for market research—to figure out ways to
increase enrollments or to influence the university’s ranking and its components, to
increase the university’s attractiveness for potential students.
We introduce the terms active and passive forecasting methods in a new sense.
Active forecasting provides a control for the administrator by pointing at some
actions to influence future developments, while a passive forecast assumes no
control. The two leading forecasting approaches, DM and marketing, are active
forecasting methods, pointing administrators at actions to be taken to achieve
a targeted forecast. Regression analysis and simulation are not always active
forecasting methods. However, time series and ratio methods are passive forecasting
methods.
For future research, the counting performed here is a rough proxy, improving the
synonyms used can give more accurate counting.
Appendix 1
148
Forecasting in Higher Education: Summary of Selected Applications over Time

Name of first Author (year) Forecasting method type Area of application Organizational level Country
Edwards (1932) Survey Enrollment forecast University USA
Schmid and Shanley Ratio and cohort-survival Enrollment projection University USA
(1952)
Baisuck and Wallace Simulation and regression Student enrollment University and state levels USA
(1970)
Sinuany-Stern (1976) Multiple regression models Questionnaire on reasons for Multi-campus community USA
students enrolling college
Hoenack and Weiler Ten sequential equations Total forecast: undergraduate University USA
(1979) enrollment
Sinuany-Stern (1980) Multiple regression and ratio Students by year and level Total and each state’s USA Indiana
university
Bessent and Bessent Markov analysis Students’ progression by year Department, by year of USA
(1980) study
Sinuany-Stern (1984) Multiple regression and ratio Enrollments and costs for Multi-campus community USA
method budgeting college
Burkett (1985) Cohort-survival ratio Freshman enrollment prediction University USA
Kwak et al. (1986) Markov analysis Student enrollment University USA
Banerjee and Igbaria Questionnaire/ survey Computer capacity planning National USA and Canada
(1993)
Sinuany-Stern and Yelin Regression Hardware resources (CPU, University Israel
(1993) memories, etc.)
Hwang et al. (1998) Fuzzy time series versus Forecasting student enrollment University USA
Markov, and first-order
time-variant model
Murtaugh et al. (1999) Cohort-survival, proportional Students retention probabilities University USA
hazards and regression
Z. Sinuany-Stern
Shah and Burke (1999) Markov analysis Undergraduate student National Australia
progression
Chen (2008) ARIMA Enrollment forecast University USA
Nicholls (2009) Markov models Planning and benchmarking University Australia
doctoral progression
Aladag et al. (2010) Neural network, fuzzy time Forecast of enrollment University USA
series–DM
Jantan et al. (2010) Data mining Prediction of academic talent National Malaysia
Min et al. (2011) Cohort-survival, nonparametric Student graduation 9 HEIs, 19 years USA
survival analysis
Trusheim and Rylee Cohort-survival, percentages by Enrollment prediction and University USA
(2011) type of student tuition income
Lavilles and Arcilla Simple moving average, Student enrollment in electronic By e-course, within a USA
(2012) exponential smoothing courses university
Saltzman and Roeder Simulation Student flow, and enrollment to College of business USA
(2012) colleges
Tang and Yin (2012) Grey models better than Education expenditure and National USA
5 Forecasting Methods in Higher Education: An Overview
exponential smoothing–DM student enrollment

Agboola and Adeyemi Ratio method and fixed annual Faculty and student forecast National Nigeria
(2013) %
Allen (2013) Ratio model and Markov model Enrollment projection at various University and discipline USA
levels
Ge et al. (2013) Fuzzy time series and Enrollment University USA
exponential smoothing
Amber et al. (2015) Multiple regression, genetic Electricity forecasting National United Kingdom
algorithm–DM
Ge and Xie (2015) Adaptive residual correction Forecasting cost of university University China
based on Grey model–DM education
(continued)
149
(continued)
150
Name of first Author (year) Forecasting method type Area of application Organizational level Country
Anggrainingsih et al. Exponential smoothing and time Predicting # of website visits University Indonesia
(2015) series
Pamungkas and Rofiqoh Moving average Enrollment University Indonesia
(2015)
Sweeney et al. (2016) Factorization machines, random Retention in next-term, grade National USA
forest and personalized linear prediction to improve student
regression–DM performance
Hamsa et al. (2016) Decision tree and fuzzy genetic Student academic performance BS and MS students of India
algorithm–DM prediction computer science
Cohen (2017) Data mining classification Predicting course dropout Individual student Israel
Witteveen and Attewell Hidden Markov models Students graduation National USA
(2017)
Xiao and Chankong System dynamics Forecast of demand and supply National China
(2017) of students (medical talents)
Miguéis et al. (2018) Data mining Students’ academic performance University European country
(details not
mentioned)
Moon et al. (2018) Time series analysis, random University building electrical University Korea
forest–DM load forecast
Sass et al. (2018) Bayesian structural equation Student retention University USA
model, Markov, Monte Carlo
samples, factor analysis–DM
Tsevi (2018) Cohort-survival, qualitative International students’ success University USA
study via interviews, using the
theory of persistence
Z. Sinuany-Stern
Chen et al. (2019) Time series analysis Student enrollment University USA
Wade (2019) Cohort survival, random and Student retention rate National USA
fixed effects models, linear
regression
Chou (2019) Questionnaire and regression Predicting self-efficacy in National Taiwan
English test preparation
Hasan et al. (2020) Analytic and data mining, Predict student performance in Course Malaysia
random forest video e-course
151
Appendix 2
152
Number of articles on forecastinga in higher education (HEa ) by method

No. Method Number of articles during 2000–2020 Percentage of:
Method, forecast a Method in topicb , forecast
b
Method in topic (1) and HE a in topicb (2) a and HE a in title (3) Column (2) from (1) Column (3) from (1)
1 Ratio method 3378 20 0 0.592 0.000
2 Cohort survival 222 4 0 1.802 0.000
3 Survival analysis 31, 779 787 12 2.476 0.038
4 Markov 116, 354 822 19 0.706 0.016
5 Sequential equations 4 0 0 0.000 0.000
6 Time series 161, 813 2100 67 1.298 0.041
7 Trend analysis 8732 106 3 1.214 0.034
8 Moving average 14, 359 267 13 1.859 0.091
9 Exponential smoothing 1738 70 6 4.028 0.345
10 ARIMA 5464 158 11 2.892 0.201
11 Auto regression a 30, 492 364 17 1.194 0.056
12 Fuzzy time series 680 146 3 21.471 0.441
13 Regression a 887, 584 26, 621 969 2.999 0.109
14 Linear modeling 1798 27 0 1.502 0.000
15 Simulation 2, 280, 840 19, 565 330 0.858 0.014
16 System dynamic a 23, 929 360 28 1.504 0.117
17 Discrete event a 294, 444 2210 47 0.751 0.016
18 Agent-based simulation 3840 54 2 1.406 0.052
19 Data mining 66, 410 2724 193 4.102 0.291
20 Neural networks 166, 709 3804 137 2.282 0.082
21 Decision trees 23, 178 1097 84 4.733 0.362
22 Random forests 6088 210 10 3.449 0.164
Z. Sinuany-Stern
23 Bayesian 143,585 1798 59 1.252 0.041
24 Analytic a 189,999 3013 179 1.586 0.094
25 Genetic 1,015,427 6173 119 0.608 0.012
26 Grey 51,412 623 39 1.212 0.076
27 Density estimation 9879 50 0 0.506 0.000
28 Knowledge discovery 9123 361 20 3.957 0.219
29 Mapping study 1819 49 3 2.694 0.165
30 Questionnaire 447,487 11,361 2849 2.539 0.637
Total methods 5,998,566 84,944 5219 1.416 0.087
Search was performed in Web of Science for 2000–2019 (in September 2020)
a Synonyms used for forecasting: forecast, prediction, predictive, predicting, predict, projection, projective, projecting, forecast; and higher education:
academic college, university, “tertiary education;” and (auto regression: autoregression, autoregressive; or linear regression: multiple regression, regression;
or system dynamic: systems dynamic, system dynamics; or discrete event: Monte Carlo, DES; or analytic: analytics)
b Topic means that the search words appear in the article’s title, abstract, or keywords
153
154
Appendix 3
Number of articles on forecastinga in higher educationa by area of application

No. Area of application Number of articles during 2000–2020 Percentage of:
Area, forecast a and Area in topicb , forecast a
Area in topicb (1) HE a in topicb (2) and HE a in title (3) Column (2) from (1) Column (3) from (1)
1 Transportation 2, 687, 092 32, 053 1318 1.193 0.049
2 Industry 1, 535, 172 24, 964 846 1.626 0.055
3 Transfer of knowledge 224, 253 5056 208 2.255 0.093
4 Marketing 7, 730, 644 90, 043 3149 1.165 0.041
5 Projection 274, 406 13, 867 439 5.053 0.160
6 Enrollment 697, 521 45, 926 3484 6.584 0.499
7 Faculty 217, 093 14, 531 756 6.693 0.348
8 Teaching 1, 803, 689 52, 620 2379 2.917 0.132
9 Student performance 6366 766 107 12.033 1.681
10 Performance 3, 565, 201 38, 688 1712 1.085 0.048
11 Retention 249, 534 2378 175 0.953 0.070
Total areas 18, 990, 971 320, 892 14, 573 1.690 0.077
Search was performed in Web of Science for 2000–2019 (in September 2020)
a Synonyms used for forecasting: forecast, prediction, predictive, predicting, predict, projection, projective, projecting, forecast; higher education: academic
college, university, “tertiary education”

b Topic means that the search words appear in the article’s title, abstract, or keywords
Z. Sinuany-Stern
References
Agboola, B. M., & Adeyemi, J. K. (2013). Projecting enrollment for effective academic staff
planning in Nigerian universities. Educational Planning, 21(1), 5–17.
Aladag, C. H., Yolcu, U., & Egrioglu, E. (2010). A high order fuzzy time series forecasting model
based on adaptive expectation and artificial neural networks. Mathematics and Computers in
Simulation, 81(4), 875–882.
Allen, D. M. (2013). Multi-purpose enrollment projections: A comparative analysis of four
approaches. The University of Maine, ProQuest Dissertations Publishing, 2013. No.3579435.
Al-Razgan, M. S., Al-Khalifa, A. S., & Al-Khalifa, H. (2014). Educational data mining: A sys-
tematic review of the published literature 2006–2013. Lecture Notes in Electrical Engineering,
285, 711–719.
Amber, K. P., Aslam, M. W., & Hussain, S. K. (2015). Electricity consumption forecasting models
for administration buildings of the UK higher education sector. Energy & Buildings, 90, 127–
136.
Anggrainingsih, R., Aprianto, G. R., & Sihwi, S. W. (2015). Time series forecasting using
exponential smoothing to predict the number of website visitor of Sebelas Maret
University. 2nd International Conference on Information Technology, Computer, and
Electrical Engineering. Retrieved October, 2020, from https://ieeexplore.ieee.org/abstract/
document/7437762?casa_token=cZiteuI6UfcAAAAA:ARZiNucSR7QJq5cn-pgHsXnZs-
xjTQtBDD10X9KnXkPluupCcR0V-6jUptTTt1VZCyJN-YNOSyM
Armstrong, J. S., & Green, K. C. (2018). Forecasting methods and principles: Evidence-based
checklists. Journal of Global Scholars of Marketing Science, 28(2), 103–159.
Baisuck, A., & Wallace, W. A. (1970). A computer simulation approach to enrollment projection.
Socio-Economic planning Science, 4, 365–381.
Baker, R. S., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future
visions. JEDM | Journal of Educational Data Mining, 1(1), 3–17.
Banerjee, S., & Igbaria, M. (1993). An empirical study of computer capacity planning in U.S.
universities. Information & Management, 24(4), 171–182.
Barthwal, R. R. (2000). Industrial Economics (2nd ed.). New Age International Limited.
Bessent, E. W., & Bessent, A. M. (1980). Student flow in a university department: Results of a
Markov analysis. Interfaces, 10(2), 52–59.
Brinkman, P. T., & McIntyre, C. (1997). Methods and techniques of enrollment forecasting. in
forecasting and managing enrollment and revenue: An overview of current trends, issues, and
methods. Editor Layzell DT, 67–80. New Directions for Institutional Research, no. 93. Jossey-
Bass, San Francisco.
Burkett, H. E. (1985). The use of a cohort-survival model to forecast the university of Mississippi
freshman enrollment, 1985–1989 (predict). Ph.D. Thesis, Education Administration, University
of Mississippi.
Chen, C.-K. (2008). An integrated enrollment forecast model. Institutional Research Applications,
15, 1–14.
Chen, Y., Li, R., & Hagedorn, L. S. (2019). Undergraduate international student enrollment
forecasting model: An application of time series analysis. Journal of International Students,
9(1), 242–261.
Chou, M.-H. (2019). Predicting self-efficacy in test preparation: Gender, value, anxiety, test
performance, and strategies. The Journal of Educational Research, 112(1), 61–71.
Cohen, A. (2017). Analysis of student activity in web-supported courses as a tool for predicting
dropout. Educational Technology Research and Development, 65(5), 1285–1304.
Edwards, M. (1932). College enrollment during times of economic depression: Are the predictions
of increased enrollments because of the depression justified? Journal of Higher Education,
3(1), 11–16.
Ge, C., & Xie, J. (2015). Application of Grey forecasting model based on improved residual
correction in the cost estimation of university education. International Journal of Emerging
Technologies in Learning, 10(8), 30–33.
Ge, P., Wang, J., Ren, P., Gao, H., & Luo, Y. (2013). A new improved forecasting method integrated
fuzzy time series with the exponential smoothing method. International Journal of Environment
and Pollution, 51(3–4), 206–221.
Hamsa, H., Indiradevi, S., & Kizhakkethottam, J. J. (2016). Student academic performance
prediction model using decision tree and fuzzy genetic algorithm. Procedia Technology, 25,
326–332.
Hasan, R., Palaniappan, S., Mahmood, S., Abbas, A., Sarker, A. U., & Sattar, M. U. (2020).
Predicting student performance in higher educational institutions using video learning analytics
and data mining techniques. Applied Sciences, 10(11), 1–20.
Hellas A, Ihantola P, Petersen A, Ajanovski, V. V., Gutica, M., Hynninen, T., Knutas, A., Leinonen,
J., Messom, C. & Liao, S. N. (2018). Predicting academic performance: A systematic literature
review. In G. Rossling & B. Schalau (Eds.), ITiCSE 2018 Companion: Proceedings Companion
of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science
Education (pp. 175–199). https://doi.org/10.1145/3293881.3295783.
Hemsley-Brown, J., & Oplatka, I. (2006). Universities in a competitive global marketplace: A
Hoenack, S. A., & Weiler, W. C. (1979). The demand for higher education and institutional
enrollment forecasting. Economic Inquiry, 17(1), 89–113.
Hwang, J. R., Chen, S.-M., & Lee, C.-H. (1998). Handling forecasting problems using fuzzy time
series. Fuzzy Sets and Systems, 100(1), 217–228.
Hyndman, R., & Athanasopoulos, G. (2018). Forecasting: Principles and practices (2nd ed.).
OTEXTS Publishing. http://OTexts.com/fpp2/
Jantan, H., Hamdan, A. A., & Othman, Z. A. (2010). Human talent forecasting using data mining
classification techniques. International Journal of Technology Diffusion, 1, 29–41.
Kwak, N. K., Brown, R., & Schiederjans, M. J. (1986). A Markov analysis of estimating student
enrollment transition in a trimester institution. Socio-Economic Planning Sciences, 20(5), 311–
318.
Lavilles, R. Q., & Arcilla, M. J. B. (2012). Enrollment forecasting for school management system.
International Journal of Modeling and Optimization, 2(5), 563–566.
Makridakis, S., Wheelwright, S. C., & Hyndman, R. J. (1998). Forecasting, methods and
applications (3rd ed.). Wiley.
Makridakis, S., Wheelwright, S. C., & McGee, V. (1983). Forecasting, methods and applications
(2nd ed.). Wiley.
Miguéis, V. L., Freitas, A., Garcia, P. J. V., & Silva, A. (2018). Early segmentation of students
according to their academic performance: A predictive modelling approach. Decision Support
Systems, 115, 36–51.
Min, Y., Zhang, G., Long, R. A., Anderson, T. J., & Ohland, M. W. (2011). Nonparametric
survival analysis of the loss rate of undergraduate engineering students. Journal of Engineering
Education, 100(2), 349–373.
Moon, J., Kim, Y., Son, M., & Hwang, E. (2018). Hybrid short-term load forecasting scheme
using random forest and multilayer perceptron. Energies, 11(12), 3283. https://doi.org/10.3390/
en11123283
Murtaugh, P. A., Burns, L. D., & Schuster, J. (1999). Predicting the retention of university students.
Research in Higher Education, 40(3), 355–371.
Nicholls, M. G. (2009). The use of Markov models as an aid to the evaluation, planning and
benchmarking of doctoral programs. Journal of the Operational Research Society, 60(9), 1183–
1190.
Oplatka, I., & Hemsley-Brown, J. (2021, forthcoming). A systematic and updated review of the
literature on higher education marketing 2005–2019. In Z. Sinuany-Stern (Ed.), Handbook on
operations research and management science in higher education. Springer.
Pamungkas, T., & Rofiqoh, U. (2015, August 22–23). Applications moving average methods to
forecast the number of student SMPN 11 Batam Academic Years 2016–2020. Proceedings of
the 1st International Conference on Character Education Batam, pp. 38–42.
Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert
Systems with Applications, 33(1), 135–146.
Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated
survey. Data Mining and Knowledge Discovery, 10(3). https://doi.org/10.1002/widm.1355
Saltzman, R. M., & Roeder, T. M. (2012). Simulating student flow through a college of business
for policy and structural change analysis. Journal of the Operational Research Society, 63(4),
511–523.
Sass, D. A., Castro-Villarreal, F., Wilkerson, S., Guerra, N., & Sullivan, J. (2018). A structural
model for predicting student retention. The Review of Higher Education, 42(1), 103–135.
Schmid, C. F., & Shanley, F. J. (1952). Techniques of forecasting university enrollment: Tested
empirically by deriving forecasts of enrollment for the university of Washington. The Journal
of Higher Education, 23(9), 483–503.
Shah, C., & Burke, G. (1999). An undergraduate student flow model: Australian higher education.
Higher Education, 37, 359–375.
Shahiri, A. M., Husain, W., & Rashid, N. A. (2015). A review on predicting student’s performance
using data mining techniques. Procedia Computer Science, 72, 414–422.
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral
Scientist., 57(10), 1380–1400.
Sinuany-Stern. (1984). A financial planning model for a multi-campus college. Socio-Economic
Planning Science, 18(2), 135–142.
Sinuany-Stern, Z. (1976). Enrollment forecasting models for a multi-campus community college.
ORSA/TIMS Meeting.
Sinuany-Stern, Z. (1980). Indiana higher education enrollment projections: 1979–1990. Internal
report, State of Indiana Commission for Higher Education.
Sinuany-Stern, Z., & Yelin, H. (1993). Forecasting computer resources. Computers & Operational
Research, 20(5), 477–484.
Sweeney, M., Lester, J., Rangwala, H., & Johri, A. (2016). Next-term student performance
prediction: A recommender systems approach. Journal of Educational Data Mining, 8(1), 22–
51.
Tang, H.-W. V., & Yin, M.-S. (2012). Forecasting performance of grey prediction for education
expenditure and school enrollment. Economics of Education Review, 31(4), 452–462.
Trusheim, D., & Rylee, C. D. (2011). Predictive modeling: Linking Enrollment and budgeting.
Planning for Higher Education Journal, 40(1), 12–21.
Tsevi, L. (2018). Survival strategies of international undergraduate students at a public research
mid-western university in the United States: A case study. Journal of International Students,
8(2), 1034–1058.
Wade, N. L. (2019). Measuring, manipulating, and predicting student success: A 10-year assess-
ment of Carnegie R1 doctoral universities between 2004 and 2013. Journal of College Student
Retention: Research, Theory & Practice, 21(1), 119–141.
Witteveen, D., & Attewell, P. (2017). The college completion puzzle: A hidden Markov model
approach. Research in Higher Education, 58(4), 449–467.
Xiao, B., & Chankong, V. (2017). A system dynamics model for predicting supply and demand of
medical education talents in China. Eurasia Journal of Mathematics, Science and Technology
Education, 13(8), 5033–5047.
Chapter 6
Survey of Methods for Ranking
and Benchmarking Higher Education
Institutions
José-Luis Pino-Mejías and Pedro-Luis Luque-Calvo
Abstract The objective of this chapter is to provide managers and leaders of Higher
Education Institutions (HEI) with the knowledge that allows them to evaluate the
possibilities and limitations of rankings and benchmarking techniques, and how
these techniques can be applied correctly in the different levels of higher education
systems.
The chapter includes a presentation of six of the most known international uni-
versity rankings, a selection of multicriteria decision-making techniques necessary
to select the variables to be used, normalize their values, determine their weights,
choose the aggregation method and measure how the position of each institution
in the ranking depends on the methodological options adopted. For each technique,
examples of application to HEI are shown.
The HEI benchmarking interactive tool (HEIBIT), a tool developed to facilitate
learning these techniques in international postgraduate programs in the management
of HEI in Colombia, Cuba, Dominican Republic, Ecuador, and Spain, is also
presented.
Keywords Higher education · Rankings · Benchmarking · Efficiency · Multiple

criteria · PROMETHEE · Data envelopment analysis · Sensitivity analysis ·
Uncertainty analysis
6.1 Introduction
Universities play a significant role in the development of societies; therefore, mea-

suring and assessing their performance becomes crucial for various stakeholders,
including government, industry, and society (Olcay & Bulu, 2017).
J.-L. Pino-Mejías () · P.-L. Luque-Calvo

Departamento de Estadística e Investigación Operativa, Universidad de Sevilla, Seville, Spain
e-mail: jlpino@us.es

160 J. Pino-Mejías and P. Luque-Calvo
Among the different instruments of quality assessment in higher education,

rankings probably attract the most public attention. Rankings (or league tables) are
a key element in shaping the opinions of students, parents, and employers about
the quality of Higher Education Institutions (HEIs) (Marmolejo, 2016). In recent
years, rankings of HEIs have increased, and their usage has expanded. Currently, it
is common for HEI managers to resort to these when making important decisions or
managerial policies (Johnes, 2018). In the first section, we describe six of the most
known international university rankings, using the information from their websites.
Relative performance evaluations or benchmarking is the systematic comparison
of the performance of one organization against other organizations (Bogetoft &
Otto, 2011). The interest in the benchmarking of HEIs has increased in parallel
to the development and dissemination of university rankings (Moed, 2017).
The results of any public evaluation always influence the reputation of the units
on which they are carried out. In addition, they have a special importance when they
are used to carry out the institutional evaluation or to establish financing of those
units. Therefore, the managers of universities and evaluation agencies must know
the possibilities and limitations of each technique used in the evaluation processes.
Ranking and benchmarking are based on the use of techniques developed in the
field of multicriteria analysis, and specifically, efficiency analysis when inputs and
outputs are considered.
The first element in any multicriteria analysis is the criteria selection. At this
stage, the theoretical framework that guides the selection of the criteria according to
the objective of the classification, or the benchmarking, must be explicit and allow
selecting the indicators that will be used to measure the characteristics that they seek
to represent in their classifications. However, in practice, compilers are restricted
by the limited availability of comparable and verified data. Because “quality” and
“excellence” are difficult to define and judge, one could argue that the classification
tables indicate what can be measured instead of measuring what counts (Locke et
al., 2008).
6.2 International Rankings of Universities
World university rankings, such as Academic Ranking of World Universities

(ARWU, 2018), Times Higher Education World University Rankings (THE, 2018),
Quacquarelli Symonds World University Rankings (QS, 2018), CWTS Leiden
Ranking (CWTS, 2018), US News Best Global Universities rankings (USNEWS,
2018) or U-Multirank World University Rankings (U-Multirank, 2018), have in
common that they generate controversies about their rigor and are a source of
debate. However, in many cases, they act as a guide for public policies that have
as an objective to improve the values of the indicators, and the position in the table
leagues of universities, colleges, or degrees, without analyzing the real impact on
the institutions or in the academic programs.
6 Survey of Methods for Ranking and Benchmarking Higher Education Institutions 161
Table 6.1 Criteria, indicators, and weights of the ARWU Ranking

Criteria Indicator Percentage weighting
Quality of education Alumni of an institution winning Nobel 10
Prizes and Fields Medals
Quality of faculty Staff of an institution winning Nobel Prizes 20
and Fields Medals
Highly cited researchers in 21 broad subject 20
categories
Research output Papers published in Nature and Science (For 20
institutions specialized in humanities and
social sciences, the weight of N and S is
relocated to other indicators)
Papers indexed in Science Citation Index 20
(SCI)-expanded and Social Science Citation
(SSC) Index
Academic performance The weighted scores of the above five 10
indicators divided by the number of
full-time equivalent academic staff
As an example of indicators used in university rankings, the Academic Ranking

of World Universities (ARWU, 2018) compiled annually by the Shanghai Jiao Tong
University since 2003 uses the criteria and indicators shown in Table 6.1.
From the ARWU indicators, those who value the Nobel prizes and the Fields
medals are the indicators that generate the most discussion; even the best universities
in the world cannot be sure if, in the near future, the winners will be their students
or professors. For this reason, ARWU calculated in 2014 and 2015 an alternative
ranking based on the remaining four indicators, maintaining their respective weights
without changes.
From 2012 Shanghai Ranking Consultancy has developed Global Research
University Profiles (GRUP), a comprehensive database and benchmarking tool
covering 1200 research universities in the world. Based on the data reported
by universities and collected from third parties, GRUP presents comparisons of
universities in terms of 40 indicators (13 indicators about students, 9 indicators
about faculty, 13 indicators about resources, and 5 ARWU indicators).
The Times Higher Education World University Ranking 2019 (THE, 2018)
includes more than 1250 universities and uses the 13 performance indicators shown
in Table 6.2.
The Quacquarelli Symonds World University Rankings, 2019 (QS, 2018)
includes 1000 universities and uses the six performance indicators shown in Table
6.3.
The US News Best Global Universities rankings (USNEWS, 2018) includes 1250
universities and uses the 13 indicators shown in Table 6.4. Results from Clarivate
Analytics’ Academic Reputation Survey were used to create the two reputation
indicators used in US News’ ranking analysis.
Table 6.2 Criteria, indicators, and weights of THE Ranking

Criteria Indicators Percentage weighting
Industry Research income from industry (per 2.5
Income—innovation academic staff)
International diversity Ratio of international to domestic staff 32
Ratio of international to domestic students
Teaching—the learning Reputational survey (teaching) 1564.52.252.25
environment PhDs awards per academic
Undergrads admitted per academic
Income per academic
PhDs/undergraduate degrees awarded
Research volume, Reputational survey (research) 19.55.254.50.75
income, and reputation Research income (scaled)
Papers per research and academic staff
Public research income/ total research
income
Citations—research Citation impact (normalized average 32.5
influence citation per paper)
Table 6.3 Criteria, indicators, and weights of QS Ranking

Criteria Indicator Percentage weighting
Academic Reputation Reputational survey 40
Employer Reputation Reputational survey 10
Learning environment Faculty/Student Ratio 20
Research output Citations per faculty 20
Internationalization of Faculty International Faculty Ratio 5
Internationalization of Student International Student Ratio 5
Examples of rankings that do not obtain a single composite indicator by the

weighted sum of the indicators are the Leiden Ranking and U-Multirank. The
CWTS Leiden Ranking is based exclusively on bibliographic data from the Web
of Science database of Clarivate Analytics (CWTS, 2018); this ranking considers a
subset of only the publications in the Science Citation Index Expanded, the Social
Sciences Citation Index, and the Arts and Humanities Citation Index. The CWTS
Leiden Ranking 2018 was calculated with the data of years 2013–2016 for each one
of the bibliometric indicators shown in Table 6.5.
U-Multirank world university rankings use data from 21 different subject areas
and use the performance indicators shown in Table 6.6, which allows comparing
universities on the basis of rankings in each subject area rather than looking at the
whole institution. U-Multirank does not aim to calculate a ranking of universities or
their units but to allow comparison between universities with similar characteristics.
The U-Multirank methodology analyzes university scores on individual indicators
and places them in five performance groups (“very good” to “weak”).
Along with the six methods of ordering and/or comparing the universities men-
tioned, there is an enormous number of rankings and benchmarking methodologies
Table 6.4 Indicators and weights of US News Ranking

Indicator Percentage weighting
Global research reputation 12.5
Regional research reputation 12.5
Publications 10
Books 2.5
Conferences 2.5
Normalized citation impact 10
Total citations 7.5
Number of publications that are among the 10 percent most cited 12.5
Percentage of total publications that are among the 10 percent most 10
cited
International collaboration 5
Percentage of total publications with international collaboration 5
Number of highly cited papers that are among the top 1 percent most 5
cited in their respective field
Percentage of total publications that are among the top 1 percent 5
most highly cited papers
Table 6.5 Criterion and indicators of CWTS Ranking

Criterion Indicators
Research Number of publications
Total and average number of citations, normalized for field and publication year
Number and proportion of publications:
• That belong to the top k% most frequently cited, with k = 1,5,10 and 50
• That have been co-authored with one or more other organizations
• That have been co-authored by two or more countries
• That have been co-authored with one or more industrial organizations
• With a geographical collaboration distance of less than 100 km
• With a geographical collaboration distance of more than 5000 km
of world, national or regional scope that allow comparisons of universities and

their units, such as degrees, areas, or departments. To designate the institutions or
their components that will be evaluated, we will use the term decision-making unit
(DMU).
All the rankings are constructed based on a relatively small collection of
indicators with which the level of quality of the university can supposedly be
measured; as we have seen in the previous examples, they can refer to the missions
of the university, teaching, research, or also to other issues such as knowledge
transfer, or regional engagement.
The Berlin Principles on Ranking of Higher Education is a set of principles of
quality and good practices in HEI rankings approved by the International Ranking
Expert Group (IREG, 2006), but actual ranking practices are very different from
what the principles propose (Barron, 2017).
Table 6.6 Criteria and indicators of U-Multirank

Criteria Indicator
Teaching and Learning Bachelor graduation rate
Masters graduation rate
Graduating on time (bachelors)
Graduating on time (masters)
Research Citation rate
Research publications (absolute numbers)
Research publications (size-normalized)
External research income
Art related output
Top cited publications
Interdisciplinary publications
Post-doc positions
Strategic research partnerships
Professional publications
Knowledge Transfer Co-publications with industrial partners
Income from private sources
Patents awarded (absolute numbers)
Patents awarded (size-normalized)
Industry co-patents
Spin-offs
Publications cited in patents
Income from continuous professional development
Graduate companies
College and university rankings
International Orientation Citations per faculty
Internationalization of Faculty Foreign language bachelor programs
Foreign language master programs
Student mobility
International academic staff
International joint publications
International doctorate degrees
Regional Engagement Bachelor graduates working in the region
Master graduates working in the region
Student internships in the region
Regional joint publications
Income from regional sources
Strategic research partnerships in the region
The rankings and how they are used have received numerous criticisms that can
be summarized into four types (Daraio et al., 2015): monodimensionality, lack of
statistical robustness, dependence on the size of the DMU, and lack of consideration
of the input–output structure. In addition to those who question the very concept of
ranking.
Despite these criticisms, the influence of university rankings continues to
increase, although it must also be recognized that, in general, both the information
on the methodologies and the techniques used are improving.
Regarding the monodimensionality, the main criticism focuses on the fact that
the most important or unique criterion is usually research, with very little influence
of other criteria, for example, learning outcomes. Initiatives such as Assessment of
Higher Education Learning Outcomes (ANHELO) have attempted to determine the
possibility of comparable measurements of learning outcomes in higher education
(Tremblay et al., 2012).The rankings can show a lack of statistical robustness, i.e.,
small changes in the data can produce considerable changes in the final order, and
the removal or inclusion of a DMU can also produce unexpected rank reversals.
Uncertainty and sensitivity analyses can be used to improve stability and to reduce
the uncertainty of ranking systems (Dobrota et al., 2016).
There are many empirical approaches to solve problems that arise from the lack
of consideration of DMU size and the input-output structure, such as parametric and
nonparametric efficiency analysis techniques (Grosskopf et al., 2014).
With regard to the very concept of ranking, the main concern is that the
results will be misinterpreted if they are disseminated through a simplistic analysis.
Benchmarking initiatives, such as U-Multirank, attempt to provide meaningful
information for the general public (Marope et al., 2013).
Benchmarking can be defined as a process facilitating systematic comparisons
and evaluation of practices, processes, and outcomes conducted support improve-
ment and self-regulation (Jackson, 2001). Benchmarking can be the appropriate tool
to improve the HEIs by identifying best practices. According to the terminology of
the European Network for Quality Assurance in Higher Education (ENQA), the real
benchmarking is always improvement oriented. From this point of view, ranking of
universities can be treated as an initial step in benchmarking (Kuzmicz, 2015).
6.3 Training HEI Managers on OR Methodologies
With increasing participation in higher education among young people, govern-

ments around the world are facing increasing pressure on their finances, inducing
universities to operate with a higher degree of efficiency it is one of the causes
of the growth of university evaluation systems throughout the world (Abbott &
Doucouliagos, 2003). The creation of common spaces of higher education, such
as the European Higher Education Area (EHEA), is also a reason to create mutual
trust mechanisms in the correct functioning of HEIs.
Better management and planning in higher education require more information
and research about university inputs and outputs and the relationship between them
(Sinuany-Stern, 1991).
The objective of the chapter is to provide managers and leaders of HEI the
knowledge that allows them to assess the possibilities and limitations of rankings
and benchmarking techniques and how these techniques can be applied properly to
the different levels of higher education systems.
The managers of HEIs need to have competencies in areas such as leadership,

governance, accreditation, institutional research, international cooperation, finance,
facilities, fundraising, human resources, student life, recruitment, and retention.
In many countries, it is not necessary to have specific qualifications to be
members of the HEI director’s board. One of the options for improving the
competencies of directors of these institutions is executive masters designed for
managers or executives with several years of work experience. These programs
allow directors to further develop their skills while largely maintaining their day-
to-day work schedule.
As in the MBA, the core subjects in these programs are analytical (accounting,
economics, operations research, and quantitative analysis), functional (financial
management, human resource management, and operations management), and
ethics (social responsibility, corporate governance).
The design of all learner-oriented curriculum must be considered the heterogene-
ity of students, but this is a crucial aspect in disciplines such as MS/OR that deal
with the application of advanced analytical methods to help make better decisions.
This chapter includes the themes of ranking and benchmarking that have
been included in International Postgraduate Programs in Management of Higher
Education Institutions in Colombia, Dominican Republic, Ecuador, and Spain,
within the AUIP1 network of universities, and Training Programs of the Institute
of Statistics and Cartography of Andalusia.
In Sect. 6.3, we also describe the HEI benchmarking interactive tool (HEIBIT),
a tool developed to facilitate the use of the techniques explained in these programs,
which incorporates the methodology developed in five PhD theses of the doctoral
program in Statistics and Operational Research at the University of Seville (Spain),
(Alvarado, 2016), (Correa, 2016), (Fernández, 2015), (García, 2016) and (Morocho,
2015). The methodology has been applied at the national level in Ecuador for
assets the accreditation and categorization process carried out by the Council for the
Evaluation, Accreditation and Quality Assurance of Higher Education of Ecuador
(CEAACES) in 2008 and 2013, at the regional level to assess the efficiency of
research groups in Andalusia (Spain) (Pino-Mejías et al., 2010), and at school and
institutional level in the Universidad Técnica Particular de Loja (UTPL. Ecuador).
The conceptual framework is based on the standards and guidelines for qual-
ity assurance in the European Higher Education Area (ESG), the standards of
CEAACES, the National Agency for Quality Assessment and Accreditation of
Spain (ANECA), and the National Accreditation Council of Colombia (CAN).
When the ranking and benchmarking processes are carried out by accreditation
and quality assurance agencies, the results have relevant consequences in the HEI.
There are many university systems in which the results of the evaluation process
1 The Asociación Iberoamericana de Postgrado (AUIP) is an international nongovernmental
organization recognized by UNESCO, dedicated to the promotion of postgraduate and doctoral

studies. It is currently composed of almost 190 institutions of higher education in Spain, Portugal,
Latin America and the Caribbean.
have a direct impact on the financing of institutions and may even involve the closure
of programs and even the institution.
To provide users insight into the value and limits of any multicriteria analysis is
necessary to consider, at least, the procedures used to select the variables, normalize
its values, determine the weight of each variable, and the level of compensability
allowed for the aggregation method (JRC, 2008). Another key aspect is the study of
how the position of each institution in the ranking depends on the methodological
options adopted. Sensitivity analysis and uncertainty analysis are useful tools to
assess the robustness of rankings (Saisana et al., 2011).
When information about input and output of a process is available, Data
Envelopment Analysis (DEA) is a well-established nonparametric methodology that
is used to evaluate the efficiency of organizations, called DMUs, which employ
optimization techniques (Ahn et al., 1988). The monitoring of the evolution of
the HEI efficiency can be made using adaptations of the index of change in the
productivity of Malmquist (Wolszczak-Derlacz, 2016).
6.4 Higher Education Institution Benchmarking Interactive

Tool (HEIBIT)
The authors of this chapter have developed a learning tool, which can be accessed
at http://semiestio5.us.es/shiny/HEIBIT/.
The tool is applicable to all levels of higher education: student, faculty, depart-
ment, faculty, university, or national. It is designed to be used by the managers of the
HEI so that they can know the possibilities and limitations of the techniques of OR
/ MS. The tool can be useful to close the gap between decision-makers and analysts
but cannot replace the work of analysts who have the knowledge to apply the most
appropriate technique in each circumstance.
When the ranking or benchmarking intends to build a unique measure to order the
units to which it applies, the Handbook on Constructing Composite Indicators (JRC,
2008) provides a useful guide. The tool developed considers the methodological
recommendations of this handbook.
HEIBIT has been developed in R (R Core Team, 2018), a freely available
language and the environment under the terms of the Free Software Foundation’s
GNU General Public License, which provides a wide variety of statistical, graphical,
and OR/MS techniques and runs on a wide variety of UNIX platforms, Windows,
and MacOS.
HEIBIT is a web-based application that uses the framework of Shiny, an R
package that makes it simple to build interactive web apps straight from R. You
can host standalone apps on a webpage or embed them in R Markdown documents
or build dashboards. You can also extend your Shiny apps with CSS themes,
htmlwidgets, and JavaScript actions (Chang et al., 2018). Figure 6.1 shows the
sidebar and flowchart of the application.
Fig. 6.1 HEIBIT sidebar and flowchart
6.4.1 Indicators Selection and PCA
From a qualitative point of view, the theoretical framework provides the basis for
the selection of indicators. However, you also need quantitative tools to select the
minimum set of indicators for each dimension you want to measure.
The dimensionality reduction techniques help to select a reduced set of indi-
cators, capable of providing relevant information on each criterion. Principal
components analysis (PCA) converts a set of indicators into a smaller number of
uncorrelated components that account for most of the observed variance in the
full set of indicators. The number of components and their correlations with the
indicators help to select the indicators (Adler & Golany, 2002).
In Table 6.7, examples of application to HEI of PCA are shown.
To illustrate the operation of the application, a subset of only the first fifteen
universities of ARWU (2018) has been used. Figure 6.2 shows the ARWU criteria,
indicators, weights, and the input or output type of each indicator.
In Fig. 6.3, the values of the indicators in the HEIBIT app are shown.
The module 8 of HEIBIT allows the data to be downloaded or uploaded.
HEIBIT, using the factoextra package (Kassambara & Fabian, 2017), helps to
easily perform the descriptive analysis of the values of the indicators, the calculation
of the correlations, and the PCA; in Fig. 6.4, the correlations for the indicators of
the example are shown.
Table 6.7 Examples of application to HEI of PCA
First Author Country DMUs (Number Independent Variables Dependent Variables Purpose of the analysis
(Year) (HEI level) of DUMs) (number)
Bobe and Australia Deans (56) 1. Age 3. Educational 1. Use of university Management
Kober (National) 2. Sex background management control
(2018) 4. Tenure systems
2. Financial performance
measures
3. Non-financial
performance measures
Garcia- 11 European Students Competences 4. Generic (5) Natural logarithm of Quality assurance
Aracil and countries (24.414) 1. Organizational (5) 5. Participative (6) income
Van der (Interna- 2. Specialized (2) 6. Socio-emotional (7)
Velden tional) 3. Methodological (8)
(2008)
Han and Canada Students (5.459) Questionnaire with 5. Clicker feedback – Assessment
Finkelstein (University) items: helped me focus on what
(2013) 1. Having to respond to is should be learning in
clicker questions and the course
feedback improved my 6. I found using clicker
attention feedback in the class
2. The use of clicker worthwhile to my
feedback encouraged me learning experience
to be attentive to classes 7. Viewing the class
3. When using clicker answers and feedback to
feedback used in a class, clicker questions helped
I felt more involved me assess my
4. The use of clicker understanding of the
6 Survey of Methods for Ranking and Benchmarking Higher Education Institutions
feedback improved my course content

understanding of the
course content
(continued)
169
Table 6.7 Continued
170
First Author Country DMUs (Number Independent Variables Dependent Variables Purpose of the
(Year) (HEI level) of DUMs) (number) analysis
Miller Canada Students (170) Feedback embedded in a 2. Rephrased the – Assessment
(2009) (Course) formative computer-based question
assessment 3. Provided further
1. Directed to resource information
4. Provided the correct
answer
Salas (2014) Spain Students(3.209) Modes of teaching and Personal and educational 1.Ability to make Quality
(National) learning, and assessment: characteristics: your meaning assurance
1. Lectures 11. Gender clear to others
2. Group assignments 12. Age 2-Ability to use
3. Participation in research 13. Father’s education. time efficiently
projects 14. Secondary school 3.Ability to work
4. Facts and practical grades productively with
knowledge 15. University degree others
5. Theories and paradigms 16. Duration of the 4.Ability to
6. Teacher as the main source degree. perform well
of information 17. Internships under pressure
7. Project and/or 18. Study abroad 5.Ability to
problem-based learning 19. Years of work rapidly acquire
8. Written assignments experience since new knowledge
9. Oral presentations by graduation 6.Ability to
students coordinate
10. Multiple choice exams activities
Tytherleigh UK HEI employee Occupational stress 3. Health (19) – Comparison
et al. (2005) (National) (3.808) questionnaire 4. Supplementary
1. Perceptions of job (37) information (24)
2. Attitudes towards the
organization (9)
J. Pino-Mejías and P. Luque-Calvo
Vázquez Spain Universities (47) Efficiency of 21 DEA 3. Number of seats in – Efficiency
Cueto (2008) (National) models. library
Inputs, combinations of: Outputs, combinations of
1. Number of students 4. Number of graduate
2. Number of professors students
and researchers 5. Theses read
Webster USA Universities 1. Acceptance rate 7. Predicted graduation US News and World Ranking
(2001) (National) (145) 2. Graduation rate rate Report ranking of
3. Alumni giving 8. Average rating of colleges and universities
4. Full-time faculty quality
5. Class size, 1–19 9. Freshman retention
students rate
6. Class size, 50+ 10. SAT scores
students 11. %graduated in the top
10 percent of their
high-school class
Wilkesmann Germany Faculty (3.406) 1. Teaching Motivation 8. Perceived number of Academic teaching Quality assurance
and Lauer (National) Intrinsic structural incentives
(2020) 2. Teaching Motivation 9. Perceived amount of
Identified didactic support
3. Teaching Motivation 10. Vivid exchange on
Introjected teaching matters with
4. Teaching Motivation colleagues
External 11. Constructive
5. Gender feedback from students
6. Discipline
7. Age
Wilson et al. Australia Universities Items of Course – Validation
(1997) (National) (1.993) Experience Questionnaire
(30)
171
172
Fig. 6.2 Indicators description and weights

Fig. 6.3 Value of the indicators
The most correlated indicators for the fifteen universities are 3, 4, and 5 (highly
cited researchers, papers published in Nature and Science, and papers indexed in
SCI-expanded and SSC Index).
The indicators 1 and 2 (alumni and staff winning Nobel prizes and Fields Medals)
also have a high correlation value.
PCA is useful to identify the relationship between indicators, components,
and units. Figure 6.5 shows the projection of indicators and DMUs in the two-
dimensional space generated by the first two components.
The biplot in Fig. 6.5 can be approximately interpreted as follows:
• A DMU that is on the same side of a given indicator has a high value for this
indicator.
• A DMU that is on the opposite side of a given indicator has a low value for this
indicator.
All the indicators are positively related to the first dimension, and Ind4, Ind3, and
Ind5 are negatively related to the second component (Fig. 6.6).
Fig. 6.4 Correlations between indicators
6.4.2 Weighting and AHP
In ARWU, THE, QS, and US News, the weights wi of the indicators Ii have been
set by the organizations that elaborate these rankings, and the global score is the
weighted mean
n
wi Ii
i=1
n
i=1 wi
The weighting process is always conflicting, since the importance that each
person or institution assigned to each criterion or indicator depends on many social,
cultural, economic, or political factors. In the lower part of Fig. 6.2, the user can
select weights that will be used to calculate the weighted mean, those in the “weight”
column of the indicator information table in the same Fig. 6.2, or use the result of
applying the Analytical Hierarchy Process (AHP).
The AHP is a measurement methodology through peer comparisons based on
expert judgments. It was developed by Thomas L. Saaty in the 1970s (Saaty, 1980)
and has been extensively studied and applied since then. An early application of
AHP to HEI can be found in Saaty and Rogers (1976).
The principles of AHP can be applied in the weighting process as follows: if
we have selected n criteria (C1 , C2 , . . . , Cn ), we assume that the decision-maker is
Fig. 6.5 PCA Biplot
Fig. 6.6 Correlations between components and indicators
capable of making a pairwise comparison between the different criteria. As a reason

for this approach, Saaty wrote:
In any comparison pair, the smaller of two elements A and B, say A, is taken as the unit and
the larger one is estimated as a multiple of that unit. If we assume, as we always do, that we
can derive weights, wA and wB for the elements on a ratio scale, then the absolute number
assigned to the larger element is an estimate of this ratio and the comparison takes the form
wA
wB . In the absence of a scale for measuring every element (which, due to the paucity of
known scales, is nearly always the case) this approach is the most precise way to measure
elements with respect to a property. (Saaty, 1994)
Table 6.8 Numerical rating in AHP

Verbal judgments of preferences between criterion i and criterion j Numerical rating
Ci is equally important to Cj 1
Ci is slightly more important than Cj 3
Ci is strongly more important than Cj 5
Ci is very strongly more important than Cj 7
Ci is extremely more important than Cj 9
Intermediate values 2,4,6,8
The comparisons are made using a scale, which represents how much more one
element dominates another with respect to a given attribute. In Table 6.8, we can
see the numerical scales used in the AHP method.
Using the AHP scale, we construct a matrix W, with elements wij = w1ji , ∀i, j,
being all the elements of W positives, by the Perron-Frobenius theorem, there
is always a dominant eigenvalue λ > 0 such that its associated eigenvector has
all its components greater than zero. Therefore, we can use as vector of weights
{wi , i = 1, . . . , n} the normalized eigenvector associated with λ. In general, λ ≥ n,
and λ = n if W is consistent, i.e., when
wi
wij = , ∀i, j
wj
Saaty defined the consistency index (CI) as:
λ−n
CI =
n−1
As a measure of consistency, the ratio CR is defined as the quotient between

CI and an index obtained as the average of the CI values of a set of matrices with
elements aij = 1/aji generated randomly. A value of the consistency ratio CR 0.1
is considered acceptable. Larger values of CR require the decision-maker to revise
the scores.
If a criterion needs to be measured with more than one indicator, the same method
can be used for the weighting of each indicator. If wi is the weight of criterion Ci , and
pij the weight of the indicator Iij (j = 1, . . . ,ni ), with respect to the set of indicators
nj
of Ci, the global weight of Iij will be ψ ij = wi pij , and ni=1 j =1 ψij = 1. We
n
denote by N the total number of indicators, N = I =1 ni .
As an example of the use of AHP in Fig. 6.7, we can see the matrix of pairwise
comparisons that produces the same weights as those established in ARWU.
In HEIBIT, AHP is used as a method for weighting the criteria. When some of
the criteria are defined as outputs and others are defined as inputs, their indicators
can be normalized as outputs before the aggregation step.
In Table 6.9, examples of application to HEI of the AHP are shown.
Fig. 6.7 AHP example
6.4.3 Normalization
The indicators are usually measured in different scales, so we need to select a

normalization method. ARWU normalizes each indicator Ii with the transformation:
Ii − min (Ii )
i
∗ 100
max (Ii ) − min (Ii )
i i
Therefore, the rank for each indicator of ARWU is [0, 100].

In general, when the indicator is the measure of an input and the best DMU is,
therefore, the one that has the minimum value for the indicator; the transformation
is:
max (Ii ) − Ii
i
∗ 100
max (Ii ) − min (Ii )
i i
In THE ranking, for example, the citation indicator is normalized to reflect

variations in citation volume between different subject areas. U-Multirank applies
178
Table 6.9 Applications of AHP method to HEI

First Author Country (HEI Alternatives (Number of Criteria (Number of Purpose of the analysis
(Year) level) Alternatives) indicators)
Begičević et al. Croatia E-learning 1. Strategic readiness for 4. Legal and formal Prioritization
(2007) implementation strategies e-learning readiness for e-learning
(4) implementation (3) implementation (2)
2. Basic ICT 5. Specific ICT
infrastructure for infrastructure for
e-learning (4) e-learning (6)
3. Human resources (6)
Feng et al. (2004) China (National) Universities (26) 1. Full-time R&D 4. International Efficiency
personnel (8) communications (3)
2. R&D expense (5) 5. Awards (3)
3. Fruits of science and 6.Income of technology
technology (10) transference (1)
Grandzol (2005) USA (School) Faculty (8) 1.Experience 2.4 Collaboration (1) Selection
1.1 Education (3) 3. Technological Skills
1.2 Teaching Skills (3) 3.1 GP and Application
1.3 Business (3) Software (1)
2. Scholarly Activities 3.2 Web and Online
2.1 Research Record (3) Course Delivery (1)
2.3 Research Potential (3) 4. Flexibility in Teaching
Capabilities (1)
5. Experience with
Diverse Populations (1)
Lee (2010) Taiwan Intellectual capital 1. Administration (1) 4. Research (1) Assessing
(University) Indicators (7) 2. Curriculum (1) 5. Teaching (1)
3. Technology transfer (1) 6. Service (1)
Liberatore and USA (University) Research papers (15) 1. Objectives (1) 4. Execution-Implementation Ranking
Nydick (1997) 2. Justification (1) (1)
3. Design (1) 5. Recommendations and
implications (1)
Rad et al. (2011) Iran (National) University majors (177) 1. Finance and Economics (1) 5. Services (1) Ranking
2. Society and Religion (1) 6. Agriculture (1)
3. Industry (1) 7. Medicine (1)
4. Politics (1) 8. Environments and natural
sources (1)
Saaty and Rogers USA (National) Scenarios (7) 1. Economic 1.6 Industry Planning
(1976) 1.1 Students (3) 2. Political (with the same 6
1.2 Faculty (4) sub-criteria and indicators)
1.3 Administration (2) 3. Social (with the same 6
1.4 Government (6) sub-criteria and indicators)
1.5 Private sector (4) 4. Technological (with the
same 6 sub-criteria and
indicators)
Tsinidou et al. Greece University (1) 1. Academic staff (6) 5. Location (3) Evaluation
(2010) (Universities) 2. Administration services (7) 6. Infrastructure (7)
3. Library services (4) 7. Carrier prospects (5)
4. Curriculum structure (7)
Turan et al. Turkey Investment projects (5) 1. Teaching (4) 3. Service and Social Selection
(2016) (University) 2. Research and Development Responsibility (3)
(4) 4. Finance (1)
Wu et al. (2012) Taiwan Universities (12) 1. Teaching Resources (6) 4. Discipline and Guidance Ranking
(Universities) 2. Internationalization (7) (4)

3. Extension Education 5. General Education (3)
Service (4) 6. Administrative Support (4)
179
different standardization procedures for quantitative indicators (e.g., number of

publications or master’s graduation rate) or qualitative (reputation surveys based on
Likert scales), with the objective of classifying each university for each indicator in
a group from the best group “A” to the lowest group “E.” Given the great diversity
of scales found in the review of the indicators used in university rankings, it is
useful to employ a common framework that allows the normalization of any type of
numerical indicator.
For an indicator Ii and two DMUs u and v, if d= Ii (u)− Ii (v), we can choose a
function fi (d) as a measure of the preference of decision-maker for u or v.
The preferences are always subjective; when we start the construction of a
ranking, we cannot know a priori that each decision-maker prefers. Different
managers of the same organization may have very different preferences (Cecconi
et al., 2007). In the literature, we can find a wide variety of functions used
to represent preferences. Among all of them, we have chosen the six types of
functions that were initially used in the Preference Ranking Organization Method
for Enrichment Evaluations (PROMETHEE) (Brans & Vincke, 1985). For these
preference functions, a value of 0 means that there is no preference among the units
being compared, while a value of 1 means an absolute, indisputable preference for
one of the units.
• Type I—Usual preference function

0 d≤0
PI (d) =
1 d>0
Two units with positive differences generate a full preference for the better, even
if the difference is very small.
• Type II—U-shape preference function

0 d≤q
PI I (d) =
1 d>q
Two units with close values (difference ≤ q) are indifferent. Two units with more
different values (difference > q) generate a full preference.
• Type III—V-shape preference function
⎧
⎪
⎨0 d ≤0
PI I I (d) = d
0≤d ≤p
⎪
⎩1
p
d >p
Two units with smaller different values (difference ≤ p) generate a preference

degree proportional to the difference.
• Type IV—Level preference function

⎧
⎨0 d≤q
PV I (d) = 1
q≤d ≤p
⎩ 2
1 d>p
Two units with very close values (difference ≤ q) are indifferent.

Two units with quite different values (difference > p) generate a full preference.
In between, two units with different values (q < difference ≤ p) generate a weak
preference degree (preference degree = 1/2).
• Type V—Linear preference function
⎧
⎪
⎨ 0 d≤q
d−q
PV (d) = q ≤d ≤p
⎪
⎩ 1
p−q
d>p
Two units with very close values (difference ≤ q) are indifferent.

Two units with quite different values (difference > p) generate a full preference.
In between, two units with different values (q < difference < p) generate a
preference degree that is linearly increasing from 0 to 1 as the difference is
increasing from q to p.
• Type VI—Gaussian preference function

0 d ≤0
PV I (d) = 2
− d2
1−e 2s d>0
The Gaussian preference function has been designed as an alternative to the

linear (type V) preference function. It has a smoother shape without flat indifference
and full preference areas. At the same time, it does not have any indifference or
preference thresholds. Its shape is determined by another parameter: the s Gaussian
threshold, which defines the position of the inflection point of the preference
function curve.
Figure 6.8 shows examples of the six types of functions using as values of the
parameters q = 5, p = 10, and s = 3.
6.4.4 Aggregation via PROMETHEE
The additive approach is the most used in the aggregation phase. Multiplicative
aggregation has also been proposed (Leskinen & Kangas, 2005), (Tofallis, 2012),
and the use of outranking techniques is also possible. HEIBIT uses PROMETHEE
to obtain a global score.
Fig. 6.8 Examples of preference functions
The first step in PROMETHEE consists of choosing, for each criterion, a

preference function. This is used to compute, for each pair of alternatives u,v, a
number between 0 and 1 representing a preference degree. In our application, we
use the functions selected in the normalization phase to be associated with each
indicator Iij .
At the end of the first step, we, therefore, have a fuzzy preference relation for
each indicator Pij , which is the fuzzy relation associated with the indicator Iij, and
Pij (u, v) is the value of this relation for the pair u, v.
In the next step, these fuzzy relations are aggregated by means of a generalization
of the Borda method (Marchant, 1996), obtaining a multicriteria preference index
π(u,v) of u over v, which takes all preferences into account with the expression
n nj
π (u, v) = wij Pij (u, v)
i=1 j =1
The values π(u,v) and π(v,u) are computed for each pair of units, if U is the set
of units to be evaluated, the positive flow
1
φ + (u) = π (u, v)
n−1
v∈U,v=u
represents what extent each alternative outranks all the others.

In the same mode, the negative flow
1
φ − (u) = π (u, v)
n−1
u∈U,u=v
represents to what extend each alternative is outranked by all the others.

In PROMETHEE II, the net flow φ(u) = φ + (u) − φ − (u) allows us to aggregate
all the indicators considering the weights and to order the units.
The PROMETHEE methods can be used (De Keyser & Peeters, 1996) if the
decision-maker is aware of the fact that the weights represent compensations and
knows what can happen if one or more units are added or eliminated. We address this
issue later when we address the limitations of multicriteria techniques and sensitivity
analysis.
Table 6.10 shows examples of application to HEI of the PROMETHEE methods
are shown.
In Fig. 6.9, the result of PROMETHEE II with the sample data and the usual
preference criterion are shown.
6.4.5 Efficiency Analysis via DEA
HEIs are not classic firms whose aim is profit maximization; public HEIs, in
particular, are by definition nonprofit organizations. Hence, we cannot assess
their productivity by using the methods as cost-benefit analysis applied to the
evaluation of companies producing goods or services and generating profit (Parteka
& Wolszczak-Derlacz, 2013). The DEA has been used frequently to evaluate
the efficiency of universities or their components (for example, degrees, research
and training units, departments, laboratories, research centers). Berbegal and Solé
(2012) performed an exhaustive review of the indicators used in DEA empirical
studies in the first decade of the twenty-first century, classified by units of analysis,
financial measures, human capital, infrastructures, teaching measures, research
measures, knowledge transfer activities and overall measures (e.g., external rankings
and internal surveys). Alvarado (2016) makes a review of the applications of this
technique in universities and shows that the inputs considered most frequently are
the number of professors and researchers, total funding and research expenses.
DEA was first introduced by Charnes, Cooper, and Rhodes (Charnes et al., 1978).
It is a simple, yet powerful, method used to measure the relative efficiency of a group
of homogenous firms or DMUs. A DMU can be defined as an entity responsible for
converting input(s) into output(s) and whose performances are to be evaluated. The
184
Table 6.10 Examples of application to HEI of the PROMETHEE methods

First Author (Year) Country (HEI Alternatives Criteria Purpose of the analysis
level) (Number of
Alternatives)
Bedir et al. (2016) Turkey (School) Postgraduate 1. Content of the Course 4. Instructor’s Attitude Selection
Courses (6) 2. Day of Course 5. Academic Expectations
3. Interests of Students
Chakraborty et al. (2017) India (National) States (28) 1. Number of primary 8. Average dropout rate in Benchmarking
schools per sq. km secondary level
2. Gross enrollment ratio 9. Literacy rate
(GER) in primary level 10. Number of colleges
3. Student-teacher ratio in per sq. km.
primary level 11. GER in colleges
4. Average dropout rate in
primary level
5. Number of secondary
schools per sq. km.
6. GER in secondary level
7. Student-teacher ratio in
secondary level
Colson (2000) Belgium OR Student’s 1. Applicability 2.2. the scientific value Ranking
(National) final work (8) 1.1 its applicability to and adequacy of the
practice chosen OR methodology
1.2 its link with industrial 3. General form
applications 3.1. the quality of the
2. Contents style
2.1 the consistency of the 3.2. the general quality
technical content and the cleverness of
presentation
First Author (Year) Country (HEI Alternatives Criteria Purpose of the analysis
level) (Number of
Alternatives)
Fadlina et al. (2017) Indonesia Students (4) 1. Grade Point Average 3. Status of Short Ranking
(School) 2. Leave status repair/Short semester
4. Active Organizing
Jati (2012) Indonesia Teachers (3) 1. Publication 6. Teaching Attitude Evaluation
(School) 2. Research 7. Teaching Content
3. Conferencing 8. Teaching Method
4. Consultancy 9. Teaching Effect
5. Services
Jati and Dominic (2017) Indonesia University 1. Visibility. 3. Openness. Ranking
(National) Websites (27) 2. Presence. 4. Excellence.
Özerol and Karasakal USA (National) Full-time MBA 1. Alumni 6. International faculty Ranking
(2008) programs (9) recommendation (%)
2. Career progress 7. International mobility
3. Faculty with doctorates rank
(%) 8. International students
4. FT Doctoral rank (%)
5. FT Research rank 9. Weighted salary
Pekkaya (2015) Turkey (School) Career 1. Career opportunities 4. Job flexibility Ranking
Alternatives (8) 2. Job security 5. Personal issues
3. Job benefits and perks 6. External influences
Živković et al. (2017) Serbia Strategic goals Results of evaluating the and 5 Threats factors on Planning
(University) (6) influence of 20 Strengths the realization of
factors, 14 Weakness Strategic Goals, provided

factors, 5 Opportunities by 4 Decision-Makers
factors
185
Fig. 6.9 PROMETHEE II example
popularity of DEA is due to its ability to measure relative efficiencies of multiple-

input and multiple-output DMUs without prior weights on the inputs and outputs
(Kuah & Wong, 2011).
In the efficiency analysis, you can choose an input orientation in regard to
producing a fixed output minimizing the inputs or an output orientation when you
want to maximize the outputs by fixing the inputs.
The most basic DEA model is known as the CCR model, which was named after
the three authors. If we have m DMUs, with ne inputs indicators Xe and ns outputs
indicators Ys , ne + ns = n, each university (degree, etc.) γ can be considered as a
DMU that uses the inputs Xγ 1 , . . . , Xγ ne and generates the outputs Yγ 1 , . . . , Yγ ns ,
γ = 1, . . . ,m.
Assume the input weights ϕγ k (k = 1, . . . , ne ) and the output weights ψ γ l (l = 1,
. . . , ns ), we use the benefit of the doubt approach, so we want to obtain the best
combination of these weights by solving the linear programming below, which is
known as the multiplier form in DEA
ns
τ1 uγ = max ψγ l Yγ l
l=1
ne
s.t. ϕγ k Xγ k = 1
k=1
ns ne
ψγ l Yγ l − ϕγ k Xγ k ≤ 0
l=1 k=1
ϕγ k , ψγ l ≥ ε, for γ = 1, 2, . . . , ne , k = 1, . . . , n, and l = 1, . . . , ns.
We can fix ε ≥ 0, as the smallest value allowed for the weights.

The model is run m times to identify the relative efficiency scores of all the
γ
DMUs. For each DMU, DEA selects a set of input weights ϕk and output weights
γ
ψl that maximize its efficiency score eγ . The efficiency scores would fall between
0 and 1. Generally, we say that a DMU is efficient if it obtains the maximum
score of 1; else, it is inefficient. For every inefficient DMU, DEA identifies a set of
corresponding efficient DMUs that can be utilized as benchmarks for improvement.
If we need a unique set of weights, a cross-validation approach is to use as
common weights the mean of the m weights obtained for each indicator.
m γ
γ =1 ψl
ψl = l = 1, . . . , ns ,
m
m γ
γ =1 ϕk
ϕk = l = 1, . . . , ne
m
And use has a composite indicator for the DMU uγ
ns γ
ψl Yl
τ2 uγ = l=1
ne γ
k=1 ϕ k Xk
Fig. 6.10 CCR versus BCC production frontier
In the CCR model, a constant returns to scale (CRS) is assumed. A separation

into technical and scale efficiencies is accomplished by the variable returns to scale
(VRS) model developed by Banker et al. (1984). By way of illustration, when ne = 1
and ns = 1, Fig. 6.10 represents how the production frontier is calculated according
to the BBC and CCR models.
When ne = 1, the benefit of the doubt approach can also be used with all the
indicators treated as outputs and a dummy input equal to one for all DMUs, and the
CCR model is:
ns
max ψγ l Yγ l
l=1
ns
ψγ l Yγ l ≤ 1
l=1
ϕγ k , ψγ l ≥ ε ≥ 0
The production frontier with CCR and BCC models are convex, less restrictive
assumptions are the free disposability of input or the free disposability of outputs.
When we combine the two assumptions, we derive the assumption of the free
disposability hull (FDH). An example of the application of DEA with FDH is the
Aquameth benchmarking project, based on the collection and integration of a large
dataset covering universities in the United Kingdom, Spain, Italy, Norway, Portugal,
and Switzerland (Bogetoft & Otto, 2011).
The package Benchmarking (Bogetoft & Otto, 2015) can be used in interactive
mode in HEIBIT. In Fig. 6.11, we can see that the DEA method applies the benefit
of the doubt approach, finding the best possible weights for each unit; therefore, the
Fig. 6.11 DEA (VRS) scores of ARWU example with all indicators of type output
unit UD09 that has a 100 in the indicator IND6 (Fig. 6.3) reaches, together with the
unit UD01, the maximum punctuation.
If we want to use the institution’s endowment (in $ million) as input, we can add
the new indicator and its values (Fig. 6.12).
With this input and one output obtained as the sum of the ARWU indicators, the
production frontier is shown in Fig. 6.13.
Using the endowments as inputs and the six indicators of ARWU as output, the
efficiency scores with the DEA (VRS) method are those shown in Fig. 6.14.
In this case, with the benefit of the doubt approach, only six institutions do not
achieve maximum efficiency.
The DEA does not rank the DMUs that received a score of 1 (efficient DMUs),
so that it is not possible to obtain a full ranking. A revision of techniques for ranking
Fig. 6.12 Endowments 2018 (CAM, 2018; OX, 2018; TBS., 2018)
Fig. 6.13 Production frontier Endowment—Sum of ARWU indicators
in the DEA context can be found in Adler et al. (2002) and in Hadad and Hanani
(2011).
In Table 6.11, examples of application to HEI of the DEA methods are shown;
more applications can be found in Thanassoulis et al. (2016).
6.4.6 Dynamic Efficiency Analysis via Malmquist Index
If a DMU produces a vector of output Yt using a vector of inputs Xt in time t, and

we want to compare the efficiency in two times t1 and t2, the Malmquist index can
Fig. 6.14 Efficiency scores. Endowment—ARWU indicators
be used to assess the evolution of DMU efficiency using DEA (Färe & Grosskopf,
1996).
Let Dt1 be the efficiency with respect to Ft1, and the frontier of production at t1.
The change in efficiency with respect to Ft1 is
Dt1 (Yt2 , Xt2 )

Dt1 (Yt1 , Xt1 )
and with respect to Ft2
Dt2 (Yt2 , Xt2 )

Dt2 (Yt1 , Xt1 )
the Malmquist index by definition is the geometric mean of the two changes in
efficiency
1
Dt1 ( Yt2 , Xt2 ) Dt2 ( Yt2 , Xt2 ) 2
M (Yt1 , Xt1 , Yt2 , Xt2 ) =
Dt1 ( Yt1 , Xt1 ) Dt2 ( Yt1 , Xt1 )
An equivalent way to write this is
Dt2 ( Yt2 , Xt2 ) Dt1 ( Yt2 ,Xt2 ) Dt1 ( Yt1 ,Xt1 )

1
2
M (Yt1 , Xt1 , Yt2 , Xt2 ) =
Dt1 ( Yt1 , Xt1 ) Dt2 ( Yt2 ,Xt2 ) Dt2 ( Yt1 ,Xt1 )
Table 6.11 Applications of DEA methods to HEI
192
First Country DMUs Inputs Outputs Purpose of the

Author (HEI level) (Number of analysis
(Year) DMUs)
Aoki et al. Japan Universities 1. Number of 4. Grant-in aid 1. Maximum deviations 5. Number of books Efficiency
(2010) (National) (8 models) faculties for management 2. Number of papers 6. Grant-in-aid for
(31) 2. Number of 5. General and 3. Number of graduate students scientific research
workers administrative 4. Number of undergraduate 7. Contract research
3. Education and expenses Students funds
research expenses 6. Profit from 8. Profit from business.
donations
Agasisti Italy Universities 1. Number of 4. Financial 1. Number of graduates Efficiency
and Pérez- Spain (57 Italian, students resources 2. Amount of external resources
Esparrells (Interna- 46 Spanish) 2. Number of available attracted to research activities
(2010) tional) Ph.D. students
3. Number of
professors
(academic staff).
Avkiran Australia Universities 1. Academic Model 1 2. Student progress rate Efficiency
(2001) (National) (36) staff, FTE 1. Undergraduate enrolments 3. Graduate full-time
2. Non-academic 2. Postgraduate enrolments employment rate
staff, FTE 3. Research quantum (research Model 3
component of Federal funds) 1. Overseas fee-paying
Model 2 enrolments
1. Student retention rate 2. Non-overseas
fee-paying enrolments
Beasley UK Departments 1. General 1. Undergraduates number 4. Total number of Efficiency
(1990) (National) (52 expenditure 2. Postgraduates on taught course postgraduates
Chemistry 2. Equipment 3. Number of postgraduates who 5. Research rating: Star
and 52 expenditure are doing research 6. Research rating: A+
Physics) 3. Research 7. Research rating: A
income 8. Research rating: A-
Façanha Brazil Universities Tree factors obtained 9. Academic staff of Tree factors obtained 6. Number of master Efficiency
et al. (National) (52) from second and first-degree from degree thesis approved
(1997) 1. Area of buildings teaching 1. Number of 7.Number of doctoral
2. Area of hospitals 10. Administrative undergraduate courses dissertations approved
3. Area of laboratories personnel at support 2. Number of graduate 8. Weighted average or
4. Total number of level courses—master degree MEC’s evaluation:
students 11. Administrative level master degree courses
5. Academic staff with personnel with 3. Number of graduate 9. Weighted average of
doctoral degree high-school degree courses—doctoral MEC’s evaluation:
6. Academic staff with background degree level doctoral degree courses
master degree 12.Administrative 4. Certificates
7. Academic staff with personnel with issued—undergraduate
specialization degree undergraduate degree or degree
8. Academic staff with higher 5. Certificates
undergraduate degree 13. Budget for current issued—medical schools
expenses residence
14. Incoming students at
undergraduate level
15. Incoming medicaI
residents
Ferrari Italy Graduates Average number: 6. . . . of journals and 1. Average grade of the Efficiency
and (University) (2.236) 1. . . . of full and reviews in the library per exams
Laureti associate professors per student 2. Inverse of the length
(2005) graduate 7. . . . of university of study time
2. . . . of researchers per pieces of furniture per
graduate student
3. . . . of seats in lecture 8. . . . of university
halls per student pieces of equipment
4. . . . of lecture halls per 9. Final high-school

student mark
5. . . . of books in the
library per student
193
(continued)
Table 6.11 Continued
194
First Country (HEI DMUs Inputs Outputs Purpose of the

Author level) (Number of analysis
(Year) DMUs)
Johnes China Universities 1. Full-time staff to 4. Research expenditure 1. Prestige of the HEI Efficiency
and Li (National) (109) student ratio 5. Library books 2. Total number of
(2008) 2. % of the faculty 6. Area of the buildings research publications
with associate 3. Publications per
professor position or member of academic
higher staff
3. Proportion of all
students who are
postgraduates
Köksal Turkey Departments 1. Academic staff 3. Entering students 1. Research activities 3. Other activities Efficiency
and (School) (14) salaries and quality 4. Graduates
Nalçaci 2. Potential of the 2.Education activities
(2006) department and quality
Pino- Spain Research 1.Number of active 3.Amount received in the 1.Income from public 3.Total thesis number Efficiency
Mejías et (University) groups (86) doctors call for grants to research financing projects 4.Total other projects
al. (2010) 2.Other research groups 2.Publications in ISI 5.Total number of
staff contracts
Sinuany- Israel Departments 1. Operational 1. Grant money 4. Number of credit Efficiency
Stern et (University) (21) expenditures 2. Number of hours given by the
al. (1994) 2. Faculty salaries. publications department.
3. Number of
graduate students
Torre et Spain Universities 1. Enrolled students 3. Students graduated in 1.Number of 2. Number of national Efficiency
al. (2016) (National) (47) in undergraduate undergraduate and Master publications in the patents and patents
and Master studies studies Web of Science-ISI with a Patent
2. Academic staff Cooperation Treaty
full-time equivalent international
extension
Fig. 6.15 Efficiency, technology, and productivity change
Worthington and Lee (2008), following Coelli et al. (2005), use Fig. 6.15 to
illustrate the case of one input and one output with the constant returns of scale
and orientation output.
If the frontiers of production are Ft1 (x) = β t1 x and Ft2 (x) = β t2 x,the first term
of the Malmquist index is the change in efficiency on the period (t1,t2):
yt2 /xt2
Dt2 ( yt2, xt2 ) βt2
= yt1 /xt1
Dt1 ( yt1, xt1 )
βt1
And the second term is the frontier shift in this period.

1
βt2
yt2 /xt2 yt1 /xt1 2
βt1 βt1
=
yt2 /xt2
βt2
yt1 /xt1
βt2 βt1
In Table 6.12, examples of application to HEI of the Malmquist index method

are shown.
In general, the Malmquist productivity change index decomposes multiplica-
tively into an efficiency change component (EFFCH) and a technical change
component (TECH). HEIBIT uses the package productivity (Dakpo et al., 2018)
to obtain the Malmquist index and the two components
Table 6.12 Applications of the Malmquist index method to HEI
First Author Country DMUs Inputs Outputs Purpose of the
196
(Year) (HEI level) (Number of analysis

DMUs)
Abbott and Australia Colleges of 1. Number of 1. Number of students Productivity
Doucouliagos (National) advanced academic staff change
(2001) education 2. Number of
(25) general staff
3. Capital
expenditures
Agasisti and England Universities 1. Number of 3. The number 1. Number of graduates 2. Total amount of Comparing
Dal Bianco Italy (Inter- (127 English, students of PhD students external grants and
(2009) national) 58 Italian) 2. Total amount of 4. Number of contracts for research
financial academic staff
resources/incomes
Agasisti and Germany Universities 1. Number of 3. Expenditures 1. Number of university 2. Amount of money Comparing
Pohl (2012) Italy (Inter- (69 German, students degrees awarded by for research grants
national) 53 Italian) 2. Number of students in the given year and contracts.
professors
Barra and Italy Departments 1. Equivalent 2. Total amount Model 1 Model 4 Productivity
Zotti (2016) (University) (28) personnel of financial 1. Number of publications 4. Capacity of change
(professors resources spends (articles in international attracting resources
+0:8*associate on research journals +0:75*articles in index
professors activities national Model 5
+0:6*researchers journals+0:5*articles in 5. Research
+0:4*assistant international books productivity per cost
professors +0:25*articles in national of the
+0:2*non − aca- books) academic staff index
demic Model 2 Model 6
staff) 2. External research Outputs 1 to 5.
funding obtained by the
university
Model 3
3. Productivity index
Castano and Philippines Universities 1. Number of 3. Operating 1. Student enrollment 3. Total revenue Performance
Cabanda (National) (30) faculty members expenses 2. Graduates per year evaluation
(2007) 2. Property,
plant, and
equipment
Førsund and Norway Universities 1. Number of 3. Current 1. Final exams, short 3. Research Efficiency
Kalhagen (National) (100) academic staff expenses other studies publications evolution
(1999) (FTE) than salaries 2. Final exams, long
2. Number of 4. Building size studies
non-academic in square meters
staff (FTE)
full-time
equivalents
Hu et al. China Universities 1. Number of 1. Number of 3. Number of CSSCI Comparison by
(2017) (National) (211) R&D staff with monographs articles university type
senior title 2. Number of SSCI and geographical
2. R&D grants articles area
Johnes UK Universities 1. Number of 3. Expenditure 1. Number of first-degree 3. Income received in Productivity
(2008) (National) (112) full-time incurred on and other undergraduate funding council change
academic staff centralized qualifications awarded grants plus income
plus 0.5 times academic 2. Number of higher received in research
the number of services degree qualifications plus grants and contracts
part-time 4. Number of total other postgraduate
academic staff FTE first-degree qualifications awarded
2. Expenditure and other
on total undergraduates.
administration 5. Number of
and central FTE

services postgraduate
students
(continued)
197
198
Table 6.12 Continued

First Author Country DMUs Inputs Outputs Purpose of the analysis
(Year) (HEI level) (Number of
DMUs)
Parteka and Austria, Universities 1. Number of 3. Total 1. Number of Efficiency
Wolszczak- Finland, (266) students revenues publications evolution
Derlacz Germany, 2. Total 2. Number of graduates
(2013) Italy, academic staff
Poland, UK,
and
Switzerland
(Interna-
tional)
Thanassoulis UK Universities 1.Total 1. Full-Time Equivalent 5. Research funding Cost Efficiency
et al. (2011) (National) (121) operating cost undergraduate number 6.”Third mission” or
(FTEUN) in medicine ‘knowledge
2. FTEUN in sciences transfer” activity,
3. FTEUN in measured by income
non-sciences from other services
4. FTE postgraduate rendered
number
Torre et al. Spain Universities 1. Enrolment 1.Graduates Comparison
(2017) (National) (47 public, 2. Academic 2. Publications (Web of public-private
22 privates) staff FTE Science)
In Figs. 6.3 and 6.12, we have the outputs and input used to illustrate the DEA
application: the values of the ARWU indicators and the endowments in 2018. To
obtain the Malmquist index, we need the data of another year. In Fig. 6.16, the
values of the same input and outputs in 2017 are shown.
The values of the Malmquist index, efficiency change component, and technical
change component for years 2017 and 2018 appear in Fig. 6.17. The last row
contains the mean values.
The values of technical change lower that 1 are indications of negative “frontier
shift;” only University of Chicago increased its total factor productivity since the
growth of efficiency was enough to compensate the technical change.
6.4.7 Global Sensitivity Analysis
If we have two rankings produced by two sets of methodological decisions, to

measure the dissimilarity between the two orders, we must use a procedure that
obtains a 0 for the case in which the order does not change and reaches its maximum
when the resulting order is the inverse of the original, that is, when the first unit
passes to the last and the last to the first. If we denote Or (uγ ), the order of the DMU
γ is in the ranking r; an example of this type of function is:
1
m

R= O 1 u γ − O 2 u γ ,
m
γ =1
where |O1 (uγ ) − O2 (uγ )| is the number of positions that change, up or down, unit
γ from one ranking to another, m is the number of units, so R is the average
number of changes. With this notation, the sensitivity analysis tries to identify
the influence of methodological choices on R; a multimodeling approach can be
applied, exploring plausible combinations of the main methodological choices in
the steps on constructing the ranking (Paruolo et al., 2013) and to employ global
sensitivity methods, such as nonparametric, density-based, variance-based, moment
independent and value of information-based sensitivity measures, (Sobol, 1993),
(Borgonovo & Plischke, 2016).
For each m, the range of R is different, so we can denote R∗ (m) to the maximum
of R for each m and use R s = R ∗R(m) as the normalized measure of the change in
order.
In Table 6.13, the examples of application to HEI of uncertainty and sensitivity
analysis methods are shown.
In HEIBIT, a graphical comparison of the ranking produced by the weighted
mean (with the weights obtained from AHP), PROMETHEE II, and DEA methods
is incorporated; Fig. 6.18 shows the graphical comparison of the classifications
obtained by the three methods, using the labels of the DMUs of ARWU (2018).
For example, the values of dissimilarity index R s are shown in Fig. 6.19.
Fig. 6.16 Input and outputs 2017

Fig. 6.17 Malmquist index 2017 to 2018
The coefficient of correlation can be used as a measure of similarity of the

different rankings. In Fig. 6.20, we can see how we have high correlations for low
values of the dissimilarity index.
The Friedman test is a nonparametric statistical test similar to the parametric
repeated measures ANOVA that can be useful as a measure of how close the various
rankings are.
In Fig. 6.21 the result of the R-function Friedman test for the four rankings of the
fifteen universities are shown.
This means that the Friedman test did not find statistically significant differences
between the four rankings.
6.5 Main Drawbacks of OR/MS Techniques in Ranking

and Benchmarking of HEI Education Institutions
In a ranking method, a rank reversal is a change in the order of the units when, for
example, the set of units that are being ordered increases or decreases. The axiom
of independence of irrelevant alternatives (IIA) says that “if unit u is preferred to v
out of the choice set {u,v}, introducing a third option x, expanding the choice set to
{u,v,x}, must not make v preferable to u.”
Table 6.13 Applications of uncertainty and sensitivity analysis methods to HEI
202
First Author Country DMUs (Number of Indicators Purpose of the

(Year) (HEI level) DMUs) analysis
Dobrota and International Universities (40) Quacquarelli Symonds World University 3. Student-to-faculty ratio Ranking
Dobrota (Universi- Ranking (QS) 2013 Indicators 4. Citations per faculty
(2016) ties) 1. Academic reputation 5. International faculty ratio
2. Employer reputation 6. International student ratio
Dobrota and International Universities (20) Academic Ranking of World Universities Alternative Ranking (ARWU Robustness
Dobrota (Universi- (ARWU) indicators Excluding Award Factor)
(2016) ties)l
Dobrota and International Universities QS indicators 2. Article Impact Total Robustness
Jeremic (Universi- (field of Computer Science University Ranking by Academic Performance 3. Citation Impact Total
(2017) ties) and Information Systems (URAP) indicators: 4. International Collaboration
in QS 1. Article
Information and
Computing Sciences in
URAP) (15 Top-ranked
and 8 Indian universities)
Gnaldi and Italy Universities (56) 1. For each study program, average number of 5 Percentage of graduated Robustness
Ranalli (National) permanent teachers belonging to the basic students who were employed
(2016) subject areas which qualify the educational 3 years after graduation
profiles offered by the programs 6. Composite Indicator of
2. Ratio between the number of enrolled Research quality
students to the second academic year who 7. Economic Valorization of
acquired at least 2/3 of the available credits and Research Products Index
the total number of matriculated students 8. Ratio between number of
3. Ratio between the number of credits acquired permanent teachers taking
by enrolled students and the total number of part in “Projects of National
available credits Relevant Interest” and the
4. Ratio between the number of study program total number of permanent
which collected data on “Students’ opinions on teachers.
study programs” and the total number of study 9. Indicator of European
programs in the same academic year funds and projects obtained
for research
Lin et al. International Universities (field of Indicators of Higher Education Evaluation 2. Straight counting using Robustness
(2013) (Universi- Physics) (299) and Accreditation Council of Taiwan first author
ties) University Ranking (2011 HEEACT) with 3. Straight counting using
four counting methods of 1988–2008 physics corresponding author
papers records indexed in ISI WoS. 4. Fractional counting
1. Whole counting
Maričić et al. International Universities (500) ARWU indicators and Alternative Ranking Robustness
(2017) (Universi- (ARWU Excluding Award Factor)
ties)
Paruolo et al. International Universities (500) ARWU and THES indicators Robustness
(2013) (Universi-
ties)
Saisana et al. International Universities (50) ARWU and THES indicators Robustness
(2011) (Universi-
ties)
Shehatta and International Universities (100) ARWU, QS, THES, US News and World Comparing
Mahmood (Universi- Report Best Global University Rankings
(2016) ties) (USNWR), National Taiwan University
Ranking (NTU), and URAP
Waheed et al. Canada Universities (11) Driving forces (7 indicators) Exposure (12 ind.) Assessment
(2011) (Universi- Pressure (15 ind.) Effects (7 ind.)
ties) State (15 ind.)
Zornić et al. International Universities (field of QS indicators and University Ranking by Comparing
(2016) (Universi- Education) (15) Academic Performance (URAP) indicators

ties)
203
Fig. 6.18 Sensitivity analysis example
The Arrow theorem (Arrow, 1963) was developed for voting systems and can be
used for our problem as follows. If the number of indicators is finite and there are
at least three units, no aggregation method can simultaneously satisfy universality,
transitivity, unanimity, IIA, and nondictatorship.
Violations of IIA are particularly troubling in our context where many DMUs are
considered simultaneously; good information is available about each alternative’s
ranking with respect to each indicator being aggregated (Patty & Penn, 2018).
Fig. 6.19 Values of dissimilarity index
Fig. 6.20 Correlation coefficients
Fig. 6.21 Friedman test
The majority of currently published university rankings combine various mea-

sures to produce an overall score using an additive approach, and the choice of
normalization affects the order in which universities are ranked. The multiplicative
aggregation overcomes these difficulties (Leskinen & Kangas, 2005) but has the
shortcoming of assigning a 0 global score to the DMUs with some indicator with a
score 0.
As we have seen in the previous section, all the methodological choices for
constructing a university ranking influence the final order of DMUs; HEIBIT helps
the decision-maker to measure the influence’s degree of each choice and illustrate
the possibilities and limitations of rankings and benchmarking techniques, and how
these techniques can be applied to the different levels of higher education systems.
6.6 Conclusions and Future Development
The review that was made of the application in the HEIs of the techniques analyzed
in this chapter shows the great potential of these techniques, although it also shows
that they are still less used in HEIs than in other sectors such as, for example,
business management. One of the possible reasons for the lack of exploitation of
the OR / MS techniques on colleges and universities is the heterogeneity in their
managers’ initial training. Therefore, the offer of training programs specifically
designed for university managers, such as the postgraduate programs for which
HEIBIT has been developed, can increase the use of OR / MS techniques and have
a positive impact on improving efficiency and the effectiveness of HEIs.
Given the importance of international and national rankings on the reputation of
HEIs, the use of some of the best-known rankings to illustrate the use of decision-
making techniques favors learning their possibilities and limitations.
Tools to aid learning, such as HEIBIT, are useful to enable the application
of advanced quantitative methods to broader groups of managers, helping them
identify the correct experts to find the best alternatives for multicriteria problems
that arise in their institutions.
With respect to the future developments of HEIBIT, we are working on a module
that helps the decision-maker to choose the methodology, or the combination of
several methodologies, suitable for frequent problems in the management of HEIs.
The second line of work is to incorporate other applicable techniques in HEIBIT
to construct rankings that have shown to be more promising in the carried out review,
as an example the techniques for obtaining full rankings in the DEA context.
References
Abbott, M., & Doucouliagos, C. (2001). Total factor productivity and efficiency in Australian
colleges of advanced education. Journal of Educational Administration, 39(4), 384–393.
Abbott, M., & Doucouliagos, C. (2003). The efficiency of Australian universities: A data
envelopment analysis. Economics of Education Review, 22(1), 89–97.
Adler, N., Friedman, L., & Sinuany-Stern, Z. (2002). Review of ranking in the data envelopment
analysis context. European Journal of Operational Research, 140(2), 249–265.
Adler, N., & Golany, B. (2002). Including Principal Component Weights to improve discrimination
in Data Envelopment Analysis. The Journal of the Operational Research Society, 53(9), 985–
991.
Agasisti, T., & Dal Bianco, A. (2009). Reforming the university sector: effects on teaching
efficiency—evidence from Italy. Higher Education, 57(4), 477.
Agasisti, T., & Pérez-Esparrells, C. (2010). Comparing efficiency in a cross-country perspective:
the case of Italian and Spanish state universities. Higher Education, 59(1), 85–103.
Agasisti, T., & Pohl, C. (2012). Comparing German and Italian public universities: Convergence
or divergence in the higher education landscape? Managerial and Decision Economics, 33(2),
71–85.
Ahn, T., Charnes, A., & Cooper, W. W. (1988). Some statistical and DEA evaluations of relative
efficiencies of public and private institutions of higher learning. Socio-Economic Planning
Sciences, 22(6), 259–269.
Alvarado, D. V. (2016). Measuring the static and dynamic efficiency of universities through non-
parametric methods. Application to Ecuadorian public universities. PhD thesis (in Spanish).
https://www.educacion.es/teseo/mostrarRef.do?ref=1188876
Aoki, S., Inoue, K., & Gejima, R. (2010). Data envelopment analysis for evaluating Japanese
universities. Artificial Life and Robotics, 15(2), 165–170.
Arrow, K. J. (1963). Social choice and individual values (2nd ed.).
ARWU. (2018). Academic ranking of world universities. http://www.shanghairanking.com/

ARWU2018.html
Avkiran, N. K. (2001). Investigating technical and scale efficiencies of Australian universities
through data envelopment analysis. Socio-Economic Planning Sciences, 35(1), 57–80.
Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and
scale inefficiencies in data envelopment analysis. Management Science, 30(9), 1078–1092.
Barra, C., & Zotti, R. (2016). Measuring efficiency in higher education: An empirical study using a
bootstrapped data envelopment analysis. International Advances in Economic Research, 22(1),
11–33.
Barron, G. R. (2017). The Berlin Principles on ranking higher education institutions: Limitations,
legitimacy, and value conflict. Higher Education, 73(2), 317–333.
Beasley, J. E. (1990). Comparing university departments. Omega, 18(2), 171–183.
Bedir, N., Özder, E. H., & Eren, T. (2016). Course Selection with AHP & PROMETHEE Methods
for Post Graduate Students: An Application in Kirikkale University Graduate School of Natural
and Applied Sciences. In MATEC Web of Conferences (Vol. 68, p. 20004). EDP Sciences.
Begičević, N., Divjak, B., & Hunjak, T. (2007). Prioritization of e-learning forms: a multicriteria
methodology. Central European Journal of Operations Research, 15(4), 405–419.
Berbegal Mirabent, J., & Solé Parellada, F. (2012). What are we measuring when evaluating
universities’ efficiency? Regional and Sectoral Economics Studies, 12(3), 31–46.
Bobe, B. J., & Kober, R. (2018). University dean personal characteristics and use of management
control systems and performance measures. Studies in Higher Education, 1–23.
Bogetoft, P., & Otto, L. (2011). Benchmarking with DEA and SFA. Springer.
Bogetoft, P., & Otto, L. (2015). Benchmarking with DEA and SFA, R package version 0.26.
Borgonovo, E., & Plischke, E. (2016). Sensitivity analysis: a review of recent advances. European
Journal of Operational Research, 248(3), 869–887.
Brans, J. P., & Vincke, P. (1985). Note—A Preference Ranking Organisation Method: (The
PROMETHEE Method for Multiple Criteria Decision-Making). Management Science, 31(6),
647–656.
CAM. (2018). University of Cambridge. Reports and Financial Statements 2018. https://
www.cam.ac.uk/system/files/uoc_annual_report_2018.pdf
Castano, M. C. N., & Cabanda, E. C. (2007). Performance evaluation of the efficiency of Philippine
Private Higher Educational Institutions: application of frontier approaches. International
Transactions in Operational Research, 14(5), 431–444.
Cecconi, P., Franceschini, F., & Galetto, M. (2007). The conceptual link between measurements,
evaluations, preferences and indicators, according to the representational theory. European
Chakraborty, S., Paul, D., & Agarwal, P. K. (2017). Evaluation of educational performance of
Indian states using PROMETHEE-GIS approach. Benchmarking: An International Journal,
24(6), 1709–1728.
Chang, W., Cheng, J., Allaire, J. J., Xie, Y., & McPherson, J. (2018). Shiny: Web Application
Framework for R. R package version 1.2.0. https://CRAN.R-project.org/package=shiny
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making
units. European Journal of Operational Research, 2(6), 429–444.
Coelli TJ, Rao DSP, O’Donnell CJ, Battese GE (2005) An introduction to efficiency and
productivity analysis. .
Colson, G. (2000). The OR’s prize winner and the software ARGOS: how a multijudge and
multicriteria ranking GDSS helps a jury to attribute a scientific award. Computers and
Operations Research, 27(7–8), 741–755.
Correa, C. (2016). Measurement of the teaching activity of the university teacher through
efficiency analysis techniques. PhD thesis (in Spanish). https://www.educacion.es/teseo/
mostrarRef.do?ref=1186413
CWTS. (2018) CWTS Leiden Ranking. Retrieved December 4, 2018, from http://
www.leidenranking.com/
Dakpo, K. H., Desjeux, Y., & Latruffe, L. (2018). Productivity: Indices of productivity and
profitability using Data Envelopment Analysis (DEA) R package version 1.1.0. https://
CRAN.R-project.org/package=productivity/
Daraio, C., Bonaccorsi, A., & Simar, L. (2015). Rankings and university performance: A
conditional multidimensional approach. European Journal of Operational Research, 244, 918–
930.
De Keyser, W., & Peeters, P. (1996). A note on the use of PROMETHEE multicriteria methods.
European Journal of Operational Research, 89(3), 457–461.
Dobrota, M., Bulajic, M., Bornmann, L., & Jeremic, V. (2016). A new approach to the QS uni-
versity ranking using the composite I-distance indicator: Uncertainty and sensitivity analyses.
Journal of the American Society for Information Science and Technology, 67(1), 200–211.
Dobrota, M., & Dobrota, M. (2016). ARWU ranking uncertainty and sensitivity: What if the award
factor was Excluded? Journal of the American Society for Information Science and Technology,
67(2), 480–482.
Dobrota, M., & Jeremic, V. (2017). Shedding the Light on the Stability of University Rankings in
the ICT Field. IETE Technical Review, 34(1), 75–82.
Façanha, L. O., Resende, M., & Marinho, A. (1997). Brazilian federal universities: Relative
efficiency evaluation and data envelopment analysis. Revista Brasileira de Economia, 51(4),
489–508.
Fadlina, M., Sianturi, L. T., & Karim, A. (2017). Best Student Selection Using Extended Promethee
II Method. International Journal of Recent Trends in Engineering & Research, 3(8), 21–29.
Färe, R., & Grosskopf, S. (1996). Intertemporal Production Frontiers: With Dynamic DEA.
Springer.
Feng, Y. J., Lu, H., & Bi, K. (2004). An AHP/DEA method for measurement of the efficiency
of R&D management activities in universities. International Transactions in Operational
Research, 11(2), 181–191.
Fernández, A. (2015). Statistical techniques for the measurement of the efficiency and the total
productivity of the factors applied to the Spanish hospital system. PhD thesis (in Spanish).
https://www.educacion.gob.es/teseo/mostrarRef.do?ref=1139406
Ferrari, G., & Laureti, T. (2005). Evaluating technical efficiency of human capital formation in
the Italian university: Evidence from Florence. Statistical Methods and Applications, 14(2),
243–270.
Førsund, F. R., & Kalhagen, K. O. (1999). Efficiency and productivity of Norwegian Colleges. In
Data Envelopment Analysis in the Service Sector (pp. 269–308). Deutscher Universitätsverlag.
García-Aracil, A., & Van der Velden, R. (2008). Competencies for young European higher
education graduates: Labor market mismatches and their payoffs. Higher Education, 55(2),
219–239.
García, D. (2016). Construction of a model to determine the academic performance of students
based on learning analytics, through the use of multivariate techniques. PhD thesis (in Spanish).
https://www.educacion.es/teseo/mostrarRef.do?ref=1186695
Gnaldi, M., & Ranalli, M. G. (2016). Measuring university performance by means of composite
indicators: A robustness analysis of the composite measure used for the benchmark of Italian
universities. Social Indicators Research, 129(2), 659–675.
Grandzol, J. R. (2005). Improving the faculty selection process in higher education: A case for the
analytic hierarchy process. IR applications. Volume 6. Association for Institutional Research
(NJ1).
Grosskopf, S., Hayes, K. J., & Taylor, L. L. (2014). Applied efficiency analysis in education.
Economics Letters, 3(1), 19–26.
Hadad, Y., & Hanani, M. Z. (2011). Combining the AHP and DEA methodologies for selecting the
best alternative. International Journal of Logistics Systems and Management, 9(3), 251–267.
Han, J. H., & Finkelstein, A. (2013). Understanding the effects of professors’ pedagogical
development with Clicker Assessment and Feedback technologies and the impact on students’
engagement and learning in higher education. Computers & Education, 65, 64–76.
Hu, Y., Liang, W., & Tang, Y. (2017). Dynamic Evaluation on Research Productivity of ‘211
Project’ Universities: The DEA-Malmquist Approach. In Evaluating Research Efficiency of
Chinese Universities (pp. 219–260). Springer.
IREG. (2006). Berlin Principles on Ranking of Higher Education Institutions. http://ireg-
observatory.org/en/index.php/berlin-principles
Jackson, N. (2001). Benchmarking in UK HE: An overview. Quality Assurance in Education, 9(4),
218–235.
Jati, H. (2012). A Study on the performance appraisal method of vocational education teachers
using Promethee II. Journal of Education, 5(1), 5–22.
Jati, H., & Dominic, D. D. (2017). A New Approach of Indonesian University Webometrics
Ranking Using Entropy and PROMETHEE II. Procedia Computer Science, 124, 444–451.
Johnes, J. (2008). Efficiency and productivity change in the English higher education sector from
1996/97 to 2004/5. The Manchester School, 76(6), 653–674.
Johnes, J. (2018). University rankings: What do they really show? Scientometrics, 115(1), 585–
606.
Johnes, J., & Li, Y. (2008). Measuring the research performance of Chinese higher education
institutions using data envelopment analysis. China Economic Review, 19(4), 679–696.
JRC. (2008). Joint Research Centre—European Commission. Handbook on constructing compos-
ite indicators: Methodology and user guide. OECD Publishing.
Kassambara, A., & Fabian, M. (2017). Factoextra: Extract and visualize the results of multivariate
data analyses. R package version 1.0.5. https://CRAN.R-project.org/package=factoextra
Köksal, G., & Nalçaci, B. (2006). The relative efficiency of departments at a Turkish engineering
college: A data envelopment analysis. Higher Education, 51(2), 173–189.
Kuah, C. T., & Wong, K. Y. (2011). Efficiency assessment of universities through data envelopment
analysis. Procedia Computer Science, 3, 499–506.
Kuzmicz, K. A. (2015). Benchmarking in university toolbox. Business, Management and Educa-
tion, 13(1), 158.
Lee, S. H. (2010). Using fuzzy AHP to develop intellectual capital evaluation model for assessing
their performance contribution in a university. Expert Systems with Applications, 37(7), 4941–
4947.
Leskinen, P., & Kangas, J. (2005). Rank reversals in multi-criteria decision analysis with statistical
modelling of ratio-scale pairwise comparisons. The Journal of the Operational Research
Society, 56(7), 855–861.
Liberatore, M. J., & Nydick, R. L. (1997). Group decision making in higher education using the
analytic hierarchy process. Research in Higher Education, 38(5), 593–614.
Lin, C. S., Huang, M. H., & Chen, D. Z. (2013). The influences of counting methods on university
rankings based on paper count and citation count. Journal of Informetrics, 7(3), 611–621.
Locke, W., Verbik, L., Richardson, J. T. E., & King, R. (2008). Counting What Is Measured or
Measuring What Counts? League Tables and Their Impact on Higher Education Institutions in
England. Higher Education Funding Council for England.
Marchant, T. (1996). Valued relations aggregation with the Borda method. Journal of Multi-
Criteria Decision Analysis, 5(2), 127–132.
Maričić, M., Zornić, N., Pilčević, I., & Dačić-Pilčević, A. (2017). ARWU vs. alternative. ARWU
ranking: What are the consequences for lower ranked universities? Management: Journal of
Sustainable Business and Management Solutions in Emerging Economies, 22(1), 1–14.
Marmolejo, F. (2016). Is benchmarking more useful than ranking? January 15, Issue 396. http://
www.universityworldnews.com/article.php?story=20160113012615813
Marope, P. T. M., Wells, P. J., & Hazelkorn, E. (Eds.). (2013). Rankings and accountability in
higher education: Uses and misuses. UNESCO.
Miller, T. (2009). Formative computer-based assessment in higher education: The effectiveness of
feedback in supporting student learning. Assessment & Evaluation in Higher Education, 34(2),
181–192.
Moed, H. F. (2017). A comparative study of five world university rankings in applied evaluative
informetrics (pp. 261–285). Springer.
Morocho, M. (2015). Multilevel modeling of university academic performance. PhD thesis (in
Spanish). https://www.educacion.gob.es/teseo/mostrarRef.do?ref=1168014
Olcay, G. A., & Bulu, M. (2017). Is measuring the knowledge creation of universities possible?: A
review of university rankings. Technological Forecasting and Social Change, 123, 153–160.
OX. (2018). University of Oxford. Financial Statements
2017/18. https://www.ox.ac.uk/sites/files/oxford/field/field_document/
University%20of%20Oxford%2C%20Financial%20Statements%202017-2018.pdf
Özerol, G., & Karasakal, E. (2008). A parallel between regret theory and outranking methods for
multicriteria decision making under imprecise information. Theory and Decision, 65(1), 45–70.
Parteka, A., & Wolszczak-Derlacz, J. (2013). Dynamics of Productivity in Higher Education:
Cross-European Evidence based on Bootstrapped Malmquist Indices. Journal of Productivity
Analysis, 40, 67–82.
Paruolo, P., Saisana, M., & Saltelli, A. (2013). Ratings and rankings: Voodoo or science? Journal
of the Royal Statistical Society: Series A, 176(3), 609–634.
Patty, J. W., & Penn, E. M. (2018). A defense of Arrow’s independence of irrelevant alternatives.
Public Choice, 1–20. https://doi.org/10.1007/s11127-018-0576-7
Pekkaya, M. (2015). Career Preference of University Students: An Application of MCDM
Methods. Procedia Economics and Finance, 23, 249–255.
Pino-Mejías, J. L., Solís-Cabrera, F. M., Delgado-Fernández, M., & Barea-Barrera, R. (2010).
Assessing the efficiency of research groups using data envelopment analysis (DEA). El
Profesional de la Informacion: International Journal of Information and Communication,
19(2), 160–167. (In Spanish).
QS. (2018). Quacquarelli Symonds World University Rankings. Retrieved December 4, 2018, from
https://www.topuniversities.com/university-rankings/world-university-rankings/2018
R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for
Statistical Computing, Vienna, Austria. http://www.R-project.org/
Rad, A., Naderi, B., & Soltani, M. (2011). Clustering and ranking university majors using data
mining and AHP algorithms: A case study in Iran. Expert Systems with Applications, 38(1),
755–763.
Saaty, T. L. (1980). The analytic hierarchy process: Planning, priority setting, resource allocation
(Decision Making Series). McGraw-Hill.
Saaty, T. L. (1994). Highlights and critical points in the theory and application of the analytic
hierarchy process. European Journal of Operational Research, 74(3), 426–447.
Saaty, T. L., & Rogers, P. C. (1976). Higher education in the United States (1985–2000): Scenario
construction using a hierarchical framework with eigenvector weighting. Socio-Economic
Planning Sciences, 10(6), 251–263.
Saisana, M., d’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university
rankings and policy implications. Research Policy, 40(1), 165–177.
Salas, M. S. (2014). Do higher education institutions make a difference in competence develop-
ment? A model of competence production at university. Higher Education, 68(4), 503–523.
Shehatta, I., & Mahmood, K. (2016). Correlation among top 100 universities in the major six global
rankings: Policy implications. Scientometrics, 109(2), 1231–1254.
Sinuany-Stern, Z. (1991). The Need for Institutional Research. Higher Education Policy, 4(4), 57–
60.
Sinuany-Stern, Z., Mehrez, A., & Barboy, A. (1994). Academic departments efficiency via DEA.
Computers and Operations Research, 21(5), 543–556.
Sobol, I. M. (1993). Sensitivity analysis for non-linear mathematical models. Mathematical
Modelling and Computational Experiment, 1(4), 407–414.
TBS. (2018). The 100 richest universities: Their generosity and commitment to research 2018.
Retrieved December 22, 2018, from https://thebestschools.org/features/richest-universities-
endowments-generosity-research/
Thanassoulis, E., De Witte, K., Johnes, J., Johnes, G., Karagiannis, G., & Portela, C. S. (2016).
Applications of data envelopment analysis in education. In J. Zhu (Ed.), Data envelopment
analysis a handbook of empirical studies and applications. Springer.
Thanassoulis, E., Kortelainen, M., Johnes, G., & Johnes, J. (2011). Costs and efficiency of higher
education institutions in England: A DEA analysis. The Journal of the Operational Research
Society, 62(7), 1282–1297.
THE. (2018). Times Higher Education World University Rankings. Retrieved December 4, 2018,
from https://www.timeshighereducation.com/world-university-rankings
Tofallis, C. (2012). A different approach to university rankings. Higher Education, 63(1), 1–18.
Torre, E. M., Gómez-Sancho, J. M., & Perez-Esparrells, C. (2017). Comparing university
performance by legal status: a Malmquist-type index approach for the case of the Spanish
higher education system. Tertiary Education and Management, 23(3), 206–221.
Torre, E. M., Sagarra, M., & Agasisti, T. (2016). Assessing organizations’ efficiency adopting
multidimensional scaling, with an application to higher education. In Handbook of operations
analytics using data envelopment analysis (pp. 145–166).
Tremblay, K., Lalancette, D., & Roseveare, D. (2012). Assessment of higher education learning
outcomes. Feasibility study report, 1. OECD.
Tsinidou, M., Gerogiannis, V., & Fitsilis, P. (2010). Evaluation of the factors that determine quality
in higher education: an empirical study. Quality Assurance in Education, 18(3), 227–244.
Turan, F. K., Cetinkaya, S., & Ustun, C. (2016). A methodological framework to analyze
stakeholder preferences and propose strategic pathways for a sustainable university. Higher
Education, 72(6), 743–760.
Tytherleigh, M. Y., Webb, C., Cooper, C. L., & Ricketts, C. (2005). Occupational stress in UK
higher education institutions: A comparative study of all staff categories. Higher education
research & development, 24(1), 41–61.
U-Multirank. (2018). U-Multirank World University Rankings. Retrieved December 4, 2018, from
https://www.umultirank.org/
USNEWS. (2018). U.S. News Best Global Universities rankings. Retrieved December 21, 2018,
from https://www.usnews.com/education/best-global-universities
Vázquez Cueto, M. J. (2008). Measuring the efficiency of Spanish public universities Combination
of DEA methodology with PCA. In Revista Electrónica de Comunicaciones y Trabajos de
ASEPUMA (XV Jornadas de ASEPUMA y III Encuentro Internacional) (pp. 1–19) (In Spanish).
Waheed, B., Khan, F. I., Veitch, B., & Hawboldt, K. (2011). Uncertainty-based quantitative
assessment of sustainability for higher education institutions. Journal of Cleaner Production,
19(6–7), 720–732.
Webster, T. J. (2001). A principal component analysis of the US News & World Report tier rankings
of colleges and universities. Economics of Education Review, 20(3), 235–244.
Wilkesmann, U., & Lauer, S. (2020). The influence of teaching motivation and New Public
Management on academic teaching. Studies in Higher Education, 45(2), 434–451.
Wilson, K. L., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of
the Course Experience Questionnaire. Studies in Higher Education, 22(1), 33–53.
Wolszczak-Derlacz, J. (2016). Assessment of TFP in European and American higher education
institutions–application of Malmquist indices. Technological and Economic Development of
Economy, 24(2), 1–22.
Worthington, A. C., & Lee, B. L. (2008). Efficiency, technology and productivity change in
Australian universities, 1998–2003. Economics of Education Review, 27(3), 285–298.
Wu, H. Y., Chen, J. K., Chen, I. S., & Zhuo, H. H. (2012). Ranking universities based on
performance evaluation by a hybrid MCDM model. Measurement, 45(5), 856–880.
Živković, Ž., Nikolić, D., Savić, M., Djordjević, P., & Mihajlović, I. (2017). Prioritizing Strategic
Goals in Higher Education Organizations by Using a SWOT–PROMETHEE/GAIA–GDSS
Model. Group Decision and Negotiation, 26(4), 829–846.
Zornić, N., Dobrota, M., & Jeremic, V. (2016, September). Measuring the stability of university
rankings in the field of education. In 2016 ENTRENOVA Conference Proceedings.
Chapter 7
Balanced Scorecard in Strategic Planning
of Higher Education: Review
Zilla Sinuany-Stern and H. David Sherman
Abstract This chapter provides a review of the Balanced Scorecard (BSC) applied
to higher education institutions (HEIs). BSC was developed in the 1990s as a
comprehensive method to measure and manage organizational performance. BSC
translates an organization’s mission statement and strategy into specific, measurable
goals and monitors the organization’s performance in regard to achieving these
goals. We review publications reporting application of BSC to higher education
institutions over 25 years reported by the organizational level, country of appli-
cation, and use of other methodologies combined with BSC. The large number
of applications in many different ways suggests continued future potential for
additional applications of BSC in higher education, such as quality assurance.
Keywords Higher education · Balanced scorecard · Strategic planning · Quality

assurance · Performance measurement · Performance management
7.1 Introduction
One of the management tools used in business that academics and managers have
used successfully in Higher Education (HE) is the Balanced Scorecard (BSC)
see: Hafner (1998), Chen et al. (2006), Beard (2009), Taylor and Baines (2012),
Lassoued (2018), and others. This chapter reviews the literature of BSC in HE over
the last 25 years. In this introduction, we offer some perspective on BSC as applied
to business, not-for-profit organizations, and HE. We also describe the motivation
Z. Sinuany-Stern ()
Department of Industrial Engineering & Management, Ben Gurion University of the Negev,
Beer-Sheva, Israel
H. D. Sherman
DMSB—Northeastern University, Boston, MA, USA
e-mail: h.sherman@norhteastern.edu

214 Z. Sinuany-Stern and H.D. Sherman
for using the BSC to manage HE organizations and the reasons BSC continues to be
a valuable tool in HE for current and future use.
BSC was developed in the 1990s by Robert Kaplan and David Norton (1992)
following their study of large businesses to identify ways to assess and manage
corporate performance. BSC evolved into a respected and widely used approach to
strategic planning, which ideally 1. translates an organization’s mission statement
and business strategy into specific groups of goals, 2. develops and defines
corresponding measures for each of the goals, 3. measures and monitors the
organization’s performance in regards to achieving each of the goals, and 4. uses
the information about where individual goals are and are not achieved to initiate
management actions to help the organization meet their goals in future periods. The
BSC has proven valuable to many organizations, including businesses in a wide
variety of industries, government, and not-for-profit institutions. The way BSC is
adapted, and the way it is used in practice varies, as it is adapted to the organization’s
characteristics and management objectives. In business enterprises that are listed on
public stock exchanges, the owners of the business are the shareholders, and a key
and always present priority is financial performance, as this is the basis for providing
returns to investors by generating cash to, for example, pay dividends, or increasing
earnings to raise the value of the business common stock. The BSC applied to
business can be described as a set of performance measures/criteria that include but
also support the financial measures. Initially, the value of the BSC was to provide
a holistic view of the business/organization via performance measures that go
beyond financial performance measures and compensate for limitations in financial
performance measures. Financial measures reflect the business performance in prior
periods, so for example, the income statement is income earned last month, last
quarter, or last year. Net income is one of the key financial measures, but it does
not address the more important question of how much might be earned in future
periods—next month, next quarter, next year, or future years? Financial statements
are described as lagging indicators, as they report on the past. BSC attempts
to incorporate leading indicators that are mostly nonfinancial that suggest future
performance potential. While this will be more clear in the following discussion,
examples of these lead indicators might be number of patents on new drugs
suggesting new products that will be sold in future periods and the development of
new customers that will be the source of future sales. While no measure can predict
the future, the idea is that a business that is developing patents that are focused on
creating new products or more competitive products will generate future products
that will generate future sales and increased future income. The nature and quality of
the patents are not reflected in the number of new patents, which suggests the need
for yet other measures to evaluate the patents and an evaluation of the potential of
the patents may be yet another measure included in the BSC as a leading indicator.
The point is that these leading indicators need to be explicitly included to focus
the organization on achieving future goals. The expectation is that by achieving the
target level of these leading indicators in the BSC, management will be more likely
to achieve its overall future goals and strategic objectives.
7 Balanced Scorecard in Strategic Planning of Higher Education: Review 215
Before describing the basic BSC structure in corporations, we offer a preview of

the value to not-for-profit organizations and to HE, as the value is substantial for
reasons that differ from corporate use. Not-for-profit organizations do not have a
primary objective to generate added profits or, as it is said, “maximize shareholder
wealth.” They are not owned by shareholders, and indeed, they generally do
not have owners but are governed by a board of directors or trustees or by a
government organization. Financial measures generally are not the key measure of
their performance. The objective of a not-for-profit might be described as providing
as much service in their mission as possible at the lowest cost. The financial
performance might be considered secondary to providing their services. Instead of
profit maximization, the financial objectives might be minimizing cost along with
fund raising and maintaining cash flow to sustain the organization. Not-for-profit use
of BSC is found in museums, health care organizations, government organizations,
and HE. BSC is, in these cases, adapted to the organization goals and mission. While
careful financial management in a not-for-profit is critical helping to sustain and
increase their services, a museum or hospital, or college would not be considered to
have excellent performance just because they generated a profit. The volume, range,
and quality of their services would likely be the primary measures of their success.
Quality is a dimension that is present in the corporate organization as well as the
HE and needs to be incorporated in the BSC. Quality is a leading indicator in that it
suggests whether the product will meet customer expectations and generate repeat
sales in the future. In education, quality can include meeting the student expectation
and meeting employer expectations about the capabilities of the graduates of the
education program. High quality can generate continued demand for education. The
quality of new patents in a business will impact the success of future products. The
quality of the faculty will impact the reputation of the educational institution and
can impact the ranking and the number of students that apply for the HE educations
programs in the future.
There is also a relationship between financial measures and other leading
indicators in corporate and HE applications. An organization can boost current
period financial results by reducing current operating expenses. A business can
reduce current investment in patents which would decrease current period expenses
and increase current period net income. This may result in weaker future sales
and income due to lack of competitive new products. Similarly, Higher Education
Institution (HEI) can reduce the hiring of new faculty or defer investment in
updating existing education programs resulting in lower operating expenses in
the current period, which would generate stronger financial results in the form of
higher surplus or reduced deficit and more cash. However, this could result in a
lower ranking for the education institution where ranking includes a measure of the
faculty quality and the competitiveness of the HE programs, which can reduce the
number of future new student applications. In short, the BSC seeks to incorporate
leading indicators that capture the more important elements that management needs
to successfully manage to achieve strong future performance consistent with the
organization’s mission and strategic goals.
BALANCED SCORECARD
Examples of GOALS for each of the four perspectives
FINANCIAL PERSPECTIVE CUSTOMER PERSPECTIVE

Return on investment Market Share
Growth in Profits On time delivery
Cash Flow. Price competitiveness
Customer Satisfaction
INTERNAL PROCESS PERSPECTIVE LEARNING/INNOVATION PERSPECTIVE

Waste control New Products
Cost management Employee suggestions
Cycle time – production Continuous improvement
Rework Customer recommendations
Frequency of Accidents.
Fig. 7.1 Corporate balanced scorecard illustration (format adapter from Lorin Ipsun BSC presen-
tation, free internet templates—https://powerpointify.com/balanced-scorecard-free-powerpoint-
template)
The general structure of BSC is to develop strategic goals along four perspectives
as shown in Fig. 7.1:
1. Financial perspective—are we meeting the expectations of our Shareholder?
2. Customer perspective—are we satisfying our customers?
3. Internal processes—are we doing the right things (effectiveness)? And doing
things right (efficiency)?
4. Learning and Innovation—are we prepared for the future?
The process of identifying the goals in each of the four perspectives requires
cooperation of individuals associated with that perspective in the business, including
senior managers, functional experts like human resources and information systems,
and others that can help understand ways each perspective contributes to the
strategic goals of the organization. This can require substantial personnel time and
effort and generally requires the strong support of senior management. The costly
personnel time and effort to explain and install the BSC is one of the barriers
to adapting BSC. The need for senior management support is an element that is
required for BSC to be effective, and without this support, it may be of marginal
value. Beyond the time and other costs of adapting the BSC, there are no reports that
we are aware of where it was viewed as damaging in any way to the organization.
Ultimately management identifies the goals that need to be achieved to reflect
the organization’s strategy and achieve its mission; for each goal, a way to measure
that goal is designated, and for each measure, a target that is considered good
progress to meeting the goal is designated. These measures may be quantitative
or qualitative (e.g., have we increased market share can be a qualitative measure
and may be considered sufficient instead of an actual market share quantitative
goal). Incorporation of qualitative goals offers another attractive advantage to not-
for-profit and HE use of the BSC. The BSC is then placed into use, and the actual
results are measured for each goal compared to the target.
The BSC for the business as a single entity will often be different from
the goals and measures of the department and other parts of the organization.
Balanced scorecard has been developed to the level of the individual manager, and
performance against goals in individual scorecards have been used as a primary
basis for evaluating and rewarding individual manager performance.
An example of the value of BSC to a business is the translation of strategy into
the way the organization focuses on meeting the goals and the way the measures
in the individual manager scorecards help the organization achieve strategic goals.
Organization strategy may be described in grand terms with broad and sometimes
technical goals. The individual manager goals in the BSC have been described as
helping to “make the mission real” and as motivating the individual manager to
accomplish particular goals, which contributes to achieving the corporate mission.
For example, improving the relationship with customers can enhance the sales and
even the speed of collecting bills. Speeding up collections increases several financial
goals, such sales, profits, and cash flow. The salesperson’s BSC might include
measures of ways they improve customer relations, such as time spent with the
customer. By meeting this customer goal, the salesperson is indirectly helping to
meet financial goals as well as improving the likelihood of future sales.
Another example is that poor internal processes can lead to delays in customer
shipments, defects, customer dissatisfaction, reduced sales, and reduced profits. The
industrial engineer that has measures such as reducing cycle time and reducing
stockouts is meeting the internal process goals, which improves customer satis-
faction and can lead to increase sales and increased profitability. The industrial
engineer with cycle time and stockouts in their scorecard is helping achieve the
overall mission and improve financial performance without the need to explicitly
know or understand the corporate mission and strategic goals. The four perspectives
in the BSC are interrelated, and the success or failure to meet one set of goals in one
part of the scorecard can enhance success or failure to meet goals in other areas.
Parallel interrelationships are found in HE. For example, in a HE environment,
the institution may have objectives of raising their ranking among external rating
organizations as a means of attracting more students to apply to their institution.
Raising ranking in some cases is based on the percent that completes all 4 years in
college (a student retention goal) and the percentage of students that are employed
after graduation. The dean of the school may have little direct influence over student
employment but may have a retention goal in the dean’s scorecard. The employment
placement manager may have little influence over retention but can directly impact
the placement of graduates. For the placement manager, the placement percentage
could be the goal included in the placement scorecard. The result in each part of
the organization would ideally be motivated to reach its individual BSC goals,
which would ultimately help the HE organization achieve its goal of raising the
ranking. The dean meeting retention goals and the placement manager meeting
student placement goals are each contributing to improving the HE ranking without
their need to know how to improve the ranking. This is an example of how HE
BSC can “make the mission real.” The dean and the placement manager may well
understand that the mission is to have a high ranking to build the reputation of the
HEI, but they may not know how they can or if they can help in that mission. The
BSC directly provides a way for them to contribute toward achieving the mission
goal of improving the ranking and achieving the organization’s strategic goals.
Before focusing on HE, consider some of the typical goals included in the basic
four perspectives for a corporate entity. A for-profit education organization might
have similar elements as any corporation. As noted earlier, the not-for-profit is not
likely to have financial as the top goal. The financial elements were presented as top
perspective as in Fig. 7.1, and while the perspective may be displayed in the same
form, a not-for-profit might have the customer (students) as the top perspective and
financial as one of the other perspectives.
Figure 7.1 is an example of a corporate BSC and the types of goals that might
be included. Each organization would develop its own goals based on its strategic
objectives, so that two competing organizations may well have very different
scorecards. For each of the goals, a specific way of measuring the progress to
the goal that is considered appropriate, by the group developing the scorecard, is
selected. A target for that goal would be set as the goal over some period of time. For
example, in the internal process goal of rework, the measure that might be used is the
number of second quality or defective units requiring rework as a percentage of total
production. The target might be that no more than 5% are seconds requiring rework.
During the measurement period, such as a quarter of the year, the defects would be
measured, and if the percentage were above 5%, that would be flagged as a weakness
requiring some remedial actions by management. Each of the performance measures
on the scorecard would be tracked as in this example, and that would be applied to
each level of scorecards. For example, a business might have four goals in each
perspective, each with a designated way to measure progress to that goal, a measure
of how well the organization did in achieving each of the 16 goals would be reviewed
on a regular basis and used to evaluate progress toward achieving strategic goals and
implications where individual goals are not achieved.
Section 7.2 presents the adaptation of BSC to HEIs, Sect. 7.3 presents and
analyzes the literature on BSC in HE, and Sect. 7.4 concludes.
7.2 BSC Adapted to HE
BSC provides an integrated perspective on goals, targets, and measures of progress

expressed by tangible measured objectives, which represent the balance between the
shareholders/constituencies and customers—as shown in Fig. 7.2. Moreover, Tables
7.1, 7.2, 7.3 and 7.4 present four to eight example measurements, or metrics, which
can be selected for each perspective in HEIs. There is no objective basis for the
number of measures to include in any perspective, so if only two are identified,
that is sufficient. A very large number of goals in each category may be awkward,
but there are successful examples with more than eight in each perspective, and
1.FINANCIAL
Strategy measure target
State income $ +4%
Tuition $ +5%
2. CUSTOMER 3. INTERANL PROCESSES

Strategy measure target VISION & MISSION Strategy measure target
Students # +5% Faculty FT # FTE +2%
Intnl. St. # +1% Faculty PT # FTE +3%
4. LEARN. & INNOV

Strategy measure target
Ph.Ds # +6%
Grants $ +7%
Fig. 7.2 BSC—four perspectives with university performance measures
there may be BSCs for different parts of the organization and different management
levels resulting in many goals included in all the BSCs within an organization.
Each metric has a specific and unambiguous target or set of targets. At the end
of the measurement period, it should be clear as to whether the organization has
successfully met the target for quantitative and qualitative measurement methods.
Although BSC was designed for business organizations, it has been successfully
applied to governmental and not-profit organizations such as Higher Education
Institutions (HEIs). The goals in Fig. 7.2 are just illustrative. Readers should not
evaluate them as a possible comprehensive set of measures for an HE or even as
strong measures of the perspective in which they are listed.
Kaplan and Norton (2004) claim that successful implementation of BSC is based
on five principles:
1. Translating the strategy to operational terms.
2. Involving all levels of the organization from the top level to the department level.
3. The strategy should be part of everyone every day.
4. Strategy should be continuous and long-run development process.
5. The top leaders should lead BSC strategic process.
Hafner (1998) was the first to report on the successful use of BSC at the
University of California system, which includes nine campuses. The central vision
was achieving excellence; around it the four BSC perspectives were measured for
each perspective. Tangible and intangible goals were set, each with its performance
measure.
In 2002 Lawrence and Sharma claimed that the use of business approaches,
such as BSC, in HEIs caused by government cuts and strive for efficiency, turns
HE into a commercial good—commodification, where students become customers,

and universities compete for students locally and globally. This managerialism and
commercialism of HE degrade the value of HE. Nevertheless, they use BSC for
strategic planning in a university in Fig. 7.2 to promote efficiency and accountability
in the university. This concern about the BSC points to the importance of how an
organization defined its mission and goals and how it establishes the goals and
measures that are used in performance measurement and management system. For
example, increasing students in Fig. 7.2 customer perspective would increase tuition
and thereby help the financial performance perspective. However, if the admissions
office recruited students below the admission standards just to boost tuition revenue,
it may appear as good current period performance but could be leading to decline in
the HE’s reputation. This would be dysfunctional use of the BSC and suggests that
the goals in Fig. 7.2 do not reflect all the goals of the institution, such as maintaining
the quality of students.
Systems other than BSC, such as KPI (key performance indicators) and critical
success factors, share this need to select measures that are consistent with their
mission, and that motivate personnel to meet the individual goals in a manner that
supports and advances the overall organization strategy. The developers of BSC have
noted that just because one has a set of measures in each of the four perspectives
does not in itself make a valid or effective BSC. To some extent, this could be viewed
as a biased comment suggesting one needs to hire the inventors as consultants to
do it right, the cynical view. However, it is important that the scorecard elements
be fully reviewed by management to determine that the measures are appropriate
and sufficiently comprehensive so that meeting the individual goals will help the
organization achieve its mission and strategic objectives.
Unlike business organization, which maximize financial measures (e.g., profit
and stock price), higher education concern is centered around academic measures.
The starting point in the strategic planning process is the mission/vision statement
of the HEIs, from which the objectives and measures are derived. The vision and
mission used by HEIs, usually, are not focused on profit. Examples of HEIs vision or
mission are: building the world’s best university (Hafner, 1998); preparing students
to become managers and leaders who will add value to their organizations and
communities (Binden et al., 2014); becoming internationally recognized in research
and teaching (Stewart & Carpenter-Hubin, 2001), etc.
7.2.1 BSC Perspectives in HE
BSC perspectives and measures are adapted to achieve the vision and mission of
the HEI. Al-Hosaini and Sofian (2015) reviewed 29 publications with applications
of BSC in HEIs. They list the main perspectives of each publication and found
that most of the publications used the original four perspectives of business BSC
suggested by Kaplan and Norton (1996), as shown in Fig. 7.2. However, four
publications also use four perspectives, but their titles were adapted to HE. For
example, Philbin (2011) used:
1. Financial
2. people development
3. Institute capability
4. Research output.
However, three publications used more than four perspectives.
Ruben (1999) suggested a HE dashboard with five major perspectives:
1. Teaching/Learning (program courses and student outcomes)
2. Service/Outreach (university, profession, alums, state, students, families, and
employers)
3. Scholarship/Research (productivity and impact)
4. Workplace Satisfaction (faculty and staff)
5. Financial (revenues and expenditures)
Each major indicator was divided into sub-indicators, more tangible and multi-
dimensional. Ruben indicated that the indicators and sub-indicators should be
adapted to the specific institution and its relevant missions.
Hafner (1998) used five perspectives:
1. Human resources
2. Facility management
3. Environment health and safety
4. Information technology
5. Financial operations
Eltobgy and Radwan (2010) suggested six perspectives to evaluate HEI in Egypt
Ministry of Higher Education via BSC, as follows:
1. Educational and learning excellence
2. Scientific research excellence
3. Community participation, environment development, and stakeholders
4. Human and material resources
5. Financial resources
6. Institutional capacity and quality management.
Morley et al. (2010) used Equity Scorecard, which provides each perspective
relevant data by three social variables: gender, socioeconomic status, and age.
Another example is Chalaris et al. (2015), who suggested to use four other
perspectives adapted to HEIs in the context of quality assurance:
1. Teaching and research work
2. Students–partners
3. Internal processes
4. Human and financial resources.
These differences in the perspectives used and even the order of listing perspec-
tives are reflections of the need to adapt the BSC to the organization and the strategic
goals. The adaptation is done within the organization.
7.2.2 Goals and Measures in HE for 4 BSC Perspectives
As shown in Sect. 7.2.1, most applications of BSC to HE use the original Kaplan and
Norton BSC framework with the 4 perspectives as presented in Fig. 7.2. However,
the specific goals within each perspective were adapted to HE. Tables 7.1, 7.2,
7.3 and 7.4 present examples of goals and measures for each of the 4 perspective
as applied to HE. Perspective 1 includes financial goals; perspective 2 includes
mainly degree programs and courses; perspective 3 includes mainly undergraduate
and international student’ programs; while perspective 4 includes mainly research
outcomes and improved teaching (Hafner, 1998; Binden et al., 2014; Cullen et al.,
2003, Ruben (1999), and others).
Table 7.1 Financial Goal Measure

perspectivea —strategies
examples Increase research income Research income
Increase income from overseas students Overseas income
Increase donations from local graduates Donations income
Cost-effectiveness Cost per student
Increase state funding level State income
Keep budget in line Budget target
Program funding
Leverage Ratio
a Based on Binden et al. (2014), Hafner (1998), Cullen et al.
(2003), Ruben (1999), Ndoda and Sikwila (2014), and others
Table 7.2 Customer perspectivea —strategies examples

Goal Measure
Increase full-time under/grad. students Undergrad. Students desired #
Enrolment rates by gender and program Numbers and ratios
Increase number of international students Desired # of international students
Develop partnership with foreign HEIs # of foreign HEIs by quality
Students satisfaction Courses/teachers evaluation
Degree completion time by program # of years
Pass rates on professional exams Rates and numbers
Improve students’ job placement # of job placements and their average salary/satisfaction
Raise the international profile of faculty Rate and number
a Based on Binden et al. (2014), Hafner (1998), Cullen et al. (2003), Ruben (1999), Ndoda and Sikwila
(2014), and others
Table 7.3 Internal organization perspectivea —strategies examples

Goals Measures
Integrated/new degree programs Reducing # of non-core courses
Simplifying administrative support Reducing # of reports and their length
Simplifying committees’ structure/efficiency Reducing # of reports and their length
Up to date academic curriculum Every X years (X = 4 or else desired frequency)
Programs accreditation Positive evaluation
University/program ranking Various ranking scales
a Based on Binden et al. (2014), Hafner (1998), Cullen et al. (2003), Ruben (1999), Ndoda and
Sikwila (2014), and others
Table 7.4 Learning and Growth Perspectivea —strategies examples

Goals Measures
Advancing research # of publications; # of reports; # of patents
acquired; # of consultancy reports;
International research level # of publications in ISI; No. of scholar
visitors; No. of awards; No. of conferences;
Increase number of research students # of Ph.D. students
Improve teaching Average evaluation of lecturers by peers
Employees satisfaction Average evaluation
Skilled workforce # of faculty with Ph.D. degree
Increase faculty efficiency Student faculty ratio
Effective technology used Investment in new technologies
Physical infrastructure Area in sq. m. of built: Classes; Teaching
labs; Research labs, etc.
Operating Practices Manuals Licenses and Number
Certificates
Strategic alliances reports Number
External examiners’ reports Number
Library collections Number of Books and Journals
IT hardware and licensed software Number
a Based on Binden et al. (2014), Hafner (1998), Cullen et al. (2003), Ruben (1999), Ndoda and
Sikwila (2014), and others
7.2.3 External and Internal Audiences in HE
Stewart and Carpenter-Hubin (2001) applied BSC to Ohio State University (OSU).
They highlight the differentiation between externally and internally driven con-
stituencies in HEIs and their concerns, focus, and data acquisition: The main three
externally driven audiences in HE are:
1. Consumers: students and parents
2. Governing bodies: legislators and accrediting agencies
3. Revenue generators: Alumni, foundations, and donors.
The concerns of externally driven audiences are undergraduate education, gradu-

ate education, and the reputation of the HEI. Reflecting these audiences, the format
of assembling data for planning are report cards, rankings, and indices.
The main internally driven audiences in HEI are:
1. Faculty
2. Academic administrators
3. Nonacademic administrators.
The concerns of internally driven audiences are resource allocation, priorities,
and various organizational agendas. Their format of assembling data for planning is
faculty or institutional committees and reports (ibid.).
7.3 Literature Review of BSC in HE
In 1999 Schneiderman wrote an article with the title: “Why balanced scorecards
fail,” during the same year Ruben (1999) suggested HE should use BSC for
measuring university excellence. Evidently, Ruben’s call for using BSC in HE was
followed by many administrators, managers, and researchers as presented in Table
7.5.
This section summarizes the literature on BSC in HE over more than two decades
(1995–2019). Counting the number of articles devoted to BSC in HE over time,
we searched in Google Scholar Advanced Search and reported in Table 7.5 the
counts. In the first column, two sets of keywords were used: “balanced Scorecard”
and “higher education” (or university or college). During 1995–2019 the search
located 368 papers with the search words in the title of the articles and 69,650
articles with the search words anywhere in the article. Evidently, as shown in
Table 7.5 Number of articles on BSC in HEa versus all BSC articles over time
1. 2.
Articles’ # Articles’ # 4.
with BSC with BSC 3. Articles’ # 5.
and HE and HE Articles’ # with BSC Articles’ #
termsa in termsa anywhere with BSC anywhere reviewed
the article in the in article in the in Table
Years title article title article 7.6
1995–1999 11 (3%) 1650 (2.4%) 885 (4.3%) 3270 (3.3%) 2 (3.3%)
2000–2004 42 (11.4%) 10,500 (15.1%) 4470 (22%) 16,200 (16.7%) 4 (6.6%)
2005–2009 81 (22%) 16,900 (24.3%) 5010 (24.6%) 22,300 (23%) 10 (16.4%)
2010–2014 114 (31%) 21,400 (30.7%) 5390 (26.4%) 35,000 (36%) 22 (36%)
2015–2019 120 (32.6%) 19,200 (27.5%) 4620 (22.7%) 20,400 (21%) 23 (37.7%)
Total 368 (100%) 69,650 (100%) 20,375 (100%) 97,170 (100%) 61 (100%)
Source: Author (Google Scholar advanced search assessed on February, 2020)
a Search words: “balanced scorecard” and (“higher education” or “college” or “university”)
Table 7.5, the number of publications on BSC is growing over time significantly.
Obviously, the second column shows huge numbers since it includes articles where
BSC or HE could be negligible; for example, in some articles—only one reference
is related to BSC. However, both numbers give indication on the growing use of
BSC methodology (minimum of 368 articles), and its mention (69,650) in articles
on BSC in HE during 1995–2019. The third and fourth columns of Table 7.5 provide
the parallel data on the number of articles on BSC in general, not necessarily in the
context of HE (during 1995–2019, counting 20,375 articles with BSC in their title,
versus 97,170 articles with BSC anywhere in the article).
There are many publications on BSC in HE—368 articles where “balanced
scorecard” appear in their title. Thus, in the next sub-sections’ literature review,
there is no attempt to cover all the articles on BSC in HE. Thus we covered only 61
articles in Table 7.6—covering a wide range of countries where BSC was directed to
HE on various levels of HE. Included are more recent papers—from the last 5 years
(2015–2019)—23 papers.
This review of the literature covers over 20 years (1998–2019) of applications
of BSC in HE—as listed in Table 7.6 chronologically, highlighting three criteria for
each article: Institutional level where BSC was designed for (Sect. 7.3.1), country
of application (Sect. 7.3.2), and highlight and further analysis or other OR/MS
methodology integrated with BSC (Sect. 7.3.3)—presented in columns 1, 2, 3,
respectively.
7.3.1 Articles of BSC in Various Levels of HE Systems
As shown in Table 7.6, we cover a wide range of institutional levels: department,

school, college, university, group of universities, national, and multinational as
shown in Table 7.6. Most of the publications in this review address the university
level, whether it is applied to a single university (20 articles), a group of universities
(5 articles), the national level (15 articles), or multinational level (2 articles) in total
42 articles.
University level: In this review, 20 articles were devoted to BSC on the single
university level, for example, Stewart and carpenter-Hubin (2001), Rompho (2008),
Al-Ashaab et al. (2011), Sudirman (2012), Yakhou and Ulshafer (2012), Umayal
Karpagam and Suganthi (2012), Ndoda and Sikwila (2014), Ahmad (2015), Chalaris
et al. (2015), Pietrzak et al. (2015), Chimtengo et al. (2017), Hejazi et al. (2017),
and El Junusi et al. (2019).
College Level: In this review two articles were devoted to BSC on the college
level which is similar to the university level: Brown (2012) and Gamal and
Soemantri (2017). Additional article was devoted to an Institute of Technology—
by Chen et al. (2006).
School/Faculty Level: In this review, six articles were devoted to BSC on the
school level by Cullen et al. (2003), Papenhausen and Einstein (2006), McDevitt et
al. (2008), Al Kaabi and Jowmer (2018), and Lassoued (2018). Most of the articles
Table 7.6 Publications on BSC in HE
226
Author (year) 1. Institutional level 2. Country 3. Highlight/Further analysis

Hafner (1998) Univ. of Cal.—9 campuses USA Cause and effect relationship
Ruben (1999) University General Methodological
Bremser and White (2000) Department (accounting) USA Top down BSC design experiment
Stewart and Carpenter-Hubin (2001) University—Ohio State Univ. USA International Excellence
Cullen et al. (2003) Faculty of Business UK Quality Management
Self (2003) University library USA Focusing on 26 metrics
Chen et al. (2006) Institute of Technology Taiwan Strategy map
Papenhausen and Einstein (2006) Business School USA Survey
Asan and Tanyas (2007) Eng. Manag. Grad. Prog. Turkey Hoshin Kanri method
Umashankar and Dutta (2007) Private and public HEIs— - national India How to adapt BSC in India
McDevitt et al. (2008) Business Sch. With 6 Dept. USA 6 phase BSC process
Farid et al. (2008) HEIs national Iran Methodological
Kettunen (2008) University Europe Methodological, quality assurance
Rompho (2008) University Thailand Questionnaire and strategy map
Beard (2009) 2 HEIs USA Comparison
Raghunadhan (2009) Public and corpo. HEIs National India Comparison
Eltobgy & Radwan, 2010) HEIs National Egypt Strategy Map and Radar chart
Morley et al. (2010) HEI public and priv. National Ghana and Tanzania Equity scorecard
Al-Ashaab et al. (2011) University UK Industry–university collaboration
Philbin (2011) Institute in Univ. supported by Ind. UK Multidisciplinary institute, strategy map
Zangoueinezhad and Moshabaki (2011) University General Methodological Fuzzy AHP
Aljardali et al. (2012) HEIs National Lebanon Interviews and information sys.
Al-Zwyalif (2012) Private univ. national Jordan Questionnaire Descript. statist.
Brown (2012) College USA Strategy Map
Sudirman (2012) University Indonesia Strategy map
Del Sordo et al. (2012) HEIs National Italy Methodological
Z. Sinuany-Stern and H.D. Sherman
Taylor and Baines (2012) 4 HEIs UK Comparison by Interviews
Yakhou and Ulshafer (2012) University USA Activity based costing
Umayal Karpagam and Suganthi (2012) University India Strategy map
Jairak and Praneetpolgrang (2013) Governance IT Thailand IT and importance anal.
Lewis et al. (2013) Libraries in 4 universities USA Implementing BSC together
Maria et al. (2013) Info. Comm. Tech dept. Indonesia Service dept. within a university
Sayed (2013) 30 HEIs International Review
Valderrama et al. (2013) University by Department Spain DEA and strategy map
Weerasooriya (2013) 11 HEIs National Sri Lanka Questionnaire correlation, quality
Binden et al. (2014) HEIs National Malaysia Review and suggest framework
De la Mano and Creaser (2014) Libraries International Questionnaire and review
Ndoda and Sikwila (2014) University Africa Questionnaire Factor anal.
Al-Hosaini and Sofian (2015) Review International Table with 29 papers
Ahmad (2015) University Pakistan Faculty Recruit.
Chalaris et al. (2015) University Greece Quality assurance
Hladchenko (2015) 4 Universities Germany and Austria Comparison
Ismail and Al-Thaoiehie (2015) University national Saudi Arabia Questionnaire
Pietrzak et al. (2015) University Poland Strategy Map
Swierk and Mulawa (2015) University Poland Strategy map and international review
Elola et al. (2016) University national Spain Questionnaire, Strat. Map, structural Eq.
Lin et al. (2016) Tech. and Voc. HEIs Taiwan Fuzzy Delphi, ANP, sustainable develop.
7 Balanced Scorecard in Strategic Planning of Higher Education: Review
Mohammed et al. (2016) Comparing 25 IHEs Iraq Questionnaire-descriptive stat. quality

Brown (2017) School of Nursing USA Strategic planning process
Chimtengo et al. (2017) University Malawi Questionnaire correlation
Gamal and Soemantri (2017) Private college Indonesia Questionnaire multiple regression
Ozdemir and Tuysuz (2017) 87 universities national Turkey Fuzzy ANP and DEMANTEL
Reda (2017) University and its academic dept. General Review, quality assurance
227
(continued)
Table 7.6 (continued)
228
Author (year) 1. Institutional level 2. Country 3. Highlight/Further analysis

Hejazi et al. (2017) University Iran Questionnaire and strategy map
Usoh and Peterson (2017) Public HEIs national Indonesia Questionnaire
Al Frijat (2018) Accounting department Jordan national Questionnaire, regression analysis
Al Kaabi and Jowmer (2018) Faculty of management Iraq Sustainable BSC
Fijalkowska and Oliveira (2018) University General Review
Lassoued (2018) College of Business United Arab Emirates SWOT
El Junusi et al. (2019) University Indonesia Internationalization, ranking
Kiriri (2019) University national Kenya Suggest national framework
Source: Author
Z. Sinuany-Stern and H.D. Sherman
were devoted to the school of business, where BSC is often taught. However, Brown
(2017) used BSC for Nursing school.
Academic Department Level: In this review, only one article was devoted to a
single academic department where BSC often taught—accounting: Bremser and
White (2000). Although Al Frijat (2018) completed a questionnaire-based survey
to assess on the national level the potential for improving accounting education
in Jordan via BSC. Asan and Tanyas (2007) applied BSC to one Engineering
Management Graduate Program. Some apply BSC to the whole university down
to the department level, for example, Valderrama et al. (2013).
Service Department Level: In this review three articles were devoted to service
departments: Self (2003) applied BSC for a university library. Lewis et al. (2013)
report about the cooperation among libraries in four USA universities on implement-
ing BSC. De la Mano and Creaser (2014) wrote a review on libraries worldwide
that use BSC for measuring performance and strategic planning—they used a
questionnaire; their sample includes 49 libraries—including academic libraries.
Maria et al. (2013) used BSC for the Information communication technology service
department in a university in Indonesia to improve its performance.
Philbin (2011) used BSC for strategic planning of a research institute serving
various departments in a university supported by the industry.
Sometimes BSC is applied for a specific issue. For example, Al-Ashaab et
al. (2011) used BSC to measure the impact of industry-university collaboration.
Applications of BSC devoted to specific functions within HEIs were given in the
literature: Ahmad (2015) applied BSC to faculty recruitment to improve the least
developed departments of HEIs in a region.
Several Universities: BSC was devoted to several universities by five articles:
Hafner (1998) in the nine State Universities of California. Ozdemir and Tuysuz
(2017) applied BSC in 87 universities in Turkey. Weerasooriya (2013) used BSC
for 11 universities in Sri Lanka. Hladchenko (2015) compared the use of BSC
in two Austrian and two German Universities. Mohammed et al. (2016) used
questionnaires for 25 HEIs in Iraq.
National Level: BSC was devoted by 15 articles to HEIs on the national level,
for example, Umashankar and Dutta (2007) for India, Farid et al. (2008) for Iran,
Eltobgy and Radwan (2010) for Egypt, Aljardali et al. (2012) in Lebanon, Al-
Zwyalif (2012) for Jordan, Del Sordo et al. (2012) for Italy, Binden et al. (2014)
for Malaysia, Ismail and Al-Thaoiehie (2015) for Saudi Arabia, Elola et al. (2016)
for Spain, Usoh and Peterson (2017) for Indonesia, and Kiriri (2019) for Kenya.
Mostly on the university level, some were applied to public HEIs, some for private,
and some for both. Lin et al. (2016) applied BSC to technical vocational HEIs in
Taiwan.
Multinational Level: Kettunen (2008) suggested to use BSC for evaluating
European universities quality performance. Morley et al. (2010) designed an equity
scorecard for Ghana and Tanzania HE.
7.3.2 Country of BSC Applications
USA is the leading single country in the applications of BSC in HE. Swierk and
Mulawa (2015) listed 39 universities that used BSC, of which 33 universities were in
the USA—they extended the list of Rompho (2008), which included 22 universities
which used BSC, of which 19 were in the USA.
We made an effort to cover variety of countries in this partial review (of 61
articles in Table 7.6), 11 articles reflect applications of BSC in USA HEIs (for
example, Hafner, 1998; Bremser and White, 2000; Stewart and Carpenter, 2001;
Beard, 2009; McDevitt et al., 2008; Brown, 2012, 2017; Lewis et al., 2013), which
is far more than any other single country in our review.
BSC was applied for HEIs in many other countries. Several publications
suggested various frameworks to fit the application of BSC to specific countries.
For example, Farid et al. (2008) in Iran, Raghunadhan (2009) in India, Morley
et al. (2010) in Ghana and Tanzania, Eltobgy and Radwan (2010) in Egypt, Del
Sordo et al. (2012) in Italy, Al-Zwyalif (2012) in Egyptian private universities,
Aljardali et al. (2012) in Lebanon, Binden et al. (2014) in Malaysia, Chalaris et
al. (2015) in Greece, Hladchenko (2015) in Austria and Germany, Ismail and Al-
Thaoiehie (2015) in Saudi Arabia, Pietrzak et al. (2015) in Poland, Mohammed et
al. (2016) in Iraq, Hejazi et al. (2017) in Iran, Usoh and Peterson (2017) in Indonesia
(public HEIs on the national level), and El Junusi et al. in Indonesia (in a specific
university)), Ozdemir and Tuysuz (2017) in Turkey, Ismaand Kiriri (2019) in Kenya.
Umashankar and Dutta (2007) suggests to use BSC in India. Raghunadhan (2009)
discusses the use of BSC in Indian public universities, and Umayal Karpagam
and Suganthi (2012) reports about the application of BSC in a specific university
in India. Weerasooriya (2013) examined the performance of entire management
faculties/schools in Sri Lankan Universities.
In summary, grouping articles in Table 7.6 by continent: 27 articles were devoted
to Asia, 11 articles to Europe, 11 to America (actually the USA), and 5 to Africa—
the rest of the articles are methodological or reviews of BSC in HE (7 articles).
7.3.3 BSC Integrated with Other Methodologies
Some applications of BSC in HE are combined with additional methodologies as

given in the last column of Table 7.6.
Questionnaires and statistical methods were used in many studies often on the
national level (over 13 articles). Rompho (2008) used questionnaires for HEIs in
Thailand—correlations were used to verify the relationships of the strategy map’s
elements. Al-Zwyalif (2012) performed a study aims at identifying the Jordanian
private universities’ awareness of the importance of the implementation of BSC—
a questionnaire was used to verify several hypotheses via t-test; also descriptive
statistics was presented. Ndoda and Sikwila (2014) used questionnaires in Africa,
utilizing factor analysis. Elola et al. (2016) used questionnaires in Spanish private
and public HEIs, utilizing structural equations for strategy map. Al Frijat (2018)
used questionnaires and statistical analysis for accounting departments in Jordan.
Others also used questionnaires: Ismail and Al-Thaoiehie (2015) in Saudi Arabia,
Mohammed et al. (2016) in Iraq.
Questionnaires were also used on the single institution for extracting evaluations
from its constituencies. For example, Chimtengo et al. (2017) evaluated the
correlations between each of the 4 perspectives and the performance evaluation of
their constituencies at a university in Malawi. Hejazi et al. (2017) used questionnaire
for a university for analyzing a strategy map.
Strategy Map is a simple graphic that shows a logical, cause-and-effect con-
nection between strategic objectives—how the various perspectives: financial,
customer, internal process, learning, and growth are linked, and how they create a
balance between these perspectives to focus the organization on meeting its strategy
objective (Kaplan and Norton (2004). Examples of applying BSC with strategy
maps were reported in Chen et al. (2006), Eltobgy and Radwan (2010), Philbin
(2011), Brown (2012), Pietrzak et al. (2015), Elola et al. (2016), Hejazi et al. (2017),
and others.
Radar chart, presenting the values of the various objectives of BSC, was used by
Eltobgy and Radwan (2010).
SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis was inte-
grated with BSC by Lassoued (2018) Emirates College of Business in Abu Dhabi.
Fuzzy AHP (Analytical Hierarchy Process, by Saaty, 1994) was suggested
by Zangoueinezhad and Moshabaki (2011) in their methodological article for
evaluating the performance of HEIs annually based on the knowledge-based indices
of the 4 BSC perspectives.
Fuzzy ANP and DEMANTEL were used by Ozdemir and Tuysuz (2017), which
integrated BSC with Fuzzy ANP (Analytic Network Process—which enables
to prioritize the strategies, it is a generalization of AHP by Saaty, 1994) and
fuzzy DEMANTEL (Decision Making Trial and Evaluation Laboratory—which
determine the relationships among the strategies). The model was designed for
Turkey HEIs—an application to 87 HEIs is reported.
Fuzzy Delphi and ANP were applied by Lin et al. (2016) to technical and
vocational HEIs in Taiwan for achieving sustainable development competencies,
considering uncertainty.
Sustainable BSC (SBSC) was also used by Al Kaabi and Jowmer (2018) at
the Faculty of Management and Economics at Mustansiriyah University in Iraq.
The sustainable aspect they suggest is, for example, attract high-quality students,
develop academic excellence and innovation, and building funds for donations.
DEA (Data Envelopment Analysis) measures the efficiency of units based on
their multiple inputs and multiple outputs (Charnes et al., 1978). DEA combined
with BSC have been used by Valderrama et al. (2013), where the decision-making
units are academic departments.
ABC—Activity Based Costing (assigning more overhead costs into direct costs
compared to conventional costing) was combined with BSC by Yakhou and Ulshafer
(2012) for strategic planning in HEIs.
Asan and Tanyas (2007) presented a methodology that integrates the BSC with
Hoshin Kanri (ibid.), a seven-step process-based approach used in industry from
the corporate level (strategy) through the mid-level (tactics), down to the floor level
(operations).
7.3.4 BSC Use for Quality Assurance
Most of the articles presented here use BSC for strategic planning in HEIs
by settings multi-dimensional financial and nonfinancial targets; namely, sets of
quantitative academic measures of the desired future performance of HEI in relation
to its current performance (see examples in Table 7.1). However, several studies used
BSC for quality assurance (QA) purposes for various levels of HE. For example,
Kettunen (2008) suggested to use BSC on the international level for evaluating
European universities quality performance. Cullen et al. (2003) applied BSC for
quality assurance at the school level (school of business). Chalaris et al. (2015)
applied BSC for QA on the university level for HEIs in Greece for the national QA
process. They highlight the importance of using management information systems
to support the process of BSC in HEIs. Mohammed et al. (2016) suggested to use
BSC for QA of HEIs in Iraq on the national level.
Beard (2009) gives examples of two HEIs which used BSC successfully and
even won Malcolm Baldrige National Quality Award of the European Foundation
for Quality Management (EFQM) and BSC.
Reda (2017) highlights the congruence and role of BSC in QA practices in
HEIs. He claims that in order to have meaningful QA evaluation, there is a need to
emphasize quality measures in each of the 4 perspectives. Three areas that are noted
as important for QA in HE: 1. teaching and learning, 2. research, and 3. community
services. Each area within each prospective has its specific objectives, and for each
objective its measures, targets, and initiatives.
7.4 Advantages and Disadvantages of BSC in HE
Many wrote about the advantages and disadvantages of BSC in HE. Already in 2002,
Lawrence and Sharma claimed that treating HE as a private good and students as
customers is a potential degradation of HE function in society. Moreover, through
the application of market-based methodologies, as BSC, to promote efficiency in
HE, the very essence of HE is jeopardized. Nevertheless, the use of BSC in HE
increased greatly over the years (as shown in Table 7.5).
Taylor and Baines (2012) interviewed leaders of four UK representative HEIs

who used BSC. They provide a critique of BSC in HE. Its advantages are:
1. BSC is tied directly to the institutional goals and plans.
2. BSC provides link to all levels of the institution.
3. BSC is simple.
4. BSC provides “lag” (past) and “lead” (future)performance measures.
The disadvantages of BSC in HE are:
1. To imbed BSC in all levels of the institution significant work/cost/time are
required.
2. The 4 perspectives are not general but require local adaptation.
Overall, they concluded that BSC was a success—they all saw the Scorecard as a
methodology to integrate overall corporate strategy with operational planning across
the institution or at the level of individual departments and schools (ibid). This
success is also due to the dedication of very top academic/administrative leaders
of each of the HEIs involved and the quality assurance process required by state
authorities in the UK. They referred to a survey of the UK Committee of University
Chairmen from 2004, which reported that 20 HEIs regularly monitored performance
against their corporate plan (ibid). The success of the performance measurement
approach in general and BSC is dependent on high-quality management information
systems.
Sayed (2013), in a critical review of the applicability of BSC to HEIs, based on a
review of 30 universities around the globe, found that universities are limited in their
ability to mobilize resources as BSC often recommends. Moreover, the adaptation of
BSC by university administrators is slow, often due to their insufficient background,
and somewhat altruistic mission.
BSC is a strategic tool adapted from business by the HEIs’ administration and other
stakeholders to capture the multi-dimensional aspects of universities’ performances,
financial and no-financial, mainly for strategic planning. Beyond strategic planning,
BSC has been used for comparison/benchmark and quality assurance of HEIs. As
shown in our review, BSC is spread worldwide.
Our literature review, on BSC in HE, is, to our knowledge, the largest and
most comprehensive so far—covering over 60 articles devoted to BSC in HE. We
acknowledge that there are many other HE articles not included in this review and
hope the value of this review will help HE managers consider whether BSC would
help them achieve their institution’s mission. We also believe that while there are
limitations and hurdles to adapting BSC, the related question is what method will be
used to steer the institution to meeting its mission and would BSC be advantageous
compared with the current methods or others under consideration. Our somewhat
systematic lists the articles in a table chronologically, classifying the articles by the
level of the HEI, the country of application, the highlight of the paper, and other
methodology used in conjunction with BSC. While we covered only 61 out of 368
papers in Google Scholar, where HE and BSC are mentioned in their title, there
are many more where HE and BSC are mentioned anywhere in the article—69,650
articles.
Our findings show that BSC growth in HEIs is consistent over the last 20 years.
However, the rate of growth in the number of publications is moderate in the
last five years. The number of applications (which are often not reported in
the literature) may continue to grow due to many articles in which BSC is the
suggested framework that should be used on the national level in many countries,
mainly for quality assurance. Based on our review, it appears that BSC has been
implemented mainly at the university level. Statistical analyses with questionnaires
were the most used methodology in conjunction with BSC. We believe there is
great potential for increasing the integration of BSC with OR/MS methods such
as, DEA for measuring academic efficiency, ranking methods, and TQM (total
quality management) for quality assurance, simulation, and big data. These OR/MS
methods generate measures that can be included in the performance measures in the
four perspectives of the BSC and can also drive the initiatives to help an organization
achieve the goals related to specific performance measures such as the use of DEA
to measure and manage operating efficiency.
Educational costs and benefits need to be considered while implementing per-
formance management (Chen et al., 2006). With the budget pressure, competition,
and ranking in HEIs, the moves in the direction of performance management are
inevitable. BSC aims to provide a concise solution to manage a complex process
of assessment, evaluation, and reflection at various levels within the institution and
often in relation to other HEIs. Thus, for the success of implementing BSC, there is
a need for a unified information system to utilize the BSC data collection overtime
on all levels of HEIs: department level, school level, university level, and national
level.
Ratnaningrum et al. (2020) reviewed the BSC literature from 71 articles pub-
lished in Q1 rank Scopus indexed journals in the areas of business, management,
and accounting (21 journals). Out of 45 empirical papers, 5 were devoted to the
public sector, of which 2 were applied to HE (Asan & Tanyas, 2007 and Lawrence
& Sharma, 2002). They conclude that, although BSC has limitations, their findings
show that BSC is beneficial enough. It is reinforcing the view that the BSC remains
worth considering for a performance management system.
The numerous BSC applications to HE underlines two paramount characteristics
of the BSC that we offer at the conclusion of this study along with one general
observation:
1. BSC is a comprehensive performance measurement and management method-
ology that can be adapted to organizations at various levels, from a multi-site
HE system to individuals within these organizations. A simple reminder of this
is that the initial corporate BSC model has financial as the primary perspective
but that a not-for-profit HE can revise and adapt the BSC to include customer
or another perspective as primary and define customer as students as well as
other stakeholders such as donors and government funding sources. This is
demonstrated by the wide diversity of HE applications worldwide.
2. The effectiveness of the BSC in helping HE (and any organization) is highly
dependent on the way it is developed and applied to HE and the way it is
used in these organizations. The performance measures and targets in each
perspective need to be congruent with the mission and strategic goals. The
measure needs to clearly reflect the critical elements, including lagging indicators
such as financial results reflecting past performance and leading measures to
drive future performance. In addition to the financial measures and other common
quantitative measures, quality measures, qualitative measures, and other mea-
sures critical to the success of the organization can and should be incorporated
into the perspective. There is no requirement that any one perspective, such
as financial, dominate the BSC if that perspective is not a key element of the
mission. Senior management leadership and support are critical in adapting the
BSC. Performance evaluation and reward systems can be tied to the achievement
of BSC target goals. Under these conditions, the BSC can motivate results that
advance the organization consistent with its strategic goals and mission as it has
with many other organizations.
Observation—The BSC can be highly effective and add much value. It has
been noted that one very valuable way for management to discuss and evaluate
organization strategy and mission is to focus on the four perspectives and the
measures.
We fully acknowledge the effectiveness of the BSC when properly applied
and adapted to an organization and specifically to HEI and the evidence of
many users reported in the literature. However, we are not suggesting that the
BSC is the only comprehensive methodology. Other comprehensive performance
measurement systems that recognize and incorporate the multiple dimensions for
which management is accountable can also be effective, and in all cases, the way
they are adapted and used has a significant influence on how well they benefit the
organization.
References
Ahmad, S. (2015). A balanced scorecard approach to recruitment in higher education. Journal of

Strategy & Performance Management, 3(1), 17–39.
Al Frijat, Y. S. (2018). Activating balanced scorecard importance as a way to improve the
accounting education in Jordanian universities. International Business Research, 11(9), 66–78.
Al Kaabi, B. R., & Jowmer, B. (2018). The use of sustainable balanced scorecard as a tool for
strategic planning and resource efficiency improvement: An empirical study in the Mustansiriya
university. The Journal of Social Sciences Research, 1, 213–221.
Al-Ashaab, A., Flores, M., Doultsinou, A., & Magyar, A. (2011). A balanced scorecard for
measuring the impact of industry–university collaboration. Production Planning & Control,
22(5–6), 554–570.
Al-Hosaini, F. F., & Sofian, S. (2015). Review of balanced scorecard framework in higher education
institution (HEIs). International Review of Management and Marketing, 5(1), 26–35.
Aljardali, H., Kaderi, M., & Levy-Tadjine, T. (2012). The implementation of the balanced
scorecard in Lebanese public higher education institutions. Procedia—Social and Behavioral
Sciences, 62, 98–108.
Al-Zwyalif, I. (2012). The possibility of implementing balanced scorecard in Jordanian private
universities. International Business Research, 5(11), 113–120.
Asan, S. S., & Tanyas, M. (2007). Integrating Hoshin Kanri and the balanced scorecard for
strategic management: The case of higher education. Total Quality Management and Business
Excellence, 18(9), 999–1014.
Beard, D. F. (2009). Successful applications of the balanced scorecard in higher education. Journal
of Education for Business, 84(5), 275–282.
Binden, W., Mziu, H., & Suhaimi, M. (2014). Employing the balanced scorecard (BSC) to
measure performance in higher education—Malaysia. International Journal of Information and
Communication Technology Research, 4(1), 38–44.
Bremser, W. G., & White, L. F. (2000). An experimental approach to learning about the balanced
scorecard. Journal of Accounting Education, 18(3), 241–255.
Brown, C. (2012). Application of the Balanced Scorecard in Higher Education Opportunities
and Challenges: An evaluation of balance scorecard implementation at the College of St.
Scholastica (pp. 40–50). Society for College and University Planning (SCUP).
Brown, C. (2017). Balanced scorecard implementation in a school of nursing: A case study
analysis. Planning for Higher Education, 45(4).
Chalaris, M., Chalaris, I., Gritzalis, S., & Tsolakidis, A. (2015). Modeling and transforma-
tion of the evaluation mechanism of Greek higher education institutes using balanced
scorecard technique. International Journal on Integrated Information Management, 2, 28–
40. Retrieved from http://ejournals.teiath.gr/index.php/JIIM/article/view/3061/2812. https://
doi.org/10.15556/IJIIM.02.01.004
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making
Chen, S. H., Yang, C. C., & Shiau, J. Y. (2006). The application of balanced scorecard in the
performance evaluation of higher education. The TQM Magazine, 18(2), 190–205.
Chimtengo, S., Mkandawire, K., & Hanif, R. (2017). An evaluation of performance using the
balanced scorecard model for the university of Malawi’s polytechnic. African Journal of
Business Management, 11(4), 84–93.
Cullen, J., Joyce, J., Hassall, T., & Broadbent, M. (2003). Quality in higher education: From
monitoring to management. Quality Assurance in Education, 11(1), 5–14.
De la Mano, M., & Creaser, C. (2014). The impact of the balanced scorecard in libraries: From
performance measurement to strategic management. Journal of Librarianship and Information
Science, 48(2), 191–208.
Del Sordo, C., Orelli, R. L., Padovani, E., & Gardini, S. (2012). Assessing global performance in
universities: An application of balanced scorecard. Procedia Social and Behavioral Sciences,
46, 4793–4797.
El Junusi, R., Musahadi, M., & Yuningrum, H. (2019). Balanced scorecard: Strategy towards world
class university. Economia: Journal Ekonomi Islam, 10(1), 87–116.
Elola, L. N., Tejedor, J. P., & Tejedor, A. C. P. (2016). Analysis of the causal relationships in
the balanced scorecard in public and private Spanish universities through structural equation
modelling. The Business & Management Review, 7(5), 18–29.
Eltobgy, H., & Radwan, M. M. (2010). Monitoring Egyptian higher education institutions perfor-
mance development, the balanced scorecard approach. Report (pp. 1–15) Higher Education
Enhancement Projects Management Unit, Ministry of Higher Education.
Farid, D., Nejati, M., & Mirfakhredini, H. (2008). Balanced scorecard application in universities
and higher education institutes: Implementation guide in an Iranian context. Annals of
University of Bucharest, Economics and Administrative Series, Nr., 2, 31–45.
Fijalkowska, J., & Oliveira, C. (2018). Balanced scorecard in universities. Journal of Intercultural
Management, 10(4), 57–83.
Gamal, A., & Soemantri, A. I. (2017). The effect of balanced scorecard on the private college
performance (case study at the University of WR Supratman Surabaya). Archives of Business
Research, 5(5), 126–134.
Hafner, K. A. (1998). Partnership for performance: The balanced scorecard put to
the test at the University of California. Retrieved October 25, 2020, from http:/
/www.balancedscorecardreview.com/pages/bsc-in-practice/case-studies/university-of-
california-united-states-103.html
Hejazi, R., Rahmani, A., & Mosallanejad, A. (2017). The determination of measures of balanced
scorecard model in university; case: Alzahra university (with emphasis on growth and learning
perspective). Organizational Culture Management, 14(4), 961–980.
Hladchenko, M. (2015). Balanced scorecard—A strategic management system of the higher
education institution. International Journal of Educational Management, 29(2), 167–176.
Ismail, T. H., & Al-Thaoiehie, M. (2015). A balanced scorecard model for performance excellence
in Saudi Arabia’s higher education sector. International Journal of Accounting Auditing and
Performance Evaluation, 11(3–4), 255–280.
Jairak, K., & Praneetpolgrang, P. (2013). Applying IT governance balanced scorecard and
importance-performance analysis for providing IT governance strategy in university. Informa-
tion Management & Computer Security, 21(4), 228–249.
Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard-measures that drive performance.
Harvard Business Review, January–February, 70–79.
Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard: Translating strategy into action.
Harvard Business School Press.
Kaplan, R. S., & Norton, D. P. (2004). Strategy maps: Converting intangible assets into tangible
outcomes. Harvard Business School Publishing Corp.
Kettunen, J. (2008). A conceptual framework to help evaluate the quality of institutional perfor-
mance. Quality Assurance in Education, 16(4), 322–332.
Kiriri, P. N. (2019). Management of performance in higher education institutions: The application
of the balanced scorecard. European Journal of Education, 1(3), 168–175.
Lassoued, K. (2018). Balanced scorecard implementation in higher education: An Emirati
perspective. Corporate Ownership & Control, 15(3), 205–216.
Lawrence, S., & Sharma, U. (2002). Commodification of education and academic labor. Using
the balanced scorecard in a university setting. Critical Perspectives on Accounting, 13(5–6),
661–677.
Lewis, V., Hiller, S., Mengel, E., & Telson, D. (2013). Building scorecards in academic research
libraries: Performance measurement and organizational issues. Evidence Based Library and
Information Practice, 8(2), 183–199.
Lin, M. H., Hu, J., Tseng, M. L., Chiu, A. S. F., & Lin, C. (2016). Sustainable development in
technological and vocational higher education: Balanced scorecard measures with uncertainty.
Journal of Cleaner Production, 120, 1–12.
Maria, E., Wijaya, L. S., & Fibriani, C. (2013). Evaluation of implementation on information and
communication technology in higher education institutions in Indonesia using the IT balanced
scorecard (case study: Satya Wacana Christian University, Salatiga). Journal of Arts Science &
Commerce Vol. IV, 3(1), 49–57.
McDevitt, R., Giapponi, C., & Solomon, N. (2008). Strategy revitalization in academe: A balanced
scorecard approach. International Journal of Educational Management, 22(1), 32–47.
Mohammed, A. H., Taib, C. A., & Nadarajan’s, S. R. (2016). Quality management practices,
organizational learning, organizational culture, and organizational performance in Iraqi higher
education institution: An instrument design. International Journal of Applied Business and
Economic Research, 14(14), 1–19.
Morley, L., Leach, F., Lussier, K., Lihamba, A., Mwaipopo, R., Forde, L. D., & Egbenya, G.
(2010). Widening participation in Higher Education in Ghana and Tanzania: Developing an
equity scorecard: An ESRC/DFID poverty reduction programme research project. University
of Sussex, Center for Higher Education & Equity Research. was assessed on July 22,
2021 from https://assets.publishing.service.gov.uk/media/57a08afbed915d622c000a0d/60335-
FinalReport.pdf
Ndoda, G. R., & Sikwila, M. N. (2014, September). Ubuntu-praxis: Re-modelling the balanced
scorecard model at a university, an Afrocentric perspective. Research in Higher Education
Journal, 25, 1–27.
Ozdemir, A., & Tuysuz, F. (2017). An integrated fuzzy DEMATEL and fuzzy ANP based balanced
scorecard approach: Application in Turkish higher education institutions. Journal of Multiple-
Valued Logic and Soft Computing, 28(3), 251–287.
Papenhausen, C., & Einstein, W. (2006). Implementing the balanced scorecard at a college of
business. Measuring Business Excellence, 10(3), 15–22.
Philbin, S. P. (2011). Design and implementation of the balanced scorecard at a university institute.
Measuring Business Excellence, 15(3), 34–45.
Pietrzak, M., Paliszkiewicz, J., & Klepacki, B. (2015). The application of the balanced scorecard
(BSC) in the higher education setting of a polish university. Online Journal of Applied
Knowledge Management, 3(1), 151–163.
Raghunadhan, T. (2009). Strategy: A pedagogy for efficient, accountable, and socially responsive
higher education. Global Business and Management Research, 1(1), 36–49.
Ratnaningrum, Y., Aryani, A., & Setiawan, D. (2020). Balanced scorecard: Is it beneficial enough?
A literature review. Asia Journal of Accounting Perspectives, 13(1), 65–84. https://doi.org/
10.22452/AJAP.vol13no1.4
Reda, N. W. (2017). Balanced scorecard in higher education institutions: Congruence and roles to
quality assurance practices. Quality Assurance in Education, 25(4), 489–499.
Rompho, N. (2008). Building the balances scorecard for the university case study: The University
in Thailand. Journal in Professional Accountancy, 4(9), 55–67.
Ruben, B. D. (1999). Toward a balanced scorecard for higher education: Rethinking the college
and university excellence indicators framework. Higher Education Forum, 99(2), 1–10.
Saaty, T. L. (1994). How to make a decision: The analytic hierarchy process. Interfaces, 24(6),
19–43.
Sayed, N. (2013). Ratify, reject or revise: Balanced scorecard and universities. International
Journal of Educational Management, 27(3), 203–220.
Schneiderman, A. M. (1999). Why balanced scorecards fail. Journal of Strategic Performance
Measurement. http://atpadvisory.com/pdfs/Why%20BSCs%20Fail.pdf (the journal ceased pub-
lication in 2000).
Self, J. (2003). From values to metrics: Implementation of the balanced scorecard at a university
library. Performance Measurement and Metrics, 4(2), 57–63.
Stewart, A. C., & Carpenter-Hubin, J. (2001). The balanced scorecard: Beyond reports and
rankings. Planning for Higher Education, 29(2), 37–42.
Sudirman, I. (2012). Implementing balanced scorecard in higher education management case
study: Hasanuddin University of Indonesia. International Journal of Business and Social
Science, 3(18), 199–204.
Swierk, J., & Mulawa, M. (2015). The balanced scorecard for higher education—The case of Maria
curie-Sklodowska university. Barometer Regionalny, 13(3), 169–178.
Taylor, J., & Baines, C. (2012). Performance management in UK universities: Implementing the
balanced scorecard. Journal of Higher Education Policy and Management, 34(2), 111–124.
Umashankar, V., & Dutta, K. (2007). Balanced scorecards in managing higher education institu-
tions: An Indian perspective. International Journal of Education Management, 21(1), 54–67.
Umayal Karpagam, P. L., & Suganthi, L. (2012). A strategy map of balanced scorecard in academic
institutions for performance improvement. IUP Journal of Business Strategy, 9(3), 7–16.
Usoh, E. J., & Peterson, J. (2017). Strategic planning and performance measurement for public
universities in Sulawesi, Indonesia; quantitative approach. PEOPLE: International Journal of
Social Sciences, 3(3), 174–197.
Valderrama, T., Cornejo, V., & Bordoy, D. (2013). Balanced scorecard and efficiency: Design and
empirical validation of a strategic map in the university by means of DEA. American Journal
of Operations Research, 3, 30–52.
Weerasooriya, W. M. R. B. (2013). Performance evaluation using the balanced scorecard: The case
of Sri Lankan universities. World Review of Business Research, 3(4), 125–137.
Yakhou, M., & Ulshafer, K. (2012). Adapting the balanced scorecard and activity-based costing to
higher education institutions. International Journal of Management in Education, 6(3), 258–
272. https://doi.org/10.1504/IJMIE.2012.047270
Zangoueinezhad, A., & Moshabaki, A. (2011). Measuring university performance using a
knowledge-based balanced scorecard. International Journal of Productivity and Performance
Management, 60(8), 824–843.
Chapter 8
System Analysis of Technology Transfer
Policies and Models in Higher Education
Arnon Bentur, Daphne Getz, and Oshrat Katz Shacham
Abstract The current chapter presents a critical evaluation of the modes of

knowledge and technology transfer from academia, based on the evaluation of data
which spans over the last three decades and a case study of technology transfer in the
fields of artificial intelligence, data science, and smart robotics. A need emerges for
reevaluating and revisiting university policies with regards to its third mission. Such
policies should be set to guide the activities of the Technology Transfer Offices of
universities, to balance between technology commercialization, which is more linear
in nature, and technology transfer with industry, which is more holistic, interactive,
and entrepreneurial in nature. Greater emphasis on technology transfer and more
intimate cooperation with industry may result in an increase in research funding as
well as in improved level and significance of research. More than that, such policies
are more likely to be met by support of the academic faculty, and they should be an
inherent part of the development of entrepreneurial activities in universities, which
include education that is suitable to the needs of the industry and society, as well
as more significant and effective research. It is recommended that the achievements
of the universities regarding the third mission should be quantified and ranked on
the national and international levels. They should be based on multiple indices to
reflect various modes of achieving such goals, in view of the range of mechanisms
of university–industry interactions.
Keywords Technology transfer · Commercialization · Higher education
A. Bentur · D. Getz () · O. K. Shacham

Samuel Neaman Institute, Haifa, Israel
e-mail: bentur@technion.ac.il; daphne@sni.technion.ac.il; oshrat@sni.technion.ac.il

242 A. Bentur et al.
8.1 Introduction
The most valuable contribution of academia to society and economy is the human
capital: engineers and scientists, educated at the universities, as well as the open
knowledge produced through fundamental research. These are, perhaps, the most
important roles of academia to the knowledge-driven modern society of the twenty-
first century. They are directly related to its core missions of education and research.
In the midst of the previous century, following the Second World War, the indirect
and tacit process of innovation was considered to be sufficient for transferring
knowledge to industry via an innovation process described by Bush (1995) as a
linear process, made up of the following stages:
Basic research → Applied research → Development → P roduction/P roducts
This linear innovation model is no longer adequate, since the interactions

between basic research, applied research and development are much more complex,
with the boundaries between them becoming blurred, as is the case with boundaries
between engineering and scientific disciplines.
This change is the driving force for reevaluation of the role and mode of operation
of universities, from one where the only two missions were education and research,
to one where a “third mission” is added, labeled sometimes as “technology transfer,”
whereby the university should be actively and directly involved in getting its know-
how into the industry and society. This should take place without relying only on
the tacit and spontaneous influences associated with the core missions of education
and research. The significance of this approach shows up in discussions that
highlight the need for universities to be much more entrepreneurial, with the term
“entrepreneurial university” being frequently used (U.S. Department of Commerce,
2013; Eesley & Miller, 2018; Etzkowitz & Leydesdorff, 2000; Gibb, 2012; Graham,
2014; Ouchi, 2011; Rauw et al., 2014; UCLA, 2011; Vaquero-García et al., 2016;
Yoon & Lee, 2013).
The current chapter is based on a critical evaluation of the modes of knowledge
and technology transfer from academia (Bentur et al., 2019), based on the evaluation
of data that spans over the last three decades and case study of technology transfer
in the fields of artificial intelligence, data science, and smart robotics.
8.2 The “Third Mission” and the Entrepreneurial University
The implementation of the “third mission” is quite a challenge, as it requires new

modes of action to be taken, involving a cultural change. All of these should be
carried out with care, to make sure that they do not come at the expense of the
contribution to society which is based on the first two core missions of education and
research. In the case of technological universities, the third mission is accomplished
8 System Analysis of Technology Transfer Policies and Models in Higher Education 243
by developing special modes of interaction, to provide direct transfer of technology

to the industry. This involves a range of mechanisms, all falling under the umbrella
of “university–industry relations.”
The active steps that a university must take in the transfer of know-how, which
is an essential part of its expected third mission, requires a holistic approach, which
includes several components, in addition to the basic knowledge dissemination, such
as the ones presented in Fig. 8.1.
The items in Fig. 8.1 have been proposed by the European Commission (Finne
et al., 2009), to quantify knowledge transfer from public research organizations in
Europe. They are grouped here into three categories:
Knowledge dissemination: The traditional role of universities, such as open scien-
tific publications, conference meetings, education of engineers and scientists.
Technology Transfer: Direct interactions between university researchers and indus-
try through mechanisms such as contract research, joint research and consulting,
government subsidies to generate generic technologies by interaction between
university and industry partners, or consortium of several universities and
industries.
Technology Commercialization: Dominant mode of action of the Technology
Transfer Offices (TTO) and Technology Licensing Offices (TLO) of universities,
whereby faculty are required to disclose inventions and the TTO/TLO unit is in
charge of commercializing it by modes such as patenting followed by licensing
and creation spin-offs.
The terminology currently used in universities, “Technology Transfer,” reflects
essentially the efforts to commercialize universities’ intellectual property (IP), and
it is somewhat misleading, as it implies that this is the only mode of transfer. The
transfer of know-how involves all three categories in Fig. 8.1 whereas the proactive
technology transfer associated with the third mission of universities is based on
both, “technology transfer” and “technology commercialization,” as defined above.
The Bayh–Dole Act of 1980 in the USA was enacted based on the perception
that university ownership of IP, produced by publicly funded research, will result
in mobilizing universities to be proactive in utilizing the fruits of research for the
benefit of society, consistent with the idea of encouraging universities into being
active in promoting the “third mission.” Many of the developments since then, at the
universities, such as the establishment of TTOs and TLOs, as well as the increase
in patenting activities, have been ascribed to this Act. However, later discussions
have placed doubt on this view (Kenney & Patton, 2009; Sampat, 2006; Valentin &
Jensen, 2007), suggesting that the Bayh–Dole Act was accelerating processes that
were already incubating.
244
Fig. 8.1 Mechanisms of knowledge dissemination, technology transfer, and commercialization (Bentur et al., 2019)
A. Bentur et al.
8.3 Technology Commercialization
The mechanism of technology commercialization is to a large extent linear in nature,

involving successive steps:
Scientif ic discovery → I nvention disclosure →

Evaluation/patent application → Licensing/startup f ormation
The evolution with time of the commercialization activity is presented in Fig. 8.2
and expressed in terms of the number of TTOs/TLOs established in US universities
over time. The rise in the number of TTOs/TLOs formed during the 1990’s is related
by some to be the result of the Bayh–Dole Act enacted in 1980.
Over that time period, there has been a continuous increase in licensing activities
and commercialization income, which accompanied an increase in research funding.
Thus, an in-depth analysis of the trends cannot be simply interpreted as a rise in
the intensity of commercialization activities and benefits, and it should rather be
analyzed taking into account the rise in research expenditures, as highlighted in
Fig. 8.3. The data in this figure shows that the commercialization income remained
practically constant relative to research funding over the last two decades.
Analysis of the nature of IP indicates that most of the commercialization income
is coming from a small fraction of the inventions. The commercialization income
is largely obtained from running royalties and not cashed in equities (Love, 2014;
Merrill & Mazza, 2011; Reslinski & Wu, 2016).
The distribution of commercialization of income between US universities during
the period 1991–2017 is presented in Fig. 8.4 in relative terms, % of R&D
Fig. 8.2 Development of TTOs in US universities, 1965–2015 (Bentur et al., 2019), based on
AUTM data, extension of data presented by Valdivia (2013)
Fig. 8.3 Trends of total research expenditure and relative values (% of total research expenditure)
of research funded by industry and commercialization income for US universities, reported in the
AUTM surveys, 1991–2017 (Bentur et al., 2019), based on AUTM data (AUTM data surveys: the
number of universities surveyed increased from 72 to 115 during 1991–1993, thereafter increasing
at a mild rate up to 160 in 2003, and from there on remained almost constant, to 163 in 2017)
Fig. 8.4 Distribution of commercialization income in relative terms, % of research expenditures,

over the period 1991–2017 (Bentur et al., 2019), based on AUTM data
expenditures. The data indicates that most universities earn only a few percentages
of income relative to their R&D expenditures, with 50% of the universities less than
1%, and about 85% less than 5%.
The distribution in income presented in Fig. 8.4 has not changed in almost
20 years (Bentur et al., 2019).
This financial data indicates that only a few of the inventions are commercialized,
and only a few of the latter lead to income of some significance. In addition, only a
few universities earn significant income from IP commercialization. Valdivia (2013)
analyzed such trends and demonstrated that commercialization income is derived
mainly from blockbusters, and the probability of such blockbusters is dependent
on the overall research expenditures of the university. It was also shown that the
blockbuster income from commercialization tends to be skewed also toward life
sciences, as shown for data collected by Nature Biotechnology (Huggett, 2014).
8.4 Technology Transfer
8.4.1 Mechanisms
The flow of know-how from university research to industrial development and

innovation has many avenues with considerable impact, as has, for example, been
highlighted by Cohen et al. (2002) and quantified based on a survey of industries,
Fig. 8.5.
It can be seen that mechanisms which are defined as technology transfer in
Fig. 8.1 are relatively high on the list in Fig. 8.5: informal interactions, consulting,
contract research, and cooperative R&D projects. They are much higher on the list
than the Technology Commercialization mechanisms: patents and licenses.
The impact of “Technology Transfer” mechanisms is largely indirect and the
IP created is either jointly owned by the university and industry or industry only.
This is in contrast with the “Technology Commercialization” mechanism where the
output can be more readily quantified in terms of the number of patents, licenses,
and income. Thus, the quantification of “Technology Transfer” is based, usually, on
inputs (e.g., funding of research projects) rather than outputs.
The impact of “Technology Transfer” is the result of many mechanisms, with
some interacting with each other. Some are formal mechanisms that can be
readily identified, e.g., research contracts, and some are informal, driven by human
interactions between researchers from academia and industry who cooperate on
specific research. Thus, there is a need to resolve such mechanisms first and
thereafter to try quantifying them.
Schaeffer et al. (2018) investigated the interaction of mechanisms with the
notion that there are significant informal interactions that have value and should
be considered, and the presence of a contract, defining the formal relation, is not
sufficient to evaluate the impact of the interactions. There are formal interactions
that follow the linear model of innovation, where there is little surrounding tacit
knowledge transfer, such as patent licensing agreements, and there are ones where
continued and intense personal interactions may take place, such as in industrial
research contracts, where face-to-face interactions are quite common.
Fig. 8.5 Importance to industrial R&D of sources of information generated in public R&D (Bentur
et al., 2019), adopted from Cohen et al. (2002)
These case studies led Schaeffer et al. (2018) to identify strong complementar-
ities between the formal and informal mechanisms, which reinforce one another,
consistent with the report of Grimpe and Hussinger (2016). Informal links often
result in the establishment of formal interactions such as research agreements and
startup formation. Formal relations are often followed by personal interactions
which result in additional collaborations. It is the cumulative effects of these
interactions which lead to efficient technology and knowledge transfer, and this is
fed by good science and evolves into activities of mixed teams of academic and
industrial researchers. In many ways, this is an entrepreneurial activity whereby
the university needs to develop wider strategies and modes of operation for its
TTO/TLO, beyond the conventional mode of operation, where TTO/TLO staff is
considered just as intermediaries in charge of patenting, licensing and establishment
of new companies.
The survey carried out by Link and Siegel (2005) indicated that faculty members
who are active in securing research grants are preferring the informal modes of
technology transfer. This may imply that there is tension between the grant-active
faculty and university incentives to take part in formal technology transfer modes,
perhaps because the former is more basic in nature and the latter are more applied.
8.4.2 Quantification Mechanisms of Technology Transfer
Various attempts have been made to quantify these mechanisms, each of them on its
own, and a cumulative index incorporating all of them. Some examples highlighting
different modes of quantifications will be presented here.
The most direct index to quantify the intensity of technology transfer associated
with university–industry relations is the volume of research carried out by universi-
ties in cooperation with industry, based on industry funding. This is best described
in terms of the proportion of research supported by industry relative to total research
funding, as shown in Fig. 8.6 using OECD data-base for OECD countries, USA and
Europe, since 2000.
It can be seen that over the period since 2000, the proportion of industry-funded
research in academia has not increased and even showed a slight decline, especially
in the USA.
The trends for industrial-supported research, commercialization income, and
total research funding for the US universities included in the AUTM survey are
presented in Fig. 8.3. Total research funding has been increasing continuously over
the years. However, the relative proportion of industrial funding has been decreasing
while the relative commercial income seems to be stable since 2000, although very
variable over this period. Such trends imply that the intensity of technology transfer
and technology commercialization since 2000 have been stable at best or even
declining, contrary to the general belief that they have been intensifying.
Fig. 8.6 Proportion of research supported by industry in academia, average values in USA,
Europe, and OECD countries (Bentur et al., 2019), OECD data
The distribution of relative industrial research funding in the top 100 US

universities indicates that there are very few universities with more than 10%
industrial funding, falling quickly to levels below 10% for most universities in the
top 100 list (AUTM data; Bentur et al., 2019).
The trends above clearly highlight that the intensity of university–industry
relations as quantified in terms of industry-funded research have not been intensified
over the years, in spite of the national and international programs and policies to
encourage more industrial research at universities.
Since industrial-supported research is not the only mechanism for university–
industry relations and technology transfer, it is of interest to explore and quantify
other mechanisms.
Tijssen et al. (2016) defined seven indices by which to characterize technology
transfer and rank universities accordingly (Table 8.1).
Table 8.1 Indices quantifying university technology transfer (Bentur et al., 2019), after Tijssen
et al. (2016)
Index Description
%UIC Share of university–industry co-publications (as a % of research
publication output)
%MA UIC Share of multiple-affiliation university–industry co-publications with at
least one author listing a university address and a company address (as a
% the total UIC output)
%LOCAL UIC Share of university–industry co-publications with at least one partner
company within a 50-km range of the city in which the university is
located (as a % the total UIC output)
%DOMESTIC UIC Share of university–industry co-publications with at least one partner
company located in the same country as the university (as a % of the
total UIC output)
%CO-PATENT Share of granted international patents with a co-assignee from the
business sector (as % of all granted international patents)
%NPLR share Share of non-patent literature references, i.e., publications cited in the
reference list of international patents (as a % total research publication
output)
%NPLR–HICI Share of non-patent literature references within the world’s top 10%
most highly cited international patents across all technology areas
Data sources used by Tijssen et al. (2016) to quantify these indices included:
• Leiden Ranking 2014 (publication years 2009–2012) for %UIC
• CWTS (publication years 2009–2012; Web of Science) for %MA UIC
• U-Multi-rank 2014 (publication years 2009–2012; Web of Science) for %LOCAL UIC
• UIRC Scoreboard 2014 (publication years 2009–2012) for %DOMESTIC UIC
• U-Multi-rank 2014 (PATSTAT INCENTIM, Univ. Leuven; patent years 2002–2011) for %CO-
PATENT
• U-Multi-rank 2014 (Web of Science; PATSTAT Univ. Leiden; patent and publication years
2002–2011; citing years 2002–2014) for %NPLR Share
• PATSTAT Univ. Leiden (patent and publication years 2002–2011; citing years 2002–2014) for
%NPLR–HICI
They analyzed several universities with an attempt to develop a comprehensive

index that consists of some combination of the seven indices in Table 8.1. One such
index was obtained from the combination of the indices in the table, giving each of
them an equal weight, labeling it as RIU—Equal weights index.
Statistical analysis could reduce or remove redundancies between the indices,
where lower weights could be assigned to those that add little additional infor-
mation. Such redundancies could be detected by applying statistical analysis to
pairwise correlation coefficients between the selected metrics. Based on such
analysis, different weights were assigned to the indices in Table 8.1, to define a
single joint index based on non-equal weights using the following weights for the
indices in Table 8.1:
%UIUC: 0.90
%MA UIC: 0.77
%Local UIC: 0.35
%Domestic—UIC: 0.60
%CO—Patent: 0.29
%NPLR: 0.74
%NPLR-HICI: 0.13
The index based on these weights is labeled U-I R&D Index.

Ranking of selected universities based on these indices was presented, indicating
that the universities can vary in the intensity of university–industry relation for the
various mechanisms. This suggests that there is a need for a pluralistic approach that
is not based on a single measure.
The need for a composite index taking into account the various mechanisms
of technology transfer and commercialization was highlighted by the European
Commission Expert Group on Knowledge Transfer Indicators (Finne et al., 2011),
outlining three types of indicators: Knowledge transfer through trained people
(essentially similar to Knowledge Dissemination in Fig. 8.1), Institutional cooper-
ation in R&D and other phases of innovation (essentially similar to Technology
Transfer in Fig. 8.1), and Commercialization of research (essentially similar to
Technology Commercialization in Fig. 8.1). A comprehensive metrics approach
along such concepts was developed in the UK for evaluating the performance of
universities, based on an extensive list of mechanisms of technology transfer and
commercialization (Holi et al., 2008), such as:
Indices for Technology Transfer:
• CPD—Continued Professional Development
• Consulting (number of contracts, income)
• Collaborative research (number of contracts, income)
• Industrial research funding (number of contracts, income)
Indices for Technology Commercialization:

• IP (income, number of patents and licenses)
• Startups (number, turnover, investment)
These indices and more are part of the indices used in the UK to evaluate and
provide public support to universities for their knowledge exchange strategy within
the framework of the Higher Education Innovation Funding (HEIF). The data for
such indices and evaluation is collected officially and presented by the Higher
Education Statistics Agency (HESA). It is usually normalized to take into account
the size of the university, and one of the common methods for normalization is to
present the data per full-time equivalent of academic staff, FTE, namely each value
is divided by the FTE value. The availability of such data is used to characterize
each university, using a spider diagram.
This UK approach seems to be a sensible one as it provides a comprehensive
quantification of the technology transfer of different universities and allows to
resolve the differences in nature of such activities between the universities, iden-
tifying strengths and weaknesses. It also allows to promote the notion that the view
of technology transfer should be a pluralistic one, with each university seeking the
path where it has special added value. The application of this approach requires that
all universities apply a similar code of data collection and presentation.
8.5 Case Study: Knowledge Transfer Processes in the Fields

of Artificial Intelligence, Data Science, and Smart
Robotics
Artificial Intelligence (AI) and Data Science (DS) are emerging fields whose
significant influence on society is predicted to keep on growing. OECD experts
claim that “AI is transforming societies and economies. It promises to generate
productivity gains, improve well-being and help address global challenges, such
as climate change, resource scarcity and health crises” (OECD, 2019). Cooperation
between academia and industry is a significant factor in promoting AI and DS, as
evidenced by a series of publications published by the Samuel Neaman Institute in
2018 and 2019 (Getz et al., 2018a,b, 2019).1
1 In 2018, The Samuel Neaman Institute was commissioned by the National Council for Research
and Development (MOLMOP) at the Ministry of Science and Technology to perform a com-
prehensive mapping of activities in the Israeli academy, industry and government sectors in
artificial intelligence, data science and smart robotics and to explore the possibilities for promoting
and developing these fields in Israel. For this purpose, the research group conducted interviews
with nearly 90 specialists from around the Israeli ecosystem and surveyed employees from 160
companies. The research team published several different research reports, all can be found
on the Neaman Institute website: https://www.neaman.org.il/Artificial-Intelligence-Data-Science-
and-Smart-Robotics
Since Data plays a crucial part in the development of these fields, massive
databases have always been considered valuable resources in search engines, social
networks, and other companies. For many years, such companies invited academics
to analyze their data. This helped power theoretical research in academia and realize
predictive applications. It also created a symbiotic relationship between academia
and industry, without which, the fields of artificial intelligence and data science
would not have gotten as far as they did (Benson, 2018).
Recent research shows that the relationships between academia and industry
in the fields of AI and DS are still evolving. Collaborative and participative
relationships replacing the traditional technology donor-recipient relationship in
these fields benefit both industry and academia: Industry benefit from getting
access to state-of-the-art knowledge, creating innovative solutions for industrial
problems, and improving competitiveness. Academia benefit from getting access
to real data, educating students/researchers on real-world problems and applied
research, and getting funds to support research activities (Khamis, 2015). There
are also downsides to such relationships: Patel et al. (2019) suggested that the
significant increase in the amount and depth of interactions and engagements
between academia and industry presents increased complexities for universities to
understand and manage the interaction and in particular the impact it has on the
university culture and education mission.
There are many forms of academia–industry collaboration which are acceptable
in universities, for example, receiving research funds from companies (such as
Google and Microsoft) or a day per week consultancy work, done by researchers
for companies. Temporary leave of absence which researchers take to work in
industry for a given period can also be beneficial for universities because they often
result in upgraded teaching skills. However, the challenge to academia comes when
professors leave to join companies either en masse or individually.2 This happens
because demand for AI and DS experts in the industry has grown significantly
and because the industry can offer researchers generous compensation packages
together with distinct data sets and ample computational power. This can create
some problems for universities, which are sometimes resolved through splitting
researchers’ time between industry and academia. (Time spent in both venues can
vary from extreme 80/20 or 20/80 to an even 50/50 split (Peng, 2019)). However,
such a split can lead to conflict of interests that needs to be mitigated through
total separation between campus research and off-campus commercial activities.
There are other, more synergetic approaches. For example: in 2001–2011, Intel
established research labs in prominent computer science departments in American
universities. The labs offered research funding and open intellectual property.
Engineering resources and data sets provided opportunities to create powerful
collaborations. The unavoidable influences on research agenda were monitored and
properly disclosed (Etzioni, 2019). In recent years commercial research labs run
2 See (Gan-El, 2018) and (Levy, 2017) for examples from Israel (HEB) and (Sample, 2017) and
(Metz, 2019) for examples from the USA and the UK.
by Google Research, DeepMind, IBM Research, Nvidia, OpenAI, and others are
becoming more significant. The distinction between industry and academy research
groups is becoming blurred as the former extend their interest to cutting-edge
research (Peng, 2019). Google, for example, had 107 papers accepted at the NeurIPS
Conference in 2018, accounting for 10.5% of the total papers accepted by the
conference (Microsoft Research, 2018).
In Israel, academy–industry collaboration has resulted in some extremely suc-
cessful technologies, such as advanced driver assistance and warning systems
technology used by Mobileye (acquired by Intel in 2017) and robots that learn and
replicate tasks by observing humans, a technology which is used by the Technion’s
spinoff Deep Learning Robotics (DLR). The multidisciplinary nature of AI requires
not only academia and industry working together but also clinicians and researchers
working together in sharing their specific domain expertise to create clinically
meaningful and translatable results (Lin, 2020). Examples of Israel’s success stories
involving AI and DS technologies in the life science domain are DayTwo, which
uses customized nutrition recommendations technology developed at the Weizmann
Institute of Science and Zebra Medical Vision which is based on the medical
imaging insight and analysis platform developed by Mor Research Applications,
the technology transfer company of Clalit Health Services (HMO). It is important
to mention that the Israeli health system has gathered a large body of electronic
data about patients, conditions, and treatments over the years on its interconnected
EMR (Electronic Medical Records) systems. Up until recently the use of this data
(in its anonymized form) was subjected to individual negotiation between the HMO
or hospital which held the data and companies that wanted to use it. The use of such
data usually involved a high fee (sometimes in equity). Recently representatives
of industry and government (including Ministry of Health, HMOs, hospitals, the
Innovation Authority, and others) started drafting guidelines for communication
between public and commercial bodies on medical data that will regulate the subject,
including financially, at the national level.
A current example of academia–industry collaboration in Israel is the new joint
Intel-Technion research center dedicated to artificial intelligence technologies at the
Technion, Israel Institute of Technology. As part of this collaboration, the company
supports research projects of Technion faculty members engaged in computational
learning and artificial intelligence together with Intel researchers. The research
covers a variety of areas, including natural language processing, deep learning,
and hardware optimization for different learning algorithms (Hattori, 2018). The
Technion, like other Israeli universities, benefits from the industrial and academic
experience of visiting professors from industry such as Dr. Kira Radinsky, chairman
and CTO of Diagnostic Robotics (formerly chief scientist and the director of
data science of eBay) who teaches deep learning and natural language process-
ing and also Mentor students for graduate degrees. Another example of Israeli
academia–industry collaboration is the Check Point Institute for Information Secu-
rity (CPIIS).3 The institute is located at the Tel Aviv University School of Computer
Science, and its activities include supporting selected research projects, providing
post-doctoral fellowships, providing stipends for graduate students, providing a
laboratory environment for experimental research, and organizing workshops and
symposia. The CBG4 (Cyber@Ben-Gurion University of the Negev) is an umbrella
organization at Ben Gurion University, being home to various cybersecurity, big
data analytics and AI applied research activities. It is a salient academia–industry–
government collaboration with the Israeli National Cyber Bureau, the Israeli Police,
and Telekom Innovation Labs as partners.
8.6 Policy Implications: Technology Transfer Versus

Technology Commercialization
8.6.1 Income Generation Versus Technology Diffusion
Optimization of “technology commercialization” policies is often the main driver

for TTO’s/TLO’s existence. It is largely driven by the attempt to maximize
revenues for the university and with this goal in mind, IP protection policies are
being developed. However, the data presented suggest that the income received is
relatively small. Merrill and Mazza (2011) reported that the average “rate of return”
on funded research is about 4.1% and the median is 0.9%.
More than that, estimates of the net income from commercialization, after
considering the expenses involved, such as staff of TTO and patent expenses,
suggest that out of 155 TTOs reporting to AUTM, 130 did not break even and
were at a loss; it is estimated that over the last 20 years 87% of the universities
did not break even (Valdivia, 2013). This is further echoed by Love (2014), based
on a survey which concluded a negative rate of return (e.g., loss of about 3%), and
similar trends reported by Thursby and Thursby (2007) based on surveys.
These gloomy trends seem to have remained approximately constant over most
of the last two decades.
Thursby and Thursby (2007) discussed several issues in view of the limited or
negative financial gains by licensing and based their analysis on responses to surveys
they have taken. Most important was the input of the university stakeholders with
regards to the goals of technology transfer: Administration and TTO/TLO officials
see it as highly important while faculty take the position that financial gains is not
the only goal, as seen from the importance of indicators to success. The opposite
is the response to the issue of sponsored research funds: faculty see it as highly
important while administration and TTO/TLO officials place it as mildly important.
3 http://cpiis.cs.tau.ac.il/
4 https://cyber.bgu.ac.il/
In the consideration of faculty perception, there is a need to consider the

variability of the faculty personality and attitude. Lavie (2019) suggested to
approach the faculty taking into account three different profiles: (i) faculty who
have the entrepreneurial intentions as well as the skills and can proceed in realizing
the potential of their invention on their own, (ii) faculty who have no interest
in entrepreneurial activities to develop further their inventions and prefer to let
the organization proceed with it without their intervention, (iii) faculty who have
entrepreneurial aspirations but do not have entrepreneurial skills, who want to lead
the application of their inventions in spite of their lack of the capabilities needed.
The policies for the first and second groups should be quite different, while the
third group, is the most problematic. Consideration of these differences suggests
that flexible policies need to be developed, and the TTO/TLO should be autonomous
in implementing them, considering the profiles of the faculty members.
In the assessment of technology diffusion, attention should be given to the
possible negative impact of policies based on maximizing income which leads
to stringent IP protection considerations. Issues of IP protection and commercial
interests have often been cited as impediments in academy–industry negotiations
in research contracts (e.g., Deloitte, 2019), which result in a disappointing level of
cooperation, as might be observed in the data in Fig. 8.6. It has been suggested to
look more into generic standard contract practices, but this is not always an option
considering the differences in the nature of technologies, industries, and markets
(e.g., Deloitte, 2019).
8.6.2 New Policies and TTO/TLO’s Missions
New policies and missions for TTO/TLO require reconsideration of strategies of

universities with respect to maximizing their income. Increasing the probability for
a blockbuster might be achieved by several means such as redirecting and focusing
the research into the most attractive fields for the emergence of blockbusters
and motivating faculty by providing special incentives for promotion. These,
however, will probably be met with strong internal resistance as well as the risk
of jeopardizing other university goals.
In view of such insights, new ideas about strategies have been emerging. Valdivia
(2013) discussed alternative strategies to manage IP and proposed several policy
recommendations:
• Government should provide more funding to Small Business Technology Trans-
fer (TSTTR) programs, designating funds specifically for university startups.
• Congress should authorize a patent use exemption for non-profit research
organizations for exclusive experimental use. Also, enable federal agencies to
use march-in rights under the Bayh–Dole Act to extend nonexclusive licenses
for research tool patents that have been subjected to pricing excess.
• Create an equity rule for the distribution of funds among universities.
Additional ideas and review of several models of the mode of operation of

technology transfer units have been reported by Allen and O’Shea (2014):
Gateway approach: intended to minimize obstacles for commercialization by
joining the offices dealing with research, contracts, technology transfer, relations
with industry, and support for entrepreneurship.
Centric approach: The university develops long-term alliance with experts in
commercialization and IP to get into the market attractive ideas.
Academic Entrepreneurship Model: The office of technology transfer is actively
participating in setting relations between faculty members and venture capital
firms (VCs) and allows them to advance in the entrepreneurship process. The
university gets a small share and the TTO is not involved in running the process.
The emphasis is on getting a process going and making an impact on the
community.
Easy Access IP Model: The model was developed in Europe and it provides the IP
freely. It does not allow for income to the university but enables a great deal of
inventions to reach society. As the emphasis is on impact, the university hopes to
generate sources for financial support of research with the growth of companies
based on this IP.
8.6.3 IP Protection Policies and the Bayh–Dole Act
In view of this situation, there is considerable debate whether there is a proper

balance in university policies and mode of action of TTO/TLO, between technology
commercialization and technology transfer. This has been echoed in several reviews
of the developments following the Bayh–Dole Act of 1980, questioning whether
universities have overemphasized technology commercialization which takes place
largely by the “linear model,” at the expense of technology transfer, which is more
likely to promote interdisciplinary activities made up by groups of scientists from
the academia and industry working together.
Sampat (2006) stated that the purpose of the Bayh–Dole Act was not to provide
revenues to the universities or change the nature of the research from basic to
entrepreneurial and applied, but rather to encourage technology transfer to the
industry and private sector, which could be developed from basic-academic research
that was not commercially oriented. Since the technology transfer from university is
taking place by several modes, in addition to patenting and licensing, it is essential
to evaluate the effect of the Bayh–Dole Act by considering the whole array of
mechanisms. The issue is whether the changes with regards to patenting affected
the traditional mode of knowledge transfer such as publications, conferences, and
informal channels. There is the question of whether universities are patenting
“science” rather than embryonic technologies. The societal costs and benefits of
the Bayh–Dole Act depend less on whether university research is patented or not,
but more on how it is licensed, which is the policy issue that needs to be addressed.
From a societal point of view, one would like to patent public-funded university
research output only in the case that without it the research output would not
be effectively utilized. Also, exclusive licensing should be granted only when a
nonexclusive one will not enable commercialization.
It is, thus, best if universities take at their initiative steps for managing patenting
with the perspective of the public interest rather than their own financial interest.
Consideration for changing the current model of IP ownership which follows the
Bayh–Dole Act was suggested by Kenney and Patton (2009).
Some insight on the impact of IP protection policy might be gained from a study
by Valentin and Jensen (2007). They compared technology transfer in Denmark,
where a law like the Bayh–Dole Act was enacted (Danish Law on University
Patenting—LUP), to Sweden, where the traditional European policy of academic
researcher ownership of IP was maintained. It was concluded that the LUP resulted
in a moderate increase of inventions channeled through the university-owned patent
system, but the larger part of the inventive potential of the academia, previously
mobilized into company-owned patents, seems to have been compromised as a
result of the law. A likely explanation is that exploratory research, which is usually
a target in university–industry joint projects, especially in biotech firms, matches
poorly with LUP IP requirements: The pre-LUP system, allocating IP rights to
the industrial partner in return for research funding and publication rights to the
academic researcher, may have offered more effective industrial contracting for
research. It is concluded, however, that LUP is not uniform in its effect on joint
university–industry research: it will operate better for joint R&D issues closer
to commercial technologies, i.e., exploitive research, but less well for explorative
research.
8.7 Concluding Remarks
The spirit of the concluding remarks echoes quite well with a statement made by Lita
Nelson, the previous director of MIT TLO, in a recent interview (Engell, 2016):
It’s already changing with an emphasis on entrepreneurship. The tech transfer offices have
recognized that except for a few pharmaceuticals—and very few of those, that have been
brought along by hospital funds to the point where the big pharma wants it—on the whole
what comes out patented from university research is too risky and too early for knocking on
the door of established companies.
Any university that counts on its tech transfer to make a significant change in its finances
is statistically going to be in trouble.
So, the universities have to start figuring out how do I ripen the technology to the point
where it enters the big company stream of commerce.
Based on the survey findings, insights, and discussions presented here, a need
emerges for reevaluating and revisiting university policies with regards to its third
mission. Such policies should be set to guide the activities of the TTOs and TLOs,
which need to balance between technology commercialization, which is more linear
in nature, and technology transfer with industry, which is more holistic, interactive,
and entrepreneurial in nature.
Goals should be set for the academia’s contributions to the national economy and
societal needs, and IP protection policies should be adjusted.
Greater emphasis on technology transfer and more intimate cooperation with
industry may result in an increase in research funding as well as in improved level
and significance of research. More than that, such policies are more likely to be
met by support of the academic faculty, and they should be an inherent part of the
development of entrepreneurial activities in universities, which include education
that is suitable to the needs of the industry and society, as well as more significant
and effective research.
It is recommended that the achievements of the universities regarding the third
mission should be quantified and ranked on the national and international levels.
They should be based on multiple indices to reflect various modes of achieving such
goals, in view of the range of mechanisms of university–industry interactions.
Also, there is a need to consider the promotion of the academic staff, their
achievements, and contributions with regards to the third mission.
Nowadays, we are in the era of the fourth Industrial Revolution, Schwab (2017),
which challenges the academia to enter and contribute to new areas in advanced
research and advanced education in order to remain relevant during this revolution.
The education syllabus and teaching methodologies should be modified and updated
to meet the challenges and opportunities of this revolution. The research agenda
should include these new areas of interest. But, more than that, the interactions
between academia, industry, and society should be substantially more active,
holistic, and entrepreneurial in nature.
This calls for a comprehensive approach from all the stakeholders, as outlined
concisely below:
Academia
• Mechanisms: move from commercialization to transfer of technology
• Finances: move from royalties’ income to industrial research funding
• TTO/TLO: entrepreneurial approach of nurturing intimate industrial relations
• IP policies: relaxing, with view of long-term industrial support and donations
Industry
• IP policies: flexibility for publishing in return for IP rights
• Nature of research: support explorative research, not just exploitive/applied
research
Government
• Policies: develop policies with long-term objective of having a sustainable
ecosystem
• Incentives: set criteria for incentives to universities for achieving third mission
goals, based on multiple indices
References
Allen, T., & O’Shea, R. (2014). The roadblock to commercialization. In The Financial Times.
https://www.ft.com/content/a1ea7c72-18cc-11e4-933e-00144feabdc0
Benson, G. (2018, February 22). How industry collaboration with academia advances the
field of AI—AI News. AI News. https://artificialintelligence-news.com/2018/02/22/industry-
collaboration-academia-advances-field-ai/
Bentur, A., Zonnenshein, A., Nave, R., Barzani, E., Zatkovesky, I., & Dayan,
T. (2019). University-industry relations: Evidence based insights. Neaman
Institute, Technion. https://www.neaman.org.il/Files/University-IndustryRelations-
EvidenceBasedInsights_20190603185633.795.pdf
Bush, V. (1995). Science: The Endless Frontier, A Report to The President, Office of Scientific and
Research Development. 1945. Reprint, North Stratford: Ayer Co.
Cohen, W. M., Nelson, R. R., & Walsh, J. P. (2002). Links and impacts: The influence of public
research on industrial R&D. Management Science, 48(1), 1–23.
Deloitte. (2019). Strategies in academic and health system technology transfer to industry. Report
to the Israeli National Council for Research and Development, Deloitte.
Eesley, C. E., & Miller, W. F. (2018). Impact: Stanford University’s economic impact via
innovation and entrepreneurship. Trends® in Entrepreneurship, 14(2), 130–278.
Engell, J. (2016, May). Exit interview: Lita Nelsen on MIT tech transfer, Startups & Culture. Xcon-
omy. https://xconomy.com/boston/2016/05/31/exit-interview-lita-nelsen-on-mit-tech-transfer-
startups-culture/
Etzioni, O. (2019). AI Academy under Siege. Inside Higher Ed. https://www.insidehighered.com/
views/2019/11/20/how-stop-brain-drain-artificial-intelligence-experts-out-academia-opinion
Etzkowitz, H., & Leydesdorff, L. (2000). The dynamics of innovation: From National Systems and
“mode 2” to a triple Helix of university–industry–government relations. Research Policy, 29(2),
109–123.
Finne, H., Arundel, A., Balling, G., Brisson, P., & Erselius, J. (2009). Metrics for knowledge
transfer from public research organizations in Europe: Report from the European Commission’s
expert group on knowledge transfer metrics.
Finne, H., Day, A., Piccaluga, A., Spithoven, A., Walter, P., & Wellen, D. (2011). A Composite
Indicator for Knowledge Transfer. Report from the European Commission’s Expert Group on
Knowledge Transfer Indicators.
Gan-El, O. (2018, February 22). There is a war on workers in the AI field. I’ve already seen
everything. Globes. https://www.globes.co.il/news/article.aspx?did=1001224848
Getz, D., Katz Shacham, O., Klein, R., Rozenberg, S., & Barzani, E. (2019). Artificial Intelligence
Data Science and Smart Robotics. Streamlining the transfer of knowledge from academia and
research institutes in the healthcare system to industry. Neaman Institute, Technion (HEB).
https://www.neaman.org.il/EN/Artificial-Intelligence-Data-Science-and-Smart-Robotics-First-
report-Streamlining-the-transfer-of-knowledge-from-academia-and-research-institutes-in-the-
healthcare-system-to-industry
Getz, D., Katz Shacham, O., Klein, R., Tzezana, R., Rosenberg, S., Shoham, A., Barzani, E.,
Leck, E., & Tziperfal, S. (2018a). Artificial Intelligence, Data Science, and Smart Robotics-
First report summary. Neaman Institute, Technion. https://www.neaman.org.il/Files/Summary-
ENG-Artificial-Intelligence-Data-Science-and-Smart-Robotics_20190103155717.804.pdf
Getz, D., Katz Shacham, O., Klein, R., Tzezana, R., Rosenberg, S., Shoham, A., Tziper-
fal, S., Barzani, E., & Leck, E. (2018b). Artificial Intelligence, Data Science, and Smart
Robotics Companies-Survey. July 2019. https://www.neaman.org.il/en/Artificial-Intelligence-
Data-Science-and-Smart-Robotics-Companies-Survey
Gibb, A. (2012). Exploring the synergistic potential in entrepreneurial university development:
Towards the building of a strategic framework. Annals of Innovation & Entrepreneurship, 3(1),
16742.
Graham, R. (2014). Creating university-based entrepreneurial ecosystems: Evidence from emerg-

ing world leaders. Massachusetts Institute of Technology.
Grimpe, C., & Hussinger, K. (2016). Formal and informal knowledge and technology transfer
from academia to industry: Complementarity effects and innovation performance. Industry and
Innovation, 20(8), 683–700.
Hattori, K. (2018). Technion and Intel inaugurate joint center for artificial intelligence. American
Technion Sosiety. https://ats.org/news/technion-and-intel-inaugurate-joint-center-for-artificial-
intelligence/#v
Holi, M. T., Wickramasinghe, R., & Van Leeuwen, M. (2008). Metrics for the evaluation of
knowledge transfer activities at universities (p. 5). Library House.
Huggett, B. (2014). Reinventing tech transfer. Nature Biotechnology, 32(12), 1184.
Kenney, M., & Patton, D. (2009). Reconsidering the Bayh-Dole Act and the current university
invention ownership model. Research Policy, 38(9), 1407–1422.
Khamis, A. (2015). Machine intelligence and its role in strengthening Industry-Academia Collab-
oration. April. https://doi.org/10.13140/RG.2.1.5007.8241
Lavie, P. (2019). Personal communication, Technion. Israel Institute of Technology.
Levy, R. (2017, January 11). The industry that steals the doctorates from the academy. Themarker.
https://www.themarker.com/technation/1.263652
Lin, M. D. (2020). Accelerating the translation of artificial intelligence from ideas to routine clini-
cal workflow. Academic Radiology, 27(1), 121–122. https://doi.org/10.1016/j.acra.2019.08.019
Link, A. N., & Siegel, D. S. (2005). Generating science-based growth: An econometric analysis of
the impact of organizational incentives on university–industry technology transfer. European
Journal of Finance, 11(3), 169–181.
Love, B. J. (2014). Do university patents pay off-evidence from a survey of university inventors in
computer science and electrical engineering. Yale Journal of Law & Technology, 16(2), 285.
Merrill, S. A., & Mazza, A. M. (Eds.). (2011). Managing university intellectual property in the
public interest. National Academies Press.
Metz, C. (2019, September 6). When the A.I. professor leaves, students suffer, study says.
The New. https://www.nytimes.com/2019/09/06/technology/when-the-ai-professor-leaves-
students-suffer-study-says.html
Microsoft Research. (2018). NeurIPS conference analytics. https://www.microsoft.com/en-us/
research/?post_type=msr-blog-post&p=480543&preview=tru
OECD. (2019). Artificial intelligence in society. OECD Publishing. https://doi.org/10.1787/
eedfee77-en
Ouchi, W. G. (2011). An ecosystem for entrepreneurs at UCLA, An invitation for campus-wide
input. UCLA. https://ucla.app.box.com/s/f0sagdnxi76ui1oqmratjqmhand45dcg
Patel, S., Rexford, J., Zorn, B., & Morrisett, G. (2019). Evolving academia / Industry relations in
computing research (Issue June). https://arxiv.org/pdf/2001.06684.pdf
Peng, T. (2019). Are commercial labs stealing academia’s AI thunder. Medium.
https://medium.com/syncedreview/are-commercial-labs-stealing-academias-ai-thunder-
dd51cf4bd8d6
Rauw, B., Smith, L., Honig, S., & Medovoi, O. (2014). UCLA ecosystem for entrepreneurs, Part
III: A strategic blueprint for entrepreneurship at UCLA submitted for campus discussion.
UCLA. https://ucla.app.box.com/v/Entrep-Ecosystem-III
Reslinski, M. A., & Wu, B. S. (2016). The value of royalty. Nature Biotechnology, 34(7), 685.
Sampat, B. N. (2006). Patenting and US academic research in the 20th century: The world before
and after Bayh-Dole. Research Policy, 35(6), 772–789.
Sample, I. (2017, November 1). “We can’t compete”: Why universities are losing their best
AI scientists. The Guardian. https://www.theguardian.com/science/2017/nov/01/cant-compete-
universities-losing-best-ai-scientist
Schaeffer, V., Öcalan-Özel, S., & Pénin, J. (2018). The complementarities between formal and
informal channels of university–industry knowledge transfer: A longitudinal approach. The
Journal of Technology Transfer, 45, 1–25.
Schwab, K. (2017). The fourth industrial revolution. Penguin Random House.
Thursby, J. G., & Thursby, M. C. (2007). University licensing. Oxford Review of Economic Policy,
23(4). Proceedings from The ISPIM Innovation Conference—Innovation, The Name of The
Game (June 2018), Stockholm: Sweden.
Tijssen, R. J., Yegros-Yegros, A., & Winnink, J. J. (2016). University–industry R&D linkage
metrics: Validity and applicability in world university rankings. Scientometrics, 109(2), 677–
696.
U.S. Department of Commerce. (2013, October). The innovative and entrepreneurial university:
Higher education, Innovation & Entrepreneurship in focus. US Department of Commerce,
Economic Development Administration.
UCLA (2011, September). UCLA ecosystem for entrepreneurs, Part II: Transition to a new
technology transfer process. UCLA. https://ucla.app.box.com/v/Ecosystem-Report-II
Valdivia, W. D. (2013). University start-ups: Critical for improving technology transfer. Center for
Technology Innovation at Brookings. Brookings Institution.
Valentin, F., & Jensen, R. L. (2007). Effects on academia-industry collaboration of extending
university property rights. The Journal of Technology Transfer, 32(3), 251–276.
Vaquero-García, A., del Río, M. D. L. C., & Álvarez-García, J. (2016). Best university practices
and tools in entrepreneurship. In M. Peris-ortiz, J. A. Gomez, F. Velez-Torres, & C. Rueda-
Armengot (Eds.), Education tools for entrepreneurship (pp. 183–198). Springer.
Yoon, H., & Lee, J. J. (2013). Entrepreneurship education and research commercialization of
engineering-oriented universities: An assessment and monitoring of recent development in
Korea. International Journal of Engineering Education, 29(5), 1068–1079.
Chapter 9
Models for Planning and Budgeting
in Higher Education
Zilla Sinuany-Stern
Abstract The budget of an organization reflects its plan in monetary terms over
a given period—usually for a year. This chapter is devoted to planning and
budgeting in higher education (HE). We present various budgeting procedures,
such as incremental budgeting, along with costing methods in HE institutions
(HEIs). A variety of optimization models with budget constraints and bounds on
the allocations to organizational units are given, where the bounds are a result of
the planning process and past budgets. We present linear and nonlinear objective
functions (quadratic), with and without constraints, intertwined with a simulation
scheme in an HEI. The optimization models are presented in a single and multilevel
hierarchy, and over time. Moreover, several aggregative long-run financial models
are presented. We conclude that the quadratic model fits HE better than the linear
model since it provides allocations around the midpoints of the upper and lower
bounds of the allocations rather than the extreme linear model’s allocation. We
determine that the quadratic model procedure is preferred, as its optimal solution
is intuitive and does not require mathematical formulation and skills. Moreover, the
relationship between the mathematical models and known budgeting procedures are
analyzed, and we conclude that the optimization/simulation scheme described here
results in a combination of several budgeting procedures—as actually happens in
practice.
Keywords Planning · Budgeting · Higher education · Simulation ·

Optimization · Hierarchical · Linear · Quadratic
Z. Sinuany-Stern ()
Department of Industrial Engineering & Management, Ben Gurion University of the Negev, Beer
Sheva, Israel

9.1 Introduction
The budget reflects the plan of an institution in monetary terms over a given
period (Lasher & Sullivan, 2004). It has two parts: the operational budget, usually
for 1 year, and the capital budget, often for more than 1 year. Budgeting is the
process of identifying, gathering, summarizing, and communicating financial and
nonfinancial information about an organization’s future activities. The budgeting
process provides managers with the opportunity to match organizational goals
with the resources necessary to accomplish those goals. Sometimes, prior to the
budgeting process, strategic planning is performed, where the long-term goals are
established (usually a 5- to 10-year period) that form the basis for preparing the
annual operating and capital budgets.
This chapter is devoted to planning and budgeting in higher education (HE).
Historically, HE has been defined as a nonprofit organization. The leading HE
institutions (HEIs) are comprehensive universities, which include teaching and
research facilities, typically awarding Bachelor’s, Master’s, and Doctorate degrees
in a variety of disciplines and professions. The main production workers of an
HEI are full-time (FT) tenured academic faculty, whose activities’ quantity and
quality determine the level and prestige of the HEI. Besides teaching, FT faculty
must perform research and must receive grants and other resources to enhance their
research. Some contribution to the community is also expected of tenured faculty.
This multitasking, mainly between teaching and research, with vague borders and
fuzzy requirements, creates tension among faculty and between HEIs, and even
competition. International rankings of HEIs and the h-index of researchers are its
extreme symbols. All these elements determine the difference between secondary
and postsecondary education.
Successful modern universities strive to have a wide national and international
impact on society and economics. For example, on its website, the University of
California, Berkeley, which is ranked among the top universities in the world, states
as part of its mission:
. . . Berkeley also contributes to the economic welfare of the state and world, building
industries and commercializing technologies whose impact is felt around the world. Tens of
thousands of companies have been founded by Berkeley students and professors, creating
well over a million jobs, and generating several hundred billion dollars in economic activity.
Berkeley is also a crucible for the talented workforce that drives California’s innovation
economy: offering a first-rate education to more undergraduates than Stanford, USC, and
CalTech combined, Berkeley and its graduates truly power Silicon Valley . . . (Berkeley,
2020)
Although public HEIs have been defined as nonprofit organizations, as the altruistic
statement above reflects, we can see that even in its mission a state university such as
Berkeley is proud to include for-profit goals. Moreover, the above mission statement
has a marketing flavor. Slaughter and Leslie (1997) call it: academic capitalism, and
the entrepreneurial university. Moreover, Etzkowitz (2001) declares that the rise of
entrepreneurial science in HEIs is proof that we are in the midst of the second
academic revolution, the first academic revolution being the creation of mass HE.
9 Models for Planning and Budgeting in Higher Education 265
Bogomolova et al. (2017, 2018) claim that, even in Russia, universities are involved
with industry in their budgeting; moreover, they prove that total university R&D is
positively correlated with the main macroeconomic indicators in both Russia and
the USA.
Smith (2019) claims that a university is a business; moreover, university bud-
geting terminology is the same as in business, providing an overall description
of all aspects of the budgeting process. The goods and services of a university
are instruction, research, dormitories, cafeterias, and other services. The workers
of the university include academic, administrative, and technical staff—all of
whom receive salaries from the university. The university constructs buildings for
classrooms, offices, libraries, sport facilities, etc. It uses marketing to attract students
and to improve its rating. Moreover, universities are involved in export/import,
namely, in the form of international students and researchers. To finance all these
needs, a university acquires resources from student fees, government allocations,
donors, and graduates. These various constituencies expect transparency regarding
resource allocation and accountability in reference to outcomes and achievements.
The annual budget is the overall summary of all these inputs and outputs.
Appendix 1 presents an annotated literature review of over 40 publications
devoted to planning and budgeting in HE by type of model used, area of application,
HE level (department, university, state, etc.), and country of application. Most of
the articles include mathematical models for planning and budgeting including
optimization, simulation, or forecasting. Some publications are devoted to known
budgeting procedures such as ZBB and PPBS (see Sect. 9.2). Most applications
were applied at the “university level,” with 18 in the USA.
In this chapter, we present a general scheme for an optimization and simulation
model for planning and budgeting in HE (Sinuany-Stern, 1984b). Planning comes
before budgeting in various systems and subsystems of HE, utilizing various
operations research and management science models. Various budgeting procedures
(such as incremental budget and allocation formulas) and optimization models with
budget constraints and bounds on the allocations to organizational units are given,
where the bounds are a result of the planning process with a linear objective function
(ibid.) versus a quadratic objective function (Sinuany-Stern, 2014). Moreover,
several aggregative long-run models are presented (Sinuany-Stern, 1983a). We
show the relationship between the mathematical optimization models and known
budgeting procedures.
The strength of this work is in combining a variety of mathematical models
with known budgeting procedure models, relating to various aspects of planning
and budgeting at different levels of HE, and over time.
This chapter is organized as follows: Sect. 9.2 presents the main types of
budgeting systems used in HE. Section 9.3 lists expenses and income typical
for HEIs and methods to estimate these and their projections. In case there is a
deficit, a simulation/optimization budgeting scheme is presented. Various types of
optimization models for budget allocation are given in Sect. 9.4, when resources are
scarce. Section 9.5 presents aggregated university long-run budget planning models
under various assumptions, and Sect. 9.6 concludes the chapter.
9.2 Types of Budgeting Systems
There are five main budgeting methods relevant to HE: Incremental, ZBB, PPBS,
formula budgeting, and responsibility-centered budgeting.
9.2.1 Incremental Budgeting (IB)
Incremental budgeting (IB) is the oldest and most widely used budgeting method.
It is often based on the most recent budget by adjusting each item in the new
budget by a certain percentage from its historical value, without relating to its
priorities or outcomes. The imbedded assumption is that costs and allocations will
be incremental, as in the past. IB is often used by HEIs for its simplicity (Barr &
McClellan, 2018). However, it does not account for major changes in plans; thus,
often it is combined with other budgeting approaches as needed; namely, certain
parts of the budget are incremental (such as for existing departments), while others
are not incremental (such as for new departments or new projects).
9.2.2 Zero-Based Budgeting (ZBB)
Zero-based budgeting (ZBB) is the opposite of IB. It allocates the budget based on
program efficiency and necessity, rather than on budget history. ZBB reviews every
program and expenditure at the beginning of each budget period and justifies each
line item in order to receive more funding. The new budget always starts from a
zero value for each line item. It requires considerable work and time to justify and
compute how much to allocate for each line item—if at all. Moreover, implementing
ZBB is a long and costly procedure in which internal conflicts may arise within the
institution (Goldstein, 2012), and in practice, it is impossible to implement in its
pure form (Barr & McClellan, 2018, p. 85). Ekanem (2014) reports on implementing
ZBB in a Nigerian university.
9.2.3 Planning–Programming–Budgeting System (PPBS)
Budget planning models have been used in HE since the 1960s (Terrey, 1968).
Weathersby and Balderston (1972) reported about the use of PPBS (Planning–
Programming–Budgeting System) for budget planning in various parts of the USA
and other countries. For example, during 1966–1971, PPBS was used by the
University of California for budget planning (Balderston & Weathersby, 1972), and
in 2009, Gibson (2009) reported on the use of PPBS in HE. As a top-down budgeting
method, it fits the hierarchical HEI budget. The main attributes of this system are that
it relates means to ends (outputs), qualitative dimensions are considered, activities
are aggregated to programs, and various alternatives are evaluated quantitatively
and over time. Thus, PPBS helps in decision-making. Its disadvantages are that the
notion of a program is vague in HEIs (e.g., department), the top-down and bottom-
up processes are cumbersome and require strong centralized leadership, outcomes
of programs are not always easily measured, and a special management information
system needs to be built to implement the system. Barr and McClellan (2018, p.
214) stated that PPBS is a quite complex budgeting model for HEIs.
9.2.4 Formula Budgeting
The future of a program is calculated by the relationship between its costs and
program activity level—often via historical data or a tariff developed per unit of
activity. For example, the state planning and budgeting committee can develop sets
of formulas for HEIs in that state for teaching per student by type of institution
and by type of department. For research, this can be done by the number of papers
published and number of citations, by type of grant and its amount, and specific
allocations for special projects (e.g., in the case of Israel, see Frank, 2012). This
method is intuitive after the tariff is agreed upon. It has some equitable properties
that are often violated by dividing institutions into two or more types due to
seniority or other criteria. Formula budgeting is easy, and has fewer conflicts,
especially after a formula has been agreed upon for some years. It seems fairer
and more straightforward (Brinkman, 1984; Fonte, 1987). However, drastic changes
in enrollment may cause problems, especially when a formula is enrollment based
(Thomas, 1999). The formula may change to include reservations of various
constituencies over time, till it becomes so complicated that its fairness is questioned
again. Formula budgeting is often used at the state level to allocate budgets to HEIs
(Barr & McClellan, 2018).
9.2.5 Responsibility-Centered Budgeting (RCB)
RCB means, in its purest form, that each academic unit (department, school,
college) is responsible for all the costs it produces and receives all the revenues it
generates; namely, academic departments and support units are the cost centers for
fiscal purposes and are expected to be self-supporting (Zierdt, 2009). Other terms
for RCB are value-centered management, cost-centered budgeting, profit-centered
budgeting, and revenue responsibility budgeting (Barr & McClellan, 2018). RCB
has also been widely used in business. Those who understand their operations best,
such as department and school heads, should be given greater budgetary authority
and responsibility (Dubeck, 1997). Deering (2015) examined the implications
of RCB for the internal structure and strategic management of North American
universities. Although it seems that since the 1980s, universities have gradually
moved from centralized incremental budgeting to decentralized RCB (Whalen,
1991), in practice, this is not so, Deering and Lang (2017) show that RCB is
never fully deployed. To resolve this anomaly, Myers (2019) built a moral-hazard
mathematical model with an asymmetric information situation, in which the top
administration does not have full information about the efforts and revenues of a
specific faculty member. He proved that only a pure RCB is optimal as it maximizes
the HEI’s revenue and solves the incentive problem of faculty efforts. However,
if the faculty is risk-averse, then the pure RCB is not optimal, and as the revenue
becomes uncertain, the RCB becomes less pure. In practice, in a large prestigious
research HEI with a large endowment fund, there is less uncertainty of the revenues;
thus, such universities use a purer RCB in comparison to small teaching colleges
(ibid.).
These five budgeting procedures are the main methods applied in HEIs. However,
there are many more budgeting methods and variations such as initiative-based bud-
geting and performance-based budgeting (Goldstein, 2012), activity-based budget-
ing (Kenno & Sainty, 2017), the executive budget model (completely centralized),
and performance budget (e.g., the number of students and number of credit hours;
Gibson, 2009). In practice, the budget may often be a combination of two or more
methods. For example, most budget items may be based on incremental budgeting,
whereas a research budget may be based on responsibility-centered budgeting.
Soares et al. (2016) pointed at the networked organization as an ideal structure
that links various departments and functions within an HEI through sets of expecta-
tions while providing benefits to all partners based on a unified information system
that allows various alternatives of costs and outcomes and their consequences
to be analyzed. The orchestrators of a network budgeting process could be the
institutional research office, which already performs in a similar manner. Such a
network is not necessarily hierarchical. The existing shared governance structure
could be the basis for the network as a participative form of management. The next
section provides a listing of expenses and income in HEIs, highlighting the special
structure of an HE budget.
9.3 Expenses and Income in HEI

Budgets—Estimation/Simulation
Budget allocation is a product of the planning process. We make a clear distinction

between resource requirements and resource allocation. Budget allocation is the
actual annual amount committed to each component of the organization to be
applied toward its needs, taking into consideration a given limited budget that
meets its requirements either fully or partially. In budgeting terminology, resource
requirements are usually equivalent to budget requests, whereas resource allocation
is equivalent to the approved budget. This section is based mostly on Sinuany-Stern

(1983b) and Sinuany-Stern (1984a).
HE is differing in relation to other public sector institutions, even in relation
to secondary education, in its income and expenses. In addition to the teaching
function, faculty members are expected to perform independent research. Further-
more, academic freedom is very specific to HE. Often faculty must apply to internal,
but mostly to external, sources to receive funds for their research, to support their
laboratories, equipment, materials, along with technical manpower and assistants.
Some grants are limited to certain areas of research, whereas others are not, but
require the submission of detailed proposals from researchers that are carefully
reviewed. Some grants come from public sources (such as the state) or private
organizations (such as independent companies). Sometimes, there is a contract
between the researcher and a company regarding patent rights, etc., and often the
HEI charges overhead (e.g., 15% of the grant), with support given by the HEI
to the research. There is great flexibility among various types of HEIs regarding
the extent of research versus teaching, covering the spectrum of zero research (in
most community colleges) to almost 100% research (in some research institutes
that have only research students). There are large differences regarding the average
grant amount per faculty member even within the same department, and among
departments. For example, a researcher in mathematics or literature usually does
not need heavy equipment and materials like a researcher in biology or chemistry.
The researcher is usually a full-time faculty member in a tenure track and is free to
apply or find sources for research. For full-time faculty, the portion of time to be
devoted to teaching in relation to research and administrative duties is vague, and
varies among HEIs, within institutions, and even within a specific department.
To prepare a budget that reflects the coming year’s plan of an HEI, there is a
need to know the current and past year’s budgets and how well they match the
current and past year’s plans (Barr & McClellan, 2018). Without such detailed
information and a good supporting educational cost accounting information system
at the desired level connecting between costs and activities (performance), be it
in a program, a department, or a school, sophisticated cost accounting such as ABC
(activity-based costs) is not viable in HE (Massy, 1996; Soares et al., 2016). Even so,
Soares et al. (2016) report that community colleges (CCs) do use ABC successfully.
Perhaps this success stems from the fact that CC faculty focus mainly on teaching,
while university faculty focus on submitting research proposals for funding, with
great variability among them. They also change over time as opportunities may
emerge unexpectedly. It is ideal to have the equivalent of MRP (material requirement
planning) for HE academic activities as is done in industrial engineering. However,
this is impossible in HE due to the fuzziness of inputs and outputs, and their costs.
Thus, it is hard to pinpoint the marginal costs of university activities as Massy (2017)
suggests is the ideal case.
9.3.1 Budget Expense Structure
As shown in Fig. 9.1, the student enrollment forecast is the basis of budget
generation in HE (Sinuany-Stern, 1983b, 1984a; Trusheim & Rylee, 2011). The
number of faculty needed is based on enrollment and the full-time/part-time mix
policy of the HEI, which determine the number of each type of faculty since the
salary of full-time tenure track faculty is much higher than part-time faculty. Many
assumptions in planning and aspirations are imbedded in predicting the various
expenditures in each HEI.
The main types of expenses which comprise the total expenditures, shown in Fig.
9.1, are:
1. Instructional expenditures: faculty compensation (full-time and part-time,
depends on the mix and variability of salaries by discipline—mainly in private
institutions), other instructional compensation, external vendors, and e-learning
expenses.
2. Academic support expenditures: other employees’ compensation needed to
support students and faculty, labs, equipment, computer hardware and software,
internet access and technical support for academic and administrative purposes,
marketing expenses to recruit students locally and internationally, library expen-
ditures, deans’ office expenditures, and departments’ office expenditures.
3. Institutional support expenditures: utilities, supplies, and materials, depreciation,
amortization, interest, and other operating costs.
4. Students’ services expenditures: registration, counseling, academic development,
health services, financial aid, support for needy students, job placement service
for graduates, and preparation for the job market.
5. Public services expenditures: museums, sport facilities, lifelong learning, shops,
and cultural activities.
6. Overhead: physical plant operations expenditures, which depend on gross square
meters by type of facility, computer services, utilities, property taxes, financial
services, human resources, central administration, materials purchasing and
management, fundraising, and marketing.
7. Research expenditures: long-run projects, grants, and contracts.
Another approach, presented by Goldstein (2012), is to display the expenses in a
matrix form showing the functional categories (e.g., instruction, research, public
services, etc.) by type of expenses (e.g., personnel compensation, services and
supplies, and utilities). In fact, the main expense in the budget is human resources,
especially full-time and tenure track. Toutkoushian and Danielson (2002) use the
following performance measure of academic faculty: 1. Inputs: student headcounts,
percentage from under-represented racial/ethnic groups, average SAT/ACT scores
of freshmen, high school GPA or class rank of freshmen, percentage of applicants
who are admitted, percentage of admitted students who enroll, and average faculty
salaries. 2. Production process: expenditures per student, student-to-faculty ratio,
credit hours per faculty member, percentage of courses/students taught by tenure-
Student faculty Full time/part

ratio time mix
Full time and part
time
Total Number of compensations
faculty
full time per FTE faculty
need
and
part time faculty
Total Student
Demand/forecast
Total faculty
compensation
Other instructional
compensation vendor
e-learning
Other instructional
expenses
Instructional
expenses
Academic support
expenses
Institutional support
Total Expenditures
expenses
Student services
expenses
Public services
Region population
expenses
External aid Financial aid

expenses
Gross square m. Physical plant

operation expenses
External research grants and contracts Research expenses
Fig. 9.1 Higher education institution’s total expenditures

track faculty, expenditures per student by major category, revenues per student by
major category, and level of deferred maintenance. 3. Outputs: number of faculty
publications and number of degrees awarded. 4. Outcomes: reputational rankings
(e.g., university ranking by USNWR), percentage of alumni who have donated to
the institution, retention rates (from freshmen to sophomore, etc.), graduation rates
(4, 5, and/or 6 years), the average time to degree, research grant dollars received,
and student satisfaction (from surveys). A more detailed breakdown is possible
depending on the needs of the HEI. Other examples for breakdowns for type of
expenses and their subcomponents are given by Hanover (2010).
Figure 9.1 gives a simple outline of an HEI’s expenditures. For more details
and an application of cost estimation of various specific types of expenses such
as the sum of a fixed cost and a variable cost multiplied by the amount of
different variables, see details in Sinuany-Stern (1984a). For example, the total
cost of academic support is a fixed cost, plus the variable cost per faculty for
academic support multiplied by the number of faculty. It is possible to identify
from past academic detailed expenditures those which are fixed regardless of faculty
number, and those that are dependent on the number of faculty to estimate the
fixed and variable costs (ibid) Fig. 9.1 shows the functional relations for the main
expenditures of an HEI. Another way is to use linear regression models to estimate
the intercept (fixed costs) from the data and the slope (variable cost per unit, e.g.,
academic department). For example, Weathersby (1967) used linear regression to
estimate fixed and variable costs per student, based on the actual past expenses of
departments.
9.3.2 HEI Budget Income
There are several types of main income in HEIs, most of which are specific to
them:
1. Tuition—differentiate between local students and international students, and by
degree.
2. Student fees such as room and board.
3. Financial aid (from external sources).
4. State and government support.
5. Donations and gifts from alumni, organizations, etc.
6. Investment income (e.g., from past endowment funds).
7. Grants from national and international sources.
8. Sponsored research from government and other sources.
9. Contracts with business and industry from patents, other royalties, auxiliary
enterprises, incubators, etc.
10. Educational activities and auxiliaries such as services of academic departments,
conferences, museums, sport facilities, lifelong learning, shops, and cultural
activities.
9.3.3 Determining Versus Predicting in Budget Planning
The HEI must decide how much control its administration has on the plan, and
how much uncertainty exists. For example, enrollment projection for a top-ranking
university with a very high enrollment of outstanding students allows that university
to determine the number of students according to its planned capacity, which
reduces risk. However, most universities do not have such full control of the number
of students. They can increase their control on enrollment by investing in marketing
(advertisements, etc.), while passive HEIs may use various methods to forecast
enrollment. Sinuany-Stern (forthcoming-a) presents many forecasting methods such
as the ratio method, time-series, regression, and data mining.
Linking enrollment predictions and tuition income models is common, as applied
by Trusheim and Rylee (2011) for the University of Delaware. The enrollment
matrix is constructed from past enrollment data by semester over several years
by type of entering students (freshman, transfer, readmitted, and continuing). The
forecasting model is based on a cohort survival method, with prediction based on
average historical percentages for each group, where the head of the university
can dictate a total target. The tuition model is integrated, where the enrollment
breakdown of the full-time/part-time ratio by the resident/nonresident ratio is used
to calculate the desired income from tuition based on full-time tuition versus part-
time tuition, and resident tuition versus the higher nonresident tuition. The desired
ratios are transmitted to the admissions office to consider in their admitting process.
This model is a good example of how the policy of HEIs can affect enrollment and
the income from tuition.
Regarding the forecast of the number of faculty, there is more control since the
tenure track faculty is determined by the administration via changing the student–
faculty ratio by adjusting class size in the worst case. The full-time/part-time faculty
ratio is another parameter the administration can use to control total expenditures.
For example, De La Torre et al. (2016) suggest a mixed-integer linear programming
(MILP) model for long-run planning of academic staff in a public university: its size
by department and by type of categories, and the cost of each type. The objective is
to minimize total academic staff costs, and different scenarios are tested. The various
costs and expenses and incomes can be predicted similarly. For example, Sect. 9.5
provides seven types of models for estimating the total expenses and total income of
an HEI over time (see details in Table 9.3) such as linear growth (a constant amount)
and exponential growth (a constant percentage).
According to state-level budgeting, often the number of students is controlled by
the state, and the graduation rate is also a measure used to determine a state’s allo-
cation to an HEI. Often the cost per student varies significantly among disciplines,
as in the Israeli case [to this date by the Council for Higher Education (CHE; Frank,
2012)], although the allocation for research is based on the achievements of each
university by examining two criteria: publications by their impact factor and the
number of competitive grants secured by the faculty. Thus, only seven universities
are entitled to full research allocations, while over 50 state HEIs are not (ibid.).
Other allocations for enhancing specific minority groups and special activities are
also allocated, e.g., annual allocations by the Israel Science Foundation (ISF), which
awards grants to individual researchers based on their proposals, which are reviewed
by international reviewers. Another example is a program for improving the quality
of teaching, which I championed as the chair of the Israeli Quality Assurance
Committee (when I was a member of CHE during 2012–2017). Consequently, HEIs
in Israel are allocating some funding to enhance teaching quality based on several
criteria. This is a case of spending a small amount of money to achieve significant
improvement in teacher and student satisfaction.
On the HEI level, to calculate the cost per student by academic department, it is
necessary to consider the Induced Course Load Matrix (ICLM), which provides the
number of contact hours each department provides to each of the other departments
(Sinuany-Stern et al., 1994, p. 553). Using this approach to calculate the average
cost per student by the department at a specific university, the decision was made
to close one department. However, using cost per student as a main criterion can
be misleading, as is shown in this case. When using data envelopment analysis
(DEA) in which several academic monetary and non-monetary criteria are used
(e.g., number of publications and grant money received by faculty), the department
was found to be efficient and, indeed, eventually that department was reopened
(ibid.). The cost per student was used to measure departmental efficiency in
relation to DEA, which considers multiple inputs and multiple outputs of academic
departments such as research outputs (ibid. and Johnes et al., 2017). As shown by
Sinuany-Stern (forthcoming-b), DEA is one of the OR methods often used in HE,
mainly for measuring efficiency, benchmark analysis, and pointing at improvements
needed (see Sinuany-Stern & Hirsh, forthcoming).
In some cases, the importance of marginal costs can be crucial. For example,
while provost of Ariel University, I was aware that the Physical Therapy (PT)
department had a very high demand. However, the department head wanted to
limit student numbers since there was a shortage of professors with PhDs in the
field. When the issue was brought to me, I found that the largest possible class in
the department was 50 students. Moreover, the marginal cost was extremely high
because of the laboratory equipment and field training requirements—even with
state support, this cost was higher than the marginal income per student. Thus,
we were losing money on each student. As a result, I decided not to increase
freshmen enrollment beyond 50 in that department. I convinced the Board that such
restrictions would cause high demand which will cause some spillover of good
students to similar departments (such as health management) with low marginal
cost, and create prestige for the new department and for the new university. In the
end, we were able to meet the planned enrollment by accepting more students in
departments with lower marginal costs such as Health Management. In this case,
the difference between the marginal costs was so obvious, that there was no need to
perform exact calculations since there are no expensive training requirements in the
related departments.
Regarding the investment in a large research facility, Philbin and Mallo (2016)
adapted the known Management Successful Programs (MSPs® ) to support the
strategic decision of investing in a research facility that can serve various additional
departments and external purposes that may put the organization ahead in a specific
scientific area. This analysis is done before such items enter the budgeting process.
Special projects that span several years often are not included in the operational
budget, such as facilities building, as they are part of the development (capital)
budget. In this case, due to their risks and other expected expenses and income
over time, the algorithms for generating the discrete efficient frontier to a capital
budgeting problem can be used (Rosenblatt & Sinuany-Stern, 1989), where the
projects are ordered according to their ratio of risk and expected rate of return, with
the ranges of the risk-averse weight (the weight ranging from 0 to 1).
9.3.4 Incentives for Maximal Performance in HE Via

Budgeting
Budgeting formulas may include incentives for encouraging specific performance

in HE. For example, the state may wish for public HEIs to increase the number of
students and the graduation rate. Moreover, for a comprehensive public university,
the state expects HEIs to increase their research output in number and quality. Thus,
the budgeting formula would be designed to foster these objectives by accounting
for an increase in enrollment and students’ success rate of graduation, and for more
publications in high impact factor journals (as is the case of the Israeli formula to
this day, as shown by Frank, 2012). Another example at the state level is given
by Toutkoushian and Shafiq (2010), showing a model that proves that need-based
financial aid to students is more beneficial than giving state appropriation to the
college. However, solving one problem sometimes causes another, in this case, the
brain drain problem, where students who can get state aid may go to study in another
state. Thus, they suggest conditioning financial aid to students who attend an in-
state HEI (ibid.). Another example is the US federal government that gives loans
and direct grants to needy students.
Often a survey is needed to learn about some practices in HEIs for planning
purposes, for example, Henry-Moss et al. (2019) explored the planning of space for
lactation in community colleges in the USA and their funding. They used a survey,
whose findings were used to plan better facilities to accommodate nursing mothers
on campus. Such facilities are incentives for such women to return to school.
Among the incentives for research and teaching quantity and quality that
I initiated as the Provost of Ariel University (2000–2008) was a program for
encouraging faculty to excel in teaching and research. As reported by Davidovitch
et al. (2016) and Davidovitch and Sinuany-Stern (2014), 60% of the faculty at the
university were awarded an increase of up to 20% of their salary based on their
teaching and research performance. There was a list of criteria used and points given
per criterion (e.g., number of publications, grants, the average number of students in
courses, and average student evaluation scores). The top 20% of the faculty scoring
the most total points received a 20% pay increase, the next 20% of faculty achievers
received a 15% increase, and the next 20%, a10% increase. Achievements were
measured for the past year, and the salary increase was given for 1 year. Publications
in journals counted for three years to emphasize their importance, and to account for
fluctuations in the number of publications from 1 year to the next; the same was done
for complete books of faculty members published by a recognized publisher.
Priest et al. (2002) covered the issue of incentive-based budgeting systems in
public universities, exploring diverse areas of HE: incentives to faculty, the effect of
incentives on teaching, incentives to increase revenue, and efficiency implications at
the state level, providing examples from universities in North America. Elson (2017)
highlights the issue of financing for gender equality via the budget, in compliance
with human rights standards (see also, Steinþórsdóttir et al., 2016).
On the international level, OECD collects data on HEIs from its national
members and other countries regarding financial and other performance measures,
which indirectly give various countries incentives to improve their performance
accordingly. As they claim:
Governments are increasingly looking to international comparisons of education opportu-
nities and outcomes as they develop policies to enhance individuals’ social and economic
prospects, provide incentives for greater efficiency in schooling, and help to mobilise
resources to meet rising demands. (OECD, 2019)
Examples of such indicators in HE are attainment of HE of 25–64-year-olds by

gender, by degree, by field of study; labor market outcomes: employment %
(unemployment) by level of education, relative earnings, private and public cost
benefits, and return on investment in HE by gender. Although no budgeting is
directly involved in the OECD report (2019), there is a hidden incentive for countries
to allocate more budget or more attention to improve certain performance measures.
There are many possibilities to encourage improvement in HEIs by using various
formulas or budgeting rules to foster the desired objective in some budget items.
However, these types of incentives should apply for long periods to gain the full
cooperation of the targeted population since it takes time, sometimes even years, to
implement a new procedure and to learn its implications (ibid.).
9.3.5 Planning in HE Versus Budgeting
Indeed, the budget is a presentation of an HEI plan in monetary terms. However,

planning is a much wider term. Planning can be done at any level of the HE
system, with or without connection to the budget, financially linked or not. In this
subsection, examples of planning that are not tied directly to the budget process are
presented.
Planning in subsystems: The above procedures seem to fit the total university, but
its principles can also fit subsystems such as library budget allocation—a subsystem
that has received great attention in the literature (see examples by Adekanmbi &
Boadi, 2008; Kao et al., 2003; Walter, 2018). Another example is planning the
future load of information systems for educational purposes, for which Gorbunov
et al. (2017) used a simulation model, and Sinuany-Stern and Yelin (1993) used a
regression analysis to forecast computer hardware resources (CPUs, memory, etc.)
to avoid bottlenecks at a large university (see also Banerjee & Igbaria, 1993). In
addition, Hamid et al. (2018) analyzed space capacity by type (classrooms, labs,
offices, etc.) using government standards at a public university in Malaysia by a
capacity index.
Sometimes planning in HE does not have monetary implications. For example,
Magnanti and Natarajan (2018) used discrete optimization to allocate students
to multidisciplinary projects. Timetabling for courses or exams are well-known
planning models in which space or manpower such as faculty and courses or exams
are placed in time to achieve a specific objective under some constraints. Oude
Vrielink et al. (2019) performed a systematic review on optimization models and
practices of timetabling in HEIs, showing that, since 1990, the number of articles
published on timetabling in HE has grown exponentially.
Garcia (2019) used a multi-objective max-flow model for faculty planning
to assist future hiring decisions. Aviso et al. (2019) suggested an optimization
model with a P-graph approach for the case of a teaching-oriented HEI that was
planning a transition to include research activities. The framework is input/output,
reflecting interdependencies among various categories applied to a university in the
Philippines (ibid.).
9.3.6 Simulation/Optimization Scheme of Budget Planning

in HEI
In HEIs, the most common budget planning is performed mainly at the university
level (as reflected in Appendix 1). Budget planning often involves a combination
of several operations research (OR) methodologies such as forecasting simulation
and resource allocation. For example, student enrollment may be based on the
past year’s enrollment and the population born 18 years before multiplied by the
participation ratio of this cohort in the past. Instructional costs may be based
on the predicted enrollment and the student–faculty ratio, and the full-time/part-
time mix of faculty for calculating faculty compensation, as shown in Fig. 9.1.
Some institutions predict only the incremental changes of some categories such as
institutional support, whereas others may use ZBB or PPBS for evaluating capital
budget projects, and still others may use responsibility-centered budgeting for the
business school.
An overall picture of the budget planning system, as developed in this chapter, is
given in Fig. 9.2. The enrollment forecasts influence, future income, and resource
requirements. A cost simulation model is utilized for estimating the resource
requirements to the desired level of detail (as in Figs. 9.1 or 9.2) by using
Demographic Economic
Marketing factors factors
policy
Cost parameters
Tuition Enrollment Resource

projection/ requirements Operation policy
policy CSM upper
Financial policy bounds
Income projection:
Income Y
Tuition & fees Final
sufficient? budget
Investment
Industry/Business contracts N
Other services
Deficit? N
N
Priority Policy
optimization? change
Y
Decision maker's Optimal resource
Bounds
preferences allocation
Y N Final
Risk
analysis? budget
Fig. 9.2 Simulation optimization budgeting planning scheme in a university
cost parameters, and environmental and policy factors. If the expected income is
sufficient to cover the resource requirements, then the budget can be approved.
Otherwise, the decision-makers can allocate the constrained funds by revising their
policy and enforcing their preferences, or by securing additional funds, approving a
deficit budget, or using an optimization model. The optimization budget allocation
models outlined in Sect. 9.4 maximize the overall objective of the institution based
on the decision-makers’ preferences (for example, the number of served students),
where the sub-expenses from the simulation stage are the upper or lower bounds
of each type of expense. The other bound can be the past expenses (incremental
growth is possible). After the optimal solution is reached, the final budget can be set;
otherwise, risk analyses are performed to test the sensitivity of the optimal solution
for various changes in various assumptions and forecasts answering various “what
if” scenario changes. The next section provides a variety of optimization budget
allocation models, linear and non-linear, for short term and long term planning, and
for various levels of HEIs.
9.4 Optimization Models for Budget Allocation
In this section, several types of main optimization budgeting models are presented:
linear, quadratic, goal programming, chance constraint, multi-period, and multilevel
hierarchical network models. This section is based on Sinuany-Stern (1984b, 1993,
2014).
The basic budget allocation problem is stated as follows: We must allocate a
given annual budget to organizational units (e.g., expense items from Fig. 9.1, and/or
departments from Fig. 9.2). The allocations of the units may have lower and upper
bounds. The upper bounds can be the expense items presented in Fig. 9.1, and the
lower bounds can be a certain percentage below each upper bound or previous year
allocation (Sinuany-Stern, 2014). If different groups and managers have different
estimates for the various expenses, then the lowest estimate of each expense item
will be its lower bound, and the highest estimate will be its upper bound.
Here are some definitions:
Xj —the allocation for unit, j = 1, . . . ,n (the decision variables).
n—the number of organizational units.
B—the given total budget B > 0.
Type of constraints:
Budget constraint: nj=1 Xj ≤ B.
Bound constraints (upper and lower bounds):
Lj ≤ Xj ≤ Uj , ∀ j = 1, . . . , n.
Lj is the lower bound of allocation j, Lj >0

Uj is the upper bound
n of allocation j, Uj >0
n
j =1 Lj ≤ B ≤ j =1 Uj , so that the problem is feasible and nontrivial.
Weights of the objective function:
Wj is the weight (e.g., reward or level of output such as the number of students)
associated with one unit of Xj . For simplicity, let us assume that the organizational
units are ordered decreasingly, so that Wj > Wj + 1 ,
∀j = 1, . . . , n − 1 . . . . For example, if the units are academic departments, then
the weights can be the percentage of students in each department. Such weights are
used in all the linear models throughout this section, at all hierarchical levels. The
weights may represent preferences of decision-makers such as AHP (Sinuany-Stern,
1984b).
The bounds represent two extreme scenarios for the expected allocations of each
unit: the minimal allocation and the maximal. In practice, often the lower bounds are
derived from historical allocations of the budget, while the upper bounds are derived
from budget requests of the units. The bounds may reflect forecasts calculated
by various parties in the organization. The minimal scenario of these forecasts
may serve as the lower bound, and the maximal as the upper bound (e.g., student
forecasts). In Sect. 9.3, a number of methods are shown from which the bounds
can be generated, including forecasting methods, planning models, and simulation
approaches to determine the needs of various units in a university that help to set the
bounds on the allocations (see also, Sinuany-Stern, 1983b, 1984a).
9.4.1 Linear Model with One Period and One Hierarchical

Level
The objective function is linear, maximizing the overall reward from the sum of
allocations:

Max nj=1 Wj Xj , where Wj is a positive weight associated with one unit of Xj .
In this case, there are four sub-models:
1a. Linear model with no constraints: The solution is to allocate infinity to each unit.
1b. Linear model with only budget constraints: The solution is to allocate all the
budget B to the first unit (since it has the highest weight).
1c. Linear model with budget constraints and bounds on the allocations: The
solution procedure is: 1. Allocate the lower bounds to all units as the initial
allocation. 2. Add allocations up to the upper bound starting with unit 1. Check
if there are leftover funds of the total budget, and if so, continue to the next unit
in order till reaching unit j*, where we exceed the budget. Then this unit will be
allocated only to the remaining budget, and all units thereafter will receive no
additional allocation beyond the lower bound.
1d. Linear model with only bound constraints (no budget constraints): The solution
is to allocate the upper bound for each unit.
The detailed formulations of each model are given in Table 9.1. These formula-
tions are not necessary for non-analytical decision-makers. The solutions are simple
and intuitive (as listed above), and do not require mathematical calculations. The
optimal solutions of linear objectives with linear constraints are at extreme points of
the constraints, mostly either at the bounds or infinity. See Example in Appendix 2.
The last column of Table 9.1 provides the change in the allocations when a budget
cut occurs. Budget cuts affect only one unit, for a linear objective function. Again,
the solutions are intuitive and require simple arithmetic calculations.
In summary, the solutions of linear objective functions are extreme and seem
unfair (see Sinuany-Stern, 2014 on fairness). Thus, the following subsection
presents a nonlinear objective function—the quadratic objective function.
Table 9.1 Formulations and solutions of linear models with one period and one hierarchical level

The objective is to find Xj to maximize : nj=1 Wj Xj
Sub-model Constraints (const.) Solution Solution for budget cut of ε
1a None Xj = ∞, No change in the solution
∀j = 1, . . . , n
1b Budget
n const. X1 = B X1 =
j =1 Xj ≤ B Xj = 0, ∀ j = 2, . . . , n B−ε
Xj = 0, ∀ j = 2, . . . , n
1c Budget
n const. X j = U j , ∀ j = 1, . . . , Only Xj∗
j =1 Xj ≤ B j∗ −1 reduces
Bounds const. Xj = Lj , ∀ j = j∗ + 1, by: ε
Lj ≤ Xj ≤ Uj , ..., n
∀j = 1, . . . , n Xj ∗ =
B − nj=1,j =j ∗ Xj
1d Bounds const. Xj = Uj No change in the solution
Lj ≤ Xj ≤ Uj , ∀j = 1, . . . , n
∀j = 1, . . . , n
9.4.2 Quadratic Model with One Period and One Hierarchical

Level
The quadratic model attempts to minimize the distance of each allocation from its
bounds in order to avoid extreme solutions. This subsection is based on Sinuany-
Stern (2014). Table 9.2 presents several types of quadratic models within our general
framework as described above, considering a budget constraint and lower and upper
bounds on the allocations of the various units. The objective function is common to
all the quadratic models presented in this Sect. 9.4.2, as follows:
n
2 2
Min qj Xj − Uj + Xj − Lj
j =1
As a minimization objective function, the weights, qj , reflect a penalty on

deviation from the bounds. For example, the reciprocal value of the weight can
be used, qj = 1/Wj , giving higher preference to units with lower qj , which are in
fact units with higher Wj . For example, if Wj represents the number of students,
then units with more students (greater Wj ) will have preference in the allocation,
and also in the quadratic model since their qj will be lower. For example, if
there are two units in a college where unit 1 has 200 students and unit 2 has
100, thenW1 = 200 300 and W2 = 100/300; consequently, q1 = W1 = 3/2 and
1
q2 = W12 = 3/1.
Following are three sub-models derived here for the quadratic objective model
with the constraints used for the linear model.
2a. Quadratic objective function with no budget constraint
Model 2a presents a special type of goal programming approach, where there
is no budget constraint and the upper and lower bounds are soft constraints with a
quadratic penalty, rather than linear with the deviation from the bounds. The total
penalty is the sum of the weighted deviations to be minimized—the weights are
qj , ∀ j = 1, . . . , n. The optimal solution in this case is the midpoint between the
bounds. Thesum ofthe optimal allocation can be used as a suggested total budget:
L +Uj
B = nj=1 j 2 .
2b. Quadratic model with only budget constraint
Model 2b adds the budget constraint to Model 2a. As shown in Table 9.2, the
solution is quite intuitive: Start with the initial assignment of the midpoint to each
unit j. If the sum of the midpoints is less than the budget B, then the sum left
over from the initial assignment of the budget, B, is reallocated to each unit j
proportionally to 1/qj and added to the original midpoint assignment.
If there is a shortage after the initial assignments; namely, B <0, then the shortage
in the budget is distracted from the original midpoint assignment.
2c. Quadratic model with budget and bounds constraints
Model 2c is similar to model 2b, but in addition, there is a need to verify that the
bounds are met: If the upper bound of unit j is violated, then set Xj = Uj , and if the
lower bound of unit j is violated, then set Xj = Lj . The overall leftover sum/deficit
after accounting for the bounds will be divided among the units that did not reach
any of their bounds.
Table 9.2 Solution of linear model with one period and one hierarchical level

2 2
The objective is : find Xj to minimize nj=1 qj Xj − Uj + Xj − Lj
Sub-model Constraints (const.) Solutiona Solution for budget cut of
2a None Midpoint allocation: No change
L +U
Xj = j 2 j , ∀j =
1, . . . , n
2b Budget const.:
Midpoint allocation + its Cut across the board:
n
j =1 Xj = B proportional share from Xj =Xj −
leftover suma : 1/q
n j
Xj = j =1 1/qj
Lj +Uj 1/q , ∀j = 1, . . . , n
2 + n j B
j =1 1/qj
2c Budget
n const.: As in 2b, but if the bound As above, cut across
j =1 Xj ≤ B is violated, then force the the board, but verify
Bound const.: bound, and the leftover the bounds as in the
Lj ≤ Xj ≤ Uj , sum/deficit will be previous column
∀j = 1, . . . , n divided proportionally to
1/qj for other units j that
are within the bounds
a Left over from midpoint allocation:B = B −
n Lj +Uj
j =1 2
The detailed formulations of each model are given in Table 9.2. These formula-
tions are not necessary for non-analytical decision-makers. The solutions are simple
and intuitive and those of quadratic objectives with linear constraints are mostly at
the midpoints of the constraints. See example in Appendix 2.
A quadratic objective function was also used by Bogomolova et al. (2018)
to find the optimal annual growth rate of a budget allocation for research in a
university, which minimizes the sum of the square deviation of the allocation from
the prediction over time.
9.4.2.1 Fairness of the Quadratic Versus the Linear Models for Budget
Allocation
As shown in Table 9.1, the linear budgeting models provide extreme budget allo-
cations; almost all units will have allocation on one of the bounds (lower or upper
bound). However, as shown in Table 9.2, the quadratic budgeting models provide
more fair solutions, mostly between the bounds. Sinuany-Stern (2014) defined a
fairness index for budget allocation with bounds and proved mathematically that the
quadratic model is fairer than the linear model. This gives the various participants
in the budgeting process some security that they have a fair chance in the budget
allocation, and more so when the lower bound is based on the needs of the units and
is tied with the performance level, be it enrollment or research outputs. Moreover,
budget cuts are distributed more evenly among the various units in the quadratic
model, while the budget cut in the linear model tends to cut the allocation of only
one unit.
9.4.2.2 Dynamic Quadratic Budget Allocation with Only Bounds

Constraints
In cases when, historically, a unit was overbudgeted and the lower bound represents
the real needs of the unit, the quadratic allocation is at the midpoint between the
lower and upper bounds. However, if there is no budget constraint (the budget is the
sum of the midpoint solutions), then to reduce the specific units’ allocations without
a drastic cut, we apply the quadratic model over time, so we can speed the approach
of the allocation to the lower bound by setting the allocation of year t as the upper
bound of the consecutive year dynamically from year to year. Meanwhile, the lower
bound will remain unchanged. After one year, 50% of the gap between the bounds
will be achieved (the midpoint), after 2 years 75% of the gap between the original
bound will be achieved (the midpoint of the second year’s bounds), and after 3 years,
87.5% of the original gap will be corrected (see details by Sinuany-Stern, 2014). The
2t −1)Lj +Uj
general formula for the allocation of unit j over time t is: Xj t = ( 2t (e.g.,
for t = 1 the allocation is at the midpoint between the original bounds). A similar
method can be used for a unit that is underbudgeted. This dynamic approach of
correcting the allocations over time may motivate overbudgeted units to improve
their performance to justify a higher allocation (if at least the lower bound is
positively related to the performance level). Any decision-maker will be satisfied
to achieve 75% of a budget correction within 2 years.
9.4.2.3 Goal Programming and Absolute Value of the Objective Function
Case 2a is like a goal programming model with no hard constraints (no budget
constraint), where the goals are the bounds, and the deviation from the goal is
quadratic.
The absolute value model, where the objective function is the absolute value of
the deviations from the bounds rather than a quadratic function without constraints
provides a similar solution to the quadratic model, namely, allocation to the
midpoint of each unit.
9.4.3 Linear Two-Level Hierarchical Budgeting Model Over

Time
This model deals with budgeting over T years in a hierarchical system with
two levels (e.g., college level and department level). This subsection is based on
Sinuany-Stern (1984b). The objective function is linear with the allocations, X, and
the weights, W, representing the preference (importance) of the relevant unit (can
be proportional to the number of students in each unit at each level). The allocation
of year t is Xt .., the allocation of college k in year t is Xtk , and the allocation to
department j at campus k is Xtkj . All allocations X . . . have lower and upper bounds,
Lt ,.., Ut .., where the anticipated income of year t is Bt ; ai is a weight given to the
ith level of the institutional hierarchy, i = 1, 2, 3; and Dt is the surplus carried from
year t to year t + 1. Where t = 1, . . . ,T; k = 1, . . . ,K; j = 1, . . . n. T is the planning
horizon (in years), K is the number of campuses, and n is the number of units at
each campus.
The objective function is to find Xtkj , Xtk , and Xt such that:
T K n
Max a1 Tt=1 Wt ..Xt .. + a2 Tt=1 K
k=1 Wt k .Xt k . + a3 t =1 k=1 j =1 Wt kj Xt kj
Subject to:
Balance constraints in the two organizational levels:
Xt .. = K k=1 Xt k , t = 1, . . . ,T

Xt k . = nj=1 Xt kj , t = 1, . . . ,T; k = 1, . . . ,K
Budget constraint:
Xt .. + St − S
t −1 = Bt
t t
m=1 X m ≤ m=1 Bm , t = 1, . . . ,T
Lower and upper bounds on allocations:
Lt . . ≤ Xt . . ≤ Ut . . , t = 1, . . . ,T
Ltk . ≤ Xtk . ≤ Utk . , t = 1, . . . ,T; k = 1, . . . ,K
L ≤ Xtkj ≤ Utkj , t = 1, . . . ,T; k = 1, . . . K; j = 1, . . . ,n
tkjT K J
t =1 Wt .. = 1 , k=1 Wt k . = 1 , j =1 Wt kj = 1
The various weights, W, are normalized so that only ai reflects the weights
given to each level. This model can be solved as a linear programming model of
a capacitated network model. The two-level hierarchical linear network budgeting
model over time presented here is sometimes too cumbersome mainly because
the bounds setting over each period for two hierarchy levels (usually a 5-year
plan) is taken. For example, if at a small comprehensive university, we have: five
colleges, five academic departments in each college, and five functional areas in
each college, then for a 5-year plan, we need to estimate at least 500 bounds!
(5x(5 + 5)x5x2 = 500) see general formula in Sinuany-Stern (1993, p. 301).
Therefore, in the long run, a very aggregated model is more practical (as shown
in Sect. 9.5). Moreover, in practice usually, budget allocation is done for one year
ahead, while long-run planning is usually done for the aggregated level to learn the
long-run implications of current allocations over time, not for actual allocations for
the long run by units and subunits.
9.4.4 Linear Multilevel Hierarchical Linear Network

Budgeting Model
This section deals with a single year’s budget allocation in a multilevel hierarchical
system with N levels, where the objective function is linear, with upper and lower
bound constraints on each allocation, and a constrained budget B to be allocated.
Due to the hierarchical structure, naturally, there are also balanced constraints,
where the sum of the allocations in one level is the allocation of the upper level
(based on Sinuany-Stern, 1993) as shown in Fig. 9.3.
Let us assume that we have N hierarchical levels (e.g., three levels: the university
level, the college level, and the department level). For each level I, there are Ji
units under it, the weight given to level i is ai , where i = 1 . . . N; Xj1 ,...,ji are
the allocation to unit ji in level i, and Wj1 ,...,ji is the unit weight of the allocation
to unit j at level i—the weight can be proportional to the number of students. The
Level 1
1 j J1
Level 2 1 J2
j
Level i 1 j J1
Level N
1 j JN
Fig. 9.3 Multi-hierarchical structure
objective function is linear:

N J1 Ji
max ai ,..., Wj1 ,...,ji Xj1 ,...,ji
i=1 j1 =0 ji =0
Subject to:
Budget constraints:
J1
Xj1 ≤ B
j1 =0
Balance constraints at all organizational levels from level i + 1 to level i:

Ji+1
Xj1 ,...,ji = ji+1 =0 Xj1 ,...,ji+1 ; for all i = 1, . . . N and j = 1 . . . Ji
Upper and lower bound constraints on each allocation:
Lj1 ,...,ji ≤ Xj1 ,...,ji ≤ Uj1 ,...,ji ; for all i = 1, . . . N and j = 1 . . . Ji
This type of network multi-hierarchy model fits a centralized budget. Linear
programming simplex can solve it or a more efficient cost-capacitated i network
algorithm. In general, the number of balance equations is T k=1 kJ . The number
of variables is the same, as is the number of lower and upper bounds.
Sinuany-Stern (1993) showed several developments and generalizations of this

model: a. If the objective function is nonlinear, in some cases, it can be linearized by
piece-wise linearization (ibid., pp. 302–304); b. multi-criteria objective considering
can be used via AHP (analytical hierarchy process; Saaty & Vergas, 2012) for
evaluating the preferences, Wj, as suggested by Sinuany-Stern (1984b, pp. 301–
302); and c. including zero, one variable is possible when projects are to be
considered in the allocation process (ibid., p. 305).
9.4.5 Chance Constraint
In each of the above models, a chance constraint can be introduced to account for
randomness. For example, if the budget level is distributed normally with expected
value B and standard deviation σ , then thefollowing budget constraint reflecting
n
j =1 Xj ≤ B + σ • z(0.95) = B
a 95% one-sided confidence interval is: "
where z(0.95) =1.645 is the value of the standard normal distribution where
Prob.(Z < z(0.95)) = 0.95. The original budget constraint will simply change to:
n
Xj ≤ B " .
j =1
Namely, in each of the above models and sub-models, the change will be in the
budget constraint by replacing B by B , and the change will be in the solutions.
Wherever B appears, it will be replaced by B .
9.5 Aggregated University Long-Run Budget Planning

Model
Long-run financial planning is usually based on aggregated incomes and expenses

(Sinuany-Stern, 1983a). In the extreme aggregation: It is the total income of an
HEI at period t, and Et is the total expenditure of an HEI at period t. Usually, the
period used in budget planning is a “year” although any time unit can be used:
semester, quarter, etc. In long-run budget planning, an HEI may carry funds or a
deficit from year t-1 to year t. In this case, we consider, r, the rate of return on
the university’s excess (or deficit) funds (the rate of interest). For simplicity, we can
assume that r is the same for investment and for deficit, since often in HEIs there are
endowment funds whose annual returns are part of the HEI income, and a deficit in
the budget can be taken from those funds, while excess budget is invested back into
the endowment funds. From one year to another, the following difference equation
describes the dynamics of the budget progression and its deficit/over Dt , where
Dt = It − Et + (1 + r) Dt −1
There are many types of mathematical models for describing the progression
of income and expenses of budgets over time, as shown in Table 9.3, where
seven models are described. These mathematical models are simple enough for
administration in HEIs, and as presented in the table, the differences between them
is very clear:
Model 1 presents an incremental increase in the income and expenses of an HEI;
namely, linear constant growth in income and expenses from one year to the next.
Model 2 presents a constant percent of growth in income and expenses, namely,
an exponential growth of income and expenditures over time. In both models, the
constant between two consecutive years does not change (a, b), while in
Model 3, the change varies over time (at , bt ).
Model 4 has another general formula for the change of income and expenses over
time [I(t), E(t)], and
Model 5 allows for another growth parameter such as student growth or decline.
While
Models 1–5 are highly aggregated,
Model 6 allows a breakdown of the income (e.g., tuition, state, endowment,
grants) and expenses (e.g., instructional costs, student services, financial aid,
physical plant operation costs).
Model 7 considers a continuous rate of change over time; thus, it is in an integral
form, whereas the other six models are summative (see Sinuany-Stern, 1983 for
more details about additional models).
The above difference equation expressing for Dt , the deficit/surplus is applicable
to each of the above seven models.
The connection between the traditional budgeting procedures and the models in
Table 9.3 are given in the first column: Models 1–3 and 5 are types of incremental
budgets, and Model 4 is a formula type budgeting. All seven models in the table
are applicable to all organizational levels (department, school, college, university,
and state) of an HEI. Incremental budgets are often used, while formula budgets are
more applicable at the state level.
Taking a closer look at Model 2, for example, we can analyze the progression
of the deficit/surplus by applying the geometric growth of income and expense
equations to the above definition of Dt . It becomes:
(1 + a)t +1 − (1 + r)t +1 (1 + b)t +1 − (1 + r)t +1

Dt = I0 − E0 .
a−r b−r
Thus, a break-even analysis can be done for Dt = 0, implying conditions for

convergence on the relations among the various parameters of the model; see
Sinuany-Stern (1983b) for full derivations. Models for the case of uncertainty
represented by probability density functions are also given there (ibid.).
In the long-run planning range, which can be for any number of years, it is
usually by breakdown of income and expenses, and often for 5 years. For example,
Brandeau et al. (1987) used a set of equations of various types of incomes and
expenses by department within a medical school as RCB within a private university
(Stanford) for a 5-year plan. Many of the equations predicted the annual increments
Table 9.3 Examples of analytic models for annual progression of income and expenditures
Nature of I, E
Model no. progression over time Income – It Expenditure – Et
1 Incremental linear It = It − 1 + a = I0 + ta Et = Et − 1 + b =E0 + tb
growth
2 Incremental with I = (1 + a)It − 1 = Et = (1 + b)Et − 1 =
constant rate and (1 + a)t I0 (1 + b)t E0
geometric/ exponential:
a, b
3 Incremental with variable = (1 + ai ) It−1
It = Et =
(1 + bi ) Et−1 =
rates over time, at , bt I0 ti=1 (1 + ai ) E0 ti=1 (1 + bi )
4 Formula budget as a It = I(t) Et = E(t)
function of t
5 Incremental as model 2 It = (1 + g)(1 + a)It − 1 Et = (1 + g)(1 + b)E
with additional constant = [(1 + g)(1 + a)]t I0 t − 1 = [(1 + g)(1 + b)] E0
t
growth -g (e.g., student

enrollment)
KA KB
6 Breakdown of I and E are It = k=1 Ikt Et = k=1 Ekt
given over time (KA and
KB categories)
t+1 t+1
7 Continuous rate is used It = t I (x)dx Et = t E(x)dx
over the year
a In Models 1–5, the expenses and income are discrete, i.e., they occur at one point during the year
of specific types of income/expenses with varying enrollment rate of change, etc.

combining Models 4, 5, and 6 (Table 9.3). This was basically a simulation model
testing the effect of various assumptions (not optimization) on the 5-year budget
plan (ibid.).
As Zierdt (2009) declared: “The literature suggests that institutions need to choose a
budgeting tool that is most reflective of their needs and strategic priorities. It seems
likely, therefore, that a hybrid approach to budgeting will become the norm within
institutions of higher education, especially with the increased demand for reform of
how scarce resources are allocated.”
This chapter proves that the above statement is unavoidable. We started by
outlining the five main traditional (non-mathematical) approaches used in HE bud-
geting: Incremental, ZBB, PPBS, Formula, and RCB. The second stage, estimating
expenses and incomes of HEI via several methods and formulas (ratio, regression,
increase by percentage, etc.) and a simulation/optimization budgeting scheme was
presented, where if the income is insufficient, a scenario risk analysis can be done
by analyzing various “what if” questions. If still the budget is not sufficient, an
optimization model can be used, with lower and upper bounds on the allocation
taken from the simulation model. Several linear models were analyzed, also in
hierarchical network structure (10 variations), and a quadratic model with three
variations was presented. We showed that the solutions of the optimization models
are simple and intuitive, and do not require mathematical skills. The bounds on
the allocations can be a combination of the simulation approach and some of the
above traditional budgeting procedures. The optimal solution of the linear model is
to allocate to all the units on the upper or the lower bound except for one marginal
unit, while the solution of the quadratic model is to allocate to the midpoint between
the bounds. Consequently, the allocation of the quadratic model is more intuitive
and fairer than the allocations of the linear model. For the long run, an aggregated
model was presented with many functional forms (7 variations). These models are
also connected to the above traditional budgeting procedures.
Massy’s (2017) book on reengineering the university: How to be Mission
Centered, Market Smart, and Margin Conscious is applicable today, especially
with the COVID-19 pandemic, which provides many opportunities for using short-
run and long-run planning, mainly of online and web-based tools (e.g., Zoom).
Moreover, new forms of virtual classrooms and virtual universities are expected
to evolve (see Almog & Almog, 2020).
Appendix 1: Summary of the Literature on Planning

and Budgeting in HE
Type of plan- Levels HE

Author (year) Method used ning/budgeting HE area hierarchy Country
Balderston PPBS Program planning & budgeting University USA
and
Weathersby
(1972)
Weathersby
and
Balderston
(1972)
Sinuany- Break-even Increment Long-run University USA
Stern (1983a) aggregate, budgeting planning
income vs.
expense
equations
Sinuany- Simulation, Budgeting Costing University USA
Stern (1983b) system
analysis
approach
Sinuany- Budget Budgeting & Financial Multi- USA
Stern (1984a) planning planning planning campus
procedure; model university
Projections &
Scenario
analysis
(continued)

Sinuany-Stern Network Budgeting Budget Multi- USA
(1984b) optimization allocation campus
linear with over time univer-
bounds and sity
budget
constraint
Brandeau et Set of RCB & Budgeting & Medical USA
al. (1987) equations increment planning, cost school by
predicting budget for (in a private depart-
incremental 5 years university) ment
change in
expenses &
revenues by
dept.
Whalen Decentralized Responsibility-center HEI USA
(1991) management budgeting
of HEI
Banerjee and Questionnaire/ Computer capacity planning National USA
Igbaria (1993) survey
Sinuany-Stern Regression Hardware resources (CPU, University Israel
and Yelin memories, etc.)
(1993)
Sinuany-Stern Data Efficiency and planning University Israel
et al. (1994) envelopment academic departments
analysis
(DEA)
Thomas System of Formula-based budgeting HEI UK
(1999) resource
allocation
Priest et al. Review Incentive-based budgeting Pub. General
(2002) Univ.
Kao et al. Decision Library Decision- University Taiwan
(2003) support acquisition making, library
allocation budget budget
model via allocation
circulation
data mining
Adekanmbi Questionnaire Budgeting Library Libraries Botswana
and Boadi to study resource at 6
(2008) libraries acquisition teaching
budgeting colleges
(continued)

Nicholls Markov Planning & benchmarking University Australia
(2009) models doctoral progression
Zierdt (2009) Responsibility-centered budgeting HEI USA
Trusheim and Cohort- Enrollment prediction & University USA
Rylee (2011) survival, tuition income
percentages
by type of
student
Frank (2012) Formula Formula State National Israel
mainly by budgeting allocation
discipline per
student
Tang and Yin Grey models Education expenditure & National USA
(2012) better than student enrollment
exponential
smoothing
Agboola and Ratio method Student forecast for faculty National Nigeria
Adeyemi & fixed planning
(2013) annual
percent
Ekanem Questionnaire Zero-based To verify staff University Nigeria
(2014) on ZBB— budgeting opinion on
Descriptive (ZBB) ZBB
statistics
Cheporov and Time Formula Student/faculty University Ukraine
Cheporova equations budgeting ratio is too
(2015) activity-based low
costing
Sweeney et Educational Strategy Student National USA
al. (2016) data mining, planning planning—
regression, Retention, by
factorization grade
machine prediction
Davidovitch Regression Criteria for faculty University USA
et al. (2016) incentive for teaching &
research excellence
Soares et al. General Activity-based costing for HEIs USA
(2016) evaluation community college
Philbin & MSP Strategic Investing in University United
Mallo, 2016 (managing planning new research in coop- Kingdom
successful facility eration
programs)
(continued)

Steinþórsdóttir Case study Budgeting Gender needs University Iceland
et al. (2016)
De La Torre Mix integer Planning Long-term Public Spain
et al. (2016) linear faculty size & universi-
programming composition ties
(MILP)
Gorbunov et Simulation Private Planning University Russia
al. (2017) model, using cloud vs. future info.
AnyLogic univer- Sys. Load for
(determinis- sity’s own ed. purposes,
tic) server facility
planning
Bogomolova Correlation Planning & Budgeting National USA and
et al. (2017) between budgeting science in Russia
economic universities
factors and
budgeting of
science
Kenno and Heuristic Activity- Resource University Canada
Sainty (2017) inquiry and based allocation
content budgeting
analysis
Xiao and System Demand & supply forecast National China
Chankong dynamics of students (medical talents)
(2017)
simulation
Hamid et al. Survey listing Space Facility Public Malaysia
(2018) space by type capacity, planning, universi-
of facility facility, & performance ties
(labs & efficiency strategy
classroom) by
government
standards
Walter (2018) Case study to Strategic Facility University USA
promote planning planning, library
strategic strategy
cooperation
of library
users
Bogomolova Optimization Budgeting Resource Universities, Russia
et al. (2018) model— allocation national
Quadratic
Magnanti and Discrete Allocating students to School of Singapore
Natarajan optimization multidisciplinary projects engineer-
(2018) ing
(continued)

Aviso et al. P-graph Human Planning, National Philippines
(2019) model resource resourcing,
quality
Garcia (2019) Multi- Scenario Optimal College USA
objective analyses resource of
max-flow allocation business
Myers (2019) Moral hazard Responsibility-centered University USA &
mathematical budgeting Canada
program
Oude Vrielink Timetabling Planning Course, exam, HEI General
et al. (2019) class, faculty review
Smith (2019) Review HE accounting & budgeting HEI General
Henry-Moss Online survey Lactation Service National USA
et al. (2019) facility for campus Commu-
breast facilities for nity
feeding women, better colleges
accessibility
Wardhani et Regression, Participation Strategy, University Indonesia
al. (2019) employee’s & budget planning,
questionnaire control budgeting
methods on importance
good
governance in
budgeting &
planning
Source Google scholar (assessed during 2020)
Appendix 2: Example for Linear and Quadratic Budget

Allocation Modes
Assume we have 6 schools in a university. The first 3 columns of Table 9.4 present
the number of students, the upper and lower bounds of the allocations to the schools,
where the lower bounds are based on cost analyses, where the upper bounds are
the requests of the schools. The schools are ordered according to their number of
students.
Under budget constraint and bounds constraints: the linear model allocates to
the first four schools their upper bound and to the other schools, with a lower number
of students, only their lower bound is allocated, as shown in the fifth column,
according to model 1c in Table 9.1. First, the lower bounds are allocated to each
school, afterwards the first schools get addition to their upper bound are reached
till the budget is finished, the rest of the schools do not get any additional budget
beyond their lower bound.
While the quadratic model allocates between the bounds according to model 2c
in Table 9.2 not exactly in the middle of the midpoint, as shown in the seventh
Table 9.4 Linear and quadratic budget allocation under bounds and limited budgeta
Linear allocation: limited Quadratic allocation:
Schools Student # bounds Weights of students budget of 74,000 limited budget 74,000
lower upper limited not limited not
Liberal arts 2000 30, 000 32, 000 0.360 32,000 32, 000 31, 072.0 31, 000
Engineering 1000 11, 200 12, 000 0.180 12,000 12, 000 11, 636.0 11, 600
Business 840 5600 6000 0.151 6000 6000 5830.2 5800
Social sci. 800 10, 000 10, 800 0.144 10,800 10, 800 10, 428.8 10, 400
Sciences 600 5200 7200 0.108 5200 7200 6221.6 6200
9 Models for Planning and Budgeting in Higher Education
Health sci. 320 8000 9600 0.057 8000 9600 8811.4 8800
Total 5560 70, 000 77, 600 1.000 74,000 77, 600 74, 000.0 73, 800
a All figures in the table are in thousands
295
column. After initial allocations in the midpoints, the leftover added to each school
proportionally to their number of students.
When there are no budget constraints, but there are bound constraints, then in
the linear model, each school receives its upper bound, according to model 1d, as
shown in Column 6.
While the quadratic model allocation to each school is in the midpoint between
the bounds, as shown in Column 8. To simplify the calculation the president of the
university can set the budget to be allocated according to the eighth column with
a total of $73,800,000, and leave the leftover $200,000 (74,000,000 − 73,800,000)
for unforeseen costs.
References
Adekanmbi, A. R., & Boadi, B. Y. (2008). Budgeting for library resources in colleges of education.
Library Collections, Acquisitions, & Technical Services, 32(2), 68–75.
Agboola, B. M., & Adeyemi, J. K. (2013). Projecting enrollment for effective academic staff
planning in Nigerian universities. Educational Planning, 21(1), 5–17.
Almog, T., & Almog, O. (2020). Academia – All the lies: What went wrong in the university model
and what will come in its place. Haifa University Publishing.
Aviso, K. B., Chiu, A. S. F., Demeterio, F. P. A., Lucas, R. I. G., Tseng, M.-L., & Tan, R. R. (2019).
Optimal human resource planning with P-graph for universities undergoing transition. Journal
of Cleaner Production, 224(1), 811–822.
Balderston, F. E., & Weathersby, G. B. (1972). PPBS in higher education planning and manage-
ment: Part II, the University of California experience. Higher Education, 1(2), 299–319.
Banerjee, S., & Igbaria, M. (1993). An empirical study of computer capacity planning in U.S.
universities. Information & Management, 24(4), 171–182.
Barr, M. J., & McClellan, G. S. (2018). Budgets and Financial Management in Higher Education
(3rd ed.). Jossey-Bass, A Wiley Brand.
Berkeley. (2020). Retrieved August 19, 2020, from https://opr.ca.gov/docs/20181226-
Master_Plan_Report.pdf
Bogomolova, A., Balk, I., Ivachenko, N., & Temkin, A. (2017). Impact of the macroeconomic
factors on university budgeting in the US and Russia. Journal of Physics: Conf. Series, 913(1),
1–8. https://doi.org/10.1088/1742-6596/913/1/012006
Bogomolova, A., Balk, I., Ivachenko, N., & Terlyga, A. (2018). Budget optimization modelling
for sustainable development of the university research: The example of Russia. IOP Conf.
Series: Journal of Physics: Conf. Series, 1117(1), 1–6. https://doi.org/10.1088/1742-6596/
1117/1/012012
Brandeau, M., Hopkins, D. S. P., & Melmon, K. L. (1987). An Integrated Budget Model for Medical
School Financial Planning. Operations Research, 35(5), 638–794.
Brinkman, P. (1984). Formula budgeting: The fourth decade. New Directions for Institutional
Research, 43, 21–44.
Cheporov, V., & Cheporova, G. (2015). The time equations and formula budgeting in jointed model
for higher education. American Journal of Educational Research, 3(8), 1015–1019.
Davidovitch, N., & Sinuany-Stern, Z. (2014). Compensation criteria in higher education in
Israel and elsewhere: Evaluation of research and teaching outcomes. Journal of International
Education Research, 10(4), 279–294.
Davidovitch, N., Sinuany-Stern, Z., & Druckman, E. (2016). Assessing research and teaching
performance in Israel’s higher education system: The case study of Ariel University. Creative
Education, 7(11), 1604–1614.
De La Torre, R., Lusa, A., & Mateo, M. (2016). A MILP model for the long-term academic staff
size and composition planning in public universities. Omega: The International Journal of
Management Science, 63, 1–11. https://doi.org/10.1016/j.omega.2015.09.008
Deering, D. (2015). Responsibility Center Budgeting and Responsibility Center Management:
Implications for internal structure and strategic management of North American universities.
PhD Thesis, University of Toronto, Canada.
Deering, D., & Lang, D. (2017). Responsibility center budgeting and management ‘Lite’ in
university finance: Why is RCB/RCM never fully deployed? Planning for Higher Education,
45(3), 94–109.
Dubeck, L. W. (1997). Beware higher Ed’s newest budget twist. Thought & Action, 13(1), 81–91.
Ekanem, E. E. (2014). Zero-based budgeting as a management tool for effective university budget
implementation in University of Calabar, Nigeria. European Journal of Business and Social
Sciences, 2(11), 11–19.
Elson, D. (2017). Financing for gender equality: How to budget in compliance with human rights
standards. In Z. Khan & N. Burn (Eds.), Financing for gender equality: Realizing Women’s
rights through gender responsive budgeting (pp. 25–48). Palgrave Macmillan UK.
Etzkowitz, H. (2001). The second academic revolution and the rise of entrepreneurial science.
IEEE Technology and Society Magazine, 20(2), 18–29.
Fonte, R. W. (1987). Community college formula funding: A policy analysis framework. Commu-
nity College Review, 15(2), 5–13.
Frank, G. (2012). The budget model of Israel system of higher education. CHE (Council of Higher
Education) Jerusalem.
Garcia, C. (2019). Practice summary: Managing capacity at the University of Mary Washington’s
College of Business. INFORMS Journal on Applied Analytics, 49(2), 93–171. https://doi.org/
10.1287/inte.2019.0986
Gibson, A. (2009). Budgeting in higher education. The Higher Education Management Series 3,
Silver Spring, MD.
Goldstein, L. (2012). A guide to college and university budgeting: Foundations for institutional
effectiveness (4th ed.). National Association of College and University Business Officers.
Gorbunov, A., Isaev, E., & Morgunov, A. (2017). A simulation model for educational process
planning in an institution of higher education. Modeling of Social and Economic Systems 2(40),
57–67.
Hamid, M. Y., Hamdan, W. S. Z. W., Zaidi, M. A., Yasin, M. F. M., Radzuan, N. A. M., & Ali, I. M.
(2018). The importance of space capacity and frequency audit of a Malaysia public university.
International Journal of Advanced and Applied Sciences, 5(9), 23–27.
Hanover Research. (2010). Program costing models at institutions of higher education.
Retrieved September 2, 2020, from http://www.planning.salford.ac.uk/__data/assets/pdf_file/
0017/20726/Program-Costing-Models-at-Institutions-of-Higher-Education-Membership.pdf
Henry-Moss, D., Lee, J., Benton, K., & Spatz, D. L. (2019). An exploration of lactation facilities
and planning in U.S. higher education campuses. Breastfeeding Medicine, 14(2), 121–127.
Johnes, J., Portela, M., & Thanassoulis, E. (2017). Efficiency in education. The Journal of the
Kao, S. C., Chang, H. C., & Lin, C. H. (2003). Decision support for the academic library acquisition
budget allocation via circulation database mining. Information Processing & Management,
39(1), 133–147.
Kenno, S. A., & Sainty, B. (2017). Revising the budgeting model: Challenges of implementation
at a university. Journal of Applied Accounting Research, 18(4), 496–510.
Lasher, W. F., & Sullivan, C. A. (2004). Follow the money: The changing world of budgeting in
higher education. In J. C. Smart (Ed.), Higher education: Handbook of theory and research vol.
XIX (pp. 197–240). Kluwer Academic Publisher.
Magnanti, T. L., & Natarajan, K. (2018). Allocating students to multidisciplinary capstone projects
using discrete optimization. Interfaces Journal on Applied Analytics, 48(3), 204–216.
Massy, W. F. (Ed.). (1996). Resource allocation in higher education (economics of education).
University of Michigan Press.
Massy, W. F. (2017). Reengineering the university: How to be Mission Centered, market Smart,
and margin conscious. Johns Hopkins University Press.
Myers, G. M. (2019). Responsibility center budgeting as a mechanism to deal with academic moral
hazard. Canadian Journal of Higher Education, 49(3), 13–23.
Nicholls, M. G. (2009). The use of Markov models as an aid to the evaluation, planning and
benchmarking of doctoral programs. The Journal of the Operational Research Society, 60(9),
1183–1190.
OECD. (2019). Retrieved from https://www.oecd.org/education/education-at-a-glance/
educationataglance2019-dataandmethodology.htm
Oude Vrielink, R. A., Jansen, E. A., Hans, E. W., & van Hillegersberg, J. (2019). Practices
in timetabling in higher education institutions: A systematic review. Annals of Operations
Research, 275(2), 145–160.
Philbin, S. P., & Mallo, C. A. (2016). Business planning methodology to support the development
of strategic academic programs. Journal of Research Administration, 47(1), 23–39.
Priest, D., Becker, W., Hossler, D., & St. John, E. (2002). Incentive-based budgeting Systems in
Public Universities. Edward Elgar.
Rosenblatt, M., & Sinuany-Stern, Z. (1989). Generating the discrete efficient frontier to the capital
budgeting problem. Operations Research, 37(3), 384–394.
Saaty, T. L., & Vergas, L. (2012). Models, methods, Concepts & Applications of the analytic
hierarchy process (2nd ed.). Springer, New York, NY, USA.
Sinuany-Stern, Z. (1983a). Break-even and equilibrium analyses in university financial planning.
Computers Environment and Urban Systems, 8(2), 109–119.
Sinuany-Stern, Z. (1983b). Cost simulation models for university budgeting. Computers Environ-
ment & Urban Systems, 8(4), 211–216.
Sinuany-Stern, Z. (1984a). A financial planning model for a multi-campus college. Socio-
Economic Planning Science, 18(1), 135–142.
Sinuany-Stern, Z. (1984b). A network optimization model for budget allocation in multi-campus
university. The Journal of the Operational Research Society, 35(8), 749–757.
Sinuany-Stern, Z. (1993). A network optimization model for budget planning in multi-objective
hierarchical systems. The Journal of Operational Research Society, 44(3), 297–308.
Sinuany-Stern, Z. (2014). Quadratic model for allocating operational budget in public and nonprofit
organizations. Annals of Operations Research, 221(1), 357–376.
Sinuany-Stern, Z (forthcoming-a). Forecasting methods in higher education: An overview. In
Sinuany-Stern (ed.), Handbook of Operations Research and Management Science in Higher
Education, Springer, New York, NY, USA.
Sinuany-Stern, Z (forthcoming-b). Operations research and management science in higher educa-
tion: an overview. In Sinuany-Stern (ed.) Handbook of Operations Research and Management
Science in Higher Education, Springer, New York, NY, USA.
Sinuany-Stern Z & Hirsh A (forthcoming) The relative efficiencies of higher education in OECD
countries. In Sinuany-Stern (ed.) Handbook of Operations Research and Management Science
in Higher Education, Springer, New York, NY, USA.
Sinuany-Stern, Z., Mehrez, A., & Barboy, A. (1994). Academic departments’ efficiency via DEA.
Sinuany-Stern, Z., & Yelin, H. (1993). Forecasting computer resources. Computers & Operations
Research, 20(5), 477–484.
Slaughter, S., & Leslie, L. L. (1997). Academic capitalism; politics, policies and the
entrepreneurial university. Johns Hopkins University Press.
Smith, D. O. (2019). University finances: Accounting and budgeting principles for higher
education. Johns Hopkins University Press.
Soares, L., Steele, P., & Wayt, L. (2016). Evolving HE business model: Leading with data to deliver
results. American Council on Education Center for Policy Research and Strategy.
Steinþórsdóttir, F. S., Einarsdóttir, Þ., Heijstra, T. M., & Pétursdóttir, G. M. (2016). Money talks:
Gender budgeting in the University of Iceland. Icelandic Review on Politics & Administration,
12(1), 171–194.
Sweeney, M., Lester, J., Rangwala, H., & Johri, A. (2016). Next-term student performance
prediction: A recommender system approach. Journal of Educational Data Mining, 8(1), 22–
51.
Tang H-W, V., & Yin, M.-S. (2012). Forecasting performance of grey prediction for education
expenditure and school enrollment. Economics of Education Review, 31(4), 452–462.
Terrey, J. N. (1968). Program budgeting and other newer management tools in higher education:
A description and annotated bibliography. Center for Development of Community College
Education. Paper no. 6.
Thomas, H. G. (1999). Managerial implications of adopting formula-based systems of resource
allocation: A case study from higher education. Educational Management & Administration,
27(2), 183–191.
Toutkoushian, R., & Danielson, C. (2002). Using performance indicators to evaluate decentralized
budgeting systems and institutional performance (pp. 205–226). In W. E. Becker, D. Hossler,
& E. P. John (Eds.), Incentive-based budgeting Systems in Public Universities. Edward Elgar.
Toutkoushian, R. K., & Shafiq, M. N. (2010). A conceptual analysis of state support for higher
education: Appropriations versus need-based financial aid. Research in Higher Education,
51(1), 40–64.
Trusheim, D., & Rylee, C. D. (2011). Predictive modeling: Linking enrollment and budgeting.
Planning for Higher Education Journal, 40(1), 12–21.
Walter, S. (2018). Communicating value through strategic engagement. Library Management,
39(3), 154–165.
Wardhani, R., Marwa, T., Fuadah, L., Siddik, S., & Awaluddin, M. (2019). Good university
governance: Budgeting participation and internal control. Asia-Pacific Management Accounting
Journal, 14(1), 1–18.
Weathersby, G. (1967). The development and applications of a university cost simulation model.
University of California.
Weathersby, G. B., & Balderston, F. E. (1972). PPBS in higher education planning and manage-
ment: Part I, an overview. Higher Education Quarterly, 1(2), 191–206.
Whalen, E. (1991). Responsibility Center budgeting: An approach to decentralized Management
for Institutions of higher education. Indiana University Press.
Xiao, B., & Chankong, V. (2017). A system dynamics model for predicting supply and demand of
medical education talents in China. Eurasia Journal of Mathematics, Science and Technology
Education, 13(8), 5033–5047.
Zierdt, G. L. A. (2009). Responsibility-centered budgeting: An emerging trend in higher education
budget reform. Journal of Higher Education Policy and Management, 31(4), 345–353.
Part II
New Methodologies
Chapter 10
Funding Research in Higher Education
Institutions: The Game Theory Approach
Baruch Keren, Yossi Hadad, and Yizhaq Minchuk
Abstract This chapter presents a unique model, based on game theory, that can
help decision-makers in higher education (HE) institutions determine an optimal
research budget. The model can then help them decide how to allocate that budget
among academic units such as researchers, institutions, and departments. The model
considers the management of the institution as a contest organizer and the academic
units as contestants that compete with each other to win the contest. The prize
of this contest is a desired research budget. The proposed model includes a form
of two contestants with different abilities, as well as a form with unlimited (N)
contestants with the same abilities. The model enables decision-makers to determine
the size of the optimal research budget (the prize), and the optimal distribution
mechanism (a contest or a budget division) of that prize among the contestants.
To the best of our knowledge, determining the size of the Tullock contest prize
according to the contestants’ abilities with comparison to a bargaining model has
not previously been studied. This is an application that is new to the HE budget
allocation process. The study includes a numerical example that demonstrates the
model and its applicability.
Keywords Contest · Game theory · Budget division · Higher education ·

Academic units · Research funding · Analytical Hierarchical Process (AHP)
10.1 Introduction and Scientific Background
One of the principal challenges of government is the promotion of higher education

(HE). Realization of this challenge can be met by increasing the accessibility of
the population to academia, to high-quality teaching, and to international research
B. Keren · Y. Hadad () · Y. Minchuk

Department of Industrial Engineering and Management, SCE—Shamoon College of Engineering,
Beer Sheva, Israel
e-mail: baruchke@sce.ac.il; yossi@sce.ac.il; yizhami@sce.ac.il

304 B. Keren et al.
excellence. To achieve these goals, an appropriate budget must be provided to

HE institutions and they, of course, must use it wisely. Setting the right budget
and determining its partition among the needs of the institution should be done
while taking into account all relevant considerations and constraints in order to
achieve a good balance between various aims and goals. At the system level of
HE, competition can be divided between exclusivist elite institutions and mass
institutions; the former produce high-value positional goods, where demand always
exceeds supply and expansion is constrained to maximize status, while the latter are
characterized by place-filling and expansion (Marginson, 2006).
HE institutions usually receive the majority of their funding from public sources,
but private funding also plays an important role in HE. Many governments use
competitive elements in the process of allocating public funds to institutions of HE.
Examples include the implementation of performance measures through a “funding
formula,” or resource allocation on the basis of evaluated project proposals (Liefner,
2003). Governments exert significant effort to build proper funding mechanisms
(budget size, allocation, and regulation) for HE in order to receive maximum impact
from that funding in the future (King, 2000). There are several forms of budgeting
allocations in HE that have been employed over the years. The allocation might be as
simple as division by specific criteria or may involve the use of a more sophisticated
mechanism. In recent years, many governments adopted performance-based funding
(Nisar, 2015). While one of the main purposes of this form of budget allocation is
to increase competitiveness between universities and colleges, given the complex
nature of the HE system, those strategies have limited effect (Niklasson, 1996;
Nisar, 2015). Internal allocation of resources in some universities and colleges is
also competition-based (Liefner, 2003).
Massy (1996) suggested that it is obvious that one should just put the money
where it will do the best. He questioned whether, in the most prestigious universities,
it was obvious that one should hire the best possible faculty and then step back and
let them do their jobs. He wondered why good results could not be reliably attained
simply by deciding what programs are most in need of funding and giving them the
money. Massy concluded that making the right decision is not that simple and that
even when common sense is coupled with knowledge and discipline, institutions are
not always sufficiently able to distinguish good processes from bad.
Israel, for example, has an independent body called the “Planning and Budgeting
Committee” (PBC), which is preparing and overseeing Israel’s HE budget. The
functions of the PBC were set in accordance with a 1977 government decision
and include proposing HE budgets while taking into consideration the needs of
society and the state, safeguarding academic freedom, and observing due diligence
for research and instruction. In connection with these responsibilities, the PBC
has the exclusive right to allocate regular and development budgets between the
institutions of HE. The PBC exercises these rights in order to promote efficiency in
the institutions of HE and to monitor the utilization of budgets in order to prevent
deficits. In general, there are two budget types that HE institutions receive from
the PBC—a global block grant (the main budget), and a budget that is based on
outputs (the competitive part). In 2012, the PBC established a new funding model
10 Funding Research in Higher Education Institutions: The Game Theory Approach 305
for HE institutions. In this new model, the research budget is divided among the
institutions competitively, according to their research outputs. The research outputs
are: competitive research grants (34%), other grants (15%), Ph.D. students (15%),
publications (34%), and M.Sc. students (2%). Moreover, the two leading institutions
in several areas of science can receive an addition to their research budget of up to
20% (see PBC, 2012). This budgeting model has two problems—neither the amount
of the research budget as determined by the PBC nor the division of the budget
among the institutions, are necessarily optimal amounts. Indeed, although there are
some models suggesting proper contest mechanisms (see Moldovanu & Sela, 2001,
2006; Schweinzer & Segev, 2012), but not in the HE context. Thus, one of the
questions that should be asked is whether performance-based competition yields
the desired outcome.
In current science policies, competition and output incentives are emphasized
as a means of making university systems efficient and productive (Auranen &
Nieminen, 2010). Competition (or a contest) is a well-known mechanism for
resource allocation. There are many contest-like situations—in the sports world,
the labor market, in rent-seeking, and so on. Among the well-known tools used to
predict contest outcomes are game theory tools. There are many papers based on
game theory that are applicable to contests as a means for analyzing and solving
problems in which players expend costly efforts in order to get ahead of one another
(Konrad, 2009). Our chapter applies contest theory tools in order to determine if it
is better to use performance-based funding (competition) or some form of budget
division. There are a limited number of studies that have used game theory to
analyze systems of HE. Most of those studies are based on the principal-agent
theory to describe the interaction between the government and HE systems (see,
for example, Enders et al., 2013; Nisar, 2015).
We propose to use the contest theory framework as a tool for performance-based
funding. The interaction between the contestants (universities and colleges or their
subordinate departments) and between the responsible authorities, can be modeled
as a game. This work focuses on the issue of an additional budget that should be
allocated among institutions (universities and colleges) or among departments in one
institute. At the first stage, the authorities use allocation mechanisms (a contest or a
budget division) and the contestant’s abilities (in the case of contests) to determine
the budget. The paper finds a proper benchmark that indicates when the contest
model or a budget division provides a more favorable outcome, e.g., a higher utility,
for the principal (a higher utility).
This chapter presents a game-theory-approach-based model that enables
decision-makers to determine the optimal amount of a research budget that should
be allotted, and then properly divide this amount among academic units. The
academic units in this model are contestants that compete with each other to win
the contest; the management of the institution is the contest organizer. The model
includes two alternatives—a division of the budget in which the distribution of
resources is carried out “fairly” among the contestants, or a competition in which
the winner takes all. The study includes a numerical example that demonstrates its
applicability.
306 B. Keren et al.
Contests are economic or social interactions in which two or more players

expend costly resources in order to win a prize. The resources expended by players
determine their probability of winning a prize (Chowdhury & Sheremeta, 2011).
As mentioned, a typical tool to incentivize researchers and academic units is to add
budget or other resources such as research equipment or assistants. In any incentive
fee method, the principal must determine in advance the incentive amount to be
shared (money or resources), and how this amount would be divided among the
participants (criteria) in order to maximize its utility.
In this chapter, the contest model is based on a sequential two-stage of a Tullock
contest (Tullock, 1980), under complete information. The Tullock model describes
a contest where the probability of a contestant winning the contest depends on his
effort and the effort of his rivals. The contestant who exerts the greatest effort has
the highest probability of winning, but he cannot be certain of actually winning (for
more details see Konrad, 2009). Additionally, in our model, we assume complete
information, i.e., the abilities of the contestants and the value of the additional
budget are publicly known. In the first stage of the contest, the organizer determines
the additional budget. The budget size is determined by the ability of the contestants,
a form of altruism toward the contestants, and the interest rate that the organizer
should pay for the additional budget. In the second stage, the contestants see the
size of the budget and determine the amount of effort each is willing to exert.
Chowdhury and Sheremeta (2011) explained why the variations of Tullock’s
models are widely used. First, a number of studies have provided axiomatic
justification for them (Clark & Riss, 1998; Congleton et al., 2008; Skaperdas,
1996). Second, Baye and Hoppe (2003) had identified conditions under which a
variety of rent-seeking contests, innovation tournaments, and patent-race games are
strategically equivalent to the Tullock contest. In this work, we derive the conditions
that determine which mechanism is more profitable to the organizer—a budget
division or a contest.
As its main contribution to the literature, this paper proposes a practical
application that combines models from the contest theory, the bargaining theory,
and the analytical hierarchical process (AHP) that supports higher education budget
allocation policy. Moreover, existing literature appears non-existent with regard to
determining the size of a Tullock contest prize according to the contestants’ abilities,
with comparison to a bargaining model. Although there were some works that
connected between the effort and the prize, see, for example, Chung (1996), this
is an application that is new to budget allocation in HE.
The chapter is organized as follows: Sect. 10.2 presents the background of the
study; Sect. 10.3 presents the contest model; Sect. 10.4 presents the bargaining
model; Sect. 10.5 compares between contest and budget division solutions; Sect.
10.6 deals with the case of several academic units with identical abilities; Sect. 10.7
is a numerical example; and Sect. 10.8 highlights the main conclusions.
10.2 Background
10.2.1 Budget Allocation Problem in Higher Education
The problem of allocating a budget among HE units can be solved by the use of the
linear budget allocation (LBA) model. In the LBA model, the objective function is
to maximize the sum of the benefits of the budget allocation, over all the units. Each
unit has minimum and maximum requirements regarding its budget needs. Another
requirement is that the budget allocation problem must have a feasible solution.
In other words, the total budget must be enough to supply the minimum budget
needs of all the units. This model can be easily solved by linear programming. One
problem of this model is difficulty evaluating the marginal organizational benefit
associated with the allocation.
Sinuany-Stern (2014) claimed that this marginal benefit may be estimated
subjectively by various single or multi-attribute utility approaches, as proposed
by Keeney and Raiffa (1993) or Royes (2004), or by the AHP, as was proposed
by Saaty and Vergas (2001). In our proposed model there are also parameters
that should be estimated subjectively by experts or by the AHP. Sinuany-Stern
(2014) emphasized that there are several requirements needed by a budget allocation
model: fairness, effectiveness, and efficiency in the public sector. For nonprofit
organizations, fairness is the most important quality desired of a budget allocation
model. Sinuany-Stern (2014) proposed a quadratic budget allocation model (QBA),
which assumes that the objective function is to minimize the quadratic deviations of
the allocations from their bounds. The applicability of this model was demonstrated
by an example where a dean of a school of engineering must allocate a budget for
several departments in her school. She concluded that the QBA is more fair and
effective than the LBA, although the LBA may be more efficient. Thus, the QBA is
a better fit for the public sector and for nonprofit organizations, while LBA is more
suited for-profit organizations.
10.2.2 Budget Partition and Fairness Index

n 2

n
One of the fairness indexes is fA (x) = xi / n xi2 , where xi is the
i=1 i=1
budget allocated to unit i and n is the number of units (Jain et al., 1984). If all
the units get identical budgets, the fairness index is 1. When one unit gets the
entire budget and all other units get zero, the fairness index is 1/n. Therefore,
1/n ≤ fA (x) ≤ 1. In general, a fair division is a method of dividing a resource among
several parties in such a way that all recipients believe that they have received a fair
amount according to some rules of fairness (see, Brams & Taylor, 1996).
Consider a simple example with two academic units, where one unit has two
researchers and the other unit has three researchers. Assume that a budget of B$ is
308 B. Keren et al.
planned to be divided fairly between the two units according to the research ability
of the two units. If all five researchers have equal abilities, the first unit will get 25 B$
and the second unit will get 35 B$; but the fairness index between the two units will
be 0.9615. Now assume that the two researchers of unit one published a combined
total of 10 papers per year. Also, assume that during the same time period the three
researchers of unit two published a combined total of 6 papers. In such a case, the
marginal publication ability of unit one is a1 = 1/10 and of unit two is a2 = 1/6. A
budget division according to the publication capability is 10 16 B for unit one and 16 B
6
for unit two; but the fairness index between the two units will be 0.9412. From a
practical point of view, it can be assumed that 0 < ai < 1. The value ai = 0 represents
a unit with unlimited ability (infeasible) and ai ≥ 1 represents a unit with poor ability
(not a candidate for additional budget funding). Sinuany-Stern (2014) stated that an
equal budget for all units, or fA (x) = 1, is not always efficient for the owner of the
budget. However, as will be shown in this paper, the owner of the budget can ensure
more utility by organizing a contest where the winner takes all (see Eq. 10.1), than
the utility he can get from a budget division obtained by a bargaining solution (see
Eq. 10.15).
10.2.3 Efficiency
Efficiency can be defined as profit (i.e., output minus input), or as the ratio between
output and input. In any case, the common aim is maximizing efficiency. It is
difficult, and often impossible, to measure efficiency in public sector entities and
in nonprofit organizations. This is particularly so because many of their values are
often intangible. One method used to measure efficiency in the public sector and in
HE institutions is to quantify relative efficiency through the use of data envelopment
analysis (DEA) and its variants. For a review of the application of DEA and ranking
methods, see Adler et al. (2002) and Hadad and Hanani (2011).
10.2.4 Literature References to Game Theory Applications

in Higher Education
There are few papers that directly deal with the application of game theory in HE.
Burguillo (2010) presents a method for using game theory tournaments as a way to
implement competition-based learning, together with other learning techniques. His
goals were to use game theory to help increase motivation learning performance for
the students. Burguillo’s main contribution is the use of game theory tournaments
to develop programming frameworks that can be used to support competition-based
learning.
Niklasson (1996) analyzed a 1993 Swedish reform of HE. That reform made
changes from central planning to deregulation, privatization, and performance-
related funding. He used a model based on game theory to examine the interaction
between government and universities, where government and universities are seen as
actors in an iterated prisoners’ dilemma analysis. Niklasson stated that one problem
in his study was to make the universities play against each other, instead of only
against the government. Another problem in this and similar models is to ensure
socio-economic efficiency.
Nisar (2015) explored the strategy of U.S. President Obama to make colleges
more affordable for the middle class. Nisar claimed that “paying for performance”
was a core component of Obama’s strategy, although most impact assessment
studies had shown that such policies have had a limited effect on the performance of
these institutions. Most explanations given for this failure have been on the basis of
principal-agent theory, resource dependence theory, and neo-institutionalism. Nisar
explained the failure of performance-based funding policies in terms of the inherent
complexity of the HE system. He used the concept of ecology of games (Long,
1958) and claimed that such games are not based on the rigid assumptions and pay-
off structures of classical game theory. Instead, these games are centered on specific
policy goals in which various political actors participate, each with different agendas
and ideals.
Rothschild and White (1995) developed a model where the contestants are
universities that compete for students through price and non-price means. According
to their model, some students may be desirable to a university and those students
may be the object of special price discounts given by means of scholarships.
They specified a simple production process for universities’ educational services,
based on the Cobb-Douglas production function. The results of the model are a set
of prices that would allocate students among universities efficiently. Epple et al.
(2006) developed an equilibrium model of the market for HE. They simultaneously
predicted student selection into institutions of HE, financial aid, educational expen-
ditures, and educational outcomes. They showed a strict hierarchy of colleges that
differ by the educational quality provided to the students. Later, Fu (2014) developed
a structural equilibrium model of the HE market. Students, having heterogeneous
abilities and preferences, make application decisions subject to uncertainty and
application costs. The HE institutions measure student ability and choose tuition
and admissions policies to compete for better students.
10.3 The Proposed Contest Model
The proposed contest model is stated as follows: Consider a contest with one
principal and several academic units (AUs). The principal might be a governmental
or institutional budgeting committee, a president of an academic institute, a dean
of faculty, or other groups or individuals serving similar functions with similar
responsibilities. The AUs can be academic institutions, academic departments,
310 B. Keren et al.
researchers, and so on. The AUs compete with each other to win extra resources
(such as a research budget), where all the AUs are under supervision by the same
principle. All the players are risk-neutral and the value of the extra resources is V.
The contest itself will be the execution of an important task for the institution—
one that would promote the institution economically and/or would increase its
reputation. However, this task will not necessarily produce research outputs for
the contestants that will promote them in their academic careers. Such contest
tasks might include writing research proposals, developing new curricula, social
activities, public relations activities, and other activities that are not directly linked
to the academic agenda of the researchers. Therefore, the efforts that the researchers
invest in the contest may somehow harm their ongoing academic activities, so their
participation in the contest has a price. On the other hand, winning the contest will
allow the winning researchers to use the prize money to promote their core research.
The expected utility of the principal is
Uc = x1 + x2 + α (U1 + U2 ) − c(V ), (10.1)
here:
V—the value of the prize (in monetary units) that the winner in the contest will
receive (a decision variable of the principal).
xi —the efforts (in monetary units) in the contest exerted by the AUs, i = 1, 2, . . . ,
n in order to win the contest (decision variables of the AUs). In our case, the cost
function of the efforts is linear and given by g(xi ) = xi . Thus, the efforts of the
contestants can strictly translate to monetary units.
Uc —the expected utility of the principal (in monetary units).
Ui —the expected utility of the AUs i, i = 1, 2, . . . , n (in monetary units).

n
Pi —the probability that AUi will win the contest, i = 1, 2, . . . , n, Pi = 1.
i=1
c(V)—the cost of the prize to the principal (in monetary units).
Parameters with values that should be estimated:
α—the rate of the expected utility of the contestants that the principal credits to
himself.
γ —the coefficient of the cost function of the prize.
ai —the ability parameter of AUi . A low value of this parameter means a higher
ability to produce academic outputs. In other words, it is easier for a contestant
with a low ai value to produce academic outputs. This parameter can be estimated
according to previous years’ performances.
It is well known that borrowing money has a disadvantage to scale. A larger loan
increases the risk taken by the lender, for which he asks a higher rate of interest.
The assumption here is that the cost function of the prize, c(V), has a quadratic form
that is given by c(V) = γ V2 when γ > 0. This proposed form was developed by
Baker et al. (1994) and it is based on the concept of increasing the marginal cost of
incentives, which is a common assumption in financial markets.
The utility function of the principal includes a portion of the AUs’ utility. Given
that the extra resources will help the research projects succeed, the principal has an
incentive to allot the extra resources to the AUs. The value of parameter α describes
the rate of the expected utility of the AUs that the principal credits to himself. In
other words, α → 0, no credit, α → ∞, very high credit.
The expected utility functions of the AUs are based on a variation of the Tullock
contest, given by
U1c = V P1 − a1 x1 , (10.2)
U2c = V P2 − a2 x2 , (10.3)
where 0 < ai < 1 is the ability factor of AU i = 1, 2 (a small value of ai means high
ability).
The probabilities are given by
x1
P1 = x1 +x2 ,
x2
P2 = x1 +x2 .
The stages of the game are as follows: In the first stage, the principal chooses the
value of the extra resources to be allotted (V), in order to maximize his expected
utility. In the second stage, the AUs see the value of the extra resources and
determine the efforts they want to exert in the contest.
In order to analyze the subgame perfect equilibrium of the contest, a backward
induction is used beginning with the second stage. The second stage is a simple
asymmetric Tullock contest with a well-known solution that maximizes the prob-
lems shown in (10.2) and (10.3) (see, for example, Nti (1999), Konrad (2009)).
Thus, the efforts in the contest are
a2
x1 = V, (10.4)
(a1 + a2 )2
a1
x2 = V. (10.5)
(a1 + a2 )2
Substituting (10.4 and 10.5) in (10.2 and 10.3) yields
a22
U1c = V, (10.6)
(a1 + a2 )2
312 B. Keren et al.
a12
U2c = V. (10.7)
(a1 + a2 )2
Substituting (10.4), (10.5), (10.6), and (10.7) into (10.1) yields
a2
Uc = V + a1 2 V
(a1 +a2 )2 (a1 +a2 )
a22 a12
+α V + V − γ V 2.
(a1 +a2 )2 (a1 +a2 )2
Thus, the maximization problem of the principal is

V a 2 + a12
maxUc = max + 2 αV − γ V 2 (10.8)
V V (a1 + a2 ) (a1 + a2 )2
Differentiating (10.8) with respect to V yields
∂Uc 1 a 2 + a12
= + 2 α − 2γ V = 0
∂V (a1 + a2 ) (a1 + a2 )2
Rearranging the derivative

1 1 a22 + a12
V∗ = + α . (10.9)
2γ (a1 + a2 ) (a1 + a2 )2
2
The second derivative is ∂∂VU2c = −2γ < 0. Namely, the second-order conditions
for local maximum are satisfied.
Substituting (10.9) in (10.8) yields
2
1 1 a 2 + a12
Uc = + 2 α . (10.10)
4γ (a1 + a2 ) (a1 + a2 )2
Then, substituting (10.9) in (10.4, 10.5, 10.6, and 10.7) will yield the contestant
efforts and the utilities.
10.4 The Bargaining Model for Budget Division
Now consider a case where the principal allows the AUs to engage in bargaining
with each other for the extra resources; the extra resources between the AUs will be
allotted by the principal according to the agreement reached by the AUs. In a case
of disagreement between the AUs that prevents a budget division (the bargaining
model solution), the contest solution will be used.
The bargaining solution will yield a different utility function to the principal
because in such case the utility does not depend on the efforts of the AUs. In any
event, in the bargaining process, the principal enjoys a portion of the utilities of
the AUs. In some cases, a bargaining solution might be a better solution for the
principal than the contest solution, as will be presented later. The utility function of
the principal in the case of bargaining is
Ud = α (U1 + U2 ) − γ Vd2 . (10.11)
The value V of the extra resources is determined according to this bargaining

solution. The proposed division rule between the AUs is 0 ≤ θ ≤ 1 and accordingly,
the AUs’ utilities are
U1b = Vd θ, (10.12)
U2b = Vd (1 − θ ) . (10.13)
This type of division rule is common in many bargaining solutions (see, for
example, Anbarci et al., 2002, Skaperdas, 2006, the Nash bargaining solution, or
the Kalai–Smorodinsky solution). For example, in a Nash bargaining solution θ is
chosen to maximize the following product:

max U1d − U1c U2d − U2c . (10.14)
θ
From (10.11) it is easy to see

that
in this case, the optimal value of the extra
∗
resources to be allotted is Vd = 2γ and the principal’s utility is given by
α
α2
Ud = . (10.15)
4γ
10.5 Contest Versus Budget Division
This section includes a comparison between the two mechanisms (a contest and a
budget division), in order to select the mechanism that will yield a higher utility
for the principal. It is well known that in the case of a linear cost function in a
contest, it is not a good policy to divide the prize and establish two prizes (see,
Moldovanu & Sela, 2001; Schweinzer & Segev, 2012). Furthermore, in a case with
two contestants, the second prize is considered by the contestants as a consolation
prize. Generally, in a divided prize case, the effort of the contestants will be exerted
314 B. Keren et al.
according to the difference between the main prize and the consolation prize (see,
for example, Szymanski & Valletti, 2005).
The following two propositions derive conditions under which one mechanism
(a contest or a budget division) is more profitable for the principal.
Proposition 10.1 There exists α ∗ such that for all α < α ∗ , a contest is more profitable
for the principal than a budget division, and vice versa.
Proof: Taking the difference between (10.10) and (10.15) we get
⎡ 2 ⎤
1 ⎣ 1 a22 + a12
Uc − Ud = + α − α2 ⎦ .
4γ (a1 + a2 ) (a1 + a2 )2
Notice that
⎡ 2 ⎤
∂ 2 (U c − Ud ) 1 a22 + a12
= ⎣ − 1⎦ .
∂α 2 2γ (a1 + a2 )2
a22 +a12 ∂ 2 (Uc −Ud )

Because 0 ≤ ≤ 1 we get < 0.
(a1 +a2 )2 ∂α 2
∂(Uc −Ud )
Because Uc − Ud |α = 0 > 0, Uc − Ud |α → ∞ < 0, and ∂α =
α=0
1 a22 +a12
> 0, there exists an α ∗ that fulfils the condition that
2γ (a1 +a2 )4
Uc − Ud |α=α ∗ = 0, such that for α > α ∗ , Uc − Ud < 0 and for α < α ∗ , Uc − Ud > 0.
That yields the result of Proposition 10.1.
The intuition behind the results is as follows: for a small α, the authorities prefer
a contest because their cost for the effort transferred from the academic units is low.
Once αis large, the cost borne by the academic unit transferred to the authorities’
payoff is too high to conduct a contest.
10.6 The Case of Several AUs with Identical Abilities
This section considers a case with n AUs that have identical abilities
(a1 = a2 = . . . = an = a). For AUs with identical abilities, a budget
division is
made simply by dividing the resource equally among the AUs, Vnd . The principal’s
n
utility in this case is Ud = α Ui − γ Vd2 = αn Vnd − γ Vd2 = αVd − γ Vd2 .
i=1
Thus, result (10.15) also holds for this case and
α2
Udn = . (10.16)
4γ
In a contest setup, the principal’s utility is

n
n

Ucn = xi + α Ui − γ V 2. (10.17)
i=1 i=1
Following (10.2) and (10.3), the utilities of AU i, i = 1, . . . , n can be rewritten

as:
Ui = V P i − ax i , (10.18)
xi
where Pi =
n .
xi
i=1
By applying the backward induction approach as used in the previous section, the
solution of (10.18) is the well-known symmetric equilibrium of a Tullock contest
(see, for example, Konrad, 2009), which is given by:
n−1V
xi = i = 1, . . . , n, (10.19)
n2 a
V
Ui = i = 1, . . . , n. (10.20)
n2
Substituting (10.20 and 10.19) in (10.17) yields:
n−1V α
Ucn = + V − γ V 2. (10.21)
n a n

∂Ucn
Differentiating (10.21) with respect to V yields: ∂V = n−1
an + α
n − 2γ V = 0.
By rearranging the derivative:
1 n−1 α
V∗ = + , (10.22)
2γ an n
2
and the second derivative, ∂∂VU2c = −2γ < 0. Thus, the second order condition for
the local maximum is satisfied. Substituting (10.22) in (10.21) yields:
2
1 n−1 α
Ucn = + . (10.23)
4γ an n
The next proposition addresses a comparison between a contest and a budget

division. As mentioned before, dividing the prize in a case of linear cost function
like in Tullock contest is not optimal (Schweinzer & Segev, 2012). Thus, we focus
316 B. Keren et al.
on a comparison between the cases where the winner takes all (a contest) and a
budget division.
Proposition 10.2 There is α ∗ such that for all α < α ∗ , a contest is more profitable
for the principal than a budget division, and vice versa.
Proof: In a manner similar to that of Proposition 10.1,
∂ 2 (Ucn −Udn )
∂α 2
= 2γ n12 − 1 < 0. Because Ucn − Udn α=0 > 0, Ucn − Udn α→∞ <
1

∂ (Ucn −Udn )
0 and ∂α = 1
2γ
n−1
an2
> 0, we get that for all α < α ∗ , Ucn − Udn > 0
α=0
and for α > α ∗ , the inequality is reversed.
10.7 A Numerical Example
The president of an academic institution in Israel is interested in providing research

funds to the academic departments. These funds are intended to be used in order to
promote the quality of the research and the scope of publications, to increase the
academic reputation of the institution, and to produce economic opportunities.
The budget planned for this purpose is NIS 100,000 (V1 ) (about $30,000). Taking
into account the overheads related to this budget (15%), the effective cost of this
amount to the institute is NIS 115,000 (V2 ). From these two values and the formula
c(V) = γ V2 , the value of parameter γ can be calculated as follows: γ = V22 =
V1
115
1002
= 0.0115.
The research budget will be given to one of two academic departments, depart-
ment A and department B, on a competitive basis. Department A has 15 academic
staff members and department B has 8 academic staff members. The benefit gained
by the winning department would directly benefit the institution itself. It is estimated
that the benefit to the institution is 20% higher than the benefit gained by the winning
department. Therefore, the value of α is determined to beα = 1.2.
In order to estimate the value of the coefficient of the marginal research
capability(ai) the following four criteria were selected:
P—Peer-reviewed papers
W—Winning research proposals
C—Participation in academic conferences
S—Supervision of graduate students
In order to determine the relative importance of each criterion, the AHP
methodology (Satty, 1980) was used. The weight of each criterion, as presented in
Table 10.2, is based on the principal eigenvector of a pairwise comparison matrix,
as presented in Table 10.1. If, for example, the value between P and C in the
comparison matrix (Table 10.1) is 7, it means that from the point of view of the
decision-maker, one peer-reviewed paper equals seven participations in academic
conferences.
Table 10.1 Pairwise P W C S

comparison matrix
P 1 1/3 7 3
W 3 1 9 3
C 1/7 1/9 1 1/3
S 1/3 1/3 3 1
Table 10.2 Weights of the Category Priority (%) Rank

criteria
P 28.7 2
W 53.5 1
C 4.6 4
S 13.2 3
Table 10.3 Research outputs of the two departments

Outputs Weights (%) Department A Department B
Peer-reviewed papers 28.7 32 25
Winning research proposals 53.5 6 4
Participation in academic conferences 4.6 19 15
Supervision of graduate students 13.2 8 11
In the AHP the numeric values are derived from the subjective preferences of
the decision-makers. Therefore, some inconsistencies in the matrix of judgments
may occur. Saaty (2012) showed that a consistency ratio (CR) of 10% or less is
acceptable to continue using the AHP analysis. If the consistency ratio is greater
than 10%, it becomes necessary to revise the comparison values of the judgments
in order to obtain more consistency. In our case, the consistency ratio is CR = 4.6%
(less than 10%), which means that the decision-maker is consistent. The weights of
the four criteria based on this matrix (Table 10.1) are presented in Table 10.2.
The research outputs of the two departments during the previous year are
presented in Table 10.3.
These weights can be used to convert the four output types into a single output
type. This step is needed in order to calculate the marginal research ability, ai . In
this case, all the outputs will be converted into “article equivalents.” This output
(articles) was selected as a reference because the management of the institution
defined as a standard the requirement that each academic member produce 1.5
articles per year. The ratio between the weights of each output to the weight of
the output “articles” is the equivalent of one unit of specific output per article. For
53.5%
example, one winning research proposal is equivalent to 28.70% = 1.8641 articles.
Table 10.4 shows that department “A” produced outputs that were equivalent
to 49.91 articles, and department “B” produced outputs that equivalent to 39.92
articles.
As stated, the standard research outputs for each academic member per year
is equivalent to 1.5 articles. The ratio between the actual weighted outputs and
the standard outputs reflects the efficiency of the departments. The efficiency
318 B. Keren et al.
Table 10.4 Article equivalent research outputs of the two departments

Department A Department B
Equivalent to one outputs (articles outputs (articles
Outputs article equivalent) equivalent)
Peer-reviewed papers 1.0000 32.00 25.00
Winning research 1.8641 11.18 7.46
proposals
Participation in 0.1603 3.05 2.40
academic
conferences
Supervision of 0.4599 3.68 5.06
graduate students
Total articles 49.91 39.92
equivalent
of department “A,” which has 15 academic members, is 15×1.5 49.91

= 2.22. The
department’s research output is 122% higher than the standard output. Therefore,
its ability parameter is aA = 2.22
1
= 0.45. When calculated in a similar manner, the
ability parameter of department “B” is aB = 0.30.
Substituting the values in the model (10.9) yields V∗ = 85.101. In other words,
the amount that should be allocated as a prize to the winning department in the
competition (which lasted for 1 year of research) is NIS 85,101. Because not all the
budget is being allocated to the prize, the remainder will be used by the institution
to promote research goals in other ways. The net expected utility of the institution
itself from the competition is Uc = 83.286. In this contest model, one department
gets all the budget V∗ = 85.101 and the other department gets zero. Therefore, the
fairness index is 0.5.
The probability of department “A” to win the contest is pA = aAa+a B
B
=
0.3+0.45 = 0.4,while the probability of department “B” to win the contest is
0.3
pB = aAa+a A
B
= 0.6. It is clear that if the parties want to increase their chance
of winning in any future contest they must improve their abilities in the long run.
The optimal budget of the bargaining prize in this numerical example is Vd∗ =
∗
2γ = 2×0.0115 = 52.17. The division of this amount, Vd = 52.17, between the
α 1.2
two departments can be done according to the Nash bargaining solution or by any
similar method. Ultimately, the net expected utility of the institution itself from
a budget division is Ud = 24.00, for any division. If the division of the amount
Vd∗ = 52.17is done according to the “research outputs” of each department as
presented in Table 10.4, then VA = Vd∗ × 49.91/ (49.91 + 39.92) = 28.99 and
VB = Vd∗ × 39.92/ (49.91 + 39.92) = 23.18. The fairness index for this division is
fA (x) = 0.9875. Note that both solutions, the contest prize and the bargaining budget,
are less than the budget that was planned initially by the president of the institution.
The residual budget can be used for other research goals.
10.8 Conclusions
Academic institutions must deal with the problem of research budget size and how it
should be divided among subordinate academic units. Allocation of an appropriate
research budget would improve and increase research outputs. More high-quality
research outputs will improve the academic reputation of the institution and its
ability to raise higher budgets and recruit students. This chapter proposed a unique
competition model based on the Tullock contest, which enables decision-makers in
HE institutions to determine the optimal research budget and its distribution among
academic units: researchers, institutions, or departments.
The chapter includes a comparison between two mechanisms for determining
and allocating a budget for a prize, a contest or a budget division. The issue of
which mechanism is better was examined from the point of view of the organizer.
The conditions when a contest is preferable and when a division is preferable to the
organizer were exactly defined. It was shown that a budget division is better for the
organizer than a contest, if the utilities of the contestants are very important for the
organizer, i.e., the contestants’ utilities are a significant part of the organizer’s utility.
Otherwise, a contest is preferred. The model is expanded to unlimited contestants
where the participants have the same ability.
We used a method based on AHP methodology for estimating the research
abilities of the academic units in the institution. We followed this methodology
in order to determine research ability, ai , and thus to identify the two academic
units with the highest abilities in the institution; those were the units that would
participate in the contest. The research budget was determined according to the
abilities of the participants in the contest, the cost of raising the needed budget,
and the benefit of the contest organizer.
The model can be used by managers in various applications in which there is
a requirement to allocate a budget on a competitive basis. For example, research
and development budgets where the contestants are universities and academic units,
cultural budgets for institutions where the contestants are theaters and concert
houses, and sports budgets where the contestants are athletes and teams.
Future research may take several directions:
• Solving a contest model with more than two contestants, where each participant
has a different ability. This problem does not yet have an analytically close
solution.
• In a contest with N identical participants, it is easy to see that engaging more
participants yields a higher utility to the organizer. Our conjecture is that this is
also true for a contest with N participants, where each participant has a different
ability. This issue should be explored.
• Extension of the proposed model to a case with incomplete information and to
an all-pay contest model where the contestant with the highest effort wins the
contest with a probability 1 result. (In a Tullock contest, the contestant with the
highest effort has only the highest probability of winning).
320 B. Keren et al.
References
Adler, N., Friedman, L., & Sinuany-Stern, Z. (2002). Review of ranking methods in the data
envelopment analysis context. European Journal of Operational Research, 140(2), 249–265.
Anbarci, N., Skaperdas, S., & Syropoulos, C. (2002). Comparing bargaining solutions in the
shadow of conflict: How norms against threats can have real effects. Journal of Economic
Theory, 106(1), 1–16.
Auranen, O., & Nieminen, M. (2010). University research funding and publication performance—
An international comparison. Research Policy, 39(6), 822–834.
Baker, G., Gibbons, R., & Murphy, K. J. (1994). Subjective performance measures in optimal
incentive contracts. The Quarterly Journal of Economics, 109(4), 1125–1156.
Baye, R. M., & Hoppe, C. H. (2003). The strategic equivalence of rent-seeking, innovation, and
patent-race game. Games and Economic Behavior, 44, 217–226.
Brams, S. J., & Taylor, A. D. (1996). Fair division: From cake-cutting to dispute resolution.
Cambridge University Press.
Burguillo, J. C. (2010). Using game theory and competition-based learning to stimulate student
motivation and performance. Computers & Education, 55(2), 566–575.
Chowdhury, S. M., & Sheremeta, R. M. (2011). A generalized Tullock contest. Public Choice,
147(3), 413–420.
Chung, T. Y. (1996). Rent-seeking contest when the prize increases with aggregate effort. Public
Choice, 87, 55–66.
Clark, D. J., & Riss, C. (1998). Competition over nore than one prize. The American Economic
Review, 88(1), 276–289.
Congleton, D. R., Hillman, L. A., & Kontad, A. K. (2008). 40 years of research on rent seeking 2:
Applications: Rent seeking in practice. Springer.
Enders, J., De Boer, H., & Weyer, E. (2013). Regulatory autonomy and performance: The reform
of higher education re-visited. Higher Education, 65(1), 5–23.
Epple, D., Romano, R., & Sieg, H. (2006). Admission, tuition, and financial aid policies in the
market for higher education. Econometrica, 74(4), 885–928.
Fu, C. (2014). Equilibrium tuition, applications, admissions, and enrollment in the college market.
Journal of Political Economy, 122(2), 225–281.
Hadad, Y., & Hanani, M. Z. (2011). Combining the AHP and DEA methodologies for selecting the
best alternative. International Journal of Logistics Systems and Management, 9(3), 251–267.
Jain, R. K., Chiu, D. M. W., & Hawe, W. R. (1984). A quantitative measure of fairness and
discrimination. Eastern Research Laboratory, Digital Equipment Corporation, Hudson, MA.
Keeney, R. L., & Raiffa, H. (1993). Decisions with multiple objectives: Preferences and value
trade-offs. Cambridge university press.
King, F. A. (2000). The changing face of accountability. The Journal of Higher Education, 71(4),
411–331.
Konrad, K. A. (2009). Strategy and dynamics in contests. Oxford University Press.
Liefner, I. (2003). Funding, resource allocation, and performance in higher education systems.
Long, N. E. (1958). The local community as an ecology of games. American Journal of Sociology,
64(3), 251–261.
Marginson, S. (2006). Dynamics of national and global competition in higher education. Higher
Education, 52(1), 1–39.
Massy, W. F. (1996). Resource allocation in higher education. University of Michigan Press.
Moldovanu, B., & Sela, A. (2001). The optimal allocation of prizes in contests. American
Economic Review, 91(3), 542–558.
Moldovanu, B., & Sela, A. (2006). Contest architecture. Journal of Economic Theory, 126(1), 70–
97.
Niklasson, L. (1996). Game-like regulation of universities: Will the new regulatory framework for
higher education in Sweden work? Higher Education, 32(3), 267–282.
Nisar, M. A. (2015). Higher education governance and performance based funding as an ecology
of games. Higher Education, 69(2), 289–302.
Nti, K. O. (1999). Rent-seeking with asymmetric valuations. Public Choice, 98(3–4), 415–430.
PBC. (2012). Budgeting model of the Higher Education system in Israel (in Hebrew). http://
che.org.il/wp-content/uploads/2012/04/Model_tiktsuv.pdf
Rothschild, M., & White, L. J. (1995). The analytics of the pricing of higher education and other
services in which the customers are inputs. Journal of Political Economy, 103(3), 573–586.
Royes, G. F. (2004). A hybrid fuzzy-multicriteria-CBR methodology for strategic planning. In
Fuzzy information (Vol. 1, pp. 208–213).
Saaty, T. L. (2012). Decision making for leaders: The analytic hierarchy process for decisions in a
complex world (3rd Revised ed). RWS Publications.
Saaty, T. L., & Vergas, L. G. (2001). Models, methods, concepts and applications of the analytical
hierarchy process. Springer.
Satty, T. L. (1980). The analytical hierarchy process: Planning, priority setting, resource alloca-
tion. RWS publication.
Schweinzer, P., & Segev, E. (2012). The optimal prize structure of symmetric Tullock contests.
Public Choice, 153(1–2), 69–82.
Sinuany-Stern, Z. (2014). Quadratic model for allocating operational budget in public and nonprofit
Skaperdas, S. (1996). Contest success function. Economic Theory, 7, 283–290.
Skaperdas, S. (2006). Bargaining versus fighting. Defense and Peace Economics, 17(6), 657–676.
Szymanski, S., & Valletti, T. M. (2005). Incentive effects of second prizes. European Journal of
Political Economy, 21(2), 467–481.
Tullock, G. (1980). Efficient rent seeking. In J. M. Buchanan, R. D. Tollison, & G. Tullock (Eds.),
Toward a theory of the rent-seeking society (pp. 97–112). Texas A&M University Press.
Chapter 11
A Fast Threshold Acceptance Algorithm
for the Examination Timetabling
Problem
Nuno Leite, Fernando Melício, and Agostinho C. Rosa
Abstract In this chapter, an accelerated variant of the threshold acceptance (TA)

metaheuristic, named FastTA, is proposed for solving the examination timetabling
problem. FastTA executes a lower number of evaluations compared to TA while not
worsening the solution cost in a significant way. Each exam selected for scheduling
is only moved if that exam had any accepted moves in the immediately preceding
threshold bin; otherwise, the exam is fixed and is not evaluated anymore. If an exam
had zero accepted movements in the preceding threshold bin, it is likely to have
few or zero accepted movements in the future, as it is becoming crystallised. The
FastTA and TA were tested on the Toronto and Second International Timetabling
Competition benchmark (ITC 2007) sets. Compared to TA, the FastTA uses
38% and 22% less evaluations, on average, on the Toronto and ITC 2007 sets,
respectively. On the ITC 2007 set, the FastTA is competitive with TA attaining the
best average solution cost value in four out of twelve instances while requiring less
time to execute. Compared with the state-of-the-art approaches, the FastTA is able
N. Leite ()
Instituto Superior de Engenharia de Lisboa, Instituto Politécnico de Lisboa, Lisboa, Portugal
LARSyS—Laboratory for Robotics and Systems in Engineering and Science, Universidade de
e-mail: nleite@cc.isel.ipl.pt
F. Melício
Instituto Superior de Engenharia de Lisboa, Instituto Politécnico de Lisboa, Lisboa, Portugal
e-mail: fmelicio@laseeb.org
A. C. Rosa
Department of Bioengineering/Instituto Superior Técnico, Universidade de Lisboa, Lisboa,
Portugal
e-mail: acrosa@laseeb.org

324 N. Leite et al.
to achieve competitive results. The main contribution/value of this chapter is the

proposal of a new acceptance criterion for the TA metaheuristic, which leads to a
significantly faster variant (FastTA), and its application to solve public examination
timetabling benchmark sets.
Keywords Examination timetabling · Hybrid algorithm · ITC 2007 benchmark

set · Local search · Threshold acceptance · Timetabling · Toronto benchmark set
11.1 Introduction
Many companies and organisations deal with timetabling problems, having to build
various types of timetables on a regular basis. This is the case of transportation
and railway companies, bus/metro service, sport federations, hospitals, universities,
among others. The timetabling problem is also solved generally by companies
that need to schedule personnel or resources for a given task, e.g., work shifts, or
supermarket companies that need to distribute and deliver, themselves, the products
to their online customers. Since timetabling is a combinatorial problem, dealing
with it in medium- and large-size companies is a very complex task. Therefore,
automatic solvers are required by companies’ decision makers in order to solve their
timetabling tasks.
Timetabling problems involve fixing a time for sets of triples (events, resources,
space), satisfying a given set of hard and soft constraints. Events can comprise
lectures, examinations, surgeries, sport events, or trips. Resources can include
teachers, nurses and medical staff, sports referees, or vehicles. Space can be
classrooms, hospital rooms, or sport fields.
In this chapter, the examination timetabling problem (ETP) (Qu et al., 2009)
is solved. This problem consists in scheduling exams to rooms and time slots,
trying to space out enrolled students as much as possible, as well as satisfying other
constraints. Two published benchmark sets, the Toronto (Carter et al., 1996) (also
known as Carter’s datasets) and the Second International Timetabling Competition
(ITC 2007) (McCollum et al., 2012), comprise the standard sets mainly used by
researchers in this area.
The ETP is a NP-complete decision problem (de Werra, 1985, 1997). Methods
applied to solve this problem include operations research approaches (e.g., mathe-
matical programming), for relatively small-size problem instances, and approximate
methods (artificial intelligence (Schaerf, 1999) methods, metaheuristics (Qu et al.,
2009), and constraint programming approaches).
11 A Fast Threshold Acceptance Algorithm for the Examination Timetabling Problem 325
11.1.1 Related Work
The first studies involving the examination timetabling problem include the seminal
works by Broder (1964) and Cole (1964), which are the first works that try to solve
the problem using a computer program. Carter and Laporte (1996) and Schaerf
(1999) review recent approaches employed to solve the ETP. In Welsh and Powell
(1967), a relation between the related problems of graph colouring and timetabling
is studied. In Carter (1986), the authors employ various graph colouring heuristics to
solve the ETP. One of the most successful graph colouring is the saturation degree
heuristic proposed by Brélaz (1979). In Cheong et al. (2009), the authors study the
application of four graph colouring heuristics to the ETP, namely largest degree,
colour degree, saturation degree, and extended saturation degree. They conclude
that the saturation degree heuristic was among the two best heuristics.
Mathematical programming approaches for solving the ETP were proposed in
the research literature. In a recent work, Woumans et al. (2016) propose a column
generation approach for solving the examination timetabling problem. The authors
apply two mathematical models to solve the ETP at KU Leuven campus Brussels
(Belgium), for the business engineering degree program, and apply the models to
the sta83 and yor83 instances of the Toronto benchmark set from the literature.
However, they used a different formulation of the Toronto set in which an exam is
allowed to be scheduled more than once in the timetabling. The reported results
are good on smaller datasets such as the ETP at KU Leuven. The results were
also good for the Toronto dataset (multiple exam formulation) using one of the
proposed models, obtaining new upper bounds on the Toronto set’s sta83 and
yor83 instances. However, on larger datasets such as the yor83 instance, the
second model was not able to obtain a feasible solution within the time limit of
400 hours.
Local search approaches, such as simulated annealing (SA) (Kirkpatrick et al.,
1983), are within the most used metaheuristics used to solve timetabling problems.
The use of SA to timetabling dates back to the 1990s, with the pioneer works
of Dowsland (1990) and Abramson (1991). In a later investigation, Thompson
and Dowsland (1996) use SA to solve a variant of the ETP (a multi-objective
formulation of the ETP). In Thompson and Dowsland (1998), the same authors
propose an approach based on SA for solving examination timetabling problems.
The authors employ and compare three neighbourhoods (standard, Kempe chains,
and S-Chains). In their experiments, the authors conclude that the Kempe chains
neighbourhood, a concept extracted from the graph theory and used also on graph
colouring problems, is the most effective one. In the literature, SA-based approaches
were also used to solve other types of educational timetabling problems (school
and course timetabling), such as the works of Melício et al. (2000), Melício et al.
(2004), Zhang et al. (2010), and Bellio et al. (2016). More recently, the SA was
used to solve the ITC 2007 problem formulation (Battistutta et al., 2017). In
their work, the initial solution is generated in a random fashion and could be
infeasible. Non-feasible states are penalised by the algorithm. The authors employ
326 N. Leite et al.
a statistically principled experimental analysis with the goal of understanding the

effect of parameter selection on the algorithm performance. After this, the algorithm
parameters are set using a feature-based parameter tuning, which is able to achieve
a good generalisation on unseen instances. The study concludes that the standard
SA, when properly tuned, is able to solve the ITC 2007 problem effectively.
June et al. (2019) propose the application of constraint programming and
simulated annealing and use their method to produce the exam timetable of
Universiti Malaysia Sabah Labuan International Campus (UMSLIC).
Li et al. (2015) propose a search framework for exam timetabling problems
based on evolutionary ruin and stochastic rebuild and apply it to solve the Toronto
benchmark set.
The ITC 2007 participants have proposed methods that range from hybrid
local search (Müller, 2009), greedy randomised adaptive search procedure
(GRASP) combined with other local search metaheuristics (Gogos et al., 2008),
constraint programming solver hybridised with tabu search and iterated local
search (Atsuta et al.—2008 (McCollum et al., 2010)), tabu search approach (De
Smet—2008 (McCollum et al., 2010)), and nature-inspired heuristic approach based
on cell biology (Pillay—2008 (McCollum et al., 2010)).
Other works on the ITC 2007 examination timetabling track comprise meth-
ods ranging from population-based metaheuristics, such as bee colony optimisa-
tion (Alzaqebah & Abdullah, 2014, 2015) and cellular memetic algorithm (Leite
et al., 2018), hill-climbing variants (Bykov & Petrovic, 2013), great deluge local
search (Hamilton-Bryce et al., 2013; McCollum et al., 2009), GRASP (Gogos et al.,
2012), to hyper-heuristics (Demeester et al., 2012; Özcan et al., 2012; Burke et al.,
2014; Sabar et al., 2015). Bykov and Petrovic (2016) present a new single-parameter
local search heuristic named step counting hill-climbing algorithm (SCHC). The
algorithm is applied to the ITC 2007 examination timetabling track. Leite et al.
(2019) propose an accelerated variant of SA, named FastSA, that is able to reduce
the number of evaluations as compared to SA. The FastSA was tested on the ITC
2007 benchmark set requiring 17% less evaluations, on average, compared with SA,
and attaining a new upper bound on one instance.
Some surveys on the field of educational timetabling were published recently. Qu
et al. (2009) present a detailed survey of algorithmic strategies applied to the
ETP. Kristiansen and Stidsen (2013), Johnes (2015), and Teoh et al. (2015) survey
the application of operations research and metaheuristic approaches to academic
scheduling problems. Gashgari et al. (2018) present a recent survey on exam
scheduling techniques. Bettinelli et al. (2015) present an overview of curriculum-
based course timetabling. Pillay (2016) presents a review of hyper-heuristics for
educational timetabling.
Metaheuristics’ components were recently studied in the literature. In Dowsland
and Thompson (2012), the authors study the several components of simulated
annealing and its variations. In Santini et al. (2018), the authors investigate the move
acceptance criterion component of adaptive large neighbourhood search (ALNS)
and compare a range of alternatives. Among the best variants, the authors find
versions of criteria based on simulated annealing, threshold acceptance, and record-

to-record travel algorithms.
Table 11.22, in the Appendix, presents the main characteristics of recent method-
ologies applied to solve educational timetabling problems for Higher Education
(HE).
11.1.2 This Chapter’s Contribution
In this chapter, an approach based on the threshold acceptance (TA) Dueck and
Scheuer (1990) local search algorithm is proposed for solving the examination
timetabling problem. Threshold acceptance is a metaheuristic method that belongs
to the family of simulated annealing. TA has been applied in several areas ranging
from operations research to optimisation in statistics and econometrics Winker
(2001).
The proposed method comprises two phases, a construction phase and an
optimisation phase. We extend the study made earlier on the SA in Leite et al. (2019)
and study how the TA could be accelerated, similarly to what is done in Leite et al.
(2019). The proposed approach, named fast threshold acceptance (FastTA), is tested
on the Toronto and ITC 2007 benchmark sets and compared with the state-of-the-art
approaches.
The main contribution/value of this chapter is the proposal of a new acceptance
criterion for the TA metaheuristic, which leads to a significantly faster variant
(FastTA), and its application to solve public examination timetabling benchmark
sets.
The remaining sections of this chapter are organised as follows. In Sect. 11.2,
the notations and abbreviations used in this chapter are defined. Section 11.3
describes the uncapacitated Toronto and capacitated ITC 2007 problem instances.
Section 11.4 presents the proposed approach based on the threshold acceptance
metaheuristic for solving the examination timetabling problem. Section 11.5 reports
the experimental results on the Toronto and ITC 2007 benchmark sets and compares
FastTA’s results with the ones in the literature. Section 11.6 presents concluding
remarks and future work.
11.2 Notations and Abbreviations
For the convenience of readers, we collect below the various notations and
abbreviations employed in the text.
ei Exam identifier
E A set of exams
tk Timeslot identifier
328 N. Leite et al.
P A set of timeslots (or periods)

rl Room identifier
R A set of rooms
C = (cij )|E|×|E| Symmetric matrix of size |E| (called the conflict matrix) where
each element, denoted by cij (i, j ∈ {1, . . . , |E|}), represents
the number of students attending both exams i and j
M Total number of students
δi,j The Kronecker delta function
fc Solution cost
σ Standard deviation
s Solution identifier
N(s) Neighbourhood of solution s
Q Threshold value in threshold acceptance algorithm
r Cooling schedule rate in threshold acceptance algorithm
k Number of iterations performed at a given threshold in threshold
acceptance algorithm
α Alpha level or significance level
p p-value
11.3 Examination Timetabling Problems
In this section, the Toronto and ITC 2007 benchmark sets are described.
11.3.1 Uncapacitated Toronto Dataset
The Toronto benchmark set is a well-known set used in examination timetabling

research. It is an uncapacitated dataset, meaning that no rooms are considered in
its specification. The formulation provided here was adapted from Abdullah et al.
(2009) and Burke et al. (2004). The following terms are defined:
• E = {e1 , . . . , en } is the set of exams.
• P = {p1 , . . . , pk } is the set of time slots (or periods).
• C = (cij )|E|×|E| (called the conflict matrix) is a symmetric matrix of size |E|
where each element, denoted by cij (i, j ∈ {1, . . . , |E|}), represents the number
of students enrolled simultaneously in exams i and j .
• M denotes the total number of students.
• tk (tk ∈ P ) indicates the time slot where exam ek (ek ∈ E) is scheduled.
The Toronto problem formulation introduces a single soft constraint that disal-
lows students to take two exams in near periods (periods that differ from one to five
time slots). This constraint is denoted by the fc function, specified in (11.1). The
problem is formulated as follows:
|E|−1 |E|
1
minimise fc = · cij · prox(i, j ), (11.1)
M
i=1 j =i+1
where

25−|ti −tj | , if 1 ≤ |ti − tj | ≤ 5
prox(i, j ) = (11.2)
0, otherwise
subject to
|E|−1 |E|
0, ti = tj
cij · δti ,tj = 0, δti ,tj = . (11.3)
1, ti = tj
i=1 j =i+1
Function prox(i, j ) in (11.2) specifies how the penalty is calculated when exams ei
and ej are scheduled in time slots ti and tj , respectively. The resulting penalty is
16, 8, 4, 2, and 1, for exams sitting one, two, three, four, and five time slots apart,
respectively. For exams that sit more than five time slots apart, there is no penalty.
Equation (11.3) denotes the unique hard constraint, which forbids conflicts between
two or more exams scheduled in a given time slot.
Table 11.1 presents the characteristics of the uncapacitated Toronto benchmark
set (version I). In Table 11.1, the conflict matrix density is the ratio of the number
of non-zero elements in the conflict matrix and the total number of elements. The
Toronto dataset is available at ftp://ftp.mie.utoronto.ca/pub/carter/testprob.
Table 11.1 Specifications of Conflict

the uncapacitated Toronto
# # # matrix # Time
benchmark set (version I)
Dataset Exams Students Enrolments density slots
car91 682 16, 925 56, 877 0.13 35
car92 543 18, 419 55, 522 0.14 32
ear83 190 1125 8109 0.27 24
hec92 81 2823 10, 632 0.42 18
kfu93 461 5349 25, 113 0.06 20
lse91 381 2726 10, 918 0.06 18
pur93 2419 30, 029 120, 681 0.03 42
rye93 486 11, 483 45, 051 0.08 23
sta83 139 611 5751 0.14 13
tre92 261 4360 14, 901 0.18 23
uta92 622 21, 266 58, 979 0.13 35
ute92 184 2749 11, 793 0.08 10
yor83 181 941 6034 0.29 21
330 N. Leite et al.
11.3.2 The ITC 2007 Dataset
The ITC 2007 benchmark set (International Timetabling Competition, 2007) con-
sists of three tracks, each one related to a type of educational timetabling problem:
• Track 1—examination timetabling
• Track 2—post-enrolment-based course timetabling
• Track 3—curriculum-based course timetabling
The examination timetabling track, which is focused in this chapter, specifies
instances with characteristics and constraint types that are similar to real-world
instances found in practice. It is comprised of 12 instances whose features are
described in Table 11.2. The ITC 2007 formulation extends the Toronto formulation
by adding new kinds of hard and soft constraints.
The ITC 2007 hard constraints are the following (McCollum et al., 2012):
• No Conflicts—Exams having enrolled students in common (conflicting exams)
cannot be scheduled in the same time slot.
• Room Occupancy—The room capacity cannot be exceeded for each room and
time slot.
• Period Utilisation—For each exam scheduled in a time slot, the exam duration
cannot exceed the time available in that period.
• Period Related—Time-ordering requirements exist for pairs (e1 , e2 ) of exams,
which are specified by the following constraints:
– After Constraint—e1 must take place strictly after e2 .
– Exam Coincidence—e1 must take place at the same time as that of e2 .
– Period Exclusion—e1 must not take place at the same time as that of e2 .
Table 11.2 Basic properties of the instances of the ITC 2007s examination timetabling track
Dataset Density Exams Students Periods Rooms Period HC Room HC
Exam_1 0.05 607 7891 54 7 12 0
Exam_2 0.01 870 12, 743 40 49 12 2
Exam_3 0.03 934 16, 439 36 48 170 15
Exam_4 0.15 273 5045 21 1 40 0
Exam_5 0.009 1018 9253 42 3 27 0
Exam_6 0.06 242 7909 16 8 23 0
Exam_7 0.02 1096 14, 676 80 15 28 0
Exam_8 0.05 598 7718 80 8 20 1
Exam_9 0.08 169 655 25 3 10 0
Exam_10 0.05 214 1577 32 48 58 0
Exam_11 0.03 934 16, 439 26 40 170 15
Exam_12 0.18 78 1653 12 50 9 7
Density is the conflict density. HC refers to the numbers of hard constraints
• Room Related—An additional room requirement was defined:

– Room Exclusive—A given exam e1 must take place in a given room, with
exclusive use.
As an additional requirement, the solution must be complete, i.e., all exams have
to be scheduled in the timetable. Exams must also not be allocated in more than
one period or room. A solution is said to be feasible if it satisfies the set of hard
constraints.
The following set of soft constraints was also defined:
• Two Exams in a Row—For all students, the number of times a student has two
exams arranged consecutively should be minimised.
• Two Exams in a Day—For all students, the number of times a student has
two exams arranged in the same day (excluding adjacent exams) should be
minimised.
• Period Spread—Its aim is to spread the students’ exams over a fixed number of
periods.
• Mixed Durations—Its aim is to minimise the number of occurrences of exams
with different durations that are scheduled in the same room and period.
• Front Load—Exams having the largest number of students should be scheduled
at the start of the examination session.
• Room and Period Penalties—Its goal is to minimise the use of selected rooms or
periods in the timetable.
11.3.2.1 Mathematical Formulation
In this section, the main aspects of the ITC 2007 examination timetabling problem
mathematical formulation, proposed by McCollum et al. (2012), are given. For more
details, please refer to McCollum et al. (2012).
In the problem formulation, the following sets are used:
E: set of exams
P: set of periods
R: set of rooms
Institutional Weights and Parameters
The institutional weights, specified for each dataset (in each input file), are the
following:
w2R : weight for “two in a row”
w2D : weight for “two in a day”
wP S : weight for period spread
wNMD : weight for “no mixed duration”
wF L : weight for the front-load penalty
332 N. Leite et al.
Table 11.3 ITC 2007 Dataset w 2D w 2R wP S w NMD wF L

benchmark set institutional
weights Exam_1 5 7 5 10 100
Exam_2 5 15 1 25 250
Exam_3 10 15 4 20 200
Exam_4 5 9 2 10 50
Exam_5 15 40 5 0 250
Exam_6 5 20 20 25 25
Exam_7 5 25 10 15 250
Exam_8 0 150 15 25 250
Exam_9 10 25 5 25 100
Exam_10 0 50 20 25 100
Exam_11 50 10 4 35 400
Exam_12 10 35 5 5 25
Dataset: ITC 2007 dataset instances; w 2D : weight
for “two in a day”; w 2R : weight for “two in a row”;
w P S : weight for period spread; w NMD : weight for
“no mixed duration”; w F L : weight for the front-load
penalty
The institutional weights are depicted in Table 11.3.

In the input files, two parameters are also specified:
wrR : a weight that specifies the penalty for using room r
wpP : a weight that specifies the penalty for using period p
Primary Decision Variables
The binary (Boolean) decision variables that fix the assignment are
P
Xip =1 if exam i is in period p, 0 otherwise (11.4)
R
Xir =1 if exam i is in room r, 0 otherwise. (11.5)
There are other (secondary) variables that are used to write the constraints and to
compute the objective function (McCollum et al., 2012).
Soft Constraints Penalties
The penalties for violations of the various soft constraints are encoded as non-
negative secondary variables as follows (McCollum et al., 2012):
Cs2R = “Two in a row” penalty for student s
Cs2D = “Two in a day” penalty for student s
CsP S = “Period spread” penalty for student s
C NMD = “No mixed duration” penalty

CF L = “Front-load” penalty
CP = “Soft period” penalty
CR = “Soft room” penalty
Objective
The objective function is as follows:
Minimise

w2R Cs2R + w2D Cs2D + wP S CsP S + wNMD C NMD + wF L C F L + C P + C R .
s∈S
(11.6)
There are no separate weights for the room and period penalties, C R and C P , as
the associated weights were already included in their definitions (McCollum et al.,
2012). These are detailed next.
Soft Period Penalty
The soft period penalty, part of the overall penalty, should be calculated on a
period-by-period basis. For each period, the penalty is calculated by multiplying
the associated penalty (represented by weight wpP introduced earlier) by the number
of exams timetabled within that period. The soft period penalties C P are enforced
by
CP = wpP Xip
P
. (11.7)
p∈P i∈E
Soft Room Penalty
This part of the overall penalty should also be calculated on a period-by-period

basis. For each period, if a room used within the solution has an associated penalty,
then the penalty for that room for that period is calculated by multiplying the
associated penalty (represented by weight wrR introduced earlier) by the number
of exams using that room. The soft room penalties C R are enforced by
CR = wrR Xir
R
. (11.8)
r∈R i∈E
334 N. Leite et al.
For details about the full mathematical formulation, please consult (McCollum
et al., 2012).
11.4 Proposed Approach
This section describes the two proposed approaches based on the TA metaheuristic,
namely the standard TA approach, and an accelerated variant of TA, named FastTA.
This section is organised as follows. Section 11.4.1 describes the TA-based search
method and the neighbourhood structure used. Section 11.4.2 describes the FastTA
approach. Sections 11.4.3 and 11.4.4 describe the application of TA and FastTA to
the Toronto and ITC 2007 benchmark sets, respectively.
11.4.1 Threshold Acceptance Metaheuristic
The threshold acceptance (TA) (Dueck & Scheuer, 1990) metaheuristic belongs
to the family of simulated annealing (SA)-based metaheuristics and is structurally
similar to SA. The difference relies in that the TA uses a deterministic acceptance
criterion, whereas the SA uses a probabilistic one. The TA components are:
• X, solution set
• f , objective function to be minimised over X
• N(s), neighbourhood of each solution s in X
• L, state-space graph induced by X and by the definition of N(s)
The TA metaheuristic starts from an initial solution, or state, and proceeds to
improve it by moving in the state space of L. In each movement, a neighbouring
solution s of the current solution s is created. The algorithm accepts the neighbour
solution s , even if f (s ) > f (s), as long as f (s ) − f (s) is less than or equal
to the current threshold, Q. The threshold starts at a high value and is reduced
according to a given cooling schedule. The template of the TA algorithm is shown
in Algorithm 11.1.
Many cooling schedules were proposed in the literature (Talbi, 2009). In this
chapter, the threshold Q is updated by simulating the exponentially decreasing
temperature over time, as used in the metal annealing process. This is achieved by
the function T (t):
T (t) = Qmax · exp(−R t), (11.9)
where R is the threshold decreasing rate and Qmax is the initial threshold (Qmax
should have a large value as compared to R). Function T (t) allows for a slow
decreasing cooling schedule to be defined, given that a small value of parameter
R is defined.
Algorithm 11.1 Steps of the threshold acceptance metaheuristic (Talbi, 2009)

Input: Threshold annealing Qmax
1: s ← s0 Creation of the initial solution
2: Q ← Qmax Initial threshold value
3: repeat
4: repeat At a fixed threshold
5: Generate a random neighbour s ∈ N(s)
6: E ← f (s ) − f (s)
7: if E ≤ Q then s ← s
8: end if Accept the neighbour solution
9: until Equilibrium condition e.g., a given number of iterations executed at
each threshold Q
10: Q ← g(Q) Threshold update
11: until Stopping criteria satisfied e.g., Q ≤ Qmin
12: Output: Best solution found.
In the proposed technique, the Kempe chain (Thompson & Dowsland, 1998)
neighbourhood is used. Only feasible Kempe chain moves are considered. The
Kempe chain heuristic is further explained in detail in Leite et al. (2018).
In the next section, the FastTA search method for solving the ETP is explained.
11.4.2 Accelerated TA Metaheuristic for the ETP
We start by studying, in Sect. 11.4.2.1, the effect of the cooling schedule on the
exam move acceptance rate in TA, introducing the basic framework used by the
FastTA method. The presented study is similar to the one made for the fast simulated
annealing metaheuristic in Leite et al. (2019). The FastTA approach is described in
Sect. 11.4.2.2.
11.4.2.1 Effect of the Cooling Schedule on the Exam Move Acceptance

Rate
In the study carried out in this section, the effect of the cooling schedule on
the exam move acceptance rate in TA is analysed. The study undertaken reveals
some important properties of the local search operator, which are exploited by the
FastTA algorithm. The experiments were performed on the Toronto and ITC 2007
benchmark sets. In the text, the value of the objective function of a given solution is
sometimes referred by the terms cost and solution cost.
As mentioned in Sect. 11.4.1, the cooling schedule in TA is the means by which
the threshold is updated. In order to test the TA algorithm under different cooling
schedule intensities, two representative cooling schedules were used for the Toronto
and ITC 2007 sets: a light cooling schedule and an intensive one. The cooling
336 N. Leite et al.
Table 11.4 Threshold acceptance’s cooling schedules used in the Toronto and ITC 2007
benchmark sets
Dataset Type QMax r k QMin # Evaluations
Toronto Light 0.1 1 × 10−5 5 2 × 10−5 4, 258, 600
Intensive 0.1 1 × 10−6 5 2 × 10−5 42, 585, 970
ITC 2007 Light 1000 1 × 10−5 5 1 × 10−5 9, 210, 346
Intensive 1000 4.5 × 10−6 5 1 × 10−5 20, 467, 426
Threshold acceptance’s cooling schedule parameters: Qmax —initial threshold value, r—cooling
rate, k—number of iterations at a fixed threshold, and Qmin —final threshold value
schedules’ parameters are the following: Qmax —initial threshold value, r—cooling
rate, k—number of iterations at a fixed threshold, and Qmin —final threshold value.
The cooling schedule parameters, for the Toronto and ITC 2007 benchmark sets, are
depicted in Table 11.4.
In the light cooling schedule, the threshold is updated in a faster way leading to
a superficial exploitation of the search space. On the other way, in the intensive
cooling schedule, a slower cooling rate is used, leading to a more intensive
exploitation of the search space compared to the light cooling schedule.
The parameter selection was set as follows. We have selected an initial threshold
value that allows for the great majority of exams to be moved several times, in
order to better explore the search space. Graphically, we know that this effect is
achieved by having the first threshold bin practically filled in Figs. 11.1 and 11.2
(described below). After several experiments, we have determined that the values for
the Qmax parameter shown in Table 11.4 were appropriate values. The other cooling
schedule parameters, r, k, and Qmin , were set as follows. Parameter r (rate) admits
two values, in order to have two cooling schedules, a light one and an intensive
one. Parameter k (number of iterations per temperature/threshold) was set to a low
value, since the intensification is mainly controlled by the rate parameter. Finally,
parameter Qmin (minimum temperature/threshold) was set to have an appropriate
low threshold, in order for the TA algorithm to accept non-improving states with
low probability.
Figures 11.1 and 11.2 illustrate the evolution of the number of accepted exam
movements as the TA threshold varies, for Toronto’s car92 (see Table 11.1) and ITC
2007’s Exam_4 (see Table 11.2) instances, respectively.
The y-axis in the figures represents the exam index. The exams were ordered
according to the largest degree heuristic (Cheong et al., 2009), i.e., sorted decreas-
ingly by the number of conflicts they have with other exams. Hence, the lower
indices represent “difficult” exams, in terms of scheduling complexity, and the
higher indices represent “easy” exams.
The x-axis is divided into ten thresholds bins. A threshold bin is comprised of
adjacent TA’s thresholds. Each bin is represented by two limiting vertical grid lines,
interpreted from the x-axis. Each bin was generated in order to represent an equal
number of evaluations, i.e., the TA metaheuristic computes 1/10 of the total number
0 7,500
100 7,000
200 6,500
Exam
6,000
300
5,500
400
5,000
500
4,500
−1 −2 −3 −4 −5
10 10 10 10 10
Thresholds bins 4,000
(a)
3,500
0
3,000
100
2,500
200
2,000
Exam
300
1,500
400 1,000
500 500
0
10−1 10−2 10−3 10−4 10−5
Thresholds bins
(b)
Fig. 11.1 Evolution of the number of accepted exam moves when applying (a) light and (b)
intensive cooling schedule in the TA algorithm for Toronto’s car92 instance
338 N. Leite et al.
0
2,600
50
2,400
100
2,200
Exam
150
2,000
200
1,800
250
1,600
300
103 102 101 100 10−1 10−2 10−3 10−4 10−5
Thresholds bins 1,400
(a)
1,200
0
1,000
50
800
100
Exam
150 600
200 400
250 200
300 0
103 102 101 100 10−1 10−2 10−3 10−4 10−5
Thresholds bins
(b)
Fig. 11.2 Evolution of the number of accepted exam moves when applying (a) light and (b)
intensive cooling schedule in the TA algorithm for ITC 2007’s Exam_4 instance
of evaluations in each of the ten bins. The gray levels represented in the graph, for
each bin on the x-axis and for each exam index on the y-axis, denote the number
of accepted states for that exam. Larger values of accepted exam movements are
illustrated with darker gray levels.
For the Toronto benchmark set experiment, Fig. 11.1a illustrates the optimisation
using a light cooling schedule, which yields a cost of 3.87. In Fig. 11.1b, the
intensive cooling schedule is employed yielding a lower cost of 3.77. The same
behaviour is observed for the ITC 2007 benchmark set experiment. In Fig. 11.2a,
a light cooling schedule is used yielding a cost of 12,898. Using a more intensive
cooling schedule (Fig. 11.2b), the algorithm yields a lower cost of 12,356.
For each bin, the total number of exams’ accepted states/moves is typically lower
than the number of evaluations carried out in that bin, as only a fraction of candidate
states is accepted by the TA acceptance criterion.
From the figures, one can observe that the more complex exams (having lower
indices) only have accepted states/moves in the beginning of the optimisation
process, when higher threshold values are set. When lower threshold values are
set (e.g., below threshold 1 × 10−2 in the Toronto case), the complex exams start
to stick to their final positions, whereas easier exams (with higher indices in the
y-axis) continue to have accepted states/moves, being fixed to their final positions
only when the threshold value is near the minimum.
As observed from the difference in the top and bottom graphs from Figs. 11.1
and 11.2 (and associated solutions cost), it is crucial to schedule in an optimal or
near-optimal way the difficult exams (largest degree ones), in order for the easy
exams to be scheduled in an optimal or near-optimal way as well. In this way, a
slow cooling schedule is needed for the TA to find the optimal time slots, for the
difficult exams, in a first moment, and later for the easy exams.
11.4.2.2 FastTA Algorithm for the ETP
As illustrated in Figs. 11.1 and 11.2, the difficult exams tend to become crystallised
when lower threshold values are set in the algorithm. We can see that an exam
becomes crystallised when the number of movements in the preceding threshold
bin is zero or near zero. If one decided in the algorithm to not expand states of
(and therefore, not evaluate) crystallised exams, this would yield a faster algorithm.
However, not perturbing a crystallised exam could lead to worse results, if that
exam has bins with zero values followed with bins with non-zero values. In this
chapter, this faster algorithm variant was implemented, which stops perturbing (and
evaluating) an exam when a bin with a zero value is found for that exam. The
pseudocode of this faster TA variant, named FastTA, is specified in Algorithm 11.2.
340 N. Leite et al.
Algorithm 11.2 Template of the fast threshold acceptance algorithm

Input: Cooling schedule and starting threshold Qmax
1: prevBin ← nil; Array containing # accepted moves in the previous bin, initially nil
2: currBin ← createArray(numExams); Array containing # accepted moves in the current
bin, initialised with zeros
3: s ← s0 ; Generation of the initial solution
4: Q ← Qmax ; Starting threshold
5: repeat
6: repeat At a fixed threshold
7: Generate a random neighbour s ∈ N(s);
8: examT oMove ← selectExamFromSolution(s )
9: if prevBin = nil or (prevBin <> nil and prevBin [examT oMove] > 0) then
Evaluate neighbour solution
10: E ← f (s ) − f (s)
11: if E ≤ Q then s ← s
12: end if Accept the neighbour solution
13: if s was accepted then
14: currBin [examT oMove] ← currBin [examT oMove] + 1
15: end if
16: end if
17: until Equilibrium condition e.g., a given number of iterations exe-
cuted at each threshold Q
18: Q ← g(Q) Threshold update
19: prevBin ← updateBin(Q, currBin) If Q is in the next bin, make prevBin ←
currBin and zero currBin
20: until Stopping criteria satisfied e.g., Q ≤ Qmin
21: Output: Best solution found.
In Algorithm 11.2, lines appearing with numbers in bold face are extensions
to the standard TA algorithm, specified in Algorithm 11.1. In the FastTA, all exam
perturbations (number of accepted state transitions/moves) are recorded for each bin
and exam. Perturbing an exam consists in selecting an exam randomly and checking
if the exam’s previous bin value is zero. If the value is zero, the exam is crystallised
and is not perturbed. Otherwise, the exam is perturbed and the new state move, if
accepted by FastTA, is counted in the exam’s current threshold bin. As presented
in Sect. 11.5, the cost degradation produced by FastTA, compared with TA, is not
expressive.
The next two sections describe the application of FastTA to the Toronto bench-
mark set (uncapacitated problem) and to the ITC 2007 benchmark set (capacitated
problem).
11.4.3 Application to the Toronto Benchmark Set
This section describes the application of FastTA to the Toronto benchmark set.
11.4.3.1 Solution Representation and Fitness Function
The adopted solution representation for the Toronto benchmark set is illustrated in
Fig. 11.3.
Each solution is represented by an array of time slots, where each time slot
contains an array of the scheduled exams in that time slot. For example, in Fig. 11.3
the exams scheduled in time slot t1 are e1 , e9 , and e23 , whereas the exams scheduled
in time slot t2 are e2 , e8 , e14 , e16 , and e17 . The exams allocated to a given period
must satisfy the specified hard constraint.
The cost function used is given by Eq. (11.1).
11.4.3.2 Construction of Initial Feasible Timetables
While not explicitly stated in the original paper by Carter et al. (1996), it was
discovered experimentally by many researchers that the Toronto set’s instances have
at least one feasible solution.
The initial solution generation for the Toronto problem uses the saturation degree
(SD) graph colouring heuristic (Brélaz, 1979). The exams to be scheduled are
maintained in a priority queue, which is sorted in non-decreasing order of the
number of available periods. In the beginning, all exams have the same priority. The
heuristic successively selects a candidate exam randomly and assigns it to a random
time slot, until the queue is empty or the top priority exam could not be scheduled in
the chosen time slot. In the latter case, the process stops and is restarted generating
a new random priority queue with all exams. The process is successful if all exams
could be scheduled in a given cycle.
According to conducted tests, the algorithm is able to find a feasible solution
within less than five cycles (on average). As a consequence, no further investigation
into a more complex construction procedure, which could construct viable solutions,
was performed.
11.4.3.3 Neighbourhood Operator
In solving the Toronto timetabling problem instances, the Kempe chain neighbour-
hood (Thompson & Dowsland, 1998; Demeester et al., 2012) was used. Using the
Fig. 11.3 Solution

representation for the Toronto
t1 t2 · · · tT
benchmark set (uncapacitated e1 e2 e6
problem) e9 e8 e13
e23 e14 · · · e15
e16 e20
e17
342 N. Leite et al.
Kempe chain neighbourhood, the move operator always produces feasible solutions
for the Toronto dataset.
11.4.4 Application to the ITC 2007 Benchmark Set

11.4.4.1 Solution Representation and Fitness Function
Since the ITC benchmark set is capacitated, rooms also have to be considered in the
solution. The generic representation of a solution in this case is given in Fig. 11.4.
For example, exam e4 is scheduled in time slot t1 and room r1 , and exams e1 and e6
are scheduled in time slot t2 and room r2 .
The cost function used is given by Eq. (11.6).
11.4.4.2 Construction of Initial Feasible Timetables
In the ITC 2007 problem, the SD heuristic is also used to generate the initial
solution. For complexity reasons, each exam’s number of available time slots is
calculated taking into account only the No Conflicts hard constraint (McCollum
et al., 2012) (see Sect. 11.3.2), as an alternative to include all of the hard constraints.
The remaining hard constraints are verified when the exam is allocated.
When the SD heuristic starts, instead of generating a priority queue with
exams placed in a random fashion, the exams are initially sorted in the priority
queue according to their number of After hard constraints. In this way, the more
constrained exams are scheduled first (Leite et al., 2018).
The SD construction algorithm executes two steps:
Step 1 An exam is extracted from the priority queue, and a random period and
room, chosen from the list of available ones, are selected. If all hard requirements
are satisfied, the exam is scheduled in that time slot and room, and the exams’
priorities (number of available time slots) in the queue are updated, considering
only the No Conflicts hard constraint. Continue in this fashion until all exams
Fig. 11.4 Solution t1 t2 ··· tT

representation for the ITC
2007 benchmark set r1 e4 e9 e12 , e22
(capacitated problem) r2 e1 , e6 e18 , e10
· · · ·
· · · ··· ·
· · · ·
rR-1 · · ·
rR e20 e2
are scheduled or an exam fails to be scheduled due to the absence of free

periods/rooms or hard constraints violations. In the latter case, go to Step 2.
Step 2 If there are no free periods or rooms for the selected exam, random values
are chosen for these. When the algorithm reaches Step 2, we know that the
exam cannot be scheduled in a feasible way due to conflicts with other exams
placed in the timetable. To solve this, the conflicting exams are removed from the
timetable and the candidate exam is scheduled. The removed conflicting exams
are reinserted in the priority queue, and priorities are recomputed as in Step 1.
The algorithm then goes to Step 1 again.
In order to avoid cycles, due to repetitive assignments of variables (exams) to the
same values (period and room), the conflict-based statistics (CBS) (Müller, 2009)
structure is employed. For more details, refer to Leite et al. (2018).
11.4.4.3 Neighbourhood Operators
The neighbourhood operators used in this chapter are the same as in Leite et al.
(2018). These are:
Room move A randomly chosen exam is scheduled in another room (randomly
chosen) in the same time slot.
Slot-Room move A randomly chosen exam is scheduled in other, randomly
selected, room and time slot.
These operators may produce infeasible solutions (Leite et al., 2018). In this case,
the move is ignored by the FastTA algorithm.
11.5 Experiments
In this section, the experimental evaluation of the FastTA, on the Toronto and ITC
2007 sets, is reported. In Sect. 11.5.1, the TA variants tested in the simulations are
described.
Section 11.5.2 describes the algorithm parameter settings specified for the used
benchmark sets.
Section 11.5.3 reports the comparison results between FastTA and TA on the
Toronto and ITC 2007 benchmark sets.
Sections 11.5.4 and 11.5.5 provide a comparison of FastTA with the state-of-the-
art approaches for the Toronto and ITC 2007 benchmark sets, respectively.
The developed algorithms were programmed in the C++ language using the
ParadisEO framework (Talbi, 2009). The hardware and software specifications are:
Intel Core i7-2630QM, CPU @ 2.00 GHz × 8, with 8 GB RAM; OS: Ubuntu 18.04
LTS, 64 bit; Compiler used: GCC v. 7.3.0.
For the Toronto benchmark set, the algorithm computation time was nearly
15 hours, at most, given the chosen parameters (Sect. 11.5.2). For the ITC 2007
344 N. Leite et al.
benchmark set, the computation time was set to 276 seconds, as measured by the
provided benchmarking tool from the ITC 2007 site (International Timetabling
Competition, 2007).
All obtained solutions were validated using Qu et al.’s validator tool (Qu et al.,
2009) (for the Toronto benchmark set) and the ITC 2007s online validator tool (for
the ITC 2007 benchmark set).
All statistical tests were carried out with a 95% confidence level. For each tested
configuration, ten runs of the algorithm were performed. The statistical significance
was assessed using the Friedman test (García & Herrera, 2008). The Java tool
in García and Herrera (2008) was used to produce the results of all statistical tests.
For each examination timetabling benchmark set, the source code, the resulting
solution files for each instance/run, and the produced statistics, are publicly available
in the following Git repositories:
Toronto: https://github.com/nunocsleite/FastTA-ETP-Toronto
ITC 2007: https://github.com/nunocsleite/FastTA-ETP-ITC2007
11.5.1 Compared TA Variants
In the conducted experiments, three TA algorithm variants, named TA, FastTA100,

and FastTA80, were tested. These are described as follows:
1. TA—This is the original TA metaheuristic, as described in Algorithm 11.1.
2. FastTA100—This comprises the basic FastTA algorithm, as described in Algo-
rithm 11.2. In the FastTA100 variant, exam perturbations for the current threshold
are recorded in the respective threshold bin, i.e., if a selected exam is perturbed,
and the new state is accepted, the number of accepted moves for that exam is
incremented in the corresponding bin. An exam is only allowed to move if that
exam is not crystallised, i.e., the number of accepted moves in the previous
threshold bin is non-zero. If the exam is already crystallised, no perturbation
is made to that exam until the end of the optimisation process. This leads to a
reduction in the number of evaluations taken.
3. FastTA80—This variant is similar to the FastTA100, differing in the way how
exams are fixed. An exam becomes crystallised if two conditions are met: the
number of accepted moves in the previous threshold bin is zero and the exam is in
the set comprising 80% of the exams with highest degree, that is, it is considered
a “difficult” or largest degree exam (Leite et al., 2019). Perturbations made to
large-degree exams are often not accepted when reaching low thresholds, as these
perturbations imply a high variation in the cost. On the other side, small-degree
exam perturbations yield small fluctuations in the objective function value. The
idea of this FastTA variant is to fix large-degree exams as they stop accepting
perturbations, and allow the smallest degree exams to move freely, despite having
some threshold bins with zero counts.
These TA variants are compared with each other in Sect. 11.5.3 on the reference
benchmark sets of Toronto and ITC 2007.
11.5.2 Parameter Settings
This section describes the algorithm parameter settings specified for the Toronto and
ITC 2007 benchmark sets.
For the experiments involving the Toronto benchmark set, the cooling schedules
used for the TA and FastTA are specified in Table 11.5. Two different cooling
schedules were used: a light cooling schedule and an intensive one. The light
cooling schedule was used in the experiments presented in Sect. 11.5.3, whereas the
intensive cooling schedule was used to generate the final timetables for the Toronto
dataset used in the comparison with the state-of-the-art approaches (Sect. 11.5.4).
Two parameter configurations were used in the conducted experiments for the
ITC 2007. In the first configuration, the parameters were manually tuned for each
ITC 2007 instance. This configuration was only used in the FastTA variants’
comparison (Sect. 11.5.3). In the second configuration, a fixed set of parameters was
used for all instances, excepting the cooling rate, which was computed automatically
by the optimisation algorithm. The algorithm used to compute the cooling rate in an
automatic fashion is described in Algorithm 3 of Leite et al. (2019). The second
parameter configuration was used to compare the FastTA100 with the state-of-the-
art approaches.
Table 11.6 summarises the cooling schedule parameters used in the first configu-
ration (parameters manually tuned) for the ITC 2007 benchmark set. The parameter
values were chosen using as reference the FastTA100 algorithm. Specifically, the
cooling schedule values were adapted empirically for each dataset instance while
guaranteeing that the FastTA100 computation time was within the ITC 2007 time
limit constraint.
For the second parameter configuration, the following parameter set was used:
QMax = 350 (initial threshold), QMin = 1 × 10−6 (final threshold), k = 5 (number
of iterations at a fixed threshold), and r computed automatically using the algorithm
described in Leite et al. (2019).
The neighbourhood operators, the Room and Slot-Room moves, were selected
with equal probability. This value was found after conducting some tuning experi-
ments.
Table 11.5 Threshold Type QMax r k QMin # Evaluations

acceptance’s cooling
schedules used in the Toronto Light 0.1 0.001 5 2 × 10−5 42, 590
benchmark set Intensive 0.1 5 × 10−7 5 2 × 10−5 85, 171, 935
Threshold acceptance’s cooling schedule parameters: Qmax —
initial threshold value, r—cooling rate, k—number of iterations
at a fixed threshold, and Qmin —final threshold value
346 N. Leite et al.
Table 11.6 Threshold acceptance’s cooling schedules used to solve the different ITC 2007
instances
Dataset QMax r k QMin # Evaluations
Exam_1 100 4.00 × 10−6 5 1.00 × 10−6 23, 025, 856
Exam_2 100 7.00 × 10−6 5 1.00 × 10−6 13, 157, 631
Exam_3 100 9.00 × 10−6 5 1.00 × 10−6 10, 233, 716
Exam_4 350 4.50 × 10−6 6 1.00 × 10−6 26, 231, 263
Exam_5 300 1.10 × 10−5 5 1.00 × 10−6 8, 872, 411
Exam_6 300 6.00 × 10−6 5 1.00 × 10−6 16, 266, 081
Exam_7 300 8.00 × 10−6 5 2.00 × 10−6 11, 766, 346
Exam_8 300 3.00 × 10−6 5 3.00 × 10−4 23, 025, 856
Exam_9 300 1.50 × 10−6 5 3.00 × 10−4 46, 051, 706
Exam_10 300 1.60 × 10−6 5 3.00 × 10−4 43, 173, 476
Exam_11 300 1.20 × 10−5 5 3.00 × 10−4 5, 756, 466
Exam_12 300 1.80 × 10−6 5 3.00 × 10−4 38, 376, 421
Sum of # evaluations 265, 937, 229
Threshold acceptance’s cooling schedule parameters: Qmax —initial threshold value, r—cooling
rate, k—number of iterations at a fixed threshold, and Qmin —final threshold value
11.5.3 Comparison of FastTA and TA on the Toronto and ITC

2007 Sets
In this section, we provide a comparison between two FastTA algorithms,

FastTA100 and FastTA80, and the standard TA on the Toronto and ITC 2007
benchmark sets. In these experiments, the light cooling schedule was used for the
Toronto benchmark set (Table 11.5) and the cooling schedules of Table 11.6 for
the ITC 2007 set. An analysis and discussion of results are given at the end of this
section.
Tables 11.7 and 11.8 show the solution costs and the corresponding execution
times, respectively, for the FastTA100, FastTA80, and TA algorithms, for the
Toronto benchmark set. For each algorithm, the minimum and average values, and
the standard deviation of ten runs are shown. Tables 11.9 and 11.10 present the same
information but for the ITC 2007 benchmark set.
11.5.3.1 Discussion
Analysing the results in terms of solution cost (Tables 11.7 and 11.9), it can be
observed that, for the Toronto set (Table 11.7), the results of FastTA variants are in
general close to the ones attained by TA, but inferior to TA. However, the FastTA
variants use a significantly lower number of evaluations (column “ENP”) compared
to TA. For the ITC 2007 (Table 11.9), the FastTA100 has results that compete with
the TA, obtaining the best average solution cost in four of twelve instances, while
Table 11.7 Solution costs of FastTA100, FastTA80, and TA on the Toronto benchmark set
FastTA100 FastTA80 TA
Dataset ENP fmin favg σ ENP fmin favg σ fmin favg σ
car92 57 5.01 5.06 0.05 52 4.92 4.99 0.05 4.57 4.65 0.07
car91 60 6.22 6.33 0.05 55 6.10 6.24 0.09 5.65 5.84 0.13
ear83 74 36.94 39.21 1.28 66 37.18 38.95 1.65 35.25 37.61 1.00
hec92 73 10.55 10.97 0.25 62 10.36 11.01 0.29 10.27 10.91 0.27
kfu93 50 14.61 15.24 0.43 47 14.47 15.18 0.44 13.79 14.40 0.37
lse91 54 11.61 12.00 0.34 50 11.47 11.95 0.26 10.63 11.45 0.38
pur93 71 8.01 8.10 0.08 60 7.83 7.96 0.10 6.38 6.46 0.06
rye92 60 9.93 10.17 0.17 53 9.86 10.04 0.16 8.95 9.25 0.21
sta83 24 157.06 157.16 0.11 23 157.05 157.14 0.11 157.03 157.16 0.11
tre92 60 8.94 9.19 0.17 55 8.99 9.22 0.18 8.70 8.89 0.16
uta92 53 4.12 4.20 0.05 49 4.04 4.15 0.06 3.69 3.78 0.07
ute92 58 25.15 25.73 0.51 51 25.28 25.66 0.32 25.01 25.41 0.45
yor83 81 39.68 40.72 0.83 67 38.65 39.95 0.93 37.46 39.08 0.78
Avg. 60 53
The column “ENP” presents the percentage of evaluations not performed, on average, by the
FastTA variants in comparison with the standard TA. A higher value of ENP means a faster
algorithm. fmin and favg represent the minimum and average values of solution cost, respectively.
Solution costs are calculated by function fc , defined in Eq. (11.1). The best results are shown in
bold
Table 11.8 Execution times (in seconds) of FastTA100, FastTA80, and TA on the Toronto
benchmark set
Dataset tmin tavg σ tmin tavg σ tmin tavg σ
car92 1 1.80 0.42 2 2.10 0.32 6 6.80 0.42
car91 2 2.50 0.53 2 2.70 0.48 9 9.00 0.00
ear83 0 0.30 0.48 0 0.50 0.53 1 1.60 0.52
hec92 0 0.20 0.42 0 0.30 0.48 0 0.60 0.52
kfu93 0 0.60 0.52 0 0.50 0.53 2 2.00 0.00
lse91 0 0.80 0.42 0 0.80 0.42 1 1.90 0.32
pur93 6 6.50 0.53 7 7.00 0.00 32 32.50 0.71
rye92 1 1.00 0.00 1 1.00 0.00 3 3.60 0.52
sta83 0 0.30 0.48 0 0.40 0.52 0 0.50 0.53
tre92 0 0.70 0.48 0 0.70 0.48 2 2.20 0.42
uta92 2 2.50 0.53 2 2.60 0.52 7 8.10 0.57
ute92 0 0.50 0.53 0 0.50 0.53 1 1.10 0.32
yor83 0 0.40 0.52 0 0.30 0.48 2 2.00 0.00
The best results are shown in bold
348
Table 11.9 Solution costs of FastTA100, FastTA80, and TA on the ITC 2007 benchmark set
Dataset ENP fmin favg σ ENP fmin favg σ fmin favg σ
Exam_1 48 4740 4922.3 96.3 43 4758 4930.2 101.2 4751 4845.0 103.9
Exam_2 1 395 407.5 7.5 1 390 404.5 7.6 395 402.5 6.3
Exam_3 8 9701 10,145.0 231.9 8 9514 9906.1 265.2 9465 9960.9 240.7
Exam_4 68 11, 938 12,432.7 385.8 60 12, 165 12,841.7 394.5 11, 802 12,557.9 627.7
Exam_5 7 3100 3498.9 183.1 6 3081 3393.5 177.1 2953 3332.4 183.7
Exam_6 17 25, 985 26,129.0 120.0 16 25, 940 26,149.5 146.0 25, 790 26,037.0 208.2
Exam_7 6 4269 4512.4 132.0 6 4269 4462.3 132.0 4251 4440.6 133.6
Exam_8 21 7284 7487.5 134.8 21 7383 7533.0 147.1 7303 7526.1 141.0
Exam_9 25 944 996.7 44.9 23 961 1021.6 44.5 970 1008.5 27.2
Exam_10 13 13, 606 13,768.1 108.1 12 13, 456 13,613.7 169.1 13, 573 13,685.5 80.4
Exam_11 14 30, 590 31,672.6 781.1 14 30, 596 32,757.5 1387.9 30, 994 32,543.3 1252.2
Exam_12 23 5201 5249.2 33.4 24 5118 5241.9 66.9 5149 5220.6 51.7
Avg. 21 20
The column “ENP” presents the percentage of evaluations not performed, on average, by the FastTA variants in comparison with the standard TA. A higher
value of ENP means a faster algorithm. fmin and favg represent the minimum and average values of solution cost, respectively. Solution costs are calculated
by the objective function defined in Eq. (11.6). The best results are shown in bold
N. Leite et al.
Table 11.10 Execution times (in seconds) of FastTA100, FastTA80, and TA algorithms on the
ITC 2007 benchmark set
Dataset tmin tavg σ tmin tavg σ tmin tavg σ
Exam_1 252 267.2 9.16 259 278.4 8.49 492 502.6 7.06
Exam_2 260 266.2 4.37 267 273.3 3.74 270 274.5 2.17
Exam_3 259 266.7 4.47 268 273.3 5.91 289 300.8 8.97
Exam_4 247 256.9 6.33 286 296.2 7.32 730 746.7 8.77
Exam_5 226 242.4 16.68 233 247.0 7.47 267 286.3 11.19
Exam_6 207 238.5 20.32 195 219.7 17.16 261 272.0 5.03
Exam_7 221 243.8 19.89 247 255.0 4.88 269 272.4 2.27
Exam_8 229 250.2 15.87 228 233.1 3.51 294 296.5 1.43
Exam_9 230 244.2 11.14 206 229.4 12.17 297 305.0 4.83
Exam_10 242 249.5 9.44 229 236.4 4.81 257 263.8 3.85
Exam_11 219 226.2 6.86 227 232.5 4.7 278 282.3 4.35
Exam_12 210 228.4 11.46 199 214.0 8.68 275 280.4 3.89
The best results are shown in bold
requiring less evaluations. This difference is explained in part by the light cooling
schedule chosen for the Toronto benchmark set in these experiments, which involve
a lower number of evaluations in contrast with the cooling schedules used for the
ITC 2007 case.
In terms of the performed number of evaluations (column “ENP”), the FastTA100
and FastTA80 execute, respectively, 60% and 53% less evaluations, on average,
compared to the TA algorithm, for the Toronto benchmark set. The same indicators
for the ITC 2007 give 21% and 20% less evaluations, respectively. Here, we can see
a great difference in the number of evaluations not performed for the Toronto set.
This fluctuation is explained, as mentioned before, by the use of a light cooling
schedule. Moreover, in Sect. 11.5.4, Table 11.11, an intensive cooling schedule
is used in the comparison of FastTA100 with the state-of-the-art approaches, and
the average number of evaluations not performed is only 38% for the Toronto
benchmark set. This shows that the cooling schedule influences the algorithm’s
behaviour.
In the experiments, the sum of the mean values of the number of evaluations
obtained for each instance is 224,160.6 and 259,331.7, respectively, for FastTA100
and FastTA80, in the Toronto case. For the TA, for the Toronto set, the sum of the
number of evaluations executed for each instance is 553,670.
For the ITC 2007 case, the sum of the mean values of the number of evaluations
is 200,004,835.7 and 204,849,734.1, for FastTA100 and FastTA80, respectively.
For the TA, the sum of the number of evaluations executed for each instance is
265,937,229. These values show that the FastTA100 is the technique that performs
the lowest number of evaluations on average.
For the FastTA100 applied to Toronto, the number of neighbours not evaluated
is equal or greater than 50% on all instances except one, while not degrading the
350 N. Leite et al.
Table 11.11 Fitness values and computation times (in hours) of FastTA100 algorithm using the
intensive cooling schedule on the Toronto benchmark set
Fitness value Computation time
Dataset ENP fmin fmed fmax favg σ tmin tmed tmax tavg σ
car92 27 3.71 3.75 3.80 3.75 0.03 2.5 2.6 2.6 2.6 0.03
car91 27 4.38 4.43 4.53 4.44 0.05 2.6 2.7 2.7 2.7 0.02
ear83 62 32.72 32.90 34.13 33.09 0.53 0.4 0.4 0.4 0.4 0.02
hec92 67 10.05 10.20 10.30 10.18 0.08 0.1 0.1 0.1 0.1 0.01
kfu93 31 12.83 12.92 13.07 12.93 0.07 0.5 0.5 0.6 0.5 0.01
lse91 36 9.81 9.96 10.15 9.98 0.13 0.6 0.6 0.6 0.6 0.00
pur93 10 4.11 4.17 4.21 4.17 0.03 10.9 11.0 14.7 11.3 1.20
rye92 32 7.94 8.02 8.08 8.01 0.05 1.2 1.2 1.3 1.2 0.02
sta83 22 157.03 157.06 157.06 157.06 0.01 0.2 0.2 0.2 0.2 0.00
tre92 41 7.63 7.82 7.96 7.78 0.12 0.6 0.6 0.6 0.6 0.01
uta92 21 3.02 3.07 3.11 3.07 0.02 2.3 2.3 2.3 2.3 0.01
ute92 48 24.85 24.85 25.39 24.91 0.17 0.2 0.3 0.3 0.3 0.01
yor83 70 34.68 35.13 35.83 35.23 0.38 0.4 0.4 0.4 0.4 0.01
Avg. 38
FastTA variant in comparison with the standard TA. A higher value of ENP means a faster
algorithm. fmin , fmed , fmax , and favg represent the minimum, median, maximum, and average
values of solution cost, respectively. Solution costs are calculated by function fc , defined in
Eq. (11.1)
solution cost value significantly, which is a significant number. On the ITC 2007,
the FastTA100 is able to attain between 13 and 68% of neighbours not evaluated on
more than half of the datasets.
From Tables 11.8 and 11.10 results, we can conclude that the presented FastTA
approaches are globally faster than the TA approach.
Using as criteria the average percentage of the number of evaluations not
performed, and also the execution time, the FastTA100 is the best method. However,
in terms of solution cost, the TA is better than FastTA.
In the next two sections, the FastTA100 is compared with the state-of-the-art
approaches for the Toronto and ITC 2007 benchmark sets, respectively.
11.5.4 Comparison with State-of-the-art Approaches—Toronto

Dataset
In this section, we perform a comparison between the FastTA100 approach and

other state-of-the-art methodologies, for the Toronto benchmark set. The tested
FastTA100 variant uses the intensive cooling schedule as specified in Table 11.5.
Table 11.12 Average objective function value obtained by FastTA and by selection of the best
algorithms from the literature
Dataset Ele07 Bur08 Pil10 Dem12 Lei14 Fon15 Alz15 Lei18 FastTA
car92 4.40 3.92 4.28 3.86 3.77 4.27 4.09 3.72 3.75
car91 5.20 4.68 5.05 4.64 4.45 4.85 4.52 4.39 4.44
ear83 38.30 32.91 36.49 32.69 32.69 34.48 33.66 32.61 33.09
hec92 11.40 10.22 11.69 10.06 10.06 10.61 10.90 10.05 10.18
kfu93 14.90 13.02 14.42 13.24 13.00 13.76 13.46 12.83 12.93
lse91 11.70 10.14 10.95 10.21 9.93 10.39 10.82 9.81 9.98
pur93 4.60 4.71 4.70 5.75 4.17 — — 4.18 4.17
rye92 10.00 8.06 9.41 8.20 8.06 — — 7.93 8.01
sta83 157.50 157.05 158.07 157.05 157.03 157.37 157.16 157.03 157.06
tre92 8.70 7.89 8.45 7.79 7.80 8.04 8.09 7.70 7.78
uta92 3.50 3.26 3.39 3.17 3.15 3.31 3.23 3.04 3.07
ute92 27.00 24.82 27.45 24.88 24.81 26.04 25.33 24.83 24.91
yor83 40.40 35.16 39.74 34.83 34.73 36.83 35.69 34.63 35.23
The best results are shown in bold. “—” indicates that the corresponding instance was not
tested or no feasible solution was obtained. The compared approaches are: Ele07 Eley (2006)—
Ant algorithms, Bur08 Burke and Bykov (2008)—Late Acceptance Hill-Climbing, Pil10 Pillay
and Banzhaf (2010)—Genetic Algorithm, Dem12 Demeester et al. (2012)—Hyper-heuristic
approach, Lei14 Leite et al. (2016)—hybrid evolutionary algorithm, Fon15 Fong et al. (2015)—
Hybrid Swarm-Based Approach, Alz15 Alzaqebah and Abdullah (2015)—Hybrid bee colony
optimisation, Lei18 Leite et al. (2018)—Hybrid evolutionary algorithm, FastTA—fast threshold
Table 11.11 presents the solution cost values and computation times (in hours) for
the FastTA100 algorithm with the intensive cooling schedule, tested on the Toronto
benchmark set. The minimum, median, maximum, average, and standard deviation
values are shown for the solution cost and computation time.
In Tables 11.12, 11.13, 11.14, 11.15, the compared approaches are identified
by an acronym of the first author’s name and publication’s year. The compared
authors are: Ele07—Eley (2006), Bur08—Burke and Bykov (2008), Pil10—Pillay
and Banzhaf (2010), Dem12—Demeester et al. (2012), Lei14—Leite et al. (2016),
Fon15—Fong et al. (2015), Alz15—Alzaqebah and Abdullah (2015), and Lei18—
Leite et al. (2018).
In their papers, few of the compared authors published their source code, and few
published the solution files of the executed runs, preventing that a study based on
distributions could be done. Thus, comparisons are done using the best and average
of solution cost values.
Table 11.12 presents a comparison of the average results obtained by the state-
of-the-art approaches and the FastTA.
352 N. Leite et al.
Table 11.13 Average Algorithm Rank

ranking of Friedman’s test for
the compared methods, Lei18 1.35
considering the complete Lei14 2.50
Toronto dataset (13 FastTA 3.12
instances). The lower the Dem12 4.04
rank, the better the Bur08 4.31
algorithm’s performance
Pil10 6.15
Ele07 6.54
The column Rank
denotes the average
rank (dimensionless
value) computed
according to
Friedman’s test
Table 11.14 Adjusted Lei18 vs. Unadjusted p pH olm

p-values (Friedman)
produced by running Holm’s Ele07 8.90.10−10 5.34.10−9
test (Lei18 is the control Pil10 1.39.10−8 6.97.10−8
algorithm) Bur08 0.000,474 0.001,90
Dem12 0.001,49 0.004,46
FastTA 0.0368 0.0736
Lei14 0.173 0.173
Values marked in boldface represent sta-
tistically significant values
Table 11.15 Adjusted i Hypothesis Unadjusted p pH olm

p-values (Friedman) of
multiple algorithm 1 Ele07 vs. Lei18 8:90.10−10 1.87.10−8
comparison obtained by 2 Pil10 vs. Lei18 1.39.10−8 2.79.10−7
Holm’s procedure 3 Ele07 vs. Lei14 1.88.10−6 3.57.10−5
4 Pil10 vs. Lei14 1.62.10−5 0.000,291
5 Ele07 vs. FastTA 5.35.10−5 0.000,909
6 Pil10 vs. FastTA 0.000,336 0.005,37
7 Bur08 vs. Lei18 0.000,474 0.007,11
8 Dem12 vs. Lei18 0.001,49 0.0208
9 Ele07 vs. Dem12 0.003,17 0.0412
Only statistically significant values are shown
(marked in boldface)
11.5.4.1 Statistical Analysis
In this section, a statistical analysis of the analysed algorithms for the Toronto
benchmark set is presented. The statistical methods and definitions presented in this
section were also used in the experiments made in Sect. 11.5.5.1, involving the ITC
2007 benchmark set.
We have performed 1 × N multiple comparisons tests, where a control method
is highlighted (the best performing algorithm) through the application of the test,
and have also performed N × N comparisons, considering all existing pairs of

algorithms (Derrac et al., 2011). All hypotheses of equality can be tested by the
application of a set of post hoc procedures. In the multiple comparison tests (both
1 × N and N × N comparisons), the Friedman’s test is used. For the post hoc tests,
the Holm’s procedure was used.
The null hypothesis H0 and the alternative hypothesis H1 are defined as follows.
The null hypothesis is a statement of no effect or no difference (in our case,
significant differences between algorithms), whereas the alternative hypothesis
represents the presence of an effect or a difference (Derrac et al., 2011).
Once Friedman’s test rejects the null hypothesis, we can proceed with a post hoc
test in order to find the concrete pairwise comparisons that produce differences.
We now present a statistical analysis of the obtained average results (Table 11.12)
for the Toronto benchmark set.
Table 11.13 summarises the ranks obtained by the Friedman test. The p-value
computed by the Friedman test is 9.80 × 10−11 , which is below the significance
interval of 95% (α = 0.05). This confirms that there exists a significant difference
in the observed results, and asserts that Lei18 has the best performance.
Table 11.14 presents adjusted p-values of the post hoc Holm’s test when Lei18
is the control method. Holm’s test asserts that Lei18 improves the results of all
algorithms with α = 0.05, except for Lei14 and FastTA.
Table 11.15 presents the adjusted p-values for the same studied scenario,
obtained using the Holm’s procedure, and showing pairwise comparisons. Only
cases having significant differences are shown.
11.5.4.2 Discussion
Analysing the results of Table 11.12, we can state that FastTA obtains the best
average result on one Toronto instance, reaching results that are near to the results
of the best state-of-the-art algorithms.
The statistical test results substantiate our conclusions: FastTA ranks third among
seven algorithms (Table 11.13), where Lei18 has the best performance. Post hoc
tests assert that Lei18 improves the results of all algorithms, except for Lei14 and
FastTA.
Finally, analysing the results of the multiple algorithm comparison (N ×N) from
Table 11.15, we can confirm the same conclusions of Table 11.14 and also that Lei14
and FastTA are better than Ele07 and Pil10 and that Dem12 is better than Ele07.
From these tables, we can say that Lei18 and Lei14 are not better, with statistical
significance, than FastTA.
The computation times of the techniques presented in Table 11.12 can be
consulted in (see Leite et al., 2018, Appendix B). Analysing the competitors’
running times with the FastTA computation times (Table 11.11), we can see that
FastTA is much faster than Lei14 and Lei18 requiring only 11.3 h for pur93, and
between 2.3 and 2.7 h for the other medium-size instances, namely car92, car91,
and uta92, on average. For these instances, Lei14 requires between 24 and 48 h, and
Lei18 requires 48 h for all instances, on average.
354 N. Leite et al.
11.5.5 Comparison with State-of-the-Art Approaches—ITC

2007 Dataset
In this section, we perform a comparison between the FastTA100 approach and other
state-of-the-art methodologies, for the ITC 2007 case. The tested FastTA100 variant
uses a cooling rate computed automatically as mentioned in Sect. 11.5.2.
Table 11.16 presents the solution cost values and computation times (in seconds)
for the FastTA100 algorithm with rate computed automatically, tested on the ITC
2007 benchmark set. The minimum, median, maximum, average, and standard
deviation values are shown for the solution cost and computation time.
In Tables 11.17, 11.18, 11.19, 11.20, 11.21, the compared approaches are identi-
fied by an acronym of the first author’s name and publication’s year. The compared
authors are: Gog08—Gogos et al. (2008), Ats08—Atsuta et al. (2008) McCollum
et al. (2010), Sme08—De Smet (2008) McCollum et al. (2010), Pil08—Pillay
(2008) McCollum et al. (2010), Mul09—Müller (2009), Col09—McCollum et al.
(2009), Dem12—Demeester et al. (2012), Gog12—Gogos et al. (2012), Byk13—
Bykov and Petrovic (2013), Ham13—Hamilton-Bryce et al. (2013), Alz14—
Alzaqebah and Abdullah (2014), Alz15—Alzaqebah and Abdullah (2015), Bat17—
Battistutta et al. (2017), Lei18—Leite et al. (2018), Lei19—Leite et al. (2019).
In their papers, few of the compared authors published the algorithm source code
and solution files of the executed runs, preventing that a study based on distributions
Table 11.16 Fitness values and computation times (in seconds) of FastTA100 (rate computed
automatically) on the ITC 2007 benchmark set
Fitness value Computation time
Dataset ENP fmin fmed fmax favg σ tmin tmed tmax tavg σ
Exam_1 44 5089 5333.0 5434 5313.7 98.13 115 128.0 143 129.0 8.22
Exam_2 1 395 410.0 432 409.7 10.58 252 264.0 269 262.4 5.40
Exam_3 8 9711 10,087.5 10,766 10,091.5 311.62 214 227.0 232 225.1 5.65
Exam_4 69 12, 235 12,992.0 13,735 13,011.0 536.97 61 67.0 72 66.3 3.16
Exam_5 7 3184 3455.5 3984 3470.3 212.77 167 172.0 188 173.0 6.00
Exam_6 16 25, 975 26,100.0 26,355 26,135.5 117.72 189 219.0 237 215.5 14.92
Exam_7 6 4369 4542.5 4635 4515.1 100.56 211 225.0 239 224.1 7.82
Exam_8 27 7488 7662.0 7813 7671.2 105.44 139 164.5 174 163.6 9.52
Exam_9 30 985 1023.5 1046 1024.2 20.19 149 159.0 172 160.0 7.60
Exam_10 15 13, 487 13,707.5 13,960 13,703.7 132.66 205 219.0 229 218.5 8.30
Exam_11 15 32, 189 33,795.5 35,745 33,938.1 1243.55 175 185.0 191 184.6 5.38
Exam_12 27 5148 5232.5 5344 5233.0 60.80 150 161.0 180 162.2 8.24
Avg. 22
FastTA variant in comparison with the standard TA. A higher value of ENP means a faster
algorithm. fmin , fmed , fmax , and favg represent the minimum, median, maximum, and average
values of solution cost, respectively. Solution costs are calculated by the objective function defined
in Eq. (11.6)
Table 11.17 Minimum solution cost obtained by the ITC 2007 finalists’ approaches and the
FastTA
Dataset Mul09 Gog08 Ats08 Sme08 Pil08 FastTA
Exam_1 4370 5905 8006 6670 12, 035 5089
Exam_2 400 1008 3470 623 2886 395
Exam_3 10, 049 13,771 17,669 — 15, 917 9711
Exam_4 18, 141 18,674 22,559 — 23, 582 12, 235
Exam_5 2988 4139 4638 3847 6860 3184
Exam_6 26, 585 27,640 29,155 27,815 32, 250 25, 975
Exam_7 4213 6572 10,473 5420 17, 666 4369
Exam_8 7742 10,521 14,317 — 15, 592 7488
Exam_9 1030 1159 1737 1288 2055 985
Exam_10 16, 682 — 15,085 14,778 17, 724 13, 487
Exam_11 34, 129 43,888 — — 40, 535 32, 189
Exam_12 5535 — 5264 — 6310 5148
The best solutions are indicated in bold. “—” indicates that the corresponding dataset is not tested
or a feasible solution was not obtained. The compared approaches are: Mul09 Müller (2009)—
Hybrid local search, Gog08 Gogos et al. (2008)—GRASP metaheuristic, Ats08—Atsuta et al.
(2008) McCollum et al. (2010)—Constraint satisfaction problem solver adopting a hybridisation
of local search, Sme08—De Smet (2008) McCollum et al. (2010)—Tabu search metaheuristic,
Pil08—Pillay (2008) McCollum et al. (2010)—Approach based on cell biology, FastTA—fast
threshold acceptance algorithm
could be done. Thus, comparisons are done using the best and average of solution
cost values.
Table 11.17 presents a comparison between the ITC 2007s five finalists and the
FastTA, considering the solutions’ minimum cost. In Table 11.18, the average results
obtained by the state-of-the-art approaches and the FastTA are compared.
11.5.5.1 Statistical Analysis
We now present a statistical analysis of the obtained average results (Table 11.18)
for the complete ITC 2007 benchmark set (12 instances).
Table 11.19 summarises the rank obtained by the Friedman test (only algorithms
that have values for all instances are considered). The p-value computed by the
Friedman test, 6.18 × 10−8 , is below the significance interval of 95% (α = 0.05),
asserting that there exists a significant difference in the compared results. Byk13 has
the best performance.
Table 11.20 presents the post hoc Holm’s test adjusted p-values, where Byk13
is the control method. Holm’s test asserts that Byk13 improves the results of all
algorithms except for Bat17 and Lei19, with α = 0.05.
Table 11.21 presents the adjusted p-values for the same studied scenario,
obtained using the Holm’s procedure, and showing pairwise comparisons. Only
cases having significant differences are shown.
356
Table 11.18 Comparison of the average results obtained by the FastTA algorithm with the state-of-the-art approaches on the ITC 2007 benchmark set
Dataset Col09 Dem12 Gog12 Byk13 Ham13 Alz14 Alz15 Bat17 Lei18 Lei19 FastTA
Exam_1 4799 6330.20 5032 4008 5469 5517.30 5227.81 3926.96 6478.20 5231.60 5313.70
Exam_2 425 612.50 404 404 450 537.90 457.55 407.72 572.90 405.10 409.70
Exam_3 9251 23,580.00 9484 8012 10, 444 10,324.90 10,421.64 8849.46 13,680.50 9940.10 10,091.50
Exam_4 15, 821 — 19,607 13, 312 20, 241 16,589.10 16,108.27 15,617.82 15,493.70 12,992.50 13,011.00
Exam_5 3072 5323.00 3158 2582 3185 3631.90 3443.72 2849.00 4155.60 3490.40 3470.30
Exam_6 25, 935 28,578.13 26,310 25, 448 26, 150 26,275.00 26,247.27 26,081.35 26,873.00 26,008.50 26,135.50
Exam_7 4187 6250.00 4352 3893 4568 4592.40 4415.00 3661.64 5844.40 4534.70 4515.10
Exam_8 7599 9260.90 8098 6944 8081 8328.80 8225.81 7729.46 8942.30 7602.60 7671.20
Exam_9 1071 1255.90 — 949 1061 — — 991.57 1080.30 1017.60 1024.20
Exam_10 14, 552 16,113.33 — 12, 985 15, 294 — — 13,999.56 14,208.70 13,631.70 13,703.70
Exam_11 29, 358 — — 25, 194 44, 820 — — 27,781.50 43,693.56 31,792.20 33,938.10
Exam_12 5699 5829.14 — 5181 5464 — — 5550.20 5249.00 5204.80 5233.00
The best solutions are indicated in bold. “—” indicates that the corresponding dataset was not tested or a feasible solution was not obtained. The compared
approaches are: Col09 McCollum et al. (2009)—Great deluge metaheuristic, Dem12 Demeester et al. (2012)—Hyper-heuristics, Gog12 Gogos et al.
(2012)—GRASP metaheuristic, Byk13 Bykov and Petrovic (2013)—Hill-climbing variant, Ham13 Hamilton-Bryce et al. (2013)—Great deluge metaheuristic,
Alz14 Alzaqebah and Abdullah (2014)—Bee colony optimisation, Alz15 Alzaqebah and Abdullah (2015)—Bee colony optimisation, Bat17 Battistutta et al.
(2017)—simulated annealing, Lei18 Leite et al. (2018)—Cellular memetic algorithm, Lei19 Leite et al. (2019)—simulated annealing, FastTA—fast threshold
N. Leite et al.
Table 11.19 Average Algorithm Rank

ranking of Friedman’s test for
the compared methods, Byk13 1.33
considering the complete Bat17 3.08
dataset (12 instances). The Lei19 3.25
lower the rank, the better the Col09 4.08
algorithm’s performance FastTA 4.08
Ham13 5.92
Lei18 6.25
The column Rank
denotes the average
rank (dimensionless
value) computed
according to
Friedman’s test
Table 11.20 Adjusted Byk13 vs. Unadjusted p pH olm

p-values (Friedman)
produced by running Holm’s Lei18 2.48.10−8 1.49.10−7
test (Byk13 is the control Ham13 2.03.10−7 1.01.10−6
algorithm) FastTA 0.001,82 0.007,28
Col09 0.001,82 0.007,28
Lei19 0.0298 0.0595
Bat17 0.0472 0.0595
Values marked in boldface represent sta-
tistically significant values
Table 11.21 Adjusted i Hypothesis Unadjusted p pH olm

p-values (Friedman) of
multiple algorithm 1 Byk13 vs. Lei18 2.48.10−8 5.20.10−7
comparison obtained by 2 Byk13 vs. Ham13 2.03.10−7 4.05.10−6
Holm’s procedure 3 Bat17 vs. Lei18 3.30.10−4 0.006,27
4 Lei18 vs. Lei19 6.70.10−4 0.0121
5 Ham13 vs. Bat17 0.001,32 0.0224
6 Byk13 vs. FastTA 0.001,82 0.0291
7 Col09 vs. Byk13 0.001,82 0.0291
8 Ham13 vs. Lei19 0.002,50 0.0350
Only statistically significant values are shown
(marked in boldface)
11.5.5.2 Discussion
Analysing the results, we observe that the FastTA is able to compete with the ITC
2007 finalists (Table 11.17), being superior on all instances except three. With
respect to average results (Table 11.18), the FastTA is also able to compete with
the state-of-the-art approaches.
The employed statistical tests on the average results support our observations: the
FastTA is ranked in the fifth position among seven algorithms (Table 11.19), where
Byk13 is considered the best approach; Table 11.20, containing the adjusted p-
358 N. Leite et al.
values obtained by Holm’s test comparing the control algorithm with the remaining
algorithms (1 × N comparison), allows to ascertain that Byk13 is better than all
methods except for Bat17 and Lei19. Finally, analysing the results of the multiple
algorithm comparison (N × N) from Table 11.21, we can confirm the same
conclusions of Table 11.20 and also ascertain that both Bat17 and Lei19 are better
than Ham13 and Lei18. From these tables, we can say that only Byk13 is better than
FastTA. The obtained results are statistically significant.
11.6 Conclusions
In the present work, a two-phase hybrid metaheuristic for solving the examination
timetabling problem is proposed. In the first phase, initial solutions are generated
using the saturation degree graph colouring heuristic. In the second phase, a novel
variant of the threshold acceptance (TA) is employed. The proposed TA, named
FastTA, is able to effectively reduce the number of evaluations performed without
worsening the optimisation results in a significant way.
The standard TA and proposed FastTA were evaluated using the public Toronto
and ITC 2007 benchmark sets. The experiments state that FastTA is faster than TA,
as FastTA uses 38 and 22% less evaluations, on average, for the Toronto and ITC
2007 sets, respectively. In terms of solution cost, the FastTA is competitive with the
TA algorithm on the ITC 2007 attaining the best average value in four out of twelve
instances.
Compared with the state-of-the-art approaches, the FastTA is competitive with
the analysed algorithms on both Toronto and ITC 2007 benchmark sets. The
algorithm comparison carried out confirms that, for the Toronto set, none of the
compared algorithms is better than FastTA. For the ITC 2007 set, only one algo-
rithm, Byk13, is better than the FastTA. These results have statistical significance.
The main contribution/value of this chapter is the proposal of a new acceptance
criterion for the TA metaheuristic, which leads to a significantly faster variant
(FastTA), and its application to solve public examination timetabling benchmark
sets.
The future work can be divided into three paths. Firstly, we intend to study how
FastTA can be adapted in order to solve educational problems of other ITC 2007
tracks (course timetabling and post-enrolment course timetabling tracks), and also
to other optimisation problems. We think that one of the major tasks relies in the
problem representation within the FastTA framework, i.e., how a given solution is
perturbed and how the new states are accepted and counts are recorded in each
threshold bin.
A second research path will involve the design of other neighbourhood move-
ments and its combination with the presented ones. For example, simpler neigh-
bourhood operators, such as the ones used in Müller (2009), could be combined
with Kempe chain ones.
One of the important algorithmic components is the TA cooling rate. In this
chapter, the cooling rate is set in an adaptive way based on a simulation made at
runtime. The third research line will encompass the study of improved methods of
adapting the TA’s cooling rate based on the actual optimisation time budget.
Appendix
This appendix presents, in Table 11.22, a literature review of recent methodologies

applied to solve educational timetabling problems for Higher Education (HE).
Table 11.22 Characteristics of recent methodologies applied to solve educational timetabling

problems
Type of HE
timetabling Methodology/ # Hard # Soft
Author (year) problem algorithm Where applied Constraints Constraints
Our proposal— Exam FastTA (threshold Toronto & ITC 2&8 1&6
FastTA timetabling acceptance) 2007 (track 1)
technique sets
Leite et al. Exam FastSA (simulated ITC 2007 (track 8 6
(2019) timetabling annealing) 1)
June et al. Exam Constraint UMSLIC — —
(2019) timetabling programming and (university)
simulated
annealing
Battistutta et al. Exam Simulated ITC 2007 (track 8 6
(2017) timetabling annealing 1)
Woumans et al. Exam Column KU Leuven —&2 —&1
(2016) timetabling generation campus Brussels
(Belgium) &
Toronto sets
Bellio et al. Curriculum- Simulated ITC 2007 (track 4 4
(2016) based annealing 3)
course
timetabling
Bykov and Exam Hill-climbing- ITC 2007 (track 8 6
Petrovic (2016) timetabling based 1)
Bettinelli et al. Curriculum- Math. models ITC 2007 (track 4 4
(2015) based lower bounds, 3)
course exact and heuristic
timetabling
Li et al. (2015) Exam Search based on Toronto 2 1
timetabling evolutionary ruin
and stochastic
rebuild
Alzaqebah and Exam Bee colony Toronto & ITC 2&8 1&6
Abdullah timetabling optimisation 2007 (track 1)
(2015) sets
Cheong et al. Exam Bi-objective Toronto 3 1
(2009) timetabling hybrid (capacitated)
evolutionary
algorithm
360 N. Leite et al.
Acknowledgments This work was supported by the LARSyS—FCT Plurianual funding 2020–
2023 and partially funded with grant [SFRH/PROTEC/67953/2010], from Fundação para a Ciência
e a Tecnologia (FCT).
References
Abdullah, S., Turabieh, H., & McCollum, B. (2009). A hybridization of electromagnetic-like

mechanism and great deluge for examination timetabling problems. In M. J. Blesa, C. Blum,
L. D. Gaspero, A. Roli, M. Sampels, A. Schaerf (Eds.) Hybrid metaheuristics. Lecture Notes
in Computer Science (vol. 5818, pp. 60–72). Springer.
Abramson, D. (1991). Constructing school timetables using simulated annealing: Sequential and
parallel algorithms. Management Science, 37(1), 98–113. https://doi.org/10.1287/mnsc.37.1.
98.
Alzaqebah, M., & Abdullah, S. (2014). An adaptive artificial bee colony and late-acceptance
hill-climbing algorithm for examination timetabling. Journal of Scheduling, 17(3), 249–262.
https://doi.org/10.1007/s10951-013-0352-y.
Alzaqebah, M., & Abdullah, S. (2015). Hybrid bee colony optimization for examination
timetabling problems. Computers & Operations Research, 54(0), 142–154. https://doi.org/10.
1016/j.cor.2014.09.005.
Battistutta, M., Schaerf, A., & Urli, T. (2017). Feature-based tuning of single-stage simulated
annealing for examination timetabling. Annals of Operations Research, 252(2), 239–254.
https://doi.org/10.1007/s10479-015-2061-8.
Bellio, R., Ceschia, S., Gaspero, L. D., Schaerf, A., & Urli, T. (2016). Feature-based tuning of
simulated annealing applied to the curriculum-based course timetabling problem. Computers
& Operations Research, 65(Supplement C), 83–92. https://doi.org/10.1016/j.cor.2015.07.002.
Bettinelli, A., Cacchiani, V., Roberti, R., & Toth, P. (2015). An overview of curriculum-based
course timetabling. TOP, 23(2), 313–349. https://doi.org/10.1007/s11750-015-0366-z.
Brélaz, D. (1979). New methods to color the vertices of a graph. Communications of the ACM,
22(4), 251–256. https://doi.org/10.1145/359094.359101.
Broder, S. (1964). Final examination scheduling. Communications of the ACM, 7, 494–498. https://
doi.org/10.1145/355586.364824.
Burke, E., Bykov, Y., Newall, J., & Petrovic, S. (2004). A time-predefined local search approach
to exam timetabling problems. IIE Transactions, 36(6), 509–528.
Burke, E. K., & Bykov, Y. (2008). A late acceptance strategy in hill-climbing for exam timetabling
problems. In Proceedings of the PATAT ’08.
Burke, E. K., Qu, R., & Soghier, A. (2014). Adaptive selection of heuristics for improving exam
timetables. Annals of Operations Research, 218(1), 129–145. https://doi.org/10.1007/s10479-
012-1140-3.
Bykov, Y., & Petrovic, S. (2013). An initial study of a novel step counting hill climbing heuristic
applied to timetabling problems. In Proceedings of the MISTA’13 (pp. 691–693).
Bykov, Y., & Petrovic, S. (2016). A step counting hill climbing algorithm applied to university
examination timetabling. Journal of Scheduling, 19(4), 479–492. https://doi.org/10.1007/
s10951-016-0469-x.
Carter, M., & Laporte, G. (1996). Recent developments in practical examination timetabling.
In E. Burke, P. Ross (Eds.) Practice and theory of automated timetabling. Lecture Notes in
Computer Science (vol. 1153, pp 1–21). Berlin/Heidelberg: Springer. https://doi.org/10.1007/
3-540-61794-9_49.
Carter, M., Laporte, G., & Lee, S. Y. (1996). Examination timetabling: Algorithmic strategies and
applications. Journal of the Operational Research Society, 47(3), 373–383. https://doi.org/10.
2307/3010580.
Carter, M. W. (1986) A survey of practical applications of examination timetabling algorithms.

Operation Research, 34(2), 193–202. https://doi.org/10.1287/opre.34.2.193.
Cheong, C., Tan, K., & Veeravalli, B. (2009). A multi-objective evolutionary algorithm for
examination timetabling. Journal of Scheduling, 12, 121–146. https://doi.org/10.1007/s10951-
008-0085-5.
Cole, A. J. (1964). The preparation of examination time-tables using a small-store computer. The
Computer Journal, 7(2), 117–121. https://doi.org/10.1093/comjnl/7.2.117.
Demeester, P., Bilgin, B., Causmaecker, P. D., & Berghe, G. V. (2012). A hyperheuristic approach
to examination timetabling problems: benchmarks and a new problem from practice. Journal
of Scheduling, 15(1), 83–103. https://doi.org/10.1007/s10951-011-0258-5.
Derrac, J., García, S., Molina, D., & Herrera, F. (2011). A practical tutorial on the use of
nonparametric statistical tests as a methodology for comparing evolutionary and swarm
intelligence algorithms. Swarm and Evolutionary Computation, 1(1), 3–18. https://doi.org/10.
1016/j.swevo.2011.02.002.
de Werra, D. (1985). An introduction to timetabling. European Journal of Operational Research,
19(2), 151–162. https://doi.org/10.1016/0377-2217(85)90167-5.
de Werra, D. (1997), The combinatorics of timetabling. European Journal of Operational
Research, 96(3), 504–513. https://doi.org/10.1016/S0377-2217(96)00111-7.
Dowsland, K. A. (1990). A timetabling problem in which clashes are inevitable. Journal of the
Operational Research Society, 41(10), 907–918. https://doi.org/10.1057/jors.1990.143.
Dowsland, K. A., & Thompson, J. M. (2012). Simulated annealing. In G. Rozenberg, T. Bäck, &
J. N. Kok (Eds.) Handbook of natural computing (pp. 1623–1655). Berlin,Heidelberg: Springer.
https://doi.org/10.1007/978-3-540-92910-9_49.
Dueck, G., & Scheuer, T. (1990). Threshold accepting: A general purpose optimization algorithm
appearing superior to simulated annealing. Journal of Computational Physics, 90(1), 161–175.
https://doi.org/10.1016/0021-9991(90)90201-B.
Eley, M. (2006). Ant algorithms for the exam timetabling problem. In E. K. Burke, & H. Rudová
(Eds.) PATAT. Lecture Notes in Computer Science (vol. 3867, pp. 364–382). Springer.
Fong, C. W., Asmuni, H., & McCollum, B. (2015). A hybrid swarm-based approach to university
timetabling. IEEE Trans Evolutionary Computation, 19(6), 870–884. https://doi.org/10.1109/
TEVC.2015.2411741.
García, S., & Herrera, F. (2008). An extension on “statistical comparisons of classifiers over
multiple data sets” for all pairwise comparisons. Journal of Machine Learning Research, 9,
2677–2694.
Gashgari, R., Alhashimi, L., Obaid, R., Palaniswamy, T., Aljawi, L., Alamoudi, A. (2018). A
survey on exam scheduling techniques. In 2018 1st International Conference on Computer
Applications Information Security (ICCAIS) (pp. 1–5). https://doi.org/10.1109/CAIS.2018.
8441950.
Gogos, C., Alefragis, P., & Housos, E. (2008). A multi-staged algorithmic process for the solution
of the examination timetabling problem. In Proceedings of the 7th PATAT, University of
Montreal, Canada.
Gogos, C., Alefragis, P., & Housos, E. (2012). An improved multi-staged algorithmic process for
the solution of the examination timetabling problem. Annals OR, 194(1), 203–221. https://doi.
org/10.1007/s10479-010-0712-3.
Hamilton-Bryce, R., McMullan, P., & McCollum, B. (2013). Directing selection within an
extended great deluge optimization algorithm. In Proceedings of the MISTA’13 (pp. 499–508).
International Timetabling Competition, (2007). ITC 2007 website. http://www.cs.qub.ac.uk/
itc2007/, accessed on May 2020.
Johnes, J. (2015). Operational research in education. European Journal of Operational Research,
243(3), 683–696. https://doi.org/10.1016/j.ejor.2014.10.043.
June, T. L., Obit, J. H., Leau, Y. B., & Bolongkikit, J. (2019). Implementation of constraint
programming and simulated annealing for examination timetabling problem. In R. Alfred, Y.
Lim, A. A. A. Ibrahim, & P. Anthony (Eds.) Computational science and technology (pp. 175–
184). Singapore: Springer Singapore. https://doi.org/10.1007/978-981-13-2622-6_18.
362 N. Leite et al.
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimization by simulated annealing.
Science, 220, 671–680. https://doi.org/10.1126/science.220.4598.671.
Kristiansen, S., & Stidsen, T. R. (2013). A comprehensive study of educational timetabling,
a survey. Tech. Rep. DTU Management Engineering Report, No. 8.2013 (Accessed on
21.05.2020), Department of Management Engineering, Technical University of Denmark.
http://orbit.dtu.dk/files/60366101/A_Comprehensive_Study.pdf.
Leite, N., Melício, F., & Rosa, A. C. (2016). A shuffled complex evolution algorithm for the
examination timetabling problem. In J. J. Merelo, A. Rosa, M. J. Cadenas, A. Dourado, K.
Madani, & J. Filipe (Eds.) Computational intelligence: International joint conference, IJCCI
2014 Rome, Italy, October 22–24, pp. 151–168. 2014 Revised Selected Papers. Cham: Springer
International Publishing. https://doi.org/10.1007/978-3-319-26393-9_10.
Leite, N., Fernandes, C. M., Melício, F., & Rosa, A. C. (2018). A cellular memetic algorithm for
the examination timetabling problem. Computers & OR, 94, 118–138. https://doi.org/10.1016/
j.cor.2018.02.009.
Leite, N., Melício, F., & Rosa, A. C. (2019). A fast simulated annealing algorithm for the
examination timetabling problem. Expert Systems with Applications, 122, 137–151. https://
doi.org/10.1016/j.eswa.2018.12.048.
Li, J., Bai, R., Shen, Y., & Qu, R. (2015). Search with evolutionary ruin and stochastic rebuild: A
theoretic framework and a case study on exam timetabling. European Journal of Operational
Research, 242(3), 798–806. https://doi.org/10.1016/j.ejor.2014.11.002.
McCollum, B., McMullan, P., Parkes, A. J., Burke, E. K., & Abdullah, S. (2009). An extended
great deluge approach to the examination timetabling problem. In Proceedings of the 4th
Multidisciplinary International Scheduling: Theory and Applications 2009 (MISTA 2009) (pp.
424–434).
McCollum, B., Schaerf, A., Paechter, B., McMullan, P., Lewis, R., Parkes, A. J., Gaspero, L. D.,
Qu, R., & Burke, E. K. (2010). Setting the research agenda in automated timetabling: The
second international timetabling competition. INFORMS Journal on Computing, 22(1), 120–
130. https://doi.org/10.1287/ijoc.1090.0320.
McCollum, B., McMullan, P., Parkes, A. J., Burke, E. K., & Qu, R. (2012). A new model for
automated examination timetabling. Annals of Operations Research, 194, 291–315. https://doi.
org/10.1007/s10479-011-0997-x.
Melício, F., Caldeira, P., & Rosa, A. (2000). Solving the timetabling problem with simulated
annealing. In J. Filipe (Ed.) Enterprise information systems (pp. 171–178). Dordrecht: Springer
Netherlands. https://doi.org/10.1007/978-94-015-9518-6_18.
Melício, F., Caldeira, J. P., & Rosa, A. (2004). Two neighbourhood approaches to the timetabling
problem. In Proceedings of the Practice and Theory of Automated Timetabling (PATAT) (pp.
267–282).
Müller, T. (2009). ITC2007 solver description: a hybrid approach. Annals of Operations Research,
172(1), 429–446. https://doi.org/10.1007/s10479-009-0644-y.
Özcan, E., Elhag, A., & Shah, V. (2012). A study of hyper-heuristics for examination timetabling.
In Proceedings of the 9th International Conference on the Practice and Theory of Automated
Timetabling (PATAT-2012) (pp. 410–414).
Pillay, N. (2016). A review of hyper-heuristics for educational timetabling. Annals OR, 239(1),
3–38. https://doi.org/10.1007/s10479-014-1688-1.
Pillay, N., & Banzhaf, W. (2010). An informed genetic algorithm for the examination timetabling
problem. Applied Soft Computing, 10(2), 457–467. https://doi.org/10.1016/j.asoc.2009.08.011.
Qu, R., Burke, E., McCollum, B., Merlot, L. T. G., & Lee, S. Y. (2009). A survey of search
methodologies and automated system development for examination timetabling. Journal of
Scheduling, 12, 55–89. https://doi.org/10.1007/s10951-008-0077-5.
Sabar, N. R., Ayob, M., Kendall, G., & Qu, R. (2015). A dynamic multiarmed bandit-gene expres-
sion programming hyper-heuristic for combinatorial optimization problems. IEEE Transactions
on Cybernetics, 45(2), 217–228. https://doi.org/10.1109/TCYB.2014.2323936.
Santini, A., Ropke, S., & Hvattum, L. M. (2018). A comparison of acceptance criteria for the
adaptive large neighbourhood search metaheuristic. Journal of Heuristics, 24(5), 783–815.
https://doi.org/10.1007/s10732-018-9377-x.
Schaerf, A. (1999). A survey of automated timetabling. Artificial Intelligence Review, 13(2), 87–
127. https://doi.org/10.1023/A:1006576209967.
Talbi, E. G. (2009). Metaheuristics - From design to implementation. Wiley.
Teoh, C. K., Wibowo, A., & Ngadiman, M. S. (2015). Review of state of the art for metaheuristic
techniques in academic scheduling problems. Artificial Intelligence Review, 44(1), 1–21.
https://doi.org/10.1007/s10462-013-9399-6.
Thompson, J. M., & Dowsland, K. A. (1996). Variants of simulated annealing for the examination
timetabling problem. Annals of Operations Research, 63(1), 105–128. https://doi.org/10.1007/
BF02601641.
Thompson, J. M., & Dowsland, K. A. (1998). A robust simulated annealing based examina-
tion timetabling system. Computers & OR, 25(7–8), 637–648. https://doi.org/10.1016/s0305-
0548(97)00101-9.
Welsh, D. J. A., & Powell, M. B. (1967). An upper bound for the chromatic number of a graph and
its application to timetabling problems. The Computer Journal, 10(1), 85–86. https://doi.org/
10.1093/comjnl/10.1.85.
Winker, P. (2001). Optimization heuristics in econometrics: Applications of threshold accepting.
Chichester, UK: John Wiley and Sons Ltd.
Woumans, G., Boeck, L. D., Beliën, J., & Creemers, S. (2016). A column generation approach
for solving the examination-timetabling problem. European Journal of Operational Research,
253(1), 178–194. https://doi.org/10.1016/j.ejor.2016.01.046.
Zhang, D., Liu, Y., M’Hallah, R., & Leung, S. C. (2010). A simulated annealing with a new
neighborhood structure based algorithm for high school timetabling problems. European
Journal of Operational Research, 203(3), 550–558. https://doi.org/10.1016/j.ejor.2009.09.014.
Chapter 12
Robust Efficiency via Average
Correlation: The Case of Academic
Departments
Zilla Sinuany-Stern and Lea Friedman
Abstract Many types of efficiency methods for decision-making units (DMUs)

have been suggested in the literature. Often in the same application, several
efficiency methods are given, making it hard for decision makers to choose which
efficiency method to use. This chapter provides a simple method to choose a robust
efficiency, by calculating the average correlations of each method with all other
methods. This robust method is applied to the case of 21 academic departments
within a university, taken from the literature, with two inputs and three outputs.
A variety of data envelopment analysis (DEA) methods for measuring DMUs’
efficiencies are considered here: constant return to scale (CRS), variable return to
scale (VRS), super efficiency (SE), and cross-efficiency (CE). A few multivariate
statistical efficiency methods in the DEA context are also used: discriminant
analysis, canonical correlation, and regression analysis. For this case study, the
robust continuous efficiency scale method turned out to be the CE-CRS method.
For validating the results, we also used rankings of the efficiencies, and applied
nonparametric statistical tests. Since the two inputs used in the case study had
monetary values, we also created a one-input model using the sum of the two inputs.
Thus, we were able to use, in addition to the above-mentioned efficiency methods,
a version of stochastic frontier analysis via multiple linear regression. Even in this
case, CE-CRS turned out to be the robust efficiency method, including for the ranks
of efficiencies.
Keywords Robust efficiency · Data envelopment analysis · DEA · Higher

education · Cross efficiency · Ranking · Multivariate statistics · Stochastic
efficient frontier
Z. Sinuany-Stern () · L. Friedman

Beer-Sheva, Israel

366 Z. Sinuany-Stern and L. Friedman
12.1 Introduction
Data envelopment analysis (DEA; Charnes et al., 1978) deals with the efficiency of
the decision-making units (DMUs) of an organization, with multiple inputs and mul-
tiple outputs, when the weights (prices) are measured by various units and unknowns
(the weights are not even agreed upon). DEA provides a dichotomy, splitting DMUs
into two groups: efficient and inefficient. Over the years, the popularity of DEA has
grown tremendously; Emrouznejad and Yang (2018) reported about 10,300 DEA-
related journal articles.
In their survey of DEA applications, Liu et al. (2013) found that education is one
of the five areas of application where DEA is most used. Similarly, Sinuany-Stern
and Sherman (2014), in their volume on operations research in the public sector and
nonprofit organizations, point out that the most-used methodology in their volume
is DEA. Historically, education is the first area where DEA was applied (Bessent &
Bessent, 1980; Charnes et al., 1981).
Intensive literature reviews are devoted to the use of DEA in higher education
(HE); for example, Johnes et al. (2017) reviewed the DEA literature in tertiary
education (ibid sections 4 and 5); Thanassoulis et al. (2016, section 3) reviewed
DEA in HE; De Witte and Lopez-Torres (2017) provided a comprehensive review of
DEA in education, including HE. All the above present a large variety of efficiency
models, including DEA efficiency models used in the HE context.
Many efficiency methods have been suggested in the literature; some are DEA
based, some are multivariate statistic scales, and some combine both. For example,
De la Torre et al. (2016) integrated DEA and multidimensional scaling, discussing
the potential complementarities and advantages of combining both methodologies,
to reveal the efficiency framework and institutional strategies of the Spanish HE
system. Bougnol and Dulá (2006) highlight the validity of DEA as a ranking tool,
as applied to assess the performance of universities, using DEA rankings and uni-
versity rankings based on “Top American Research Universities” by the University
of Florida. Pino-Mejías and Luque-Calvo (2021) also used DEA multi-criteria
decision-making and principal component analysis for rank-scaling universities
based on their international rankings—they do not use inputs for DEA, but rather
provide an artificial input of a vector of ones. Sinuany-Stern et al. (1994) used DEA
and discriminant analysis to analyze the efficiency of academic departments in a
specific university in Israel.
Around the practical quest for full rank-scale efficiency, many efficiency rank-
scaling methods have been developed (Adler et al., 2002). Often, several types of
efficiencies are given in the same application, making it hard for the decision makers
to know which one to choose. This chapter provides a simple method to choose
a robust efficiency. Once several efficiency methods are considered in a specific
application, our method suggests, first, to calculate the correlations between all pairs
of efficiency scales, and then, to calculate for each efficiency method the average
correlation of each with all other efficiency methods. The efficiency method with
the maximal average correlation is to be chosen as the robust method for the specific
12 Robust Efficiency via Average Correlation: The Case of Academic Departments 367
application. Pearson correlation and/or Spearman rank correlation may be applied,

for validating the results.
The robust efficiency method is applied in this chapter to a set of data from
Sinuany-Stern et al. (1994), including 21 academic departments within a university,
based on two inputs and three outputs (out of the four original outputs). A variety
of DEA methods are used: constant return to scale (CRS), variable return to scale
(VRS), super efficiency (SE), and cross-efficiency (CE). In addition, we apply
a few other multivariate statistical methods to generate efficiency scales in the
DEA context: discriminant analysis, and canonical correlation (Adler et al., 2002).
Moreover, by adding the two original inputs we end up with one input and three
outputs; thus, multiple regression is utilized to generate a form of stochastic frontier
analysis, creating an additional efficiency scale from the regression errors.
In Sect. 12.2, robust efficiency is outlined; Sect. 12.3 lays out all the efficiency
methods used for the case study given in Sect. 12.4; in Sect. 12.5 the results are
presented; Sect. 12.6 repeats the analyses for one input; and Sect. 12.7 concludes
the chapter.
12.2 Robust Efficiency
Let us assume that there are T efficiency methods for evaluating n DMUs, in
which Et is an n-dimensional vector for efficiency method t, where t = 1 . . . T. The
procedure for finding the robust efficiency method from a set of T feasible methods
is as follows:
(a) Calculate the efficiency vector Et (n-dimensional) for all the T methods.
(b) Calculate the correlation C(Et, Es) between the Et and Es efficiency vectors for
each pair of methods, t and s.
(c) For each efficiency method t, calculate ACt, the average of its correlations with
all other methods, s = 1 . . . T (excluding t):
ACt = C (Et, Es) / (T − 1) .

s=1...T ,=t
(d) Find ACt* = max{ACt}, for t = 1 . . . n.

(e) The robust methodology is t*.
To avoid the issue of the scale range of the efficiencies, it is recommended to also
use their ranks (e.g., Appendix Table 12.10). Thus, nonparametric correlations can
be used too, i.e., Spearman rank correlation, in addition to the Pearson correlation.
The ranks may give a better look at the similarity among efficiency methods with
various scales; for example, the five efficiency scales in Table 12.2, where some
have only positive values, and some have positive and negative values. Moreover,
the robustness of the chosen best method via Pearson correlation is strengthened
when the Spearman rank correlation points to the same efficiency method.
Another method, for verifying the similarity among the ranking methods, is the
Friedman nonparametric ANOVA test for ranks—the significance of the similarity
of the ranks can be tested (Friedman & Sinuany-Stern, 1998).
Not all efficiency methods provide a full-scale ranking; some only divide the
DMUs into two groups, efficient and inefficient. Thus, to be able to apply the robust
efficiency outlined above, a screening process is needed to select the final feasible
efficiency methods to be considered for a specific application.
12.3 Efficiency Methods Used for the Case Study
We refer here to nine efficiency methods, which are shown in Table 12.1. Included
here are five of the major DEA versions (E1–E5) and four stochastic multivariate
statistical methods, which are combined within the DEA context. The main features
of each method are given later in this section, to understand the advantages of each
efficiency method and its limitations, thus highlighting the need to consider several
methods.
Table 12.1 List of efficiency methods considered for the case of academic departments
Method chosen Common weights
Et Method considereda for this case study Main featuresb Scale range for all DMUs?
E1 CRS Indirectly for E5, E6 Dich. & Det. (0,1] No
E2 SE-CRS Yes Scale & Det. (0,∞) No
E3 VRS Indirectly for E7 Dich. & Det. (0,1] No
E4 SE-VRS No due to infeasibility Scale & Det. (0,∞) No
E5 CE-CRS Yes Scale & Det. (0,1) Yes
E6 Disc-CRS Yes Scale & Stoc. (−4,4) Yesc
E7 Disc-VRS Yes Scale & Stoc. (−4,4) Yesc
E8 CCA Yes Scale & Stoc. (0,∞) Yesc
E9 Regression Only when 1 input Scale & Stoc. (∞,∞) Yesc
a CRS—constant return to scale; VRS—variable return to scale; SE—super efficiency; CE—cross
efficiency; Disc—discriminant analysis; CCA—canonical correlation

b Dich.—provides dichotomy division: efficient and inefficient; Det.—deterministic; Scale—
provides full scale; Stoc.—stochastic model

c For E6 and E7 the input weights must be negative, and the output weights must be positive; for
E8 inputs and outputs must all have the same sign; for E9 all coefficients must be positive
12.3.1 CRS Method
CRS DEA efficiency (Charnes et al., 1978) is the original DEA efficiency method.
The CRS efficiency method finds, for each DMU, the ideal weights for the inputs
and outputs which maximize the ratio between its sum of weighted outputs and the
sum of weighted inputs, where the input and output data are given. CRS measures
the total efficiency of n DMUs, where each has s outputs sharing m inputs. Given
xij , the value of input i for DMU j (for all i = 1, . . . ,m and j = 1, . . . ,n), and yrj , the
past value of output r for DMU j (for all r = 1, . . . ,s), we solve n problems, one for
each DMU. The problem, for DMU k, is to find vik , the optimal weight of input i,
and urk , the optimal weight of output r for DMU k, for all i and r, which maximize
its relative efficiency measure, hkk , where
s m
hkj = urk yrj / vik xij (12.1)
r=1 i=1
(f) The problem, for DMU k, is
max hkk (12.2)
(g) subject to
hkj ≤ 1, j = 1...n (12.3)
urk , vik ≥ ε
i = 1...m
r = 1...s
The relative efficiency is the ratio between the sum of weighted outputs and
the sum of the weighted inputs. This ratio is in the range (0,1], since the data are
positive, and their weights are non-negative (or limited by an infinitesimal value).
If, with its ideal weights, DMU k does not receive the maximal efficiency score
of 1 (100%), then DMU k is not efficient; i.e., other DMUs (or a combination
of DMUs) received the maximal score of 1 with the ideal weights for DMU k.
However, if DMU k receives the maximal possible efficiency rate of 1, then unit
k is relatively efficient. CRS is named the overall efficiency. The last column of
Appendix Table 12.8 presents CRS efficiency values for our case study. CRS is
a deterministic method. Its limitation is the weights, which vary greatly from one
DMU to another (columns V1 till U3 in Appendix Table 12.8 for our example); thus,
these efficiencies are not really comparable, although often are used as a rank scale
(e.g., Andersen & Petersen, 1993); but Banker and Chang (2006) point out that CRS
values are not a rank scale. In fact, CRS version of DEA provides a dichotomy of
the DMUs into two groups: efficient and inefficient.
12.3.2 SE-CRS Method
According to the CRS version of DEA, all the efficient DMUs receive an efficiency
value of 1. In order to allow for a range of values more than 1, Andersen and Petersen
(1993) developed SE, and specifically the SE-CRS method, where efficient DMUs
can receive efficiency values greater than 1, yet the values of inefficient DMUs
are identical to CRS, namely less than 1. Thus, SE-CRS efficiencies provide a full
scale which, unlike CRS, is not bounded at 1. For SE-CRS, the constraints of the
optimization problem of the above CRS mathematical formulation for DMU k is
not applied to constraint k; namely, in the above formulation the constraint (12.3) of
j = k is deleted. Thus, hkk may exceed the efficiency value of 1 (as shown for our
example in Table 12.2, first column). Nevertheless, it is problematic to use SE-CRS
(like CRS) efficiencies as a comparable scale for all DMUs since the weights of the
inputs and outputs vary greatly from one DMU to another. Basically, both CRS and
SE-CRS provide a dichotomy of the units into two groups, efficient and inefficient.
The efficiency of DMU k reflects the extent to which it fulfilled its own potential
with its own ideal weights, in relation to other DMUs using the ideal weights for
k. However, in practice many researchers use SE-CRS as a comparable rank scale,
including its initiators, Andersen and Petersen (1993). Banker and Chang (2006)
address this problem too.
12.3.3 VRS Efficiency Method
The VRS DEA efficiency method was developed by Banker et al. (1984). VRS
formulation is almost identical to CRS, with the difference that a constant value,
ωk , is subtracted from the numerator of the objective function and the numerator of
the constraints (the sum of weighted outputs). Efficient DMUs receive efficiencies of
1, while inefficient DMUs receive values less than 1, just like with CRS efficiency.
Similarly, the weights, vik and urk , vary greatly from one DMU to another, and so
does the constant, ωk , where ωk < 0 reflects a decreasing return to scale, ωk > 0
reflects an increasing return to scale, and ωk = 0 reflects a constant return to
scale (before the last column of Appendix Table 12.9). VRS efficiency is greater
than, or equal to, CRS efficiency. VRS efficiency is called technical efficiency
(also pure technical efficiency, and managerial efficiency). Just like CRS, VRS
does not provide a rank scale for efficient DMUs, nor to inefficient DMUs, due
Table 12.2 Academic departments’ various efficiencies and original dataa
Department SE-CRS Eff. CE-CRS Eff. Disc-CRS Disc-VRSb Canonical X1 X2 Y1 Y2 Y3
Geography 1.514 0.866 0.877 0.856 1.479 108, 800 278,200 27, 900 48 3610
History 0.765 0.459 −1.412 0.596 0.678 133, 900 744,000 3430 57 6470
Education 2.536 0.859 3.107 1.704 1.044 169, 000 559,400 109, 890 35 6500
Economics 1.268 0.535 2.450 0.286 0.835 175, 700 570,100 2480 17 18, 960
Hebrew Lit. 0.892 0.455 −0.891 −0.378 0.524 62, 300 518,400 600 25 5120
Behavioral Sci. 1.115 0.677 2.065 1.364 0.919 140, 200 616,100 27, 200 38 13, 280
Bible 0.680 0.499 −0.835 −0.484 0.911 138, 700 341,000 14, 400 40 2390
Foreign Lit. 1.377 0.651 0.097 0.202 0.923 57, 800 341,500 270 33 5020
Social Work 0.729 0.486 −0.240 −0.676 0.732 59, 400 271,600 2900 19 3610
Philosophy 0.733 0.466 −0.340 −1.003 0.603 33, 100 197,000 3740 11 1930
Biology 0.703 0.470 −2.674 −2.274 0.786 658, 300 896,900 91, 060 77 7520
Geology 0.563 0.422 −1.241 −1.319 0.746 178, 000 340,400 21, 070 31 1990
Chemistry 0.882 0.610 −1.165 0.491 1.126 588, 600 801,100 74, 210 118 8350
Math. 1.133 0.596 4.348 2.761 1.091 411, 200 968,800 12, 120 75 31, 940
Physics 0.684 0.542 −1.109 1.264 0.934 464, 200 995,000 76, 570 108 8070
Nuclear Eng. 0.887 0.612 −0.483 −0.435 1.083 200, 800 332,800 33, 920 46 2300
Chemical Eng. 0.617 0.453 −1.673 −1.045 0.793 290, 500 540,300 38, 270 52 3270
Material Eng. 0.884 0.570 0.376 −0.488 0.972 275, 900 460,800 44, 590 41 7640
Electrical Eng. 0.693 0.362 −2.527 −3.356 0.711 585, 900 721,500 25, 220 49 13, 140
Mechanical Eng. 0.665 0.426 −1.435 −0.834 0.792 540, 100 994,400 38, 290 77 15, 630
Ind.Eng. & Mgmt 1.095 0.709 2.706 2.768 1.189 220, 000 694,800 18, 990 74 17, 330
12 Robust Efficiency via Average Correlation: The Case of Academic Departments
a Data is taken from Sinuany-Stern et al. (1994). Though all the calculations here are original, there was a mistake (typo) in one of the original X1 figures for
Nuclear Engineering, which is fixed here

b VRS efficient departments are the SE-CRS efficient departments plus four more: History, Philosophy, Chemistry, and Physics
371
to the variability of VRS weights and the fact that efficient DMUs each receive an
efficiency value of 1.
12.3.4 SE-VRS Method
To provide a wider scale for VRS-efficient DMUs, above 1, SE, developed by

Andersen and Petersen (1993), is applied also to VRS. In SE-VRS, for the
optimization problem of DMU k, the ratio constraint k of the optimization problem
in the VRS mathematical formulation is not applied. Thus, hkk may exceed the
efficiency value of 1 (as shown for our example in Appendix Table 12.9, last
column).
The SE-VRS efficiency division into efficient and inefficient is identical to that
for VRS efficiency. However, for efficient DMUs, VRS efficiency values are 1,
while for SE-VRS they may exceed 1, thus allowing for a full scale; SE-VRS is not
bounded at 1. However, it is problematic to use SE-VRS (and even VRS) efficiencies
as a comparable scale, since the weights vary greatly from one DMU to another; this
method has the same limitations as CRS, SE-CRS, and VRS, as discussed above.
Moreover, SE-VRS efficiency values may often be infeasible (as shown in the last
column of Appendix Table 12.9, for three departments), and so this method was
not included in the robust analysis here. (Lee et al. (2011) suggested a resolution.)
Banker and Chang (2006) also questioned the fitness of SE as a ranking method for
the efficient DMUs (see also Zhu, 1996).
12.3.5 CE-CRS Cross-Efficiency Method
CE was developed by Sexton et al. (1986) and has often been used for scale-ranking
(e.g., Doyle & Green, 1994; Ruiz & Sirvent, 2016). Beasley (1995) applied CE
in HE. Cross-efficiency is based on the original DEA results, but implements a
secondary stage where each DMU peer-appraises all other DMUs with its own
factor weights. The average of these peer-appraisal scores is then used to calculate a
DMU’s CE. The CE score of each DMU is therefore assessed with the CRS weights
of all the DMUs, instead of only its own weights (e.g., CRS weights in Appendix
Table 12.8), which create a cross-efficiency matrix from which an average measure
is calculated for the kth DMU as follows:
n
hk = hj k /n
j =1
Table 12.2 (second column) shows CE-CRS efficiency for the academic depart-
ments’ example. The efficiencies of CE-VRS are not considered here, since the
constant added to the outputs is not limited by sign, and so some of the cross-
efficiency values may be negative and some are infeasible. Note that the issue of
multiple solutions to the CRS weights was resolved by Oral et al. (1991). The scale
range of CE is (0,1].
12.3.6 Disc-CRS Efficiency Method Based on Discriminant

Analysis
Another approach to remedy some of DEA’s drawbacks is stochastic DEA, which

synthesizes DEA and stochastic frontier analysis (SFA). Some of SFA’s variations
are based on multivariate analyses such as discriminant (Disc) analysis. The Disc-
CRS method is a multinomial statistical method, where the dichotomy of CRS into
efficient (1) and inefficient (0) groupings is applied to the linear Disc model, which
calculates the optimal common weights which reclassify the DMUs into efficient
and inefficient with the best match to the original CRS classification, via a critical d*
value that discriminates between the two groups. The resulting Disc analysis scores,
di, of the DMUs are their Disc-CRS efficiencies (Sinuany-Stern et al., 1994). We
expect the weights of the Disc-CRS linear function to be positive for the outputs, and
to be negative for the inputs (e.g., see common weights of Disc-CRS in Table 12.3).
The values of Disc-CRS are not bounded, but in practice they vary between −4 and
4, with an average around zero (see Table 12.2 column Disc-CRS).
The advantage of Disc-CRS is that it uses the dichotomy of CRS into two groups,
efficient and inefficient, and the common weights apply to all DMUs. Disc-CRS’s
limitation is that some of its output weights may be negative, or some of the input
weights may be positive. In such cases Disc cannot be used—it is infeasible.
12.3.7 Disc-VRS Efficiency Method Based on Discriminant

Analysis
The VRS efficient and inefficient grouping is applied to the linear Disc model,
similarly to Disc-CRS but with a new VRS grouping. The resulting Disc of each
DMU is its Disc-VRS efficiency (Sinuany-Stern & Friedman, 1998). Disc-VRS has
the same advantages and disadvantages as Disc-CRS (see common weights of Disc-
VRS in Table 12.3, and Table 12.2 column Disc-VRS).
Table 12.3 Discriminant and canonical analysis weights for the academic departments’ case
study
Variable X1 X2 Y1 Y2 Y3
Discriminant-CRS −1.8586 −1.1989 1.3077 0.5646 2.5359
Discriminant-VRS −2.2846 −0.1935 0.7472 1.4829 1.2785
Canonical 1.2997 3.0880 7.7256 21,257.0600 67.5641
12.3.8 CCA—Canonical Correlation Efficiency Method
Canonical correlation analysis (CCA) is a multivariate statistical method, another

form of SFA which finds weights for two groups of variables (inputs and outputs
in our case), which maximize the correlation between the sum of the weighted
outputs and the sum of the weighted inputs. CCA efficiency is the ratio between
the sum of the weighted outputs and the sum of the weighted inputs (Friedman
& Sinuany-Stern, 1997). The advantage of CCA is that it has one common set of
weights for all DMUs, and thus their efficiencies are comparable (e.g., common
weights of canonical in Table 12.3). Note that CCA is not based on the DEA
dichotomy. Thus, CCA weights are independent of DEA, although both efficiencies
are defined similarly, as the ratio between the weighted sums of the outputs and
the inputs. However, while DEA (CRS and VRS versions) provides the efficient
frontier, CCA is a sort of extended regression that finds the weights which maximize
the correlation between the weighted outputs and the weighted inputs. Thus, such a
method, which is independence of DEA-based methods is important (Table 12.2).
The scale range of feasible CCA in the DEA context is (0,∞)—if any CCA weight
turns negative then CCA cannot be used in the DEA context, as it is infeasible in
this case; since, positive input cannot contribute negatively to the overall sum of
weighted inputs, and positive output cannot contribute negatively to the overall sum
of weighted outputs.
12.3.9 REG—Regression SFA
Regression efficiency (REG) is another form of SFA model. REG can be used if
there are one input and multiple outputs (or one output and multiple inputs). When
there is one input, and the error of DMU j is positive (i.e., above the regression
predictor), it means that DMU j used more input than average. Thus, the higher the
error the lower the efficiency, and so the opposite error sign is used as an efficiency
value (as shown in the last two columns of Appendix Table 12.11). When there
are one output and multiple inputs then the error is used as-is, so when the error is
positive (i.e., above the average expected output for the given inputs), the efficiency
is higher when the error is higher. The REG method is feasible and can be used
only if all the variables’ coefficients are positive (as shown in Appendix 6). In this
case, where REG is possible, discriminant analysis and canonical correlations are
redundant.
12.3.10 Evaluation of the Efficiency Methods
There are many more efficiency methods (see Adler et al., 2002; Hosseinzadeh-
Lotfi et al., 2013; Sueyoshi, 1999). We tried, in our case study, to cover a large
variety of efficiency methods. Only five of the efficiency methods described above
were finally used for rank scaling of efficiencies. We presented SE rather than the
original DEA efficiency methods (CRS and VRS), since SE uses the same scale as
DEA for the inefficient DMUs, but extends the scale of the efficient DMUs beyond
the value 1 (while all efficient DMUs get the value 1 under CRS and VRS). Thus,
CRS and VRS are redundant. SE-VRS was not used as an efficiency method in
our case study since some of the super efficiencies were infeasible, but we used
VRS as the dichotomy for the Disc-VRS efficiency method. Since the first four
DEA and SE methods do not provide rank scales, as they are not based on common
weights, they are not comparable. Their weights vary greatly from one DMU to
another. We used CE-CRS efficiency, which provides a full efficiency scale based
on the weights of all DMUs for the evaluation of each DMU. CE-VRS was not used,
since for some DMUs within the calculation of CE, some values are negative due
to the negative constants in some of the CE values using VRS weights. The better
known multivariate statistical methods for rank scaling were also used: discriminant
analysis based on the DEA dichotomy for CRS and for VRS classification of DMUs
into efficient and inefficient groups, and the canonical correlation method. These
two methods are based on common weights and are adapted to the DEA structure of
data with multiple inputs and multiple outputs. As summarized in Table 12.1, these
five efficiency methods were selected. In Sect. 12.6, we used the sum of the inputs
as one variable, X1 + X2. Thus, we used REG as one of the efficiency methods.
12.4 The Case Study
The case study for exemplifying the robust efficiency is taken from Sinuany-Stern
et al. (1994), including 21 academic departments within a university, based on two
inputs and three outputs.
The two inputs are:
X1—operational costs
X2—faculty salaries
The three outputs are:
Y1—grant money
Y2—number of publications
Y3—contact hours given by the department
The input and output data for X1, X2, Y1, Y2, and Y3 is given in the last
five columns of Table 12.2. Note that we do not include the original output
“number of graduate students” since one department had no graduate students,
and three other departments had one graduate student at that time. Moreover, in
the original paper there was a mistake (typo) in the data: for the Department of
Nuclear Engineering’s “operational costs,” the value 2005,800 is written, while it
was actually 200,800 (although all the calculations were performed with the correct
figure there). Consequently, all the tables and the appendices here are original to
our paper. Descriptive statistics are given in Appendix Table 12.7. For DEA and
efficiency runs we transformed the data so that the maximal value of each column
(variable) was 100. Thus, the weights of the variables are comparable.
12.5 Results of Academic Department Efficiencies
Table 12.2 provides the efficiency scales of the five methods finally selected.
Table 12.2 reveals that seven departments were CRS and SE-CRS efficient, and
11 departments were VRS and SE-VRS efficient, of which three departments had
infeasible SE-VRS values. The CE-CRS range was between 0.362 and 0.866.
Note that, in comparison to the basic results from the original paper (Sinuany-
Stern et al., 1994), where only CRS efficiency was used, the exact same seven
departments turned out to be efficient, even though one output was dropped
in the current study. The CRS efficient departments are Geography, Education,
Economics, Behavioral Sciences, Foreign Literature, Mathematics, and Industrial
Engineering. VRS efficiency was not run in the previous article; we ran it here, and
found an additional four efficient departments—History, Philosophy, Chemistry, and
Physics—for a total of eleven VRS-efficient departments (see footnoteb in Table
12.2).
The discriminant analysis weights for CRS and VRS dichotomies are negative
for the inputs and positive for the outputs, as expected in the DEA context; Disc-
CRS and Disc-VRS efficiencies are feasible here (see Table 12.3 for the weights,
and Table 12.2 for the efficiencies). Note that the efficiencies of both, Disc-CRS
and Disc-VRS, always have negative and positive values (between −4 and + 4), as
explained in Sects. 12.3.6 and 12.3.7, respectively.
Similarly, the weights of the canonical correlation are positive, as expected in the
DEA context, and so the CCA efficiency method is feasible here (see Table 12.3 for
the weights, and Table 12.2 for the efficiencies). Note that always the efficiencies
of CCA are positive, as explained in Sect. 12.3.8. Moreover, in our case study, the
canonical correlation is very high (0.9082) and significant at less than 0.01 p-value.
12.5.1 Choosing the Robust Efficiency Method
The upper diagonal of Table 12.4 presents the Pearson correlations between the
efficiencies of pairs of the above five methods, which vary here between 0.4836
and 0.8297, where most of the correlations are statistically significant as shown in
Table 12.4, most of them with p-values less than 0.01.
Table 12.5 presents the average of the sum of Pearson correlations for each
efficiency method. Obviously, the CE-CRS method provides a robust solution—its
average correlation with other methods has the maximal value - 0.7558.
Table 12.4 Pearson (upper diagonal) and Spearman correlations among efficiency methodsa
Pearson:
Spearman: SE-CRS CE-CRS Disc-CRS Disc-VRS CCA
SE-CRS 0.8297* 0.6797* 0.5320*** 0.4836***
CE-CRS 0.7870* 0.6736* 0.6981* 0.8218*
Disc-CRS 0.7974* 0.7610* 0.8087* 0.4937***
Disc-VRS 0.6987* 0.7701* 0.7260* 0.5769***
CCA 0.5026** 0.8351* 0.5571* 0.6429*
*p < 0.01, ** p
< 0.05, *** p< 0.1
a CRS—constant return to scale; VRS—variable return to scale; SE—super efficiency; CE—cross-
Table 12.5 The average correlation of each efficiency model with the other models
Average of Pearson correlations Average of Spearman rank
Model type with other methods correlations with other methods
Super Efficiency CRS 0.6312 0.6964
Cross-Efficiency 0.7558 0.7883
CRS
Discriminant CRS 0.6639 0.7103
Discriminant VRS 0.6535 0.7094
Canonical 0.5940 0.6344
Correlation
12.5.2 Robustness for Ranks of the Efficiencies
Appendix Table 12.10 provides a full ranking for each of the five efficiency methods,
which follows from ordering the efficiencies of each method in Table 12.2. The
lower diagonal of Table 12.4 presents the Spearman rank correlations between
the efficiencies of pairs of the above five efficiency methods. For Spearman rank
correlations, 9 out of 10 correlations were significant with p-value less than 0.01,
and the tenth significance is 0.02. The average of the Spearman rank correlations
of each efficiency method with the other methods is given in Table 12.5. Clearly,
the cross-efficiency method again has the highest value, namely 0.7883. Thus,
CE-CRS is the robust solution. The identical conclusion from the Pearson and
Spearman average correlations strengthens the robustness of the choice of CE-CRS
as the efficiency of the academic departments in this study. Looking at CE-CRS
efficiencies in Table 12.2 shows that the number 1 most efficient department in this
case study is Geography, with CE-CRS efficiency of 0.866. But the lowest-ranked
department via the robust solution CE-CRS is Electrical Engineering in this case
study (ranked last—21).
For further verification, testing with the Friedman nonparametric ANOVA test
for ranks, we conclude that we cannot reject the hypothesis that the five ranks are
similar (chi-square statistic with 4 d.f. is 0.6952, p = 0.95). Thus, we conclude that
the ranks of the five efficiency methods are significantly similar. The robust method
is CE-CRS.
12.6 One Input and Three Outputs—Regression Efficiency
Since the two inputs X1 and X2 are monetary values, which can be added up,
the sum of the inputs X1 + X2 = SX can be taken as one input versus the three
original outputs. For this case, in addition to the DEA efficiency, we can take the
multiple regression model of SX as a linear combination of the three outputs. The
REG value is the negative of the regression error. The results of the parameters
of the regression model coefficients are given in Appendix 6. All the regression
coefficients are positive, and so REG is a feasible efficiency method.
In this case, where regression analysis is possible, discriminant analysis and
canonical correlations are redundant. Regarding SE-VRS* for the case of one input
SX, as shown in Appendix Table 12.11 (second column), three DMUs are infeasible,
so again SE-VRS* was not used. Only SE-CRS*, CE-CRS*, and REG were feasible
and used.
Looking at the correlations of the case with one input (X1 + X2) in Table 12.6,
by the criterion of the average of the relevant Pearson correlations, all of them are
highly significant (p < 0.01), in this case with one input,
For SE-CRS*, the average Pearson correlation is 0.7128 = (0.8224 + 0.6032)/2.
For CE-CRS*, the average Pearson correlation is 0.8124 = (0.8224 + 0.8024)/2.
For REG, the average Pearson correlation is 0.7028 = (0.6032 + 0.8024)/2.
Table 12.6 Pearson/Spearmana correlations among efficiency methods for X1, X2, and X1 + X2
SE-CRS CE-CRS SE-CRSb CE-CRSb REG
SE-CRS 0.8297 0.9546 0.7044 0.5337
CE-CRS 0.7870 0.8656 0.9434 0.8191
SE-CRSb 0.7636 0.8753 0.8224 0.6032
CE-CRSb 0.6312 0.9130 0.9039 0.8024
REG 0.6260 0.8922 0.7701 0.8610

a Pearson values are above the diagonal, while Spearman values are below the diagonal. CRS—
constant return to scale; SE—super efficiency; CE—cross-efficiency; REG—regression efficiency
b One input (X1 + X2)
Obviously, the greatest average Pearson correlation is again achieved with CE-
CRS* (0.8124) as a robust solution, as was the case in Sect. 12.5.1 for CE-CRS.
Looking at the Spearman rank test (below the diagonal in Table 12.6), again all
the correlations are highly significant (p < 0.01). For the case of one input,
For SE-CRS*, the average Spearman correlation is 0.8370.
For CE-CRS*, the average Spearman correlation is 0.8824.
For REG, the average Spearman correlation with the other two methods is 0.8155.
Consequently, the greatest average Spearman correlation is again achieved with
CE-CRS* (0.8824) as a robust solution, as was the case in Sect. 12.5.2 for CE-CRS.
Using the regression model did not change the robust solution, although all
the correlations were significant. Using Friedman nonparametric ANOVA, for the
three rankings of the efficiencies for the case of one output: SE-CRS*, CE-CRS*,
and REG, the chi-square value is 0.2143 (d.f. 2,21), with p = 0.8984, so the null
hypothesis that the methods’ ranks are similar cannot be rejected.
Overall, looking at the case where two inputs are considered and the case where
the two inputs are added into one input (X1 + X2), we see that the department
which was closed, Nuclear Engineering, was ranked between sixth and ninth out
of 21 departments, although under the cost per student model (Sinuany-Stern et al.,
1994) it was the worst and thus was closed. We now know that Nuclear Engineering
department was reopened a few years later, based on academic reasoning rather than
economic considerations. The Geology department, which was recommended to be
closed then due to its high cost per student, is ranked here between 14th and 20th.
12.7 Conclusion
Once several efficiency methods are used in a specific application, the robust method
suggested here is to calculate the correlations of all pairs of efficiency scales,
then to calculate for each efficiency method its average correlations with all other
efficiency methods. The robust efficiency method is the one with the maximal
average efficiency. The nonparametric Friedman ANOVA is also used to verify the
fit between the various efficiency methods.
Our analyses of the case of academic departments exemplify the need to use
multiple inputs and multiple outputs (including academic outputs), as was done
in our case. In measuring the efficiency in general, and in higher education in
particular, multiple efficiency methods may provide a robust solution that is best
correlated with a group of other efficiency methods. With the development of further
new efficiency methods (e.g., De Witte & Lopez-Torres, 2017; Hosseinzadeh-
Lotfi et al., 2013; Liu & Peng, 2008; Olsen & Petersen, 2016; Pino-Mejías &
Luque-Calvo, 2021), this method of robust efficiency is very important to allow
practitioners to verify the robust solution. It is also important to realize that, in

practice, some of the multivariate statistical methods may be infeasible for the data
used, due to an inappropriate sign in some of the weights; for example, negative
coefficients in some of the variables for regression or canonical correlation.
Appendix 1
Table 12.7 Descriptive statistics for inputs and outputs of academic departments
Variablesa Min Median Mean Max SD
X1 197, 000 559, 400 580, 195 995, 000 254,100
X2 33, 100 178, 000 261, 543 658, 300 197,954
X1 + X2 230, 100 745, 800 841, 738 1, 555, 200 427,807
Y1 270 25, 220 31, 768 109, 890 31,598
Y2 11 46 51 118 28.37
Y3 1930 6500 8765 31, 940 7418
aX is input i, Yr is output r
i
Appendix 2
Table 12.8 CRS weights (prices) of inputs and outputs for academic departments
Weights: Department: V1 V2 U1 U2 U3 CRS Effa .
Geography 0.060 0 0.010 0.018 0 1.000
History 0.049 0 0.007 0.015 0 0.765
Education 0.039 0 0.010 0 0 1.000
Economics 0.037 0 0 0 0.017 1.000
Hebrew Lit. 0.106 0 0.005 0.008 0.045 0.892
Behavioral Sci. 0.022 0.009 0.006 0.005 0.017 1.000
Bible 0 0.030 0 0.020 0 0.680
Foreign Lit. 0.068 0.012 0 0.036 0 1.000
Social Work 0.068 0.014 0.013 0.020 0.033 0.729
Philosophy 0.199 0 0.030 0.027 0.063 0.733
Biology 0 0.011 0.005 0.005 0 0.703
Geology 0 0.029 0.013 0.012 0 0.563
Chemistry 0 0.012 0.006 0.005 0 0.882
Math. 0.007 0.006 0 0.003 0.008 1.000
Physics 0 0.010 0.004 0.004 0 0.684
Nuclear Eng. 0 0.030 0.013 0.012 0 0.887
Chemical Eng. 0 0.018 0.008 0.008 0 0.617
Material Eng. 0 0.022 0.007 0.006 0.016 0.884
Electrical Eng. 0 0.014 0.004 0.004 0.011 0.693
Mechanical Eng. 0 0.010 0.003 0.003 0.008 0.665
Ind.Eng. & Mgmt 0.020 0.005 0 0.005 0.012 1.000
a CRS Eff.—cross efficiency
Appendix 3
Table 12.9 VRS weights (prices) of inputs (Vi) and outputs (Ur) for the academic departments
Dept./ weights V1 V2 U1 U2 U3 Constant ω SE-VRS Eff.a
Geography 0.060 0 0.010 0.018 0 0 1.540
History 0.046 0.001 0 0.028 0 0.346 1.100
Education 0.039 0 0.010 0 0 0 Inf
Economics 0.037 0 0 0 0.017 0 1.268
Hebrew Lit. 0.106 0 0.006 0 0.043 −0.306 0.943
Behavioral Sci. 0.020 0.009 0.005 0.005 0.015 −0.080 1.152
Bible 0 0.029 0 0.008 0.000 −0.743 0.764
Foreign Lit. 0.022 0.023 0 0.010 0.024 −0.335 1.381
Social Work 0.026 0.028 0 0.012 0.029 −0.478 0.920
Philosophy 0 0.050 0 0 0 −1.000 1.821
Biology 0 0.011 0.004 0.005 0.008 −0.156 0.759
Geology 0 0.029 0.011 0 0 −0.792 0.750
Chemistry 0.008 0.004 0.004 0.010 0 0.280 Inf
Math. 0.007 0.006 0.000 0.003 0.008 0 Inf
Physics 0.014 0 0.003 0.012 0 0.343 1.151
Nuclear Eng. 0 0.030 0.011 0 0 −0.650 0.898
Chemical Eng. 0 0.018 0.008 0.010 0 −0.251 0.630
Material Eng. 0 0.022 0.006 0.002 0.017 −0.253 0.908
Electrical Eng. 0 0.014 0.004 0.001 0.011 −0.394 0.708
Mechanical Eng. 0 0.010 0.004 0.004 0.007 −0.226 0.697
Ind.Eng. & Mgmt 0.027 0.001 0 0.017 0.003 0.250 1.412
a SE-VRS Eff.—super efficiency variable return to scale
Appendix 4
Table 12.10 Rankings of the academic departments via five efficiency methodsa
Department SE-CRS CE-CRS Disc-CRS Disc-VRS CCA
Geography 2 1 6 6 1
History 12 16 17 7 19
Education 1 2 2 3 6
Economics 4 11 4 9 12
Hebrew Lit. 8 17 13 11 21
Behavioral Science 6 4 5 4 10
Bible 18 12 12 13 11
Foreign Lit. 3 5 8 10 9
Social Work 14 13 9 15 17
Philosophy 13 15 10 17 20
Biology 15 14 21 20 15
Geology 21 20 16 19 16
Chemistry 11 7 15 8 3
Math. 5 8 1 2 4
Physics 17 10 14 5 8
Nuclear Eng. 9 6 11 12 5
Chemical Eng. 20 18 19 18 13
Material Eng. 10 9 7 14 7
Electrical Eng. 16 21 20 21 18
Mechanical Eng. 19 19 18 16 14
Ind.Eng. and Mgmt 7 3 3 1 2
a CRS—constant return to scale; VRS—variable return to scale; SE—super efficiency; CE—cross-
Appendix 5
Table 12.11 Efficiencies where input is X1 + X2 for academic departments

Department SE-CRSa SE-VRSa CE-CRSa REG REG-Ranking
Geography 1.452 1.452 0.939 18.934 1
History 0.591 0.621 0.436 −8.792 17
Education 2.093 inf 0.824 12.332 3
Economics 1.098 1.183 0.571 −3.027 15
Hebrew Lit. 0.498 0.627 0.355 −9.577 18
Behavioral Sci. 0.893 0.913 0.649 4.367 8
Bible 0.672 0.736 0.553 3.857 9
Foreign Lit. 0.838 0.963 0.600 6.168 7
Social Work 0.641 0.914 0.480 1.456 11
Philosophy 0.515 1.447 0.414 1.266 12
Biology 0.547 0.682 0.483 −20.799 20
Geology 0.511 0.644 0.458 −2.624 14
Chemistry 0.705 inf 0.648 9.482 4
Math. 1.044 inf 0.688 9.267 5
Physics 0.644 0.890 0.588 −0.278 13
Nuclear Eng. 0.762 0.769 0.653 7.756 6
Chemical Eng. 0.553 0.589 0.488 −5.697 16
Material Eng. 0.756 0.764 0.616 2.467 10
Electrical Eng. 0.537 0.542 0.407 −25.795 21
Mechanical Eng. 0.606 0.656 0.483 18.679 19
Ind.Eng. and Mgmt 1.031 1.129 0.767 17.915 2
CRS—constant return to scale; SE—super efficiency; CE—cross-efficiency; REG—regression
efficiency
a Efficiencies for the case of one input, X1 + X2
Appendix 6: The Regression Results for Calculating

Regression Efficiency
X = X1 + X2 is the dependent variable; Y1, Y2, and Y3 are the explanatory

variables.
The F-statistic shows a very significant fit, as shown in the last line under the
table.
References
Adler, N., Sinuany-Stern, Z., & Friedman, L. (2002). Review of ranking methods in the data
envelopment analysis context. European Journal of Operation Research, 140(2), 249–265.
Andersen, P., & Petersen, N. C. (1993). A procedure for ranking efficient units in data envelopment
analysis. Management Science, 39(10), 1261–1294.
Banker, R. D., & Chang, H. (2006). The super-efficiency procedure for outlier identification, not
for ranking efficient units. European Journal of Operational Research, 175(2), 1311–1320.
Beasley, J. (1995). Determining teaching and research efficiencies. Journal of the Operational
Research Society, 46(4), 441–452.
Bessent, A. M., & Bessent, E. W. (1980). Determining the comparative efficiency of schools
through data envelopment analysis. Educational Administration Quarterly, 16(2), 57–75.
Bougnol, M.-L., & Dulá, J. H. (2006). Validating DEA as a ranking tool: An application of DEA
to assess performance in higher education. Annals of Operations Research, 145(1), 339–365.
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision-making
Charnes, A., Cooper, W. W., & Rhodes, E. (1981). Evaluating program and managerial efficiency:
An application of data envelopment analysis to program follow through. Management Science,
27(6), 668–697.
De la Torre, E., Sagarra, M., & Agasisti, T. (2016). Assessing organizations’ efficiency adopting
multidimensional scaling, with an application to higher education. In S.-N. Hwang, H.-S. Lee,
& J. Zhu (Eds.), Handbook of operations analytics using DEA (International series in OR &
MS) (pp. 145–166). Springer.
De Witte, K., & Lopez-Torres, L. (2017). Efficiency in education: A review of literature and a way
forward. Journal of the Operational Research Society, 68(4), 339–363.
Doyle, J. R., & Green, R. H. (1994). Self and peer appraisal in higher education. Higher Education,
28(2), 241–264.
Emrouznejad, A., & Yang, G.-L. (2018). A survey and analysis of the first 40 years of scholarly
literature in DEA: 1978–2016. Socio-Economic Planning Sciences, 61, 4–8.
Friedman, L., & Sinuany-Stern, Z. (1997). Scaling units via the canonical correlation analysis in
the DEA context. European Journal of Operational Research, 100(3), 629–637.
Friedman, L., & Sinuany-Stern, Z. (1998). Combining ranking scales and selecting variables in
the DEA context: The case of industrial branches. Computers & Operations Research, 25(9),
781–791.
Hosseinzadeh-Lotfi, F., Jahanshahloo, G. R., Khodabakhshi, M., Rostamy-Malkhlifeh, M.,
Moghaddas, Z., & Vaez-Ghasemi, M. (2013). A review of ranking models in data envelopment
analysis. Journal of Applied Mathematics, 2013, 492421. (20 pages). https://doi.org/10.1155/
2013/492421
Johnes, J., Portela, M., & Thanassoulis, E. (2017). Efficiency in education. Journal of the
Lee, H.-S., Chu, C. W., & Zhu, J. (2011). Super-efficiency DEA in the presence of infeasibility.
European Journal of Operational Research, 212(1), 141–147.
Liu, F. H. F., & Peng, H. H. (2008). Ranking of units on the DEA frontier with common weights.
Computers & Operations Research, 35(5), 1624–1637.
Liu, J. S., Lu, L. Y. Y., Lu, W.-M., & Lin, B. J. Y. (2013). A survey of DEA applications. Omega
International Journal of Management Science, 41(5), 893–902.
Olsen, O. B., & Petersen, N. C. (2016). Stochastic data envelopment analysis—A review. European
Oral, M., Kettani, O., & Lang, P. (1991). A methodology for collective evaluation and selection of
industrial R&D projects. Management Science, 37(7), 871–883.
Pino-Mejías, J.-L., & Luque-Calvo, P.-L. (2021). Survey of methods for ranking and benchmarking
higher education institutions. In Z. Sinuany-Stern (Ed.), Handbook of Operations Research &
Management Science in higher education (In international series of OR & MS). Springer.
Ruiz, J. L., & Sirvent, I. (2016). Ranking decision making units: The cross-efficiency evaluation.
In S.-N. Hwang, H. S. Lee, & J. Zhu (Eds.), Handbook of operations analytics using data
envelopment analysis (In international series of OR & MS) (pp. 1–29). Springer.
Sexton, T. R., Silkman, R. H., & Hogan, A. J. (1986). Data envelopment analysis: Critique and
extensions. In R. H. Silkman (Ed.), Measuring efficiency: An assessment of data envelopment
analysis (pp. 73–105). Jossey-Bass.
Sinuany-Stern, Z., & Friedman, L. (1998). DEA and the discriminant analysis of ratios for ranking
Sinuany-Stern, Z., Mehrez, A., & Barboy, A. (1994). Academic departments efficiency via DEA.
Sinuany-Stern, Z., & Sherman, D. H. (2014). Operations research in the public sector and nonprofit
Sueyoshi, T. (1999). Data envelopment analysis non-parametric ranking test and index measure-
ment: Slack-adjusted DEA and an application to Japanese agriculture cooperatives. Omega
International Journal of Management Science, 27(3), 315–326.
Thanassoulis, E., De Witte, K., Johnes, J., Johnes, G., Karagiannis, G., & Portela, C. S. (2016).
Applications of data envelopment analysis in education. In J. Zhu (Ed.), Data envelopment
analysis: A handbook of empirical studies and applications (pp. 367–438). Springer. (Chapter
12).
Zhu, J. (1996). Robustness of the efficient DMUs in data envelopment analysis. European Journal
Operational Research, 90(3), 451–460.
Part III
Applications
Chapter 13
Course Evaluation Using Preference
Disaggregation Analysis: The Case
of an Information Communication
Technology Course
Ioannis Sitaridis, Fotis Kitsios, Stavros Stefanakakis, and Maria Kamariotou
Abstract The purpose of this chapter is to evaluate students’ satisfaction attending

an Information Communication Technology (ICT) course. An adapted version of
the Course Experience Questionnaire (CEQ) was used in order to evaluate students’
satisfaction, based on their experience. The 36-item scale includes measures for
Good teaching, Clear Goals, Appropriate Workload, Appropriate Assessment, and
Independence in Learning and development of Generic Skills. The data were
analyzed using preference disaggregation analysis, a method of multiple criteria
analysis, based on the principles of ordinal regression. The findings of this chapter
provide many practical implications for educators because they can identify the
strengths and weaknesses of the course. Therefore, they can improve teaching
weaknesses based on students’ satisfaction rates. Additionally, the methodology
that was used will provide curriculum designers with a tool in order to evaluate
their current courses and will support them to incorporate improved educational
approaches in the future. Multi-criteria decision analysis offers many benefits
for education practitioners, such as the identification of problems as well as the
opportunity to prioritize the necessary interventions for improving their teaching.
Last but not least, this method supports educators to organize the course more
flexibly and facilitate students communicate with teachers and participate more
actively in the educational process.
Keywords Course evaluation · Multi-criteria analysis · Disaggregation analysis ·

Satisfaction · ICT course
I. Sitaridis · F. Kitsios · S. Stefanakakis · M. Kamariotou ()

Department of Applied Informatics, University of Macedonia, Thessaloniki, Greece
e-mail: ysitar@uom.edu.gr; kitsios@uom.gr; mkamariotou@uom.edu.gr

390 I. Sitaridis et al.
13.1 Introduction
Students’ satisfaction became part of universities’ mission statement and was used
as a marketing strategy, since these institutions have to operate in an environment
where competitiveness is continuously increased, making it more and more difficult
to justify their role and attract customers, i.e., students (Elliott & Shin, 2010).
Students’ satisfaction, nowadays, attracts research interest and it is used as a
means of measuring course quality as it provides many advantages (Waugh, 1998).
Satisfied students will dedicate more time and attention to their courses, and
consequently, they carry out better in their classes. Furthermore, they strengthen
their interpersonal interactions with their fellow students and professors. In addition,
students attending courses that satisfy them tend not to dropout often (Gibson,
2010).
Certain dimensions of teaching are positively connected to student satisfaction,
such as effective teaching, independent learning, and development of skills during
the course (Law & Meyer, 2011a). Finally, an equally important dimension related
to students’ satisfaction is whether or not the academic staff is capable of responding
adequately to the needs of students (Gibson, 2010; Kitsios & Sitaridis, 2017;
Sitaridis & Kitsios, 2016a, 2016b, 2017). On the other hand, the assessment of
educational programs is substantial for their success and the quality of learning
outcomes. Students’ experience can be used as a measure of the overall quality
of the course as well as, the performance of teaching. The quality of educational
courses can be implicitly evaluated by participating students, based on their
judgments regarding various dimensions of teaching. Students’ judgments may
highlight course strengths and weaknesses, and providing data on the teaching
quality. Educators and curriculum designers can benefit from this kind of diagnostic
information (Sitaridis et al., 2018; Sitaridis & Kitsios, 2019a).
Student experience evaluation is used by course designers in order to improve
their courses. However, the main difficulty with such evaluations is that the measures
are mostly descriptive, without giving insight, in order for educators to prioritize
them and organize interventions in accord. Thus, Multi-Criteria Decision Making
(MCDM) can offer a solution to the problem.
The MUlti-criteria Satisfaction Analysis (MUSA) has never been applied to
assess academic courses before. The combination of Course Experience Ques-
tionnaire (CEQ) with multi-criteria analysis offers many advantages for course
evaluation. Educators could be knowledgeable about the dimensions that are the
main weaknesses based on students’ perceptions, so as to improve them in order to
increase the quality of their course as this method helps educators to take certain
actions in order to do so.
In the Marketing and Operations Research field, MCDM methods, based on
global and partial customer satisfaction measures, such as the MUSA are used
13 Course Evaluation Using Preference Disaggregation Analysis: The Case. . . 391
for highlighting the strong and weak points of a product or service. In our case,
the judgments of students are used for the assessment of an educational course.
The case of an ICT course is examined as an example for the implementation of
the suggested methodology, in order to measure the course quality and identify
its strengths and weaknesses. For that purpose, an adapted version of the Course
Experience Questionnaire (CEQ) is used in order to evaluate students’ satisfaction
using dimensions of teaching. The data were analyzed using MUSA method. The
methodology uses students’ expressed preferences evaluations as indices of their
level of satisfaction.
The findings of this study contribute beneficial implications about the course
performance, and the partial and total students’ satisfaction with reference to the
criteria and the importance of each one. The combination of CEQ with MUSA
offers many advantages for course assessment. Firstly, all the criteria are taken into
account simultaneously, using weights of relative importance calculated for each
criterion. Secondly, the resulting value function shows how demanding the students
are and the action diagrams illustrate the relations between criteria performance
and importance. Finally, partial satisfaction functions also indicate the participants’
degree of demand regarding each one of the criteria separately.
In contrast to previous studies, this study offers an extended review of the
literature regarding the application of the CEQ, and the findings along with diffi-
culties met in the various approaches. Especially, in this chapter, a full deployment
of the MUSA method is presented, with more details given on the aggregation
function and the calculation of weights. Whereas in Sitaridis et al. (2018), only
the global satisfaction function is illustrated, being used to give information on
the global demand of students, the Partial Satisfaction Functions are calculated,
as well, providing information on students’ demand on each distinct criterion
separately. The Relative Action Diagram is thoroughly discussed and the analysis
is reinforced with the Relative Improvement Diagram which can support decision-
making, based on the effectiveness of possible interventions according to students’
demand on each criterion. Finally, although the proposed methodology is conducted
on a sample of secondary education students, this chapter offers a great opportunity
for a comprehensive comparison between the results of other studies, not only in
higher, but in secondary education, as well, a factor not available in previous works.
The conclusion is reinforced with many arguments on the benefits of the proposed
methodology, both in secondary and in higher education.
The rest of the chapter is structured as follows. Section 13.2 presents the theo-
retical background regarding previous studies that have used the Course Experience
Questionnaire in order to evaluate courses based on students’ perceptions. Section
13.3 describes the basic principles of MUSA method, the sample of the survey,
and the development of Course Experience Questionnaire. Section 13.4 presents the
analysis of the results applying MUSA method. Finally, Sect. 13.5 concludes the
paper and presents practical implications and suggestions for further research.
13.2 Theoretical Background
13.2.1 The Course Experience Questionnaire
The CEQ was developed at Lancaster University in the 1980s in order to evaluate
teaching in higher education (Wilson et al., 1997). The original questionnaire
measured the performance of a course, taking into account students’ perceptions
on six different dimensions of teaching, namely, teaching quality, lucidity of goals,
appropriate workload, appropriate evaluation methods, development of generic
skills, and independence in learning. The rationale behind this approach is that,
students’ assessment is considered to be valid in order to predict learning outcomes,
when it is positively related to their achievements (Ramsden, 1991, p. 132). In
Australia, the CEQ has been used to provide universities with data regarding the
perceived quality of degree programs, based on students’ evaluation and help them
to make useful comparisons in order to improve teaching quality standards (Griffin
et al., 2003). Hence, there are many benefits to the use of this questionnaire for
diagnostic purposes and evaluation (Ramsden, 1991, 2003). However, it received
some criticism when it was used as a tool for management intervention in the
evaluation of teaching (Talukdar et al., 2013). The questionnaire was updated by
Wilson et al. (1997) in order to examine teaching quality, as well as the fundamental
dimensions of improved learning. The reliability of the CEQ as a measure of
teaching quality and course performance has been substantiated by numerous
researches (Byrne & Flood, 2003; Law & Meyer, 2011b; Ramsden, 1991; Stergiou
& Airey, 2012; Wilson et al., 1997) with exceptional results.
The CEQ has been used for measuring the students’ satisfaction by several
studies in higher education. Lyon and Hendry (2002) used the CEQ in order to
examine teaching improvement in a graduate course of Australian graduate students.
Grace et al. (2012) extended the CEQ adding an overall satisfaction dimension to
it, in order to incorporate satisfaction measures in the original one. The analysis of
the data, also conducted using a sample of Australian bachelor students, resulted
in a six-factor model. Additionally, a modified version of the CEQ has been
successfully used for the evaluation of tourism management courses in Greece by
Stergiou and Airey (2012), using the structured equation modeling technique. The
items examining Independence in learning were substituted with a generic scale
examining the academic environment. All scales, except academic environment,
exhibited high reliability, which resulted in a five-factor model. The main drivers
for students’ satisfaction proved to be “good teaching” and “clarity of goals.”
Rahman et al. (2012) used the CEQ in a tertiary education Engineering course,
to associate course learning environment with students’ learning approaches. The
multiple regression analysis, highlighted good teaching and assessment as the main
contributors to students’ deep learning approach.
The CEQ was only recently used in higher secondary education for the assess-
ment of Science courses (Chakrabarty et al., 2016). The confirmatory factor analysis
resulted in a 4-factor solution. Significant correlations of all factors were found to
the overall satisfaction item. The authors proposed the use of a reduced version of
the questionnaire that includes 30 items, for future research.
To sum up, the main difficulties which were met when using regression analyses
with the CEQ are that the total satisfaction measure does not often suit the frame-
work and that the six-factor model is not confirmed in different contexts. The first
difficulty arises from the fact that the overall satisfaction is conceptually different
from all the other measures of the questionnaire, resulting in problematic model fit
(Waugh, 1998). The second difficulty may be the result of the reduced number of
items measuring each construct, affecting the reliability of the scale (Chakrabarty et
al., 2016). Therefore, the frequent approach of using a single item for evaluating
satisfaction is practically deficient for adequate statistical representation of the
construct (Elliott & Shin, 2010; Grace et al., 2012). Furthermore, the scale was not
originally designed for measuring students’ satisfaction but the overall satisfaction
item was added in order to validate students’ satisfaction rather than measure it
(Ramsden, 1991, 2003). Therefore, the correlations between the constructs of the
CEQ and students’ satisfaction do not have sufficient theoretical support required for
regression analysis (Sitaridis & Kitsios, 2019b). For example, Stergiou and Airey
(2012) wonder why specific CEQ constructs correlate with overall satisfaction,
while others do not. Additionally, researchers use the CEQ for students’ satisfaction,
out of necessity, due to the lack of alternative scales (Grace et al., 2012). Finally,
the relation of the anticipated course performance to proper resource allocation
remains uncertain (Ramsden, 1991). What can be the solution to these problems
is the analysis of CEQ data using means of multi-criteria analysis.
13.3 Methodology
The 36 items of the CEQ used in this chapter were adapted in order to measure
students’ satisfaction concerning six aspects of teaching. In detail, the adapted
questionnaire involve items about Good teaching (8 items), e.g., “How satisfied
are you with the staff effort to motivate you to do your best,” Clear Goals (5
items), e.g., “How satisfied are you with the level of the difficulty in discovering
what was expected from you,” Appropriate Workload (4 items), e.g., “How pleased
are you with the level of workload during the course,” Appropriate Assessment (7
items), e.g., “How satisfied are you with the need to memorize things in order to
do well,” Generic Skills Development (6 items), e.g., “How satisfied are you with
the contribution of the course in developing your ability to work in a team setting”
and Emphasis on Independence in Learning (6 items), e.g., “How satisfied are you
with the opportunities offered in this course to choose the areas of study.” All items
examine students’ satisfaction on one of the six aspects of teaching examined by the
original CEQ. A partial satisfaction item was added for each dimension of the CEQ,
as well as an item measuring the overall satisfaction from the course.
Each one of the six aspects of teaching that were examined by the original CEQ,
which appears in Fig. 13.1, consists of a distinct criterion of students’ satisfaction
394
Satisfaction
Teaching Appropriate Development

Clarity of Appropriate Independence
assessment of generic
quality goals workload in learning
methods skills
Difficulties during Teachers learn from Improvement of

Expectations Heavy workload written Ways of learning
learning students
communication
skills
Feedback for Knowledge about Level of difficulty Students select the
Volume of work
students’ work the expectations of the course Development of ways of learning
problem solving
skills
Level of pressure Variety of
Motivation Clear objectives The syllabus is vast
for students evaluation methods
Development of
planning skills
Feedback for Feedback is Students select
Clear ide of Students spend
students’ provided only in particular areas to
expectations enough time to
performance marks Students are study
learn new things confident about
tackling unfamiliar
problems Development of
Good memory
Interesting course academic interests
Development of
Assessment of what analytic skills
Teachers explain students have
the content of the memorised
course
Students work as
team members
Real interest on
students’ opinions
Fig. 13.1 Satisfaction criteria examined by the adapted questionnaire

I. Sitaridis et al.
in the adapted questionnaire. Each item of the original questionnaire was adapted in
order to measure the extent of students’ satisfaction on the corresponding criterion,
for the purpose of this research. The adapted questionnaire items are presented in
full detail in the Appendix 1, next to the corresponding item of the original CEQ.
13.3.1 Sample
All data were obtained from 125 first and second grade high school students
participating in a public senior high school in Northern Greece. All students
attended an ICT course and were requested to assess it based on the following
criteria: teaching quality, consistency of goals, appropriate workload, appropriate
evaluation methods used, independence in learning, and generic skills development
during the course. The original 36 items of the CEQ were all used to measure
students’ satisfaction and were translated into Greek. Partial satisfaction questions
were also added as per the requirements of MUSA. All constructs are assumed to be
separate factors presenting various aspects of students’ satisfaction. Each dimension
has a mandatory partial satisfaction question at the end. Finally, there is a complete
satisfaction question about the overall course evaluation.
13.3.2 The MUSA Method
MUSA was used to assess the satisfaction of students. This method can show the
satisfaction indices of every criterion as just as the weights that students assess
for every criterion (Kamariotou et al., 2018; Kitsios et al., 2009, 2015, 2016;
Kitsios & Grigoroudis, 2016; Sitaridis et al., 2018; Stefanakakis et al., 2017;
Stefanakakis & Kitsios, 2016). This method is utilized for the assessment of a set of
marginal satisfaction functions. The global satisfaction function originates from the
consequence of the marginal satisfaction functions that product customer feedback.
The most significant goal of the method is thus the aggregation of individual
decisions into a collective value function (Grigoroudis et al., 2000; Grigoroudis &
Matsatsinis, 2018; Siskos et al., 1998).
The MUSA approach evaluates global and partial satisfaction functions Y*
and Xi * , respectively, given consumers’ ordinal judgments Y and Xi (for the i th
criterion). The hypothesis of an added utility model is the basic axis of the method,
and it is represented by the following ordinal regression analysis equation:
n
Y̌ ∗ = bi Xi∗ − σ + + σ −
i=1
n
bi = 1
i=1
where Y̌ * is the estimation of the global value function Y* , n is the number of

criteria, bi is a positive weight of the i th criterion, σ + and σ − are the overestimation
and the underestimation errors, respectively, and the value functions Y* and Xi∗ are
normalized in the interval [0, 100] (Grigoroudis & Siskos, 2002, 2010).
The eventual outcome is determined utilizing the average of the near optimal
solutions of linear programming, which aims to minimize the sum of errors based
on the following constraints:
• Ordinal regression equation for each student.
• Normalization constraints for Y* and Xi∗ in the interval [0, 100], which can be
written as follows:
y ∗1 = 0, y ∗a = 100
– xi ∗1 = 0, xi ∗ai = 100 for i = 1, 2, . . . , n, where y*m is the value of the ym

satisfaction level, xi∗k is the value of the xik satisfaction level
• Monotonicity constraints for Y* and Xi∗ :
– y*m ≤ y*m+1 ↔ ym ym+1 for m = 1, 2, . . . , a − 1,
– xi∗k ≤xi∗k+1 ↔ xi∗k xi∗k+1 for k = 1, 2, . . . ai − 1, where means “less
preferred or indifferent to”.
The size of the LP can be associated with reducing the monotonicity restrictions
arising from the reduced computational effort needed to search for the optimum
solution. This is followed by the formation of a series of transformation variables.
These variables emphasize the successive steps of the value functions Y* and Xi * :
• zm = y*m+1 − y*m for m = 1, 2, . . . , a − 1
• wik = bi xi∗k + 1 − bi xi∗k for k = 1, 2, . . . , a − 1 and i = 1, 2, . . . , n
The linearity of the model is accomplished by using these variables.
It is possible to write the variables included in the MUSA method by using the
transformation variables as follows:
m−1
y ∗m = Zt for m = 2, 3, . . . , a
t =1
k−1
∗
bi xi k = wit for k = 2, 3, . . . , ai and i = 1, 2, . . . , n.
t =1
After the introduction of the zm and wik variables, the ordinal regression equation
becomes as follows:
tj −1 n tji −1
zm = wik − σ + + σ − ∀j
m=1 i=1 k=1
The final LP formulation of the method may be described as follows:
M
[min] F = σj + + σj−
j =1
Subject to
n tij −1 tj −1
wik − zm − σj+ + σj− = 0, ∀j
i=1 k=1 m=1
a−1
zm = 100
m=1
n ai −1
wik = 100
i=1 k=1
zm ≥ 0, wik ≥ 0, σj + ≥ 0, σj − ≥ 0∀i, j, k, m
where M is the number of students.

The calculation of the variables of the initial model is centered on the optimal
solution of the previous LP, since:
ai −1
1
bi = wit for i = 1, 2, . . . , n
100
t =1
m−1
y ∗m = zt for m = 2, 3, . . . , a
t =1
k−1
∗ wit
xi k = 100 at =1
i−1
for i = 1, 2, . . . n and k = 2, 3, . . . ai
t =1 wit
The eventual outcome is determined utilizing the average of the near optimal
solutions of linear programing, which increment the weights of the n satisfaction
criteria. A critical arrangement concerns the criteria weights bi, which describe the
relative significance of the assessed satisfaction criteria (value trade-offs among the
criteria). Several normalized indicators are recommended for the MUSA method
that may uphold the top to bottom examination of the satisfaction analysis and
involve different types (Grigoroudis & Siskos, 2002, 2010). The average indicators
named; satisfaction indicators are ranged between [0, 1], which present the degree of
customer global or criteria satisfaction. These indicators can be distinguished as the
essential average performance indicators (globally or per criteria) for the company.
The normalized indicators named; demanding indicators are ranged between [−1,
1] and they are determined by the arrangement of estimated added value curves.
Moreover, these indicators represent consumers’ demanding level (globally and
per criteria) and they are recognized as an index for the degree endeavors that a
company intends to enhance. The normalized average improvement indicators are
ranged between [0, 1] and these indicators present the improvement margins on a
particular criterion. The significance of the satisfaction measurements characterizes
the effect of the amelioration endeavors and their commitment to dissatisfaction
also.
Moreover, the outcomes can be optimized utilizing two types of diagrams; the
action and improvement diagrams. These diagrams are created dependent on the
previously mentioned results (Grigoroudis & Siskos, 2002, 2010). The first type of
diagram is created combining criteria weights with satisfaction indices (Fig. 13.2).
These diagrams are similar to SWOT analysis and they can outline the strong and
weak dimensions of the company, in order to assist executives to distinguish which
satisfaction dimensions should be enhanced (Grigoroudis & Siskos, 2002, 2010).
Action diagrams are separated into four quadrants and the satisfaction dimen-
sions are presented into two actions named performance and importance. The term
“Importance” refers to weight and “Performance” is the average satisfaction index.
In other words, it is similar to the mean value of the initial function of LP. The
improvement actions for every satisfaction dimension could be applied by the
quadrant in which the dimension is represented. The characteristics of the status
quo quadrant are low performance and low importance. In this way, no action is
needed in light of the fact that consumers do not examine that these satisfaction
dimensions are important. The characteristics of the leverage opportunity quadrant
are high performance and high importance. This quadrant involves dimensions
that can be described as competitive advantage. In a few cases, these dimensions
explain the reasons why students have attended the course. In regards to transfer
resources, quadrant characteristics are high performance and low importance. These
assets might be better utilized somewhere else. For instance course’s assets can
be utilized for the purpose to enhance the satisfaction dimensions concerning the
action opportunity quadrant. Ultimately, the characteristics of the action opportunity
quadrant are low performance and high importance. More consideration should be
paid to these criteria and in this manner improvement actions should be centered
around these to improve the global student’s satisfaction level.
Transfer resources Leverage opportunity

Performance High (high performance/ low (high performance/ high
importance) importance)
Low Status quo Action opportunity

(low performance/ low (low performance/ high
importance) importance)
Low High
Importance
Fig. 13.2 Action diagram. (Adapted from: Grigoroudis and Siskos (2002))
3rd priority 2nd priority

(large effort/ low (large effort/ high
Demand
High effectiveness) effectiveness)
2nd priority 1st priority

Low
(small effort/ low (small effort/ high
effectiveness) effectiveness)
Low High
Effectiveness
Fig. 13.3 Improvement diagram. (Adapted from: Grigoroudis and Siskos (2002))
Improvement diagrams are created combining improvement with demanding

indices (Fig. 13.3). Rather than the action diagram that can just represent which sat-
isfaction dimensions should be enhanced, these diagrams can assess improvement
priorities and anticipate the yield or the degree of improvement endeavors.
Improvement diagrams can be characterized as dynamic since they can introduce
just the existing circumstance of students’ behavior. These diagrams are separated
into four quadrants and the satisfaction dimensions are illustrated into two actions as
per demanding and effectiveness. The improvement priorities for every satisfaction
dimension could be assessed by the quadrant in which the dimension is outlined.
First priority territory suggests unmediated improvement actions because these
satisfaction dimensions are effective and consumers are not demanding. Then,
second priority territory involves satisfaction dimensions that have either a low
demanding indicator or a high improvement indicator. Last of all, third priority area
incorporates satisfaction dimensions that have little enhancement margin and need
improvement actions (Grigoroudis & Siskos, 2002, 2010).
In this research, Multi-criteria User Satisfaction Analysis was used to measure
the satisfaction of students as well as to provide simultaneously the weak and strong
satisfaction points (Ipsilandis et al., 2008). Among many other Operational Research
methods, for example, goal programming, multisteps linear programming, integer
programming, or dynamic programming, this method was chosen because it will
display the weights and satisfaction indices for each criterion. Besides, the goal for
this methodology is that the action and the improvement diagrams that are created
can be utilized by instructors to know about the weak and the strong points of
satisfaction. Moreover, the improvement diagrams give them a clearer perspective
on the activities that must be improved (Grigoroudis & Siskos, 2002, 2010).
13.4 Results
13.4.1 Criteria Weights and their Variability
Criteria weights, bi , present the relative importance given to each satisfaction

dimension (criterion) by the participants. A criterion is considered important if
bi > 1/n, where bi is the weight of the criterion and n is the number of criteria used
in this study. In this case, n = 6 so the cutoff point is 1/6 = 0.16666. Taking this
into consideration, we can conclude that four criteria are viewed important: good
teaching, generic skills, evaluation, and workload. Criteria weights variability can
provide us with useful information regarding the stability analysis of the results,
give us a confidence interval for the estimated weights and also identify possible
competitiveness in the criteria set (Grigoroudis & Siskos, 2010). Table 13.1 presents
criteria weights and their variability.
Table 13.1 Criteria weights Criterion Weight Relative weight Max Min
and their variability
Good teaching 0.17333 0.132 60 8
Clear targets 0.12839 −0.756 37.033 8
Independence 0.15629 −0.205 53.55 8
Generic skills 0.17742 0.212 60 8
Evaluation 0.19532 0.566 60 8
Workload 0.16925 0.051 57.55 0
Table 13.2 Criteria Criterion Satisfaction index Demanding index

satisfaction indices and
demanding indices Global 66.996% −35.196
Good teaching 68.749% −42.293
Clear targets 31.863% 37.689
Independence 69.637% −48.814
Generic skills 62.123% −28.135
Evaluation 77.058% −58.505
Workload 73.711% −52.733
13.4.2 Satisfaction and Demanding Indices
Satisfaction indices are the mean value of the global or marginal value functions,
normalized in the interval [0, 100%]. They can be used in satisfaction analysis and
benchmarking.
In this study, participating students are 66.99% satisfied from the course, while
evaluation criterion has the highest satisfaction index (77.06%) and contrariwise,
clear targets criterion has the lowest (31.86%). Demanding indices are the average
deviation of the estimated value curves from a linear function. They can take various
values for various levels of the ordinal satisfaction scale. Overall, students are
not demanding and similarly, they are not demanding in five out of six criteria
(good teaching, independence, generic skills, evaluation, workload) as suggested
by the negative value of their respective index. On the contrary, only clear targets
criterion’s index is positive and students are demanding, regarding this criterion.
Satisfaction and demanding indices are presented in Table 13.2.
13.4.3 Global Satisfaction Function
Global satisfaction function presents the real value given by the participants to each
level of the function, in the interval [0, 100]. Both global Y* , and partial satisfaction
functions, Xi∗ are mentioned as additive and marginal value or utility functions.
These functions are monotonic, nondecreasing, and discrete, while their form
indicates whether the participants are demanding or not. The global satisfaction
function represents the participants’ preference value system (Grigoroudis & Siskos,
2002).
The global satisfaction function is presented in Fig. 13.4. It is a concave value
function which means that the users are not demanding. Global Satisfaction Values
are presented in Table 13.3.
As per MUSA methodology, the form of the global satisfaction function indicates
the participants’ degree of demanding. The concave form of the specific function,
suggests that the participants are not demanding. Participants’ degree of demanding
Global Satisfaction Function

120
100.00
100
69.07 74.37
80
59.36
60
40
20
0.00
0
1 2 3 4 5
Global Satisfaction
Fig. 13.4 Global Satisfaction Function
Table 13.3 Global Scale value Additive value

satisfaction function values
1 0
2 59.359
3 69.067
4 74.367
5 100
is also indicated by the negative value of the global demanding index (−35.293)
from Table 13.2.
13.4.4 Partial Satisfaction Functions
Similarly, the form of the partial satisfaction functions also indicates the partici-
pants’ demanding degree. As indicated by the demanding indices in Table 13.4,
participants are demanding only regarding the clear targets of the course and the
convex form of the corresponding function in Fig. 13.5 confirms this inference.
Table 13.4 presents the scale and additive value for each criterion and Figs. 13.7,
13.8, 13.9, 13.10, 13.11, and 13.12 in Appendix 2 represent the partial satisfaction
function of each criterion.
13.4.5 Relative Action Diagram
Action diagrams are developed by combining criteria weights, in the horizontal

axis (importance) and satisfaction indices in the vertical axis (performance). These
Table 13.4 Partial satisfaction function by criteria—additive values

Additive Teaching Clarity Appropriate Appropriate Generic Independence
value quality of goals workload assessment methods skills in learning
1 0 0 0 0 0 0
2 55.748 15.578 61.61 52.795 69.013 64.549
3 73.077 31.155 74.407 64.068 79.253 76.366
4 84.615 46.733 87.203 75.341 89.492 88.183
5 100 100 100 100 100 100
Relative Action Diagram

0.6
Workload Assessment
0.4
Independence Good
0.2 Teaching
Performance
0
-1.00 -0.50 -0.20.00 Generic 0.50 1.00
-0.4 Skills
-0.6
Clear Targets
-0.8
-1
Importance
Fig. 13.5 Relative action diagram
values can be normalized in the [−1, 1] interval and develop the relative action
diagram. The normalization formula for calculating relative criteria weights is
b i = bi −b 2 , where bi is the weight of each criterion and bis the mean value of
i bi −b
the weights. The corresponding formula for calculating relative satisfaction indices
is S i = Si −S 2 , where Si is the satisfaction index of each criterion and S is
i Si −S
the mean value of satisfaction indices. The relative action diagram, presented in
Fig. 13.5, is divided into quadrants. The upper right quadrant (high performance
and high importance), the quadrant of leverage opportunity, contains three criteria:
evaluation, good teaching, and workload. These criteria can be advertised as
competitive advantage against other courses.
The upper left quadrant (the transfer resources quadrant with high performance
and low importance) contains the criterion of independence, which means that
the course teachers can transfer resources from this criterion to the generic skills
criterion in the lower right quadrant which is the action opportunity quadrant,
containing criteria that are considered important by the participants but these criteria
also have low performance and need to be improved. The lower left quadrant, the
status quo quadrant, contains criteria that are not deemed as significant by the
Table 13.5 Criteria relative Relative weight Relative satisfaction

weights and relative
Criterion (importance) index (performance)
satisfaction indices
Good teaching 0.132 0.133
Clear targets −0.756 −0.869
Independence −0.205 0.157
Generic skills 0.212 −0.047
Evaluation 0.566 0.359
Workload 0.051 0.268
participants and their performance is also low. No further actions are required for
the clear targets criterion which is the only one in this quadrant. Table 13.5 presents
criteria relative weights and relative satisfaction indices and Fig. 13.5 presents the
relative action diagram.
It becomes evident by the relative action diagram that the students enrolled in
the course are satisfied and not demanding. Workload, assessment, good teaching,
and generic skills are considered significant criteria with only two out of six specific
goals and independence being unimportant for the students. Assessment, workload,
and adequate teaching, all be used as a competitive advantage to draw more students
to the course. Curriculum designers, on the other hand, must bear in mind that
completing this course is an invaluable experience for students to improve their
generic skills. The average satisfaction index for this parameter is low, so attempts
to progress in this direction should be made as a result. While students are granted
a great deal of independence, it is not respected. Resources can be shifted to other
criteria. As a “status quo” feature, the statement of the goals of the teachers and the
objectives of the courses are considered.
13.4.6 Relative Improvement Diagram
Figure 13.6 presents the relative improvement diagram. The values on the horizontal
axis are the improvement indices of the criteria. The improvement index of a
criterion is calculated as follows: Ii = bi (1 − Si ), where bi is the weight and Si
the satisfaction index of each criterion and are normalized in the [−1, 1] interval
using the following formula I i = Ii −I 2 , where Ii is the improvement index
i Ii −I
of each criterion and I is the mean value of improvement indices. The improvement
index is a measure of the improvement effort required for each criterion. The values
on the vertical axis are the demanding indices of the criteria. Combining these two
values, we create the relative improvement diagram. Table 13.6 summarizes relative
improvement indices and relative demanding indices.
The quadrants of the diagram represent the priorities that should be given to the
effects for improvement for each criterion. The lower right quadrant includes criteria
that should be of top priority for improvement, as the extent of the efforts required
Relative Improvement Diagram

0.6
Clear Targets
0.4
0.2
Demanding
0
-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-0.2 Generic Skills
Independence -0.4 Good

Workload Teaching
-0.6
Assessment
-0.8
Effectiveness
Fig. 13.6 Relative improvement diagram
Table 13.6 Relative Relative improvement Relative demanding

improvement indices and
Criterion index (effectiveness) index (demanding)
relative demanding indices
Good teaching −0.08994 −0.422
Clear targets 0.78883 0.376
Independence −0.26823 −0.488
Generic skills 0.252957 −0.281
Assessment −0.33647 −0.585
Workload −0.34715 −0.527
is low but their effectiveness is high. As expected, generic skills criterion is located
in this quadrant; the importance of this criterion is high while its performance is
low. All others criteria should come after in priority order. Upper right quadrant
represents large efforts with high effectiveness. Only one criterion is here. Clear
targets have the lowest performance of all criteria but they are also not important for
the participants of the study. This is why it should be of second priority. Lower left
quadrant also contains second priority criteria. Although the improvement efforts
that are needed are not being significantly made for now their effectiveness is also
low. All four remaining criteria, namely good teaching, independence, evaluation,
and workload are in this quadrant. Finally, upper left quadrant contains criteria of
third priority, that require high improvement efforts but are of low effectiveness.
There are no criteria in this quadrant.
13.5 Comparison Between Secondary and Tertiary

Education Students’ Course Evaluation
The CEQ was originally developed for course evaluation in higher education. As
a consequence, the great majority of previous studies concern higher education,
either as course evaluations or instrument validation studies (Fryer et al., 2012; Law
& Meyer, 2011a; Rahman et al., 2012; Ramsden, 1991; Stergiou & Airey, 2012;
Ullah et al., 2011). The use of the CEQ has been validated as an instrument for
students’ course evaluation in secondary education only recently (Chakrabarty et
al., 2016). To the best of our experience the methods of multi-criteria decision-
making, and specifically MUSA is applied for the first time in combination with the
CEQ, either in higher or in secondary education. All previous studies use traditional
methods of statistical analysis, for example, factor analysis or regression. The results
of the current study are aligned to those of previous studies both in higher and
in secondary education. However, the implementation of MUSA method offers a
number of benefits in comparison to traditional approaches, explained further in the
comparison below.
Chakrabarty et al. (2016) are the only researchers that used the CEQ in secondary
education and the findings of their study were very similar to these of the current
study. The authors administered a Bengali translated version of the CEQ to 552
high school students, studying Science, in West Bengal, India. The purpose was
to validate the instrument for evaluation purposes in secondary education. The
Exploratory Factor Analysis highlighted only three of the six factors of the original
scale, namely effective teaching, generic skills, and sufficient workload, ordered
according to the proportion of variability explained by each factor. Additionally, a
fourth factor, student support, was produced. Law and Meyer (2011a), on the other
hand, came up with similar results in an attempt to validate the CEQ in the Chinese
context using a sample of six higher education institutions. Again, only three
factors were extracted from the factor analysis, namely effective teaching, sufficient
workload, and generic skills development, ordered by the produced amount of
variability. However, the analysis, in both cases, failed to identify the rest of the
CEQ constructs.
These results support Gibson’s findings (2010) who claimed that the core
attributes of a course, including quality of teaching, curriculum, and skills are the
most important ones for the participating students. Waugh (1998), on the other hand,
concluded that teaching and skills should be interpreted separately as these attributes
have the greatest fit with the concept of students’ satisfaction.
Other applications of the CEQ in higher education, report significant correlations
among effective teaching, consistency of goals, emphasis on independence and
appropriate assessment, and students’ satisfaction. Ramsden (1991), working on
a sample of 13 higher education institutions in Australia, concluded that good
teaching and appropriate assessment are linked with students’ satisfaction and deep
learning approach. This finding was confirmed in a later research by Rahman et
al. (2012), who used the CEQ to evaluate a higher education engineering course
learning environment in Malaysia. According to Rahman et al. (2012), surface

learning approach is mostly related to inappropriate workload. Since deep learning
approach is associated to better quality of learning, the authors suggested that
increasing the efforts for effective teaching and appropriate assessment, and at
the same time reducing workload, can benefit the students’ quality of learning.
However, Ramsden (1991) concluded that these measures should make meaningful
comparisons on teaching performance only within similar fields or disciplines. Fryer
et al. (2012) used the CEQ to examine the learning approaches of Economics
and Commerce students, in Japan. The authors identified four factors in their
Factor Analysis, namely effective teaching, generic skills, appropriate assessment,
and sufficient workload. Good teaching and generic skills had strong positive
correlations to deep learning approaches.
Ullah et al. (2011) studied learning approaches in conjunction to students’
satisfaction with learning environment, in multiple departments of two universities
in Pakistan. Only four of the six factors were identified by the Exploratory Factor
Analysis. The first factor was instructional practices (incl. good teaching and
increased focus on independence items), the second was appropriate workload, the
third was generic skills (incl. clear goals items), and the fourth was appropriate
assessment. The first three of these factors exhibited significant positive correlations
to overall students’ satisfaction, whereas appropriate assessment remained totally
uncorrelated.
In the Greek context, Stergiou and Airey (2012) worked on a sample of higher
education students in tourism programs with similar results. However, the level of
importance is differentiated, as Skills Development attribute is considered the most
important one among good teaching and appropriate assessment, while secondary
education students consider the Appropriate Assessment attribute as the most
important followed by skills and good teaching. As Stergiou and Airey (2012)
claimed, the nature of the course, i.e., vocational or academic, is undoubtedly
associated with the importance of one criterion over another. In our opinion, the
level of studies may also be influential on the relative importance of each criterion.
All in all, regardless of the educational level, only small differences are observed
in the results of all previous studies. The same attributes affect students’ satisfaction
from the learning environment, in most cases. Four factors are dominant in the
results of all studies, both in higher and secondary education; good teaching,
appropriate assessment, sufficient workload, and generic skills. However, the
importance of each attribute is differentiated throughout the studies, which could be
explained due to sample differences. Additionally, those studies utilizing traditional
statistical approaches (e.g., factor analysis and regression analysis), either failed to
identify some of the six factors of the original CEQ, or had insignificant effects, and
therefore it was not possible to infer the relative importance of each one of them.
A comparison between previous researches in higher and secondary education is
summarized in Table 13.7.
408
Table 13.7 Comparison between previous researches in higher and secondary education
Methods of Type of Country of
Author (year) Education level analysis questionnaire used application Type of course GT AA AW GS CG IL
This studya Secondary MCDM 36 item CEQ Greece ICT course 3b 1b 4b 2b 6b 5b
education
Chakrabarty et Secondary Exploratory 36 item CEQ India Science course 1b 3b 2b
al. (2016) Education Factor Analysis
Stergiou and Higher Education Structural 31 item Greece Tourism 2b 3b 4b 1b 5b
Airey (2012) Equation Models/ CEQ + academic
Confirmatory environment scale
Factor Analysis
Fryer et al. Higher Education Exploratory and 36 item CEQ Japan Economics and 2b b b 1b
(2012) Confirmatory Management
Factor Analysis
Rahman et al. Higher Education Multiple 36 item CEQ and Malaysia Technical Institute 2b 1b 3b
(2012) Regression WIHIC
Law and Higher Education Exploratory 36 item CEQ Hong Various b b b
Meyer (2011a, Factor Analysis Kong departments
2011b)
Ullah et al. Higher Education Exploratory 36 item CEQ Pakistan Business and 1b b 3b 2b
(2011) Factor Analysis Management,
Technology,
Social Sciences
Ramsden Higher Education Exploratory 30 item CEQ Australia Various 1b 3b 2b 4b
(1991) Factor Analysis departments
GS = generic skills, GT = good teaching, AA = appropriate assessment, AW = appropriate workload, IL = independence in learning, CQ = clear goals
a High importance attributes according to the Relative Action Diagram Fig. 13.5
b Identified criteria with a relative ranking based on statistical measures where available
I. Sitaridis et al.
13.6 Conclusion
Course evaluation is considered an important process for both tertiary and secondary
education. In recent years, many researchers have paid attention to the use of
Course Experience Questionnaire. The CEQ evaluates students’ satisfaction, based
on six separate criteria that illustrate the relationships among them and the course
performance. It is primarily used in higher education while only applications are
met in secondary education. However, the traditional statistical analysis of the CEQ
data cannot provide satisfactory explanations for constructs relationships, and it is
challenging for researchers to make fast and reliable decisions of each criterion’s
success in reference to the requirements of students.
The MUSA method used in this study has the benefits of recognizing both the
strong and the weak points of students’ satisfaction, with valuable implications for
all stakeholders. The results of the study offer educators a tool to recognize the
essential factors that adversely impact on the satisfaction of students and ultimately
impact the achievement of students. More specifically, the students of the course
under evaluation, proved to be non-demanding, according to the global satisfaction
function. A potential reason for this phenomenon may be that students do not
consider that the ICT course is fundamental in comparison to other courses in the
curriculum. Possible causes behind this result could be that the course was taught
only for 2 h per week. This inference is corroborated also by the fact that all partial
satisfaction functions show that students are demanding only when it comes to the
clarity of goals. Unfortunately, they are not demanding in good teaching, sufficient
workload and assessment, development of skills, and independence in learning.
These results are justified by the reduced time per week devoted to this course,
since the time constraints made it difficult for students to be seriously concerned
about their performance in comparison to other courses (e.g., math and chemistry)
that dominate the curriculum.
Nevertheless, according to the Relative Action Diagram, educators should con-
tinue their efforts towards good teaching, balanced workload, and flexible evaluation
procedures which are the strong assets of the course. They should also take care
of the development of generic skills of students attending this course, which is
important for them, and can offer a great amount of satisfaction, according to the
Relative Action Diagram. The Clarity of targets seems to be of great effectiveness
and demand, therefore, educators have to spend adequate time explaining what is
expected from students, what will be achieved, and how these achievements are
related to professional and technical skills. All other criteria are of lower demand,
however, regarding their effectiveness, generic skills development appears to require
some improvement, as well.
The contribution of this chapter is based on the implementation of a multi-
criteria method for course assessment. The MUSA method has never been used for
educational course evaluation before. This method helps practitioners to understand
the role each criterion performs in the course’s success, but also the importance
to the students of each criterion. The produced diagrams can support educators to
easily identify the critical factors that affect the performance of a course, and to take
students’ evaluation into consideration, in order to design their interventions.
Moreover, integrating CEQ with multi-criteria analysis has many advantages for
course evaluation. First, all criteria are considered concurrently, and a significance
weight is determined for each criterion. Second, MUSA produces a value function
based on student’s overall and partial satisfaction judgments. The value function
will explicitly provide data about demand, distinguishing neutral, demanding and
non-demanding students. Third, the action diagram which helps executives in the
decision-making process is presented by MUSA. The action diagram is actually
a map of segregated segments focused on the importance of each to the assessed
course. The findings are similar to SWOT analysis since the criteria are split into
four groups, namely status quo, leverage opportunity, transfer resources, and action
opportunity. This classification of these criteria can provide useful information for
decision makers. They could be aware of those criteria that are the competitive
advantage, the criteria of high importance that need an urgent intervention and the
criteria that are considered as valuable resources that do not have a valuable impact
on the overall performance of the course.
In conclusion, MUSA provides curriculum designers with a tool in order to
evaluate and improve their courses and supports them to better design their courses
in the future. Last but not least, the method can also increase students’ interest,
as it offers good possibilities to start a participatory model of decision making,
through fertile discussion between students and teaching staff. The relative action
and improvement diagrams are important tools for the support of rational decision-
making in education. The analysis of students’ evaluations of educational courses
or programs can help decision makers understand which dimensions of satisfaction
need to be changed in order to schedule appropriate interventions to increase course
performance.
Future work could examine if there is a difference in the level of students’
satisfaction taking demographic factors such as the sex and specialty of studies
into account. Regarding the evaluations of ICT courses, beyond general skills
development, the development of students’ ICT related skills could be additionally
analyzed, to evaluate the significance of these skills, in this setting, in comparison
to the other dimensions of the tool.
Appendix 1: The Original and Adapted CEQ Items
Original CEQ Adapted Questionnaire item

Teaching quality
1 Teachers here make a real effort to How satisfied are you with the effort
understand the difficulties we are facing teachers here make to understand the
during learning difficulties you are facing during learning?
2 Teachers of this course put a lot of time How satisfied are you with the time
into commenting on our work devoted by teachers of this course to
comment on your work?
3 The teachers of this course motivate us to How satisfied are you with the motivation
do the best work given by the teachers of this course to do
your best work?
4 Teachers here normally give helpful How satisfied are you with the usefulness
feedback on our performance of the feedback provided by teachers on
your performance?
5 Teachers of this course work hard to make How pleased are you with the effort of the
subjects interesting teachers of this course to make subjects
interesting?
6 Teachers here are extremely good at How satisfied are you with the teachers’
explaining content to us explanations about the content?
7 This course really tries to get the best out How satisfied are you with efforts during
of all the students this course to get the best out of all the
students?
8 Teachers of this course are really How satisfactory is the real interest of
interested in students’ opinions teachers in students’ opinions?
Clarity of goals
1 Teachers here make it clear right from the How satisfied are you with teachers
start what they expect from us making it clear right from the start what
they expect from you?
2 It is always easy here to know the How satisfied are you with the ease to
standard of work expected always know the standard of work
expected?
3 The aims and objectives of this course are How satisfied are you with the clarity of
not made very clear the aims and objectives of this course?
4 It is often hard to know what is expected How satisfied are you with the chance to
of me in this course know what is expected of you in this
course?
5 I have a clear idea of where I am going How satisfied are you with the clarity of
and what is expected of me understanding where you are going and
what is expected of you in this course?

Appropriate workload
1 The workload of this course is too heavy How satisfied are you with the amount of
work required from this course?
2 It seems to me that the syllabus of this How satisfied are you with the size of the
course is vast syllabus of this course?
3 The volume of work to get through in this How satisfied are you with the amount of
course means I cannot comprehend it all effort needed to get through in this course?
thoroughly
4 In this course, we are given enough time How satisfied are you with the time
to understand the things that we have to available to understand all the things you
learn have to learn in this course?
5 In this course, it is really easy to know the How satisfied are you with the ease to
standard of work expected always know the standard of work
expected?
Appropriate assessment methods
1 Teachers of this course frequently give the How satisfied are you with the frequency
impression that they have nothing to learn the teachers of this course give the
from us impression that they have something to
learn from you?
2 There is a lot of pressure on me as a How satisfied are you with the pressure on
student here you as a student of this course?
3 Usually, feedback on our work is provided How satisfied are you with the ways of
only in marks getting feedback on your performance
other than marks?
4 To do well in this course we all really How satisfied are you with the
need a good memory requirements to memorize things during
this course?
5 Teachers of this course are more interested How satisfied are you with teachers
in testing what I have memorized than examining what you have really
what I have understood understood?
6 Most of the teachers here ask questions on How satisfied are you with the questions
facts asked by teachers in this course?
Generic skills
1 This course has improved my written How satisfactory is the improvement of
communication skills your written communication skills during
this course?
2 This course has helped me to develop my How satisfactory is the development of
problem-solving skills your problem-solving skills during this
course?
3 This course has helped me in developing How satisfied are you with the
the ability to plan my own work development of your ability to plan your
own work during this course?
4 As a result of pursuing this course, I feel How satisfied are you with the
more confident about tackling unfamiliar development of your confidence to cope
problems with nontrivial problems?
5 This course has sharpened my analytic How satisfied are you with the degree this
skills course sharpened your analytic skills?
6 This course has helped me in developing How satisfied are you with the
my ability to work as a team member development of your ability to work as a
team member in this course?

Independence in learning
1 We often discuss with our teachers here How satisfied are you discussing with
on our ways of learning your teachers here on your ways of
learning?
2 Students of this course are given a great How satisfied are you with the
deal of opportunity in selecting for opportunities given in this course in
themselves their ways of learning selecting your ways of learning?
3 There is very little choice in this course in How satisfied are you with the choices
the ways I am assessed given in this course in the ways you are
assessed?
4 There are few opportunities to choose the How satisfied are you with the choice you
particular areas I want to study have on a particular area you want to
study?
5 Students of this course are given a lot of How satisfied are you with the different
choice in the work that they have to do choices you are offered about the work
you have to do in this course?
6 This course has developed my academic How satisfied are you with the
interest as far as possible development of your academic interests in
this course?
Appendix 2: Partial Satisfaction Functions
Good Teaching Partial Satisfaction Function

120
100
100 84.615
73.077
80
55.748
60
40
20
0
0
1 2 3 4 5
Good Teaching
Fig. B.1 Good teaching partial satisfaction function

Clear Targets Partial Satisfaction Function

120
100
100
80
60 46.733
40 31.155
15.578
20
0
0
1 2 3 4 5
Clear Targets
Fig. B.2 Clear targets partial satisfaction function
Independence Partial Satisfaction Function

120
100
100 87.203
74.,407
80
61.61
60
40
20
0
0
1 2 3 4 5
Independence
Fig. B.3 Independence partial satisfaction function

Generic Skills Partial Satisfaction Function

120
100
100
75.341
80 64.068
52.795
60
40
20
0
0
1 2 3 4 5
Generic Skills
Fig. B.4 Generic skills partial satisfaction function
Evaluation Partial Satisfaction Function

120 100
100 89.492
79.253
80 69.013
60
40
20 0
0
1 2 3 4 5
Evaluation
Fig. B.5 Evaluation partial satisfaction function
Workload Partial Satisfaction Function

120 100
100 88.183
76.366
80 64.549
60
40
20 0
0
1 2 3 4 5
Workload
Fig. B.6 Workload partial satisfaction function

References
Byrne, M., & Flood, B. (2003). Assessing the teaching quality of accounting programmes: An
evaluation of the course experience questionnaire. Assessment and Evaluation in Higher
Education, 28, 135–145. https://doi.org/10.1080/02602930301668
Chakrabarty, A., Richardson, J., & Sen, M. (2016). Validating the course experience questionnaire
in West Bengal higher secondary education. Studies in Educational Evaluation, 50, 71–78.
https://doi.org/10.1016/j.stueduc.2016.06.007
Elliott, K., & Shin, D. (2010). An alternative approach to assessing this important concept student
satisfaction : An alternative approach to assessing this important concept. Journal of Higher
Education Policy and Management, 24, 37–41. https://doi.org/10.1080/136008002200001351
Fryer, L., Ginns, P., Walker, R., et al. (2012). The adaptation and validation of the CEQ and the R-
SPQ-2F to the Japanese tertiary environment. The British Journal of Educational Psychology,
82, 549–563. https://doi.org/10.1111/j.2044-8279.2011.02045.x
Gibson, A. (2010). Measuring business student satisfaction: A review and summary of the major
predictors. Journal of Higher Education Policy and Management, 32, 251–259. https://doi.org/
10.1080/13600801003743349
Grace, D., Weaven, S., Bodey, K., et al. (2012). Putting student evaluations into perspective: The
course experience quality and satisfaction model (CEQS). Studies in Educational Evaluation,
38, 35–43. https://doi.org/10.1016/j.stueduc.2012.05.001
Griffin, P., Coates, H., McInnis, C., et al. (2003). The development of an extended course
experience questionnaire. Quality in Higher Education, 9, 259–266. https://doi.org/10.1080/
135383203200015111
Grigoroudis, E., & Matsatsinis, N. (Eds.). (2018). Preference disaggregation in multiple criteria
decision analysis. Essays in Honor of Yannis Siskos, Springer Int Pub.
Grigoroudis, E., & Siskos, Y. (2002). Preference disaggregation for measuring and analysing
customer satisfaction: The MUSA method. European Journal of Operational Research, 143,
148–170. https://doi.org/10.1016/S0377-2217(01)00332-0
Grigoroudis, E., & Siskos, Y. (2010). MUSA: Multicriteria satisfaction analysis. In Customer
satisfaction evaluation (pp. 91–121). Springer.
Grigoroudis, E., Siskos, Y., & Saurais, O. (2000). TELOS: A customer satisfaction evaluation
software. Computers & Operations Research, 27(7–8), 799–817.
Ipsilandis, P., Samaras, G., & Mplanas, N. (2008). A multicriteria satisfaction analysis approach in
the assessment of operational programmes. The International Journal of Project Management,
26(6), 601–611.
Kamariotou, M., Kitsios, F., & Grigoroudis, E. (2018). Strategic decision making using multicri-
teria analysis: Information systems performance evaluation in Greek SMEs. Proceedings of
the 7th International Symposium and 29th National Conference on Operational Research, pp.
184–188.
Kitsios, F., Champipi, E., & Grigoroudis, E. (2016). Cultural and creative industries innovation
strategies for new service development using MCDA. In E. Grigoroudis & M. Doumpos
(Eds.), Operational research in business and economics, springer proceedings in business and
economics (pp. 69–84).
Kitsios, F., Doumpos, M., Grigoroudis, E., et al. (2009). Evaluation of new services development
strategies using multicriteria analysis: Predicting the success of innovative hospitality services.
Operational Research—An International Journal, 9(1), 17–33.
Kitsios, F., & Grigoroudis, E. (2016). Comparing hospitality innovation strategies: New service
development using multicriteria analysis. Proceedings of 5th International Symposium and
27th National Conference on Operation Research, pp. 127–132.
Kitsios, F., Grigoroudis, E., Giannikopoulos, K., et al. (2015). Strategic decision making using
multicriteria analysis: New service development in Greek hotels. International Journal of Data
Analysis Techniques and Strategies, 7(2), 187–202.
Kitsios, F., & Sitaridis, I. (2017). An application of non weight MCDM for the evaluation of GEM
entrepreneurial ecosystems. Proceedings of 6th International Symposium and 28th National
Conference on Operational Research, pp. 35–39.
Law, D., & Meyer, J. (2011a). Adaptation and validation of the course experience questionnaire in
the context of post-secondary education in Hong Kong. Quality Assurance in Education, 19,
50–66. https://doi.org/10.1108/09684881111107753
Law, D., & Meyer, J. (2011b). Relationships between Hong Kong students’ perceptions of
the learning environment and their learning patterns in post-secondary education. Higher
Education, 62, 27–47. https://doi.org/10.1007/s10734-010-9363-1
Lyon, P. M., & Hendry, G. D. (2002). The use of the course experience questionnaire as a
monitoring evaluation tool in a problem-based medical programme. Assessment and Evaluation
in Higher Education, 27, 339–352. https://doi.org/10.1080/0260293022000001355
Rahman, S., Mokhtar, S., & Yasin, R. (2012). Learning environment and learning approaches
among engineering students. Proceedings of the 2012 IEEE Global Engineering Education
Conference (EDUCON). IEEE, pp 1–6.
Ramsden, P. (1991). Experience questionnaire a performance Indicator of teaching quality in
higher education : The course experience questionnaire. Studies in Higher Education, 16, 129–
150. https://doi.org/10.1080/03075079112331382944
Ramsden, P. (2003). Student surveys and quality assurance. Proceedings of the Australian
Universities Quality Forum, pp. 126–135.
Siskos, Y., Grigoroudis, E., Zopounidis, C., et al. (1998). Measuring customer satisfaction using a
collective preference disaggregation model. Journal of Global Optimization, 12(2), 175–195.
Sitaridis, I., & Kitsios, F. (2016a) A Taxonomy of barriers to entrepreneurship. Proceedings of 5th
International Symposium and 27th National Conference on Operation Research, pp. 133–137.
Sitaridis, I., & Kitsios, F. (2016b). Student’s perception of barriers to entrepreneurship. Pro-
ceedings of 1st Panhellenic Scientific Conference: “Greece-Europe 2020—Education, Lifelong
learning, Research, Innovation and Economy”, pp. 524–535.
Sitaridis, I., & Kitsios, F. (2017). Entrepreneurial intentions in information technology students:
The theory of planned behaviour, the role of gender and education. Journal for International
Business and Entrepreneurship Development, 10(3), 316–335.
Sitaridis, I., & Kitsios, F. (2019a). The effect of computer self-efficacy on entrepreneurial
aspirations of students. Proceedings of 2019 IEEE Global Engineering Education Conference
(EDUCON), pp. 685–688.
Sitaridis, I., & Kitsios, F. (2019b). Course experience evaluation using importance-performance
analysis. Proceedings of 2019 IEEE Global Engineering Education Conference (EDUCON),
pp. 859–862.
Sitaridis, I., Stefanakakis, S., & Kamariotou, M., et al. (2018). Students’ experience and Course
evaluation: An MCDA analysis approach. Proceedings of 7th International Symposium and
29th National Conference on Operational Research, pp. 116–120.
Stefanakakis, S., & Kitsios, F. (2016). E-service evaluation: User satisfaction measurement and
implications in health sector. Proceedings of 5th International Symposium and 27th National
Conference on Operation Research, pp. 53–58.
Stefanakakis, S., Trevlakis, S., & Kitsios, F. (2017). Analyzing quality characteristics: The case of
an e—Health application. Proceedings of the 6th International Symposium and 28th National
Conference on Operational Research, pp. 246–251.
Stergiou, D. P., & Airey, D. (2012). Using the course experience questionnaire for evaluating
undergraduate tourism management courses in Greece. Journal of Hospitality, Leisure, Sport
& Tourism Education, 11, 41–49. https://doi.org/10.1016/j.jhlste.2012.02.002
Talukdar, J., Aspland, T., & Datta, P. (2013). Australian higher education and the course experience
questionnaire: Insights, implications and recommendations. Australian Universities Review, 55,
27–35.
Ullah, R., Richardson, J. T. E., & Hafeez, M. (2011). Approaches to studying and perceptions of
the academic environment among university students in Pakistan approaches to studying and
perceptions of the academic environment among university students. Compare, 41, 113–127.
https://doi.org/10.1080/03057921003647065
Waugh, R. F. (1998). The course experience questionnaire: A rasch measurement model
analysis. Higher Education Research and Development, 17, 45–64. https://doi.org/10.1080/
0729436980170103
Wilson, K. L., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of
the course experience questionnaire. Studies in Higher Education, 22, 33–53. https://doi.org/
10.1080/03075079712331381121
Chapter 14
The Application of a Balanced Scorecard
in Higher Education Institutions: A Case
Study of Wuls
Michał Pietrzak
Abstract The purpose of this chapter is to share the experience of a Balanced

Scorecard (BSC) development and application by a public university. This case
study is used to describe the process of developing a strategy and translating it
into a customized Balanced Scorecard. The project presented in this chapter did
not lead to amazing success, but it did not completely fail. This project was adapted
to the specific conditions of a particular time and place and spanned 10 years, even
though changes in the model were made over time due to new laws at the national
level. Some of the lessons learned are discussed in the last section of this chapter.
The administrators of Higher Education Institutions can directly use or adapt the
BSC framework presented in this chapter. Moreover, they can find some useful
tips and tricks regarding the basic steps in the process of strategy and Balanced
Scorecard development. Moreover, interested parties could deliberate upon some
problems and challenges pinpointed in this case study and consider such issues in
advance. This study will be helpful in designing customized strategic management
systems adapted to the specific circumstances of a time and place, instead of simply
imitating and replicating solutions developed for business firms.
Keywords Balanced Scorecard · BSC · Strategic management · HEIs ·

Academic governance
14.1 Introduction
The Balanced Scorecard (BSC) was developed by Kaplan and Norton 29 years
ago and was initially positioned as a performance measurement system (1992).
Since that time, the conception has evolved. According to Kaplan and Norton
(2004), managers appreciate the comprehensive performance measurement system.
M. Pietrzak ()
Institute of Economics and Finance, Warsaw University of Life Sciences, Warsaw, Poland
e-mail: michal_pietrzak@sggw.edu.pl

420 M. Pietrzak
However, they also desire to use this new system to solve the problems they face—
how to execute a strategy. Thus, BSC today is perceived mainly as a methodology
to support strategic management (Kaplan & Norton, 1996, 2001, 2004, 2006,
2008). The Balanced Scorecard has been adopted by many companies worldwide.
According to the survey of Bain&Co. conducted among 1268 managers, BSC is
among the TOP 25 most popular management tools, with a 29% of usage rate (Rigby
& Bilodeau, 2018).
While typically applied in business, BSC has also gained popularity in public
agencies and non-profits (Niven, 2008). This trend is particularly observable in
the public sector, where the concept of New Public Management (NPM) has been
spread widely in many OECD countries (Hood, 1991; O’Flynn, 2007). This concept
relies on markets, rather than planning; performance measurements; monitoring and
management systems; and empowered and entrepreneurial management instead of
collegial decisions (Ferlie et al., 2008). Moreover, Higher Education Institutions
(HEIs) have become the places of application for the NPM (Schimank, 2005;
Tahar, 2013; Wilkesman & Schmid, 2012). Consequently, HEIs have started to
implement modern managerial tools developed for business. Some of them have
started to use BSC (Asan & Tanyas, 2007; Juhl & Christensen, 2008; McDevitt et
al., 2008; Papenhausen & Einstein, 2006; Tapinos et al., 2005; Taylor & Baines,
2012; Umashankar & Dutta, 2007).
This chapter presents a case study of one such application using the example of
a Polish university. The case study object is one of the 20 largest universities in
Poland. A Polish example could be particularly interesting for countries experienc-
ing political and socioeconomic transitions, as well as those that have witnessed the
massification of higher education. Moreover, some traits of public universities are
quite common worldwide, so a case study of Balanced Scorecard’s application in
a public university could shed light on some typical issues and obstacles that are
widespread in such higher education institutions. The results of the BSC application
can be considered ambiguous. The project did not lead to amazing success, but it did
not fail completely. As time passed, our team ceased pursuing the perfect application
of BSC for practical reasons. Our version of the BSC strategic model was adapted to
the specific conditions of a particular time and place. However, the project survived
for 10 years, even though changes in the model took place over time due to new
laws at the national level related to higher education in Poland. Some of the lessons
learned are discussed in the last section of this chapter.
This case study could be useful for administrators of HEIs, particularly publicly
owned ones, who can use or adapt the BSC model presented here, as well as
can find some tips and tricks regarding strategic management and the Balanced
Scorecard’s application. Interested readers could also consider some possible
problems and obstacles in advance. This case study could also facilitate some
thoughts about public policy regarding HE and about the development and cascading
of a nationwide strategy. Therefore, this chapter will be interesting for politicians
and decision makers in the HE sector. Lastly, this chapter seeks to invigorate critical
discussion among academics, the ultimate executors of any strategy in HE.
14 The Application of a Balanced Scorecard in Higher Education Institutions:. . . 421
14.2 A Short Presentation of the Warsaw University of Life

Sciences
The origins of the Warsaw University of Life Sciences (WULS) date back to
1816. With its over 200-year tradition, this school is one of the oldest universities
in Poland. Presently, the university consists of 13 faculties (Agriculture and
Biology; Veterinary Medicine; Forestry; Horticulture and Biotechnology; Civil and
Environmental Engineering; Wood Technology; Animal Breeding, Bioengineering,
and Conservation; Economics; Food Technology; Human Nutrition; Production
Engineering; Sociology and Education; and Applied Informatics and Mathematics).
Due to the latest national reform of the HE system, a matrix structure was
introduced. While the faculties are responsible for teaching, research activity is
managed across 18 institutes. However, this is only a recent change, which had no
impact at the time when the strategy and BSC were prepared (Report by professor
Wiesław Bielawski, rector of the WULS, 2020).
There are about 18,000 students (including 590 foreign students, as well as
PhD and postgraduate students) enrolled at WULS, who are taught by nearly 1200
academic staff. WULS’ share in the whole sector is around 1.5%, and within the
group of life sciences universities, its share is nearly 32%, as measured by the
number of students. WULS cooperates with over 275 foreign universities from
50 countries all over the world. Since 1990, WULS has actively participated in
European programs as a university from an associate, candidate, and—since 2004—
EU member-state country. The university is a member of international organizations
such as the Euroleague for Life Sciences (ELLS), the European Universities
Association (EUA), the Interuniversity Consortium for Agricultural and Related
Sciences (ICA), and many others. The 70-hectare campus of WULS, located in
the southern part of Warsaw, has a historical part, featuring an eighteenth-century
palace, and a contemporary part, where most of the faculties’ buildings along with
the 14 dormitories (4000 rooms), library, and sports center are located (Facts and
Figures, 2020; Report by professor Wiesław Bielawski, rector of the WULS, 2020).
The faculties of WULS are differentiated not only by research area but also by
other dimensions such as size (measured by the number of academic staff employed)
and outputs, as measured by the number of students enrolled or the number of
score points assigned to research publications (Table 14.1). For many years, the
governments and bureaucrats responsible for HE have made attempts to determine a
“proper measure” of research output, while still taking into account research quality.
Recently, two special lists were published by the Ministry of Science and Higher
Education. Based on the first list, score points were assigned to nearly 30 thousand
journals worldwide. For having a paper published in any journal from that list, a
researcher can receive from 20 to 200 points. The second list consists of editing
houses: For a monograph published by an editing house from this list, 80 to 200
points are available (split up in the case of an edited book). Score points are an
422 M. Pietrzak
Table 14.1 Differences between the faculties of WULS as of 2019

Academic Students Research vs.
staff Students Score per staff teaching
Faculty [persons] [persons] points [#] ratio [#] orientation [#]
Veterinary Medicine 155 1227 13,157 8 11
Civil and Environmental 155 1844 10,154 12 6
Engineering
Economics 145 2943 10,514 20 4
Agriculture and Biology 134 1113 15,598 8 14
Human Nutrition 112 1447 14,444 13 10
Food Technology 92 1158 8469 13 7
Forestry 78 1297 7431 17 6
Animal Breeding, 74 941 4081 13 4
Bioengineering, and
Conservation
Horticulture and 58 1093 3709 19 3
Biotechnology
Production Engineering 57 834 5390 15 7
Wood Technology 49 577 3670 12 6
Applied Informatics and 47 1054 3157 22 3
Mathematics
Sociology and Education 42 789 1973 19 3
Total/mean 1198 16,317 101,747 14 6
Total/mean in Poland 93,100 1,230,300 na 13 na
Source: Report by professor Wiesław Bielawski, rector of the WULS, 2020, pp. 13–14, 104, 242–
243, and author’s calculations (last two columns)
element of both the individual assessment of the staff and the evaluation of the
HEI. In the second case, the portion of public funds can be adjusted accordingly.
Nevertheless, the dominant stream of governmental money is still tied to teaching
activity.
According to Warning (2004), universities make the trade-off between an
emphasis on teaching or research activity. Following this, I calculated two indexes:
the number of students per academic staff and the score points per academic staff.
When we divided the second by the first, we derived an indicator called research vs.
teaching orientation (the low value of this measure indicates relatively more focus
on teaching, while a high value indicates a greater focus on research). One can
find substantial differences in both these orientations across the faculties of WULS
(Table 14.1 and Fig. 14.1). Some of these differences are evidently more oriented
toward teaching, while others rely much more on research. Still, some of them exist
in the middle without a clear orientation.
Appl. Informacs 24 y = 35,259x-0,538
&Mathemacs R² = 0,7571
22
Economics 20
Horculture 18
Sociology &Biotechnology Forestry
&Educaon
16 Producon
Engineering
1 2 3 4 5 14
6 7 8 9 10 11 12 13 14 15 16
Food Human
12 Technology
Animal Breeding, Nutrion
Bioengineering Wood
&Conservaon
STUDENTS PER STAFF RATIO

10 Technology
Civil &Environmental
Engineering
8
Agriculture
6 Veterinary &Biology
Medicine
Teaching Research
14 The Application of a Balanced Scorecard in Higher Education Institutions:. . .
ORIENTATION ON
Fig. 14.1 Diversity of the faculties of WULS. (Notes: (1) The sizes of the bubbles depict the scale of the faculties as measured by academic staff; (2) Colours:
dark grey denotes A or A+ category (the best); light grey denotes B category (medium); no one faculty belong to the C category (the worst). Categories are
assigned by the government on the basis of an evaluation of scientific activities done over four years. Source: Based on data from Table 14.1 and the author’s
423
notes from strategic meetings)

424 M. Pietrzak
14.3 Context of BSC’s Introduction
To explain the context of BSC’s implementation at WULS, this section provides a

short overview of the situation of HE in Poland. Higher education in Poland has
a nearly 660-year-long tradition, with the origins of the Jagiellonian University
(Kraków) dating back to the middle ages. In the last two centuries, the Polish HE
sector has been under the strong influence of the German model of Humboldtian
university based on Humboldt’s ideas of a holistic combination of research and
teaching, as well as autonomy and academic freedom from outside constraints.
After WWII, in Poland, as in other countries of the region, attempts were made
to introduce the Soviet model of HE (publicly held entities centrally planned
by the national bureau/ministry, dual control of the state administration and the
communistic party apparatus, the use of political loyalty criterion, separating part
of the research into an academy of science, and strong involvement in manpower
planning). Nevertheless, the strongly embedded Humboldtian ideals, as well as the
independent attitudes of many academics ensured some margin of freedom and
autonomy. For example, the only private university in the whole communistic bloc
survived in Poland—the Catholic University of Lublin (Clark, 1983; Leja, 2013).
Thus, during the transition after emancipation from Soviet dominance, the
laws were changed to reinforce Humboldtian autonomy combined with a laissez-
faire policy of government. The Higher Education Act, 1990 was very liberal.
This law reinforced academic freedom and autonomy, not only in government vs.
HEI relations but also in the internal regimens of HEIs. For example, faculties
received a high degree of autonomy and independence from rectors. Moreover, the
establishment of private HEIs was allowed, as well as the partial introduction of
tuition fees in public entities (only for part-time students).
The number of private HEIs then soared tremendously, from only one non-public
(catholic) university before 1990 to 328 HEIs held privately by the academic year
of 2010/2011. In this peak period, private HEIs accounted for 71% of the whole
system, as measured by the number of entities (Godłów-Legi˛edź, 2016). However,
private HEIs are typically small when measured by a number of students. Poland
also witnessed a rapid massification of HE. This was not only due to the soaring
number of entities established in the private sector but also due to the broadly open
gates to public universities, particularly in the form of very popular part-time studies
(payable). The percentage of young people studying in the two decades after the
transition increased tremendously from nearly 13% in 1988 to slightly over 51% in
2007, when the number of students peaked and approached 2 million (Pietrzak &
Gołaś, 2018). The highest ratio of nearly 54% was recorded in the academic year
2010/2011 (Higher education institutions and their finance, 2019).
However, such rapid massification came at a price. The quality of academic
education inflated, and job security and rates of return from HE shrank compared to
the first years of the boom. Moreover, the problem of demographic collapse began
to increase. Just a decade after the aforementioned peak, the number of students
dropped sharply by 30% (Pietrzak & Gołaś, 2018) and still continues to decrease.
Recently, the percentage of young people studying was recorded as 46% (Higher
education institutions and their finances, 2019), but one should note that the base
is lower due to demography. This situation shrank the amount of private money in
the system. During the shortage in the supply of tuition-free places in public HEIs,
payments for private education or public part-time study solved (in some way) this
access problem. However, this demographic collapse transitioned the sector into an
oversupply situation. Every year, some privately held HEIs are canceled from the
register or go bankrupt. Notably, the peak in the number of private HEIs has a 3-
year delay compared to the peak in the number of students. Nevertheless, 10 years
after the peak, the number of private HEIs also dropped by nearly 30% (Godłów-
Legi˛edź, 2016; Records of private universities, 2020). Thus, today, the market share
of private HEIs has fallen to 64%, as measured by the number of entities, and to
26%, as measured by the number of students (Higher education institutions and
their finance, 2019).
The problem with demographic collapse overlaps with the relatively low expen-
ditures on HE in Poland. Such expenditures, calculated as a ratio of the GDP, are
only slightly below the OECD average. However, given the relatively low GDP
per capita and the relatively high number of students, the annual expenditures per
student remain very low. For example, in 2013, the annual per-student expenditure
in Polish HEIs was less than USD 9000 compared with the OECD averages of nearly
USD 16,000 (Peer Review, 2017).
The revenues of public HEIs in Poland in 2018 accounted for nearly 22 billion
PLN,1 from which 15.7 billion (72% of total revenues) came in the form of subsidies
from the central government. These subsidies are divided into funds for maintaining
teaching capacity (62% of total revenue) and funds for research activity (10%
of total revenue split nearly equally between a direct subsidy connected with the
assessment of the university and grants received on the basis of a competition for
projects). The rest (28%) of the revenues come from tuition fees from part-time
students (11%), international grants, business activities conducted by the HEIs, and
other sources (Higher education institutions and their finance, 2019). Governmental
control over public HEIs is twofold. Firstly, laws regulate the functioning of HEIs.
Secondly, the government steers the money flow. The second aspect is not executed
in any discretionary manner by the ministry but is instead regulated by general rules.
Moreover, the large money flows are quite inertial due to their connection with such
parameters as the number of students. Thus, paying for individual HEI performance
is possible in only a limited scope.
Due to the inflation in the status of graduates mentioned above, some social and
political pressures have prompted more central control over the HE sector. As a
sociological survey showed, the social prestige of the profession of a university
professor has decreased. This occupation was, in the past (e.g., 1975), ranked at the
highest level (90 points on a 0–100 scale), while presently, it is ranked at 80 points
and has been overtaken by such professions as fireman (88 points), nurse (81 points),
1 At the end of the year 2018, the exchange rate was 1 PLN = 0.2660 USD.
426 M. Pietrzak
and coal miner (81 points) (Which professions do we respect?, 2019). There are
growing concerns about the labor market mismatches and increasing unemployment
of graduates. Critics emphasize that despite variations across HEIs, public higher
education provisions generally remain supply-driven and poorly aligned to the
needs of the labor market. The design and delivery of curricula are based more
on academic capacity rather than the demands of the economy (Peer Review, 2017).
Thus, the idea of an entrepreneurial university has started to become more
popular. Consequently, many opinion leaders and politicians have begun to advocate
the need to abandon the Humboldtian model for the more promising (in their minds)
model of the entrepreneurial university. The important reforms of the HE sector in
Poland undertaken in the last decade (Act, 2011; Act, 2018) were designed more
or less in this vein (Kwiek, 2012; Kwiek, 2017; Pietrzak & Gołaś, 2018). The Act,
2018 yielded some radical changes, which are new and remain difficult to fully
assess. From the perspective of our case study, the most crucial act was Act, 2011
and its further amendments, which imposed the obligation of introducing formal
strategies for HEIs.
14.4 Choosing the General Approach
The preliminary projects of changes introduced by Act, 2011 were known in

advance. The rector of WULS, anticipating these new obligations, delegated the
issue of elaborating a strategy to the Rector’s Commission of Development (RCD).
The RCD is a collegial body (it consists of about 20 people from many units
of the university) with competence in advising the rector. I was invited to this
commission in the late 2009 as an expert due to my experience in the area of strategic
management and Balanced Scorecard (BSC). In the following years, I became a
member and vice-chairman of this body.
I suggested that RCD work on a strategy using the project management formula.
However, before starting the project, some preliminary decisions and assumptions
had to be made. The most important issues were the scope, the methodology,
and positioning of the project (and the strategy’s execution process) within the
organizational structure.
I conducted a survey among top executives of the university. The survey was
about the maturity of the strategic management process. I have used to use this
survey in business consulting. It consisted of 17 questions related to strategic
thinking and execution practices. The possible answers were “no, we certainly have
not used such a practice at all”; “no, we rather have not used such a practice”; “yes,
we rather have used such a practice”; and “yes, of course we have used such a
practice”, with nonlinear scoring of 0, 1, 5, and 10 points, respectively. Thus, the
results ranged from 0 to 170 points. Typically, in business organizations, which I
consulted the results were around 70–90 points. The average result for WULS was
30 points, indicating evident immaturity in the strategic management process.
Based on this result, I suggested two scenarios for the project: an optimal
one and an easy one (regarding the scope and assignment of tasks across the
organizational structure). In both cases, I suggested the BSC as a methodological
basis. In the optimal scenario, I assumed deep involvement of the faculties in
strategic formulation and execution.
As mentioned above, during the study, faculties held a strong and independent
position, with deans elected by the academic community (staff and students) of a
particular faculty. A typical diversified public university in Poland at that time was
a federation of relatively autonomous faculties. This indicated that the execution of
a strategy for the whole university should rely strongly on the engagement of the
deans and the rest of each faculty’s staff. Moreover, from a strategic point of view,
particular faculties operate under different “market” environments consisting of
different types of “customers” and “competitors.” They thus have different strengths
and weaknesses against their particular environments. This fact highlighted the clear
analogy between faculties and strategic business units, as well as between the rector,
other top executives, central administrative staff, and headquarters in diversified
corporations.
In the strategic management literature, it is strongly emphasized that strategic
business units are the parts of any corporation where value is directly created. The
headquarters do not directly create value; they create costs. If any value is created
by the headquarters, it can only be created indirectly through better coordination
of subordinated business units and looking for synergies between them (De Witt
& Mayer, 2005; Kaplan & Norton, 2004; Koch, 1995). From this point of view,
faculties should have their own strategies. However, the question arises: How can
we interconnect the strategies of both layers—the university and the faculty? Four
options were identified: a top-down approach, a bottom-up approach, a down-up
approach, and a federative/coalitional approach (Fig. 14.2).
In the top-down approach (Fig. 14.2a), vertical coordination is achieved by
imposing a strategy for subordinate units. In this method, strategic alignment
can be achieved relatively easily. However, this method does not work when the
lower layers experience substantial autonomy and are able to resist the expected
changes. Due to the strong autonomy of faculties, the top-down approach was
not suitable for WULS, like many other Polish universities at that time. On the
opposite end is the bottom-up method (Fig. 14.2b), in which faculties elaborate
autonomous strategies on their own. Then, the necessary work in the central layer
is to synthesize and integrate particular strategies into a strategy for the whole
university. The potential advantage of this approach is engagement of the faculties
and their staff, the establishment of more achievable and locally oriented goals,
and lower resistance to changes. However, this method requires considerable work
to implement, complicated by the prerequisite of the high strategic maturity of
the lower layers, problems in achieving internal cohesion, and political issues (the
faculties were already very strong and independent, making the rector unwilling to
give them even more power).
The down-up (top-down/bottom-up) approach (Fig. 14.2c) mixes the two meth-
ods described above. At the top layer, the global assumptions and general strategic
428 M. Pietrzak
Fig. 14.2 Possible approaches to the vertical coordination of strategies between the whole
university and faculty layers. (Source: Own)
intent are defined, while at the bottom, detailed analyses are done, and abilities
and limitations are recognized. In this way, strategies in both layers (university
and faculty) can emerge as the discussions and negotiations between the two layers
progress. Alignment and internal consistency are achieved through engagement and
open discussion of the two layers.
Such an approach is strongly recommended in the literature of Hoshin Kanri
(the Japanese method of strategic management) and is called “catchball.” This
term signifies the give-and-take process between and among organizational layers
and refers to collaborative goal setting basing on such dialog, which is crucial
for people’s commitment (Babich, 2002; Cowley & Domb, 1997). The word
“catchball” assumes “tossing ideas” (e.g., strategies, metrics, and goals) back and
forth to finally find those that are mutually agreeable (Cowley & Domb, 1997).
“Catchball” can shed light on possible problems in the capabilities needed to
execute a given strategy. If a capability required does not exist, then a breakthrough
may be needed. However, the commitment to achieving this breakthrough must be
counterbalanced by maintaining control of business fundamentals (Babich, 2002;
Cowley & Domb, 1997). The alignment and internal integrity achievable using
“catchball” is far beyond the reach of both the top-down and the bottom-up
approach. However, “catchball” can be a time-consuming process. Moreover, it
requires an attitude of honesty and openness at the bottom, as well as a willingness

to listen and trust on the top, which can represent a challenge for the corporate
culture of some organizations (Cowley & Domb, 1997).
By a federative approach, I mean acceptance of the loose ties binding the major
building blocks united under one umbrella. As Clark (1983) explains, universities
operate more like federations or coalitions rather than unitary bureaucratic systems.
From this point of view, the whole university is actually a kind of holding company
for groups engaged in various areas of expertise.
In the strategic management literature, in opposition to the active attempts to
integrate strategies in the company and strategic business unit layers (compare
the three methods discussed above), an approach typical for holding structures is
also described. De Witt and Meyer (2005) called this the financial control style.
Instead of seeking to strictly coordinate the strategies of highly independent units,
the headquarters of a holding structure typically limit themselves to imposing some
financial goals and restrictions and then simply control them. As mentioned above,
the public HEIs in Poland relies heavily on financial support from the government.
In the money flow from the government to faculties, the rectors and senates operate
as gatekeepers.
Moreover, for HEIs, the symbolic side of the university (culture) can also be
considered a kind of a “glue” connecting autonomous faculties (Clark, 1983). As
Etzioni (1961) stated, one should make a distinction between coercive organizations
(e.g., prisons), instrumental organizations (e.g., business firms), and the normative
organizations (e.g., universities and churches). In the last group, a common attach-
ment to ideas predominates over forced compliance and monetary rewards (Clark,
1983). Thus, even without tight ties specific to hierarchical business organizations,
and without fully integrated strategies, autonomous faculties could still build a
federation under one university umbrella based on financial and cultural relations
(compare Fig. 14.2d).
Therefore, as noted previously, two scenarios were proposed: an optimal one and
an easy one. In the optimal scenario, I assumed deep involvement of the faculties
in strategy formulation and execution by following the down-up (“catchball”)
approach. However, this ideal approach would come at a cost. I expected that a
large amount of time, the involvement of leaders and staff, and support from external
consultants would be needed for this method. I prepared a very preliminary draft of
such a project and expected that it would take 1.5–2 years to fulfill the tasks based
on the serious involvement of many people.
I also presented an alternative in the form of an “easy” scenario. In this approach,
the strategy would be prepared only at the university level by the efforts of the
RDC members and invited experts from the staff. The issues of the faculties’
strategies would be saved for future decisions. In this way, due to the efforts of a
limited number of people (mostly members of RDC), we could develop a strategy
within half a year, and we would produce a document based on a state-of-the-art
methodology (BSC model), which will fulfill the obligations of the law. Moreover,
by doing this, we could also ensure the option to further develop the strategic
management system. In other words, this work not only would allow us to follow
430 M. Pietrzak
external obligations but would also give us the right (but not the obligation) to
further expand the strategic management system if needed. It was decided that
WULS would follow the second scenario, despite not being the most optimal
method.
14.5 Defining the Project
After the main approach to the strategy was clarified, the RCD started to work on
the project “Development of the WULS strategy based on Balanced Scorecard.”
Authorities of WULS declared that they were not fully satisfied with the current
documentation of the strategic assumptions for the university’s development. Thus,
they formulated in the project charter an expectation that the transparency and
consistency of the assumptions underlying strategic development had to be updated
and improved. Additional expectations concerned building a methodological basis
for the eventual future development of the strategic management.
Regarding the project scope, these results were planned to be achieved by
conducting a SWOT analysis, defining the mission, vision, and expression of these
assumptions in the form of a strategy map of WULS along with a set of measures
for strategic goals. To fulfill the abovementioned expectations and to complete these
tasks, the following roles were defined as the responsibility assignment:
• Project sponsor (rector): Ownership of the project, final acceptance of products /
milestones of the project, and building support for the project.
• Steering committee (RCD): Strategic control of the project and pre-approval of
the project outputs / milestones.
• Project manager (former rector): Direct organization, coordination, and control
of the project, supplying results, and editing the final document.
• Project expert (the present author): Providing know-how in the field of strategic
management and BSC and editing final document.
• Project team (members of RCD and invited specialists from the staff): Obtaining
and analyzing the necessary information.
• Project secretary: Documenting the work of the team.
14.6 Mission, SWOT, and Vision
For any analysis, the preliminary step should be to answer the following questions:
What is the analysis for, and what should be the scope of the analysis? Thus,
before a strategic analysis of any organization is carried out, the raison d’être
of such an organization and the basic scope of its activities should be defined.
In other words, the whole process of strategic management should begin with
a formulation of the mission. The mission is understood as the purpose of the
organization, which describes why it exists. The mission typically defines the scope
of the organization’s activities and/or identifies its fundamental needs, which is
fulfilled by specifying what the organization provides to customers and clients or
citizens and beneficiaries (Babich, 2002; Cowley & Domb, 1997; Kaplan & Norton,
2008). Missions are typically supplemented by value statements that define the
fundamental beliefs, attitudes, and characteristics that should drive decision-making
and behavior (Cowley & Domb, 1997; Kaplan & Norton, 2008).
Interviews were conducted with the rector and vice-rectors, the chancellor, and
living former rectors regarding their perception of the raison d’être of WULS, the
desired scope of the disciplines covered, and the beneficiaries served. They were
also asked about how they perceived the school’s fundamental values, which they
believed should drive the actions of the academic community of WULS. The results
of this survey were discussed by the project team; following this discussion, a draft
statement of the school’s mission and values was prepared. After acceptance by the
rector, this draft was voted by the senate as part of the strategy document. The final
version is presented at the top of the strategy maps in Fig. 14.5.
The next step was a strategic analysis prepared based on the SWOT (Strengths-
Weaknesses-Opportunities-Threats) framework, which is one the most popular
strategic tools for managers. This tool provides an overall picture of what a
particular factor means for an organization, as strengths, weaknesses, threats, and
opportunities must be framed by intentions (Spender, 2014). This is why the mission
should be defined first. According to the seminal paper of Lewin (1947), any social
situation can be seen through the lens of force-field analysis as the quasi-stationary
equilibrium between driving and restraining forces. If we additionally consider the
internal and external characteristics of such forces, we can easily understand the
essence of SWOT. Strengths are internal resources, processes, and other issues in
which the organization is proficient and, therefore, could serve as strategic leverage
for mission fulfillment. Weaknesses represent the opposite—obstacles to mission
fulfillment, which close the windows of opportunities and expose the organization
to threats. Opportunities and threats are forces in the external environment of
organizations and are perceived as supporting and hindering, respectively, the
mission fulfillment.
The project team identified 216 SWOT issues, including 54 strengths, 84
weaknesses, 31 opportunities, and 47 threats. Overall, this assessment was rather
pessimistic (weaknesses and threats accounted for 61% of all issues) and internally
focused (strengths and weaknesses were 64%). The typical problem with SWOT
analysis is that the analytics often define issues in a very general manner—For
example, an organization might note that “our strength is our people.” However,
what makes those people a strength? What do we expect to get from them? A much
432 M. Pietrzak
Table 14.2 Excerpt from the SWOT issue list and the explanation of how each issue was used
in defining the strategic objectives in the strategy map
Strengths Weaknesses Opportunities Threats
• Localization • Localization in • Low costs • Strong position of
in capital city capital city where the of studying in private HEIs in very small
where big costs of living are Poland relative to cities which never had
corporations and high (fact)—relative the countries of academic traditions before
central cost advantage of the “old EU” 1990 but which are proximal
administration smaller academic (fact)— to the parents’ households
offices are also centers against us international cost (fact)—cost advantage even
located (fact)— (implication) advantage against regular public
competitive • Limited capacity against western curricula (no fees) when
advantage against of dormitories European HEIs taking into account the cost
smaller academic (fact)—difficulties in (implication) of living in the big city and
centers attracting candidates even more in the case of
(implication) from remote regions payable part-time curricula at
(implication) public HEIs (implication)
↓ ↓ ↓ ↓
Strategic objective I7: Decreasing the total costs of studying
Source: Own elaboration based on notes from project team meetings
more useful statement of strength could claim that due to the depth of experience,
the sales force gives an organization credibility in the market, which makes it more
likely to attract new customers (Smith, 2007). In such a statement, there are two
components: the fact (experience of salespeople) and the implications of the fact
(the credibility and likelihood of new accounts). We followed the suggestions of
Smith (2007) and formulated each SWOT issue statement in this way (compare
Table 14.2). We also divided all SWOT issues across perspectives of the BSC.
Following both these suggestions aided subsequent steps of the project.
The vision depicts an ideal future that is inspiring and empowering. The essence
of such a statement is to articulate what an organization wants to be when it matures
in a 3 to 10 years horizon (Cowley & Domb, 1997; Kaplan & Norton, 2008; Smith,
2007). Kaplan and Norton (2008) proposed an approach to the vision statement,
which they call “enhanced vision.” This approach provides a comprehensive picture
of how an organization wants to be perceived based on the basic BSC architecture
considered from four perspectives.
We followed this idea but modified the original model of the BSC. According to
Niven (2008), the BSC, with some modifications, could be a useful tool for public
sector organizations. It could be particularly useful in HEIs, which is widely shown
in the literature (Asan & Tanyas, 2007; Juhl & Christensen, 2008; McDevitt et al.,
2008; Papenhausen & Einstein, 2006; Tapinos et al., 2005; Umashankar & Dutta,
2007). Regarding universities in the United Kingdom, Taylor and Baines (2012)
emphasized that the BSC model should be adapted to the specific mission and
circumstances of any HEI.
Due to the specific nature of public universities, it is difficult to treat the financial
perspective as the ultimate definition of success. Thus, the commercial logic of the
original BSC model should be modified. One suggestion of the authors of the BSC
concept (Kaplan & Norton, 2004) is that the mission should move to the top of
the framework, thereby replacing the financial perspective as the ultimate definition
of success. This is because the success of public and nonprofit organizations is
defined by public/social impacts according to their missions. Such organizations
make attempts to meet the needs of people who benefit from the services supplied by
them, who could be called their customers (or, more accurately, stakeholders). Thus,
Kaplan and Norton proposed to elevate the customer (or stakeholder) perspective
to the same level as the financial one (renamed as the fiduciary perspective). In
this way, both perspectives should be treated as equally important. The customer
and fiduciary perspectives are supported by two further ones: the internal process
perspective and the learning and growth perspective.
Niven (2008) offered a different modification. His proposal is only slightly
similar to that of Kaplan and Norton, as he also suggested moving the mission to
the top of the BSC framework. However, in comparison to the commercial model
of BSC, Niven proposed a reverse framework. In Niven’s adaptation, the customer
perspective is elevated while the financial perspective moves to the bottom of the
framework. Elevating the customer (or stakeholder) perspective signals that any-
thing done from other perspectives is done to support the customers/stakeholders.
On the other hand, the achievement of financial goals is not the ultimate goal but
will help serve stakeholder demands and, therefore, provide a means of fulfilling the
mission.
According to Smith (2007), the choice of a proper modification depends on the
specific culture and circumstances of any particular organization, as an organization
should choose the framework of the BSC that facilitates explaining its strategic
intent in the desired way. This reasoning also extends to the names and even the
numbers of perspectives (Smith, 2007). According to Taylor and Baines (2012),
British universities adopted the names of the original perspectives of the BSC and
used their own labels.
The project team chose Niven’s (2008) proposal. The mission was thus moved to
the top as the ultimate definition of success for WULS. The customer (stakeholders)
perspective was elevated, while the financial perspective was moved to the bottom
of the framework. Moreover, we changed the names of the perspectives: “cus-
tomer” into “stakeholder” and “learning and growth” into “potential perspective.”
Figure 14.3 presents the enhanced vision of WULS in 2020 as it was defined ten
years ago within a modified architecture of perspectives.
434 M. Pietrzak
Perspective What we want to look like in an ideal future
Better quality of candidates while maintaining their numbers

Stakeholders More research projects, including large and prestigious projects
Stronger bonds with business practice as a partner in research and advisory services
Greater student mobility and offering competences useful on the labor market
Making the university more student-friendly
Increased activity in the academic community at home and abroad, greater
Processes
international visibility of research
Transfer of knowledge to the business as an equivalent sphere of activity to
teaching and research
Academic staff—mobile, ensuring a dynamic academic career

Management of the University—continuation of the changes transforming WULS
into a modern third generation university united around its traditional core of life
Potential
sciences; effective implementation of the strategy and change management
Infrastructure—further development of research, teaching, and social facilities, the
provision of an R&D unit and adequate development of IT infrastructure
Stable financing of activities due to an increase in revenues other than subsidies

Financial from the Ministry of Science and Higher Education and due to the improvement of
cost effectiveness
Fig. 14.3 Extended vision of the Warsaw University of Life Sciences in 2020. (Source: Own)
14.7 Strategy Map
Due to the traits of the human brain and its limited capacity for processing and
memorizing information, human actions are plagued by bounded rationality (Simon,
2000; Witt, 1998, 1999, 2005). To cope with this limitation, people use cognitive
frames. The organizational strategy is always a multi-person activity based on
collaboration. Thus, leadership can be seen as a provision of cognitive frames. A
strategy expressed in any way is a kind of cognitive frame that must be transmitted
to and adopted by members of an organization to successfully execute that strategy
(Witt, 1998). Therefore, the strategy must be based on a shared mental model or
shared organizational theory-in-use (van der Heijden, 1998).
Any novel idea or conception requires acceptance from those who could
implement that idea. For it to gain acceptance, the strategy must be communicated.
Moreover, before people can execute the strategy, they must accept it (Kaplan &
Norton, 1996, 2001; Niven, 2005). Therefore, a coherent story (a narrative that
explains what is happening and how the current situation will be changed or
improved) is needed to induce the commitment needed to execute the strategy.
Strategic narration should express the aspects (see Fig. 14.4) that are crucial
for survival and development (van der Heijden, 1998). In the BSC methodology,
this story is the content of the strategy map (Kaplan & Norton, 2004). This story
can be understood as a one-page picture that shows the paths of the organization’s
movement from the present situation to the desired future. Strategy map outlines
Fig. 14.4 Detailed BSC framework for HEI adopted by WULS. (Source: Own)
what the organization has to do to successfully execute its strategy (Niven,

2008). The crucial feature of a strategy map is its simplicity—a one-page picture
visualization depicts the story that explains how the organization defines its success
and signals to employees what must be done (Niven, 2008; Smith, 2007). A strategy
map can be seen as a communication tool and cognitive pattern for members of
the organization and is helpful for building commonly shared mental models of the
organizational destination (Pietrzak, 2014a, 2017).
The strategy map consists of two elements. The first element involves the
strategic objectives put into the map as ovals with short statements inside. Objectives
are linked together through cause-and-effect relations (represented in the map by
arrows). They are not treated as independent elements here but as a whole that
expresses the overall strategic intent of the organization. Such a map creates a
visualization of the strategy, highlighting the path necessary for the successful
implementation of the strategy.
In drawing the strategy map, the project team used the results of the SWOT
analysis. These results were useful for outlining both facts and implications of the
facts. As mentioned above, the SWOT issue statements were divided across four
perspectives. In each perspective, they were further clustered into coherent groups.
Clustering was based on the “fact part” of the statements in such a way that the
grouped issues could cohesively fall under an umbrella of a common problem to be
solved. Then, a short statement about the desired direction of action was formulated
and put into an oval (for overcoming weaknesses and threats, leveraging strengths,
436 M. Pietrzak
(a) STRATEGY MAP (b) MEASURES
The mission of the WULS is to serve the economic and intellectual

development of the Polish society, focusing mainly on the rural areas,
agribusiness and natural enironment. Our purpose is to excel in scientific
research, teaching and transfering knowledge.
Our identity is built on such values as: professionalism, quality assurance,

hard-working and innovation
I1. SECURE THE STAKEHOLDERS PERSPECTIVE I1M1: Quality Index of the Admited Candidates
QUALITY OF
STUDENTS I2. CREATE
LOYALTY AND I2M2: Graduate Satisfaction Index
SATISFACTION I2M2: Graduate Loyalty Index
VALUE
PROPOSITION
I5M1: Employers Preference
I5. ENHANCE
I6. DEVELOP THE I5M2: Graduates’ Employment Rate
POSITION OF
IMAGE OF STUDENTS- I8. CONFIRM
GRADUATES ON
FRIENDLY UNIVERSITY HIGH QUALITY I6M1: Image Compliance Index
THE LABOUR
FOCUSED ON WITH
MARKET
PRACTICAL TEACHING ACREDITATIONS
I8M2: Evaluation of State Accreditation Comission
PROCESSES PERSPECTIVE P1M1: Index of Intensity of IT-based Teaching

P1. ALIGN CURRICULA
WITH EXPECTATIONS P1M2: Index of Intensity of Practical-Oriented Teaching
OF LABOUR MARKET P3. SHAPE STUDENTS-
FRIENDLY ACADEMIC P3M1: Index of Academic Staff Availability
CLIMATE
P3M2: Index of Students’ Activities
Po3. INCREASE POTENTIAL PERSPECTIVE

THE QUALITY Po3M1: Progress in Development of Teaching Quality System
Po6. DEVELOP THE
OF TEACHING
INFRASTRUCTURE
FOR EDUCATION
Po6m1: Index of Library Resource (traditional and digital)
FINANCIAL PERSPECTIVE
Fig. 14.5 Excerpt from WULS strategy map (Theme 1: Improving Teaching). (Note that the
measures were indexed as follows: The first part of the index denotes the perspective (I—
Stakeholders, P—Processes, Po—Potential) and the ordinal number of the objective in this
perspective; the second part of the index denotes the ordinal number of the measure tied to this
objective, e.g., P1M2 denotes the second measure of the first objective in the Process Perspective.
Source: Own)
and seizing opportunities; compare Table 14.22). Secondly, when the list of strategic
objectives was ready, the project team drew arrows that illustrated the expected
cause-and-effect relations between the strategic objectives. To perform this task,
the “implication of the fact” portions of the SWOT issue statements were useful.
Based on the modified framework of BSC (Fig. 14.4) and following the method
outlined above, the strategic map of WULS was drawn, and 216 of the SWOT issue
statements were transformed into 30 interrelated objectives on the map (Borecki
& Pietrzak, 2010). Figure 14.5a depicts an excerpt of this strategy map. Kaplan
and Norton (2004, 2008) suggested dividing the strategy map into several (four to
six) building blocks, which they called “strategic themes.” Each of these themes
supports strategy execution in a different but complementary way. We followed this
2 Objective I7 (an example from Table 14.2) was abandoned during one of the strategic reviews.
This objective was formulated using the logic of competition with a private HEI. However, as
explained in Sect. 14.9, the incentives created by the government generated the opposite logic. The
arbitrary limit of the students per academic staff ratio enforced by the government created a lack
of interest among public HEIs in gaining a “market share” in the teaching sector.
proposal by identifying five such components: theme3 1—Improve Teaching, theme

2—Excel at Research, theme 3—Cooperation and Internationalization, theme 4—
Enhance Cooperation with Business, and theme 5—Finance and Administration.
Figure 14.5a illustrates one such theme.
14.8 Operationalization and Further Development
To make the strategic objectives more operationalized, one or more measures were
assigned to each of them (Fig. 14.5b). The essence of any measure is to answer the
question, “How will we know that we are moving where we want to go?”. Firstly,
I prepared a long list of possible measures based on my consulting experience,
a review of the literature, and a discussion with the project manager. In the first
round of assessment, every member of the RCD offered his/her own rating (for a
particular objective) of the proposed measures. It was also possible to put into the
list own proposal of a measure. Then, a shortlist of measures was created, which
consisted of two (per objective) winners of the previous round, as well as additional
proposals included in the list by relevant members (where applicable). In the second
round, all measures from the shortlist were discussed and finally ranked again. The
winners became part of the BSC. In cases where consensus was not achieved or
where serious difficulties in measurements existed, two measures per objective were
allowed. The BSC of WULS ultimately consisted of 30 objectives and 49 measures.
Following Niven’s (2002) advice, we prepared a dictionary of measures, a
document that provides all members of the organization involved with a detailed
examination of every BSC measure. We used this dictionary to define such aspects
as the measures and their formulae (how they are calculated), the rationale for using
each measure, and each measure’s polarity and denomination.
The operationalization of the strategy did not go beyond assignment and
definition of the measures. There were no targets established with detailed timelines.
The majority of RCD members argued that for many measures, the required data
were not collected, so the information on such measures should first be collected and
then interpreted before setting targets. In the opinions of these members, gathered
information should be the baseline for establishing targets in the future. As the
years have passed, reality has shown that this was only partially true. Only some
of the objectives had targets assigned, and even in these cases, those definitions
took the form of referential values due to a lack of timelines. There were no
strategic initiatives (projects focused on target achievement) defined. Such projects
are collections of finite-duration discretionary tasks outside the organization’s day-
to-day operational processes that are designed to help an organization achieve its
targets (Kaplan & Norton, 2008).
3 Thus, the hierarchical structure of the strategy is as follows: mission and vision-strategy map-
strategic themes-strategic objectives-measures of objectives.

438 M. Pietrzak
The mission statement, the extended vision, the strategy map, and the dictionary
of the measures were subject to voting by the senate. After voting positively, this
document began to be communicated to the academic community and external
stakeholders. The strategy was printed by the WULS publishing house (Borecki
& Pietrzak, 2010) and was also available on the website of the university.
Meanwhile, the faculties of WULS were obliged to initiate their own efforts in
preparing strategies. This process, however, was not centralized. Faculty authorities
engaged in this process individually, based on the foundations of the WULS
strategy. Considering the discussion above, this process was conducted in a loose
federative manner (compare Fig. 14.2d). There were no obligations to use any
specific framework or methodology. The only central requirement for the faculties
was to develop a strategy compatible with, and supportable by, the top strategy. Up
to the end of 2013, all 13 faculties developed their own strategies. However, there
was no process to formally control the coherency of the strategies in these two layers
(university vs. faculties).
The assignment of measures to the strategic objectives allows one to report
performance and monitor the progress of strategy implementation. After its approval
in 2010, every year, the progress in strategy execution, as indicated by the measures,
was reported to the senate by the RCD. There were some changes made as time
passed. In 2013, the strategy was modernized due to an amendment of the law. In
the new map, developed after the update of the strategy, the number of objectives
was reduced to 27. However, the number of measures increased to 57. Meanwhile,
an increase was observed in consciousness of the need to reduce the number of
objectives and measures. Some of the measures were abandoned due to the yearly
reviews.
In 2018, a new law regarding HE was introduced by the new government.
Preliminarily, an idea was formed to modernize the established strategy in a
substantial way due to these changes. I prepared a draft of a new strategy map
and a set of measures. In this draft, the number of objectives was reduced to 15,
while measures were reduced to 38. However, the reform induced so many serious
changes (a number of radical changes in the organizational structure) that there was
no will to continue this idea, and the “old” version of the strategy with some slight
modifications was reported until the end of the term of the WULS authorities and
the end of the strategy horizon (the same time, 2020). New authorities were recently
elected, and reforms under the new law have already been introduced. Thus, a new
strategy in this new context will likely be prepared soon.
14.9 Objective Results: Benchmark Analysis
To assess the application of BSC in WULS, one can start with facts and figures. A
longitudinal comparison of the state of affairs “before” and “after” implementation
is the most natural way to do this: is the present conduct and performance of WULS
better than before? Which areas of functioning in this HEI have improved? Have
some of them become worse?
However, if any element has improved, is the improvement sufficient? On the
other hand, if something has become worse, is this result due to failures of the
HEI or due to external changes and limitations, e.g., legal restrictions? Thus, the
presentation of some changes in figures will be much more informative when done
in the correct context. Following this idea, I selected two benchmarks.
University of Warsaw (UW) was selected as the first benchmark. This institution
is the biggest and the best HEI in Poland. In some rankings, this HEI is sometimes
outperformed by Jagiellonian University in Kraków, but UW still remains the
most natural point of reference for other HEIs in Poland. Moreover, due to its
localization in the same city, UW is also a serious competitor for WULS. There
are several identical or similar teaching and research areas between both WULS
and UW; nevertheless, WULS has a clear focus on life sciences. There are four
other universities in Poland (Poznań, Wrocław, Kraków, and Lublin) with similar
focuses. There are also four other HEIs that had a similar focus in the past, but since
gravitated strongly into other areas or even merged with other universities; thus, they
were not taken into account. Each of the universities from the four cities mentioned
above represents about 40–50% of WULS’ potential, as measured by the number
of students. Among these schools, I chose Wrocław University of Environmental
and Life Sciences (WUELS) due to the availability of data needed to perform
comparisons.
There is a significant problem with the availability of comprehensive data on
HEIs in Poland. Theoretically, there are some legal obligations for reporting, but not
all HEIs ensure open access to their reports. Moreover, these reports have tremen-
dous diversity in their scope and the ways in which data are presented. Therefore, it
is difficult to obtain a comprehensive and comparable dataset. Nevertheless, Tables
14.3 and 14.4 present such a dataset. Although limitations remain in the dataset,
some interesting observations can be made.
Regarding potential, the relative weakness of WULS compared to UW seems to
be WULS’ excessive number of nonacademic staff (55% vs. 49%), which could lead
to an unnecessary increase in overheads. This could also be one of the reasons for
the lower dynamics of salaries and relatively high share of labor costs in the total
costs. On the other hand, WULS has a greater share of professors in its academic
staff than UW (in WUELS, this share is even higher). However, this share in WULS
is slightly decreasing. In terms of material potential, WULS clearly outperforms
in two benchmarks. Even though the net value of the buildings owned by WULS
decreased (due to depreciation), in a relative sense (e.g., per student), the value is
still much higher than that in the benchmarks. It is commonly known that WULS
has the best and the most cohesive campus in Warsaw, possibly even in the whole
country. This is due to the large investments made in the previous decade, which
were financed from selling some parcels of land owned by WULS. As the land
vs. building ratio indicates, there is still some space for potential investments of this
type in the future. Such investments, however, are not easy to conduct because of the
need to receive an allowance from the government. Nevertheless, this was done in
440
Table 14.3 Key data about WULS and its benchmarks

WULS WUELS UW WULS WUELS UW
Area Item 2009 2019 2009a 2019b 2009c 2019 CAGR (%) CAGR (%) CAGR (%)
Potential Total employees [persons] 2713 2690 1579 1640 6023 7498 −0.1 0.4 2.2
Academic staff [persons] 1243 1198 700 738 3159 3834 −0.4 0.5 2.0
Professors [persons] 269 239 177 209 435 529 −1.2 1.7 2.0
Salaries [mln PLN] 164.5 247.7 82.3 149.6 431.3 716.0 4.2 6.2 5.2
Assets [mln PLN] 2040.0 2092.0 385.1 430.6 1788.4 2468.6 0.3 1.9 4.7
Land [mln PLN] 1151.0 1162.6 92.5 91.5 435.8 541.1 0.1 −0.2 3.1
Buildings [mln PLN] 748.0 589.4 215.7 240.4 556.1 960.8 −2.4 1.8 8.1
Equipment [mln PLN] 23.3 31.6 17.9 15.2 98.8 95.1 3.1 −2.7 −0.5
Teaching Students [persons] 24,532 16,070 10,096 8076 55,387 40,493 −4.1 −2.2 −3.1
Regular students [persons] 14,272 11,056 7609 7238 31,305 31,153 −2.5 −0.5 0.0
Part-time students [persons] 10,260 4124 2487 838 24,082 9340 −8.7 −10.3 −9.0
Graduates’ (BSc, BA) salary [kPLN]d na 3.35/3.97 Na 2.50/3.12 na 3.49/4.22 na na na
Graduates’ (MSc, MA) salary [kPLN]d na 4.18/4.23 Na 3.10/3.30 na 4.34/4.45 na na na
Research External grants for research [mln PLN] na 36.1 11.1 17.7 76.4 227.3 na 4.8 11.5
Publications [#] na 2216 757 1069 6869 5199 na 3.5 −2.7
Citations (WoS) [#] 3848 27,227 na na na na 21.6 na na
Faculties with A or A+ category [#] 6 6 na 3 na 15 0 na na
Finance Revenues [mln PLN] 293.3 510.6 164.9 313.0 859.6 1651.5 5.7 6.6 6.7
Costs [mln PLN] 292.7 475.6 152.9 259.3 846.3 1559.4 5.0 5.4 6.3
Net Profit [mln PLN] −2.2 36.9 11.8 53.7 15.9 93.9 – 16.4 19.4
CAGR—Compound annual growth rate depicts the ratio of average dynamics over the year
Source: Appropriate reports of rectors, annual financial statements, Graduate Tracking System, and the author’s notes from project team meetings (regarding
citations in WULS)
a Asset values for the end of 2011
b Asset values for the end of 2017
c Asset values for the end of 2012—calculations of CAGR were adjusted as needed
d Average monthly gross salary in the first and fifth year after graduation (data for graduates from 2014 and from 2018)
M. Pietrzak
Table 14.4 Key indicators for WULS and its benchmarks
WULS WUELS UW
Area Item 2009 2019 2009b 2019c 2009d 2019
Potential Academic staff share in employee numbers [%] 46 45 44 45 52 51
Professors’ share in academic staff members [%] 22 20 25 28 14 14
Average salary [kPLN per person] 60.6 92.1 52.1 91.2 71.6 95.5
Salaries’ share in costs [%] 56 52 54 58 51 46
Salary and social security share in costs [%] 69 65 na na 60 58
Equipment per employee [kPLN per person] 8.6 11.7 11.3 9.3 16.4 12.7
Equipment per academic staff member [kPLN per person] 18.7 26.4 25.6 20.6 31.3 24.8
Buildings per student [kPLN per person] 30.5 36.7 21.4 29.8 10.0 23.7
Land vs. building ratio [#] 1.5 2.0 0.4 0.4 0.8 0.6
Teaching Students per academic staff [#] 19.7 13.4 14.4 10.9 17.5 10.6
Part-time students share [%] 42 26 25 10 43 23
Graduates’ (BSc, BA) relative salary [%]a na 67/86 na 54/74 na 64/82
Graduates’ (MSc, MA) relative salary [%]a na 85/94 na 66/79 na 81/91
Research External grants per academic staff member [kPLN per person] na 30.1 15.9 24.0 24.2 59.3
Number of publications per academic staff member [#] na 1.8 1.1 1.4 2.2 1.4
Number of citations (WoS) per academic staff member [#] 3 23 na na na na
Share of faculties with A or A+ category [%] 46 46 na 60 na 71
Finance Revenues per employee [kPLN per person] 108.1 189.8 104.4 190.9 142.7 220.3
Revenues per academic staff member [kPLN per person] 236.0 426.2 235.6 424.1 272.1 430.8
Net profit per employee [kPLN per person] −0.8 13.7 7.5 32.7 2.6 12.5
Net profit per academic staff member [kPLN per person] −1.8 30.8 16.9 72.8 5.0 24.5
Source: Appropriate reports of rectors, annual financial statements, Graduate Tracking System, and author notes from project team meetings (regarding
14 The Application of a Balanced Scorecard in Higher Education Institutions:. . .
citations in WULS)
a Gross salary in the first and in the fifth year after graduation (data for graduates from 2014 and 2018) as a percentage of average salary in the region of
residence
b Asset values for the end of 2011
c Asset values as for the end of 2017
d Asset values for the end of 2012
441
442 M. Pietrzak
the past and is thus possible in the future if needed and also creates some advantages
in material potential. Moreover, in the case of equipment, WULS exhibits a positive
dynamic (compared to the benchmarks), and, in 2019, had the highest indicator of
equipment per academic staff.
Regarding teaching, WULS performs well against its counterparts. The number
of students decreased spectacularly, but decreasing numbers were also observed
in WUELS and UW. All these changes were enforced by regulations from the
government. Due to these regulations, public subsidies are strongly tied to the
student per academic staff member ratio. The threshold of 13 was established
arbitrarily, and HEIs that do not maintain this threshold were punished by the loss
of some subsidies. Before this regulation, public HEIs experienced incentives to
treat this relation as a kind of efficiency indicator (the higher the better) due to the
additional money from the government for each regular student or the additional
tuition fees from each part-time student. This new regulation was argued to be an
attempt to improve teaching quality, but this mechanistic and arbitrary approach
seems suspect. Nevertheless, the decrease in student numbers was not a failure but
the result of the government’s enforcement of the regulation.4 The higher dynamics
in this decrease observed in WULS were due to its stronger base at the outset.
WULS does well in teaching. This factor can be objectively verified by the data
from the Polish Graduate Tracking System (GTS). GTS collects data from the Social
Insurance Institution and national register of students and graduates, covering the
entire population of graduates. The average salary of WULS graduates is evidently
higher than that of its counterpart in the life sciences area and very close (94–96%)
to the salary of graduates of the most prestigious university in Poland (UW). As
measured in relative terms, the situation is even better. Regarding the average salary
in the region of residence, graduates of WULS outperform not only WUELS but
also UW.
For research, the results of WULS are more ambiguous. Even though WULS
receives more external funds than WUELS, the amount is still nearly two times less
than that of UW, as measured per academic staff member. However, this situation
should be viewed in a broader context. In Polish science, we can observe the
phenomenon called the Matthew effect (which takes its name from the parable of
the talents from the Gospel of St. Matthew) or the accumulated advantage effect,
which involves giving greater recognition for scientific contributions to researchers
of considerable repute and withholding such recognition from researchers who have
not yet made a substantial contribution (Merton, 1968). The original observation
of Merton can also be extended into the financial domain, where the rich become
richer, and the poor become poorer (Bornmann et al., 2020). For example, in
Poland, funds for basic research are assigned mainly by the National Science Centre
according to competitions for grants. In 2011, only two out of 210 competitors,
namely UW and Jagiellonian University in Kraków, acquired 14% of the total
4 Some critics suggest that this is due to helping private competitors by giving them a substantial
“market share” for free.

funds. In 2019, these universities gained 24% of the total funds. For WULS, the
numbers were 0.5% and 0.9%, respectively. In absolute terms, the progress was even
greater—from 2.5 to 11.6 million PLN (Dynamic statistics, 2020). Nevertheless,
even such noteworthy progress was marginal compared to the gains of the biggest
HEIs.
As mentioned previously, public HEIs in Poland are assessed by their research
performance and subsidized accordingly. As a result, each faculty or other research
entity is classified as A+ (the best), A, B, or C (the worst). Six out of 13 faculties
(46%) of WULS are categorized as A or higher, WUELS has 60%, and UW
71% of such faculties. Nevertheless, WULS outperforms its benchmarks regarding
the number of publications per researcher. This indicates a very high annual rate
of increase (nearly 22%) in the number of citations (according to the Web of
Science); there was also progress made in the internationalization and quality of
these publications.
Regarding finance WULS maintains proper dynamics of cost vs. revenue, which
helped improve its profit results, which were negative in 2009. The average growth
rate of WULS’s revenues was higher than the dynamics of the Polish GDP. However,
this growth rate was still slightly lower than that indicated only by benchmarks.
To sum up this analysis of the objective results, WULS did not achieve amazing
success. However, it also did not completely fail. Overall, WULS maintained pace
with its counterparts with a tendency to outperform them in effectiveness of teaching
but it has a rather weaker position in research.
14.10 Challenges and Lessons Learned
A qualitative assessment of the implementation of this strategy and the Balanced

Scorecard by WULS can be done from at least three points of view: external,
internal, and methodological. Establishing this strategy was enforced externally
by law (Act, 2011). This new obligation was formally fulfilled. Moreover, WULS
executed this strategy quite well (regarding the strategy’s preparation and content)
compared to other public universities.
As previously mentioned, the above strategy began to be developed in 2009 and
was approved by the senate in 2010. According to the research done by Jasiczak et
al. (2011), in 2011, only 35 out of 132 public HEIs (27%) had already developed
a strategy or were just starting to do so. In total, 39% had mission statements,
while only 7% had visions. A few years later, these statistics improved slightly.
According to the research of Piotrowska-Piatek ˛ (2015), 59.3% of public HEIs
had formal strategies: 57.5% of these documents consisted of mission statements,
33.3% included visions, 27% included a SWOT analysis, and 84.8% included
strategic objectives. However, the objectives were rarely operationalized: Only
42.4% of these documents consisted of measures, and 54.5% of them defined
ways to achieve their goals. Around 12% of public HEIs used modern methods,
in such number Balanced Scorecard (Piotrowska-Piatek, ˛ 2015). Regarding BSC’s
444 M. Pietrzak
implementation in Polish HEIs, the literature is scarce. Due to my best knowledge,

there are only four cases mentioned in other papers (Ch˛eś & Lewandowska, 2013;
Jaworski & Woźny, 2015; Lewandowska & Zygarłowski, 2014; Piasecka, 2013;
Pietrzak, 2014a; Świerk, 2014; Świerk & Mulawa, 2015). Three of these case
studies involve public HEIs—namely, WULS, University of Bielsko-Biala (UBB),
and Maria Curie-Sklodowska University in Lublin (UMCS), as well as one private
HEI, TEB Akademia (Group of WSB Universities). TEB Akademia is an interesting
case of a network of private HEIs growing dynamically (now it accounts for nearly
80 thousand students, comprising over 20% of the market share among private
HEIs); however, it represents the private sector.
No papers were found on the details of BSC’s implementation in UBB; however,
analysis of the content of the strategy document for the years 2012–2020 indicates
a rather superficially elaborated methodology with only 31 strategic objectives
established (without any measures), split across 4 perspectives (Growth strategy,
2012). This model was possibly more developed in other documents that are
not available to the public. BSC’s implementation in UMCS was much more
transparent: it was briefly summarized in three papers (Piasecka, 2013; Świerk,
2014; Świerk & Mulawa, 2015) and explored in the much more profound way by
Ernst and Young (EY), a consulting company that was engaged in this project (Ernst
and Young, 2011a, 2011b, 2011c, 2011d, 2012). The documentation presented by
EY is large-scale (BSC was cascaded into faculties and into support units), deeply
elaborated, and well-formed from a methodological point of view. The strategic
planning horizon in these documents was 2021. I did not find any elaboration or
assessment of this implementation from a present-day perspective, but a new UMCS
strategy prepared for the years 2019–2025 was already approved by the senate
(Strategy of Maria Curie-Skłodowska University, 2019). It seems to be symptomatic
that in these documents BSC is not even mentioned, nor strategy map, perspectives,
cascading, etc.
To summarize this assessment from an external point of view, WULS met the
governmental requirements and prepared a formal strategy. Moreover, it did so
basing on a modern methodology borrowed from the business sector and adapted
to sectoral and local specificity. According to the short discussion above and
comparative research done by Pietrzak (2014a) and Pietrzak and Gołaś (2018),
the WULS strategy looks well compared to the strategies of other public HEIs in
Poland.
However, a skeptic could ask if there is any proof that, considering all this
involvement of public HEIs all around Poland in strategy formulation, was the
social welfare of the nation significantly enhanced? This question will be difficult
to answer. I am an enthusiast of strategic management. However, to work well this
process requires an owner. As explained above, public HEIs rely heavily on public
funds. Consequently, the government consistently attempts to somehow modify the
behavior of HEIs. The law enforced the elaboration of strategic documents. Never-
theless, there were no institutions or mechanisms to determine real accountability
under this strategy. Moreover, the government did not act as a nationwide owner
of such strategies. Notably, on a macro scale, this situation mirrored the problem
of strategy cascading at WULS. In both cases, the problem was the high degree of
autonomy of the lower layers combined with a lack of developed mechanisms for
strategic harmonization.
In the business sector, the accountability for strategies is executed twofold—by
competition on the marketplace (markets for outputs and markets for inputs) and by
internal corporate governance (board of directors, general meetings of shareholders,
etc.). These two methods failed in the case of the public HE system in Poland.
Thus, the first lesson learned is that a reasonable implementation of a strategy in
public HEIs requires the development of a nationwide system of governance and
accountability as a prerequisite.
The assessment of the WULS case from an internal point of view is also
ambiguous. Some drawbacks and limitations of our strategic efforts are already
obvious for readers. Some limitations are also outlined in the methodological
assessment section. However, BSC implementation also had some positive results
from an internal perspective. There was substantial progress in some (but not
all) important strategic directions. The project itself and reporting based on BSC
increased strategic awareness and maturity. Many staff members learned about
strategic management and modern frameworks like Balanced Scorecard. Some
innovative measures were also defined and established. Even though the imple-
mentation had drawbacks and limitations, it could be understood as an option for
future action. The experience gathered and methodological foundations constructed
could yield positive results in the future, if a new strategy is prepared. The
new institutional context in Poland seems to be more “strategy-friendly” than it
was in the past. The new law (Act, 2018) increased the power of rectors and
substantially reinforced their position against deans. It also created new bodies
in the institutional matrix of academic governance—a board of trusties entitled to
monitor the rector’s performance in strategic execution. Therefore, there are some
elementary prerequisites for the strategic accountability created.5
The second lesson learned is that to create workable strategic management,
factors beyond external enforcement by legal regulations are needed. The internal
demands of strategic intent and following through with that intent are required.
According to Clark (1983), there are three basic elements in the organization of HE
systems: the way tasks are conceived and arranged (the specific division of labor
for technologies focused on specific materials and knowledge), culture (beliefs,
norms and values, and the symbolic elements of the organization closely interlocked
with its structure), and authority (the distribution of legitimate power throughout the
system). Therefore, there are two basic possibilities for enhancing internally driven
strategic management in HEIs.
One is based on changes in the authority distribution within an HEI in such a
way that it creates a center of power not only interested in formulating a strategy
on its own but also able to fully execute it. Many changes in HE sectors in the New
5 However, one could find that these changes are in the vein of entrepreneurial university and
abandon Humboldtian ideal—what is controversial per se.

446 M. Pietrzak
Public Management vein observed worldwide in recent decades seem to follow this
direction.
The second option is based on culture, which has a powerful impact on
academics (“men of ideas”). The university is a type of organization that falls
under the Professional Bureaucracy category and relies, for coordination, on the
standardization of skills, training, and indoctrination of its professionals, who,
in turn, experience considerable control over their own work (Mintzberg, 1979).
These specifics could be attributed to the problem of information asymmetry.
Much of the work of academic professionals has traits of an experience goods
or even a credence goods. The asymmetrical information embodied in academic
work challenges the management of academics. Approaches common in business,
such as signalling and screening methods, face important obstacles in HEIs.
Thus, putting too much pressure on performance measurement and accountability
could have side effects (Pietrzak, 2018). Indeed, according to Mintzberg (1979),
real reforms in a Professional Bureaucracy do not emerge from the offices of
administrators or governmental bureaucrats looking for bringing academics under
control; rather, changes are introduced through processes of change among the
academics themselves.
Finally, from a methodological point of view, the implementation of BSC in
WULS was largely in line with the books of Kaplan and Norton (1996, 2001, 2004,
2006, 2008) and other prominent authors (Niven, 2002, 2005, 2008; Smith, 2007)
but only up to the point of deeper operationalization. Targets were defined not at the
beginning of BSC’s implementation but a few years later, and not for all objectives.
There were no strategic initiatives established with project managers, milestones,
and deadlines. Moreover, the strategy did not cascade directly to the lower layers.
Thus, the loop of the strategic management system proposed by Kaplan and Norton
(2008) was not closed. Of course, one can interpret these problems as related to the
lack of prerequisites such as governance and accountability mechanisms.
However, one could expect that these problems have deeper causes. According to
Mintzberg (1979), if stakeholders (clients, nonprofessional administrators, members
of society, and governmental representatives) of Professional Bureaucracies are
dissatisfied with their performance, then they do what seems obvious to them—
they attempt to control, supervise more tightly, and standardize processes or
outputs. Control and supervision may work in the worst cases of incompetency or
unreliability, e.g., a professor who misses too many classes may be disciplined,
but professional activity is typically too complex and too vague to be seriously
supervised by anyone other than the professionals themselves.
On the other hand, this kind of work would be difficult to standardize in both
its processes and outputs. Placing too much emphasis on standardization could
easily impede and discourage innovation among professionals/academics and force
them to satisfy the relevant standards, rather than serving stakeholders with their
creativity. Technocratic controls often cannot distinguish between proper conduct
and misbehavior, nor are they able to make an incompetent professional become a
competent one. Thus, orders, plans, standards, etc. equally constrain misbehaving
academics and dedicated ones, as well as incompetent and competent ones. If
professionals lose control over their own work, moreover, they can easily become
passive (Mintzberg, 1979).
If very deep changes in Professional Bureaucracy come not from the government
intending to bring academics under control, nor from new administrators announc-
ing reforms, can the professionals themselves be changed? Returning to Clark’s
(1983) building blocks of HE systems (work arrangement, culture, and distribution
of power), we should instead focus more on the specificity of academic work and
its arrangement. Knowledge is a very special kind of material, around which all
academic activity is organized. Thus, instead of attempts to control professionals
from the outside (strict plans, targets, and standards), we could pay more attention
to changing professionals from within. In other words, instead of forcing academics
to follow a strategy by using organizational power, perhaps we could instead focus
more on the symbolic side of the university (culture and beliefs).
Therefore, the third and the last lesson that I have personally learned, considering
the decade of experience described in the above paragraphs, is that I underestimated
the specifics of the university as an organization. It is a common opinion in the
literature that BSC must be adapted before implementing it in HEIs. This was also
the opinion of our project team, and we did our best to fulfil this requirement.
However, we applied the same method used by many others—we rearranged the
model by rearranging perspectives and changing the logic of their cause-end-
effect interrelations, as well as renaming some of them. This is a common method
of adaptation among other public agencies and even NGOs. However, such an
adaptation may not be enough. Perhaps it is too superficial? Mintzberg (1979) claims
that proper strategic considerations in Professional Bureaucracies should occur
at the level of individual professionals, who experience considerable autonomy
and often select their own clients and methods of dealing with them. Thus, such
professionals could be considered as the ultimate unit of a product–market strategy.
Therefore, we should consider rethinking our conception of what a university
strategy is and how BSC could be helpful in executing it.
Mintzberg (1987) outlined a strategy in five dimensions, a plan, a ploy, a pattern,
a position, and a perspective (5 Ps). The last option appears to be the best way
to view strategy for HEIs, where strategy represents a perspective. This raises
questions about intentions and actions in a collective context. As the ultimate
strategic executor, each academic should pursue his/her own strategy, thus requiring
some broader strategic perspectives to fit individual academic intent. From this
point of view, the mission, as a common identity, an inspiring vision, and a
clear strategy map narrating the desired direction of a whole community is more
important than time-scaled targets, performance standards, and detailed initiatives.
The interpretation of a strategy map as a form of strategic narration that integrates
the academic community seems very promising. After all, according to the Nobel
laureate in economics, Robert Shiller (2017), we are ultimately homo narrans.
448 M. Pietrzak
References
Act. (1990). Retrieved August 25, 2020, from https://isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id=

WDU19900650385
Act. (2011). Retrieved August 25, 2020, from http://isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id
=WDU20110840455
Act. (2018). Retrieved August 25, 2020, from https://isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id=
WDU20180001668
Annual financial statement of the University of Environmental and Life Sciences in Wroclaw for
the year. (2011). Retrieved October 15, 2020, from http://www.krs-online.com.pl/msig/2013/
16414.pdf
Annual financial statement of the University of Environmental and Life Sciences in Wroclaw for
the year. (2017). Retrieved October 15, 2020, from http://www.krs-online.com.pl/msig/2018/
28934.pdf
Annual financial statement of the University of Warsaw for the year. (2012). Retrieved October 15,
2020, from https://portal.uw.edu.pl/web/kwestura/za-rok-2013
Annual financial statement of the University of Warsaw for the year. (2019). Retrieved October 15,
2020, from https://portal.uw.edu.pl/web/kwestura/za-rok-2019
Annual financial statement of the Warsaw University of Life Sciences for the year. (2009a).
Retrieved October 15, 2020, from http://bip.sggw.pl/sprawozdania-2009
Annual financial statement of the Warsaw University of Life Sciences for the year. (2009b).
Retrieved October 15, 2020, from https://fs.siteor.com/sggw/files/Sprawozdania%20Finansowe
/2019/Sprawozdanie_Finansowe_Laczne_za_rok_2019.pdf
Annual report of rector of the University of Warsaw for 2009. (2010). Retrieved Octo-
ber 10, 2020, from http://archiwum.bip.uw.edu.pl/page/ViewBIP%3Fexec=nowy&pdm=
1&tmt=22&dok=6091.html
Annual report of rector of the University of Warsaw for 2019. (2020). Retrieved Octo-
ber 10, 2020, from https://www.uw.edu.pl/wp-content/uploads/2020/06/sprawozdanie_jmr
_2019_z_zalacznikami.pdf
Asan, S. E., & Tanyas, M. (2007). Integrating Hoshin Kanri and the balanced scorecard for strategic
management: The case of higher education. Total Quality Management & Business Excellence,
18(9), 999–1014.
Babich, P. (2002). Hoshin handbook. Focus and link activities throughout the organization. Total
Quality Engineering.
Borecki, T., & Pietrzak, M. (Eds.). (2010). Strategy of the Warsaw University of Life Sciences for
2011–2020. SGGW Publishing House.
Bornmann, L., Ganser, C., Tekles, A., & Leydesdorff, L. (2020). Does the hα-index reinforce the
Matthew effect in science? The introduction of agent-based simulations into scientometrics.
Quantitative Science Studies, 1(1), 331–346.
Ch˛eś, A., & Lewandowska, A. (2013). The influence of a strategy implemented using the
Balanced Scorecard method on the effective integration of change within the teaching process
at an institution of collegiate business education. 3rd International Conference the Future of
Education. Retrieved October 20, 2020, from https://depot.ceon.pl/handle/123456789/4874
Clark, B. R. (1983). The higher education system. Academic organization in cross-national
perspective. University of California Press.
Cowley, M., & Domb, E. (1997). Beyond strategic vision. Effective corporate action with Hoshin
planning. Butter-worth-Heinemann.
De Witt, B., & Meyer, R. (2005). Strategy synthesis: Resolving strategy paradoxes to create
competitive advantage. Thommson Learning.
Dynamic statistics of National Science Centre. (2020). Poland. Retrieved October 20, 2020, from
https://www.ncn.gov.pl/statystyki/?language=pl
Ernst & Young. (2011a). Analysis of external determinants of development at the Maria Curie-
Skłodowska University in Lublin. Retrieved October 10, 2020, from https://phavi.umcs.pl/at/
attachments/2013/1115/145957-zalacznik-nr-1-do-strategii-rozwoju.pdf
Ernst & Young. (2011b). SWOT analysis of the Maria Curie-Skłodowska University in Lublin.
Retrieved October 10, 2020, from https://phavi.umcs.pl/at/attachments/2014/0526/094503-
zalacznik-nr-2-do-strategii-rozwoju-analiza-swot.pdf
Ernst & Young. (2011c). Mission, vision, strategic and operational goals of the University.
Ernst & Young. (2011d). Proposed activities under the development strategy of the Maria Curie-
Skłodowska University in Lublin for selected units. Retrieved October 10, 2020, from https://
phavi.umcs.pl/at/attachments/2014/0526/094505-zalacznik-nr-4do-strategii-rozwoju.pdf
Ernst & Young. (2012). Cascading the development strategy of the Maria Curie-Skłodowska
University in Lublin for selected units. Retrieved October 10, 2020, from http://phavi.umcs.pl/
at/attachments/2014/0114/085823-strategia-rozwoju-umcs.pdf
Etzioni, A. (1961). A comparative analysis of complex organizations. The Free Press.
Facts and Figures. (2020). Retrieved August 25, 2020, from https://media.sggw.pl/teczka-prasowa/
fakty-i-liczby
Ferlie, E., Musselin, C., & Andresani, G. (2008). The steering of higher education: A public
management perspective. Higher Education, 56, 325–348.
Godłów-Legi˛edź, J. (2016). Higher education in the process of political transformation in Poland
and the quality of education. Legal and Economic Studies, 98, 197–219.
Graduates Tracking System. (n.d.). Retrieved October 20, 2020, from https://ela.nauka.gov.pl/pl/
infographics/graduates
Growth strategy for the years 2012–2020. (2012). University of Bielsko-Biala. Retrieved October
10, 2020, from http://www.jakosc.ath.bielsko.pl/files/inne/zalacznik-1-uchwala-866.pdf
Higher education institutions and their finance in 2018. (2019). Central Statistical Office—
Statistical Office in Gdańsk: Warsaw-Gdańsk.
Hood, C. (1991). A public Management for all Seasons? Public Administration, 69, 3–19.
Jasiczak, J., Kochalski, C., & Sapała, M. (2011). Preliminary results of research on the design
and implementation of development in public universities in Poland. Scientific Papers of the
University of Economics in Poznań, 167, 7–14.
Jaworski, J., & Woźny, J. (2015). Conceptual framework for use of balanced scorecard in
management of public university. Research Papers of Wrocław University of Economics, 398,
189–199.
Juhl, H. J., & Christensen, M. (2008). Quality management in a Danish business school—A head of
department per-spective. Total Quality Management & Business Excellence, 19(7–8), 719–732.
Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard—Measures that drive performance.
Harvard Business Review, 70(1), 71–79.
Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard: Translating strategy into action.
Harvard Business School Press.
Kaplan, R. S., & Norton, D. P. (2001). The strategy-focused organization. How balanced scorecard
companies thrive in the new business environment. Harvard Business School Press.
Kaplan, R. S., & Norton, D. P. (2004). Strategy maps. Converting intangible assets into tangible
outcomes. Harvard Business School Press.
Kaplan, R. S., & Norton, D. P. (2006). Alignment: Using the balanced scorecard to create
corporate synergies. Harvard Business School Press.
Kaplan, R. S., & Norton, D. P. (2008). The execution premium. Linking strategy to operations for
competitive advantage. Harvard Business School Press.
Koch, R. (1995). The financial times guide to strategy: How to create and deliver a useful strategy.
Prentice Hall.
Kwiek, M. (2012). Changing higher education policies: From the deinstitutionalization to the
reinstitutionalization of the research mission in polish universities. Science and Public Policy,
39, 641–654.
450 M. Pietrzak
Kwiek, M. (2017). Introduction: The reform of higher education in Poland and its challenges.
How the gradual de-encapsulation of a system leads to its stratification. Science and Higher
Education, 2, 9–38.
Leja, K. (2013). University management. Concepts and contemporary challenges. Oficyna a
Wolters Kluwer Business.
Lewandowska, A., & Zygarłowski, P. (2014). Implementing the strategy of a universities’ network
with the use of balanced scorecard (BSC). In K. Zi˛eba & B. Ortyl (Eds.), Contemporary
concepts of management in education higher. WSEH.
Lewin, K. (1947). Frontiers in the group dynamics. Concept, method and reality in social science;
social equilibria and social change. Human Relations, 1(1), 5–41.
McDevitt, R., Giapponi, C., & Solomon, N. (2008). Strategy revitalization in academe: A balanced
scorecard ap-proach. International Journal of Educational Management, 22(1), 32–47.
Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56–63.
Mintzberg, H. (1979). The structuring of organizations. Prentice-Hall.
Mintzberg, H. (1987). The strategy concept 1: Five Ps for strategy. California Management Review,
30(1), 11–24.
Niven, P. R. (2002). Balanced scorecard step-by-step. Wiley.
Niven, P. R. (2005). Balanced scorecard diagnostics. Maintaining maximum performance. Wiley.
Niven, P. R. (2008). Balanced scorecard step-by-step for government and nonprofit agencies.
Wiley.
O’Flynn, J. (2007). From new public management to public value: Paradigmatic change and
managerial implications. The Australian Journal of Public Administration, 66(3), 353–366.
Papenhausen, C., & Einstein, W. (2006). Insights from the balanced scorecard—Implementing the
balanced scorecard at a college of business. Measuring Business Excellence, 10(3), 15–22.
Peer Review of Poland’s Higher Education and Science System. (2017). Publications Office of the
European Union: Luxembourg.
Piasecka, A. (2013). Process of implementation of the balanced scorecard in university—Case
study. Management and Finance, 11(1), 241–252.
Pietrzak, M. (2014a). Using the strategy map as a strategic communication tool in higher education:
A case study of Warsaw University of Life Sciences. Online Journal of Applied Knowledge
Management, 2(2), 26–42.
Pietrzak, M. (2017). Balanced scorecard and Morgan’s organizational metaphors. Scientific works
of the Wrocław University of Economics: Global Challenges of Management Control and
Reporting, 474, 106–114.
Pietrzak, M. (2018). Information asymmetry, signaling and screening vs. Audit Culture—Selected
Challenges for Academic Governance. Management Issues—Management Issues, 16(4), 134–
152.
Pietrzak, P. (2014b). Strategic issues of the Warsaw University of Life Sciences in comparison with
other universities. Scientific Papers of the Warsaw University of Life Sciences, Economics and
Organization of Food Economy, 105, 95–104.
Pietrzak, P., & Gołaś, M. (2018). Strategies of public universities in Poland—The essence and
conditions. SGGW Publishing House.
Piotrowska-Piatek,
˛ A. (2015). Analysis of the development strategy of universities in Poland.
Organization and Management, 30(2), 103–116.
Records of private universities. (2020). Retrieved August 25, 2020, from https://polon.
nauka.gov.pl/opi/aa/rejestry/run;jsessionid=66615FA22CDA0B6CC175C154E124CDAC.Nws
ProdA?execution=e1s1
Report by professor Roman Kołacz, rector of the University of Environmental and Life Sciences in
Wroclaw from the activity of the university in 2009. (2010). Retrieved October 10, 2020, from
https://bip.upwr.edu.pl/akty-prawne/sprawozdania-roczne-rektora
Report by professor Wiesław Bielawski, rector of the WULS from the activity of the university in
2019. (2020). SGGW Publishing House.
Report by the rector of the University of Environmental and Life Sciences in Wroclaw from
the activity of the university in 2019. (2020). Retrieved October 10, 2020, from https://
bip.upwr.edu.pl/akty-prawne/sprawozdania-roczne-rektora
Rigby, D., & Bilodeau, B. (2018). Management Tools & Trends. Retrieved August
25, 2020, from https://www.bain.com/contentassets/caa40128a49c4f34800a76eae15828e3/
bain_brief-management_tools_and_trends.pdf
Schimank, U. (2005). ‘New public management’ and the academic profession: Reflections on the
German situa-tion. Minerva, 43, 361–376.
Shiller, R. J. (2017). Narrative Economics. American Economic Review, 107(4), 967–1004.
Simon, H. (2000). Administrative behavior. A study of decision-making processes in administrative
organizations. The Free Press.
Smith, R. F. (2007). Business process management and the balanced scorecard. Using processes
as strategic drivers. Wiley.
Spender, J. C. (2014). Business strategy. Managing uncertainty, opportunity & enterprise. Oxford
University Press.
Strategy of Maria Curie-Skłodowska University in Lublin for the years 2019–2025. (2019).
Retrieved October 10, 2020, from https://phavi.umcs.pl/at/attachments/2019/0402/091028-
strategia-27-marca-2019-ostateczna-po-senacie.pdf
Świerk, J. (2014). The role of strategy maps in managing university. Acta Universitatatis
Lodziensis—Folia Oeconomica, 299(1), 95–106.
Świerk, J., & Mulawa, M. (2015). The balanced scorecard for higher education—The case of Maria
curie-Skłodowska university. Regional Barometer, 41(3), 169–178.
Tahar, S. (2013). Resource allocation in higher education in the context of new public management.
Public Management Review, 15(5), 687–711.
Tapinos, E., Dyson, R. G., & Meadows, M. (2005). The impact of the performance measurement
systems in setting the ‘direction’ in the University of Warwick. Production Planning and
Control, 16(2), 189–198.
Taylor, J., & Baines, C. (2012). Performance management in UK universities: Implementing the
balanced scorecard. Journal of Higher Education Policy and Management, 34(2), 111–124.
Umashankar, V., & Dutta, K. (2007). Balanced scorecards in managing higher education institu-
tions: An Indian perspective. International Journal of Education Management, 21(1), 54–67.
Van der Heijden, K. (1998). Scenarios. The art of strategic conversation. Wiley.
Warning, S. (2004). Performance differences in German higher education: Empirical analysis of
strategic groups. Review of Industrial Organization, 24, 393–408.
Which professions do we respect? (2019). CBOS. Research Report, 157, 1–11.
Wilkesman, U., & Schmid, C. J. (2012). The impacts of new governance on teaching at Germany
universities. Findings from a national survey. Higher Education, 63, 33–52.
Witt, U. (1998). Imagination and leadership—The neglected dimension of an evolutionary theory
of the firm. Journal of Economic Behavior and Organization, 35, 161–177.
Witt, U. (1999). Do entrepreneurs need firms? A contribution to a missing chapter in Austrian
economics. Review of Austrian Economics, 11, 99–109.
Witt, U. (2005). Firms as realizations of entrepreneurial visions. Papers on economics and
evolution. Working Paper No. 0510, Max-Planck-Institut fuer Oekonomik, Jena.
Chapter 15
E-Learning in Times of Crisis: An
Incidental or Facilitative Event?
Nitza Davidovitch and Rivka Wadmany
Abstract As a result of the COVID-19 epidemic that erupted in 2020 the various
higher education institutions in Israel, as elsewhere, were compelled to embrace E-
Learning at short notice. This was a revolution that appeared with no preparation
and that put on the agenda the efficacy of E-Learning from pedagogical aspects
and the implications of the lecturer’s functions and the act of teaching for the
quality of students’ learning as well as for the meaning of the learning expanse
(campus—home) in teaching and learning processes. The current study examined
the opinions of students and lecturers regarding the advantages and disadvantages of
E-Learning from various aspects in a systemic, multi-institutional perspective. The
study included 2015 students studying at various academic institutions: universities,
academic teachers’ colleges, academic colleges, and private colleges. The study also
included 223 lecturers.
The research findings show that the respondents did not display a high preference
for E-Learning: less than half the students and about one-third of the lecturers
expressed a preference for E-Learning. Both groups noted the lack of personal,
social, and emotional interaction with both students and lecturers as one of the
main shortcomings of E-Learning. Most of the students and lecturers did not
grasp E-Learning as providing them with better quality teaching and learning.
The study illuminates the role of the lecturer in the digital era as a teacher,
and particularly—the role of the professional elements in charge of teaching and
learning at academic institutions, particularly in the pedagogical aspects. According
to student evaluations, the use of technological platforms and tools does not
improve teaching, as they are used by the faculty only technically with no matching
pedagogy. In order to succeed, E-Learning requires other pedagogical educational
N. Davidovitch ()
School of Education, Ariel University, Ariel, Israel
e-mail: d.nitza@ariel.ac.il
R. Wadmany
The Council for Higher Education, Jerusalem, Israel

454 N. Davidovitch and R. Wadmany
approaches aside from copying frontal teaching patterns using the Zoom platform,
as well as others: Weber, MS Teams, etc.
In addition, the study indicates the need for perceptual changes, both by the
students, who must take responsibility for their learning, and by the lecturers,
who must reexamine the teaching and learning processes and adapt their role
and areas of responsibility to the new opportunities afforded by the technological
tools. The research findings also indicate that effective teaching is teaching that
arouses in student’s inquisitiveness, motivation, and learning experiences, and note
that learning products must be adapted to include essential skills in addition to
knowledge. Further, the study illuminates the thorough discussion that must be
held by leaders of higher education and of the academic institutions concerning
the new effective designation of the campus after the COVID-19 crisis, including
distinguishing between the virtual and the realistic in academic teaching and
challenges and ways of dealing with the new circumstances.
Keywords E-Learning · COVID-19 · Crisis · Higher education · Technology ·

Pedagogical aspects · Questionnaire · Regression
15.1 Introduction
For years, many studies have examined the shortcomings and advantages of E-
Learning among students (Davidovitch & Yavich, 2015, 2016, 2018). These studies
addressed a variety of areas, for instance, the interpersonal interaction between
participants in study groups, including the lecturer, students’ satisfaction with
lecturers’ teaching, and learners’ achievements (Cohen & Davidovitch, 2020;
Wadmany, 2017, 2018).
The COVID-19 crisis (December 2019) compelled all institutions of higher
education, in Israel and around the world, lecturers and students, to transition to
full online studies, rapidly and with no preparation (Altbach & De Wit, 2020). The
second semester (2020) at Israeli universities and colleges opened via E-Learning,
learning in a time of global crisis, with the students restricted to their homes (in
quarantine) and compelled to complete their academic duties on an online system
using computers. The faculty had to undergo an essential change in the teaching
techniques they had prepared and aimed for and to transform their way of teaching
and learning in order to do their best for their students. These changes, which began
with the appearance of COVID-19, may constitute the foundation of a new way of
teaching, creative thinking, and planning, for students and lecturers (Davidovitch &
Eckhaus, 2020; DePietro, 2020).
Moreover, institutions of higher education invest many resources in developing
distance learning courses and are required to enhance lecturers’ knowledge and
proficiency in the pedagogical dimension, facilitated by technology. Many questions
are emerging regarding the learning products of technology-enhanced teaching
15 E-Learning in Times of Crisis: An Incidental or Facilitative Event? 455
(Davidovitch, 2014). Are students ready for the new era in learning? Are lecturers
prepared?
Hence, the current study explores the opinions of students and lecturers regarding
the advantages and disadvantages of E-Learning from various aspects, in a systemic,
multi-institutional perspective. This, with the aim of examining the new effective
designation of the academic campus after the COVID-19 crisis. What will be the
nature of the academic learning domain in the aftermath of the crisis?
15.2 Literature Review
15.2.1 Information and Communications Technology-based

Teaching in Higher Education: For Whom?
In recent years gradually more academic institutions, universities, and colleges (with
the exception of the Open University) are endeavoring to integrate E-Learning in
teaching at their academic institutions, perceiving it as an attractive, relevant, and
marketing oriented way of teaching and learning. Moreover, E-Learning is relevant
for students who combine work with studies and therefore require flexible study
hours and less demand for physical presence on campus. This is particularly true in
the Israeli circumstances, where students only begin their studies after completing
military service, i.e., at least three years later than their peers in other countries
(Cohen & Davidovitch, 2020).
When wishing to examine whether E-Learning is effective we must examine
the study methods rather than the technology used to implement them, because
studies indicate that technology does not change the learning rather is only a means
(Wadmany, 2017, 2018). In a critical article titled “Media will never influence
learning,” Clark (1994) writes that what affects learning is not the technology rather
the study method.
Indeed, the findings of a study conducted at the Technion show that students who
studied from a distance had more positive attitudes toward asynchronous learning
(online contents with pre-recorded lectures) than students who studied face-to-
face, because they believed that learning from a distance at their leisure might
help advance their learning skills (Barak et al., 2012). Furthermore, students who
studied from a distance were also found to express more positive attitudes via their
perceived self-efficacy. Namely, in order to succeed in distance learning, students
must believe in their ability to manage the academic process, find motivation and
cognitive resources, and carry out the necessary operations in order to do well in a
course (Barak et al., 2012).
We learn from the research literature that in order for learning to be effective,
many students need a supportive human framework and direct contact with the
lecturer. Regarding distance learning, researchers note the learning environment
as a learner-focused environment when teaching is conveyed directly from the
lecturer to the student, versus frontal teaching where the teacher constitutes the
major part of the learner’s environment. In distance learning the learner is perceived
as independent, an active learner with freedom of choice and the liberty to decide
about his learning process, one responsible for managing his own study time, with
high motivation, a high degree of self-efficacy, one who believes in himself and
his abilities, with high self-control and thus capable of functioning effectively
and dealing with technological difficulties (Wadmany, 2017, 2018; Wagner &
McCombs, 1995).
Despite the above, the research also shows that there are problems in the virtual
environment that block learners and pose difficulties (Cohen, 1999; Phelps, 2018).
One of the major problems stems from the lack of a social framework in the
process of distance learning. Some learners are not inclined to study individually and
distance learning with its lack of a social framework might not be to their advantage
(Cohen & Davidovitch, 2020). In addition, the existing implications of closing down
academic institutions and of the transition to E-Learning make it hard for learners
and their families, particularly when belonging to lower socioeconomic groups. For
instance, there is a concern of disrupting the learning continuity, separation from
one’s studies, and the lack of an adequate study infrastructure and space—all these
might hamper equal opportunity for learners (Weissblei, 2020).
15.2.2 Features of E-Learning
E-Learning is characterized mainly by learning from home or by learning that does

not require physical presence at the academic institution, while using moderated
technological means (Nir-Gal, 2002). Academic institutions are aware of the
economic and marketing potential of these advanced technologies, where such
study methods create a shift in the structure of academic teaching at the institution
(Allen & Seaman, 2007; Cohen & Davidovitch, 2020; Nir-Gal, 2002). E-Learning
facilitates a learning experience by using computers and the Internet, both at the
academic institution and elsewhere.
The interaction in the teaching process is manifested in the learner’s ability to
react to the study material and to receive feedback from the lecturer or other stu-
dents, affecting the learning efficacy. The technology itself is mainly infrastructure
(Dabbagh & Bannan-Ritland, 2005). The choice of how to utilize it depends on its
developers and users. In this way, various models of E-Learning have been formed,
beginning from the Moodle system, where a lecturer gives a frontal lecture with no
aides, through a frontal lecture accompanied by a presentation, to innovative models
that combine a range of content sources and attempt to offer new, collaborative,
and less centralist learning methods. All these can be performed in a digital format
available on the internet for collaborative viewing or learning (Davidovitch &
Eckhaus, 2020; Goldschmidt, 2013).
15.2.3 The Added Value of E-Learning: Indeed?
Some argue for the advantages of E-Learning: E-Learning is considered more

economical than frontal learning, more easily available—students can study via
any device connected to the internet, anywhere and anytime. The institution does
not have to invest funds in teaching staff, libraries, lecturers, operation, and
regular maintenance of classrooms and study spaces. Moreover, regular expenses
of electricity, water, and Internet are significantly lower when students and lecturers
are at home and do not attend the academic institution (Anderson, 2008). Faculty,
lecturers, and students save a lot of money on transportation and are spared many
expenses. With regard to the contents of the classes, the lecturer does not have to
repeat the same class several times for different groups (Allen & Seaman, 2007;
Goldschmidt, 2013). Lecturers can record the classes in advance and then share
them with different groups of students, not necessarily at the same time (Dabbagh
& Bannan-Ritland, 2005).
It is well known that E-Learning enables learning at one’s own pace. Students can
study the courses anytime and anywhere and move around while studying (Barak et
al., 2012). If the classes are recorded and uploaded to the web, students who have
a family of their own or who work and are normally overloaded with daily tasks,
can study when they have the time. This requires viewing skills, even more so than
previously. At present, most students consume contents, “meet” friends, perform
purchases—on the Internet (Anderson, 2008; Dabbagh & Bannan-Ritland, 2005).
Therefore, in distance learning, when the study contents are prerecorded for the
students, the study material should be adapted to each student’s level and rate of
progress (Cohen & Davidovitch, 2020).
Studies also show that E-Learning enables new ways of teaching (DePietro,
2020) and improves students’ achievements and test scores. The advantages of
digital teaching include easing the lecturer’s workload and the transition from
teaching tasks to tasks involving exercises and discussions with learners (similar
to the “backward course” model), increasing motivation among students, order
and organization in teaching, and self-discipline by students (Dabbagh & Bannan-
Ritland, 2005; Miller & Forkosh-Baruch, 2018).
15.2.4 Technology as an Efficient Tool for Developing

Learning Skills
Active learning is particularly essential in distance learning. The lecturer is not

available to help the student “face to face” and is required to employ a different,
digital type of discourse. Students must act independently based on the various
information sources available to them (Berger-Tikochinski et al., 2020). Active
learning was found to be positively related to the satisfaction of the students who
utilize digital learning, and it may also contribute to their success in this form of
learning (Shain, cited by Berger-Tikochinski et al., 2020).
Information literacy skills are important skills for students in any learning
environment, and particularly for students in a distance learning environment.
Students who significantly rely on digital resources in their studies are required
to develop more thorough abilities of locating information than students who learn
in the traditional method (Berger-Tikochinski et al., 2020). Educators must consider
wise use of interactive environments on the web to advance significant learning
(Ertmer & Ottenbreit-Leftwich, 2010).
Ashburn and Floden (2006) emphasize in their book “Meaningful learning using
technology” that for learner’s technology is an intellectual partner that helps them
advance their thinking, learning, and comprehension regarding the world we live
in. Technology-assisted learning can facilitate meaningful learning if based on
learners’ involvement in constructing knowledge, discourse in the study group,
self-expression, and utilization of reflective thinking. This is done through skills of
investigation, design, communication, writing, building models, and visualization.
Learning in a hi-tech environment must be based on authentic inquiry procedures,
structuring knowledge, and collaborative learning by students.
Mioduser et al. (2010) stress that the process of technology-integrated learning
is based on a variety of digital literacies, including multi-channel information
processing, navigation of the virtual space, communication, visual literacy, hyper-
literacy, personal information management, and coping with complexity. These
literacies are part of the toolkit of both lecturer and learners, and their use may
promote significant learning. In order to thoughtfully implement these types of
literacy, it is important to utilize a pedagogy of self-regulation learning (SRL)
that emphasizes organization and regulation of the learning by the learner in
cognitive processes, meta-cognitive processes, and motivational processes. The
students set themselves goals for learning, supervise and control the learning while
controlling thinking processes, motivation, and their behavior. Hence, the challenge
encountered by teaching staff is to find the right way to integrate technology and to
utilize the possibilities it affords in an online space.
In summary, it is necessary to note the difference between E-Learning in times of
emergency and E-Learning in times of routine, which requires replanning courses,
including deep pedagogical thinking. Therefore, those who experience distance
learning in an emergency are not truly exposed to high-quality distance learning
and therefore may form a negative opinion about distance learning.
In times of emergency such as the COVID-19 crisis, which make it necessary to
stop activities at educational institutions, the use of digital learning makes it possible
for learners to maintain a study continuity and to reduce the disruption of their
study routine, while also providing an emotional response and creating a supportive
social-educational framework (Altbach & De Wit, 2020). The use of online learning,
in the case of synchronous learning, affords students access to the study material
in their free time or, when scheduled in advance, makes it possible for them to
maintain contact with their peers and with the lecturers, to ask questions, participate
in discussions in the online class, and so on (Weissblei, 2020).
15.3 The Challenges of the Transition to E-Learning
E-Learning provides opportunities for changing conceptions regarding teaching and

learning and for attention to the following aspects:
1. Creating a setting with a “flexible” time and place—Students might turn the
digital option into a lack of commitment, into flexibility interpreted as a release
from an obligatory learning environment (Benade, 2017).
2. Managing time and learning—Students are required to display skills involving
time management and personal responsibility for their learning. Studies show
that the ability to manage learning and the demands of courses in a flexible
environment constitute a key to success in any learning, and particularly digital
learning that affords more opportunities in an expanded time and place (Allen &
Seaman, 2007; Hershkowitz & Kaberman, 2009).
3. Change and adjustment—The transition from traditional learning to digital
learning turns learning experiences into something completely different for the
students and lecturers. This transition generates resistance to such a change and
difficulty handling it. The transition does not let students and lecturers adjust
easily to the E-Learning environment and they need time to adapt and to shift to
a different type of pedagogy. Previously accustomed to and expecting a “face to
face” study experience, a human experience of learning and teaching—they are
compelled to be part of a process they did not anticipate. Students and lecturers
with a “traditional” spirit find it hard to adjust to the machine. They are required,
against their will, to accept the new learning circumstances with an open mind
and heart, to understand the advantages of E-Learning, such that a discussion
of these advantages with their peers might change this spirit and better prepare
students for online classes (Bates & Khasawneh, 2007; Joanna & Jason, 2016;
Kovoor, 2020; Palloff & Pratt, 2007).
4. Lack of digital literacy—Although most students have experienced in the
technological field, they lack digital literacy, in a world flooded with knowledge
in English and Hebrew. The ability to judge material, criticize, examine the
relevance for a specific topic—all these are manifestations of digital proficiency
(Bates & Khasawneh, 2007; Palloff & Pratt, 2007).
We shall demonstrate reading digital texts—reading academic texts in a
digital format is for most learners a complicated and tiring matter as a result of
orientation difficulties and the sense of fatigue by those who read a digital text.
Consequently, they may attain lower achievements than those who read printed
texts (Bates & Khasawneh, 2007).
5. Operating technology, creating a new study space—Distance learning poses a
challenge regarding providing a response for students who do not have access to
the internet or to other means necessary for distance E-Learning (Phelps, 2018;
Weissblei, 2020).
6. Lecturer skills—Lecturers lack pedagogical skills necessary for efficient and
effective use of online technologies (Palloff & Pratt, 2007). A lack of these skills
might lead to unwise use of these technologies for learning and teaching and
meaningful use of teaching options in the online expanse (Bates & Khasawneh,
2007).
7. Equality and quarantine—One of the major challenges in the context of E-
Learning in a time of crisis is equal opportunity for learners. Among the actions
taken by different countries in this context are distributing or lending end devices
to needy students, offering Internet packages at reduced prices, providing an
option of receiving printed study materials, and more. UNESCO too stresses that
quarantine and closing schools intensify the existing gaps and inequality in the
educational system and that schoolchildren, as well as academic students from
disadvantaged populations, are particularly vulnerable to the consequences of
the crisis, as in addition to the loss of learning opportunities they are also under
increased economic and social pressures (Weissblei, 2020). Moreover, many
students describe a sense of loneliness and social disconnection when learning
in an online environment. Students emphasize the lack of physical means of
reinforcement that usually exist in face-to-face learning (Cohen & Davidovitch,
2020). This sense of loneliness had a negative impact on students’ academic
achievements (Eshet-Alkalai & Geri, 2007; Hershkowitz & Kaberman, 2009).
8. Motivation, willpower, and self-demand—Self-motivation is a vital requirement
for learning, both in general and in online learning specifically. Nevertheless,
many learners who study online are found to lack motivation (Cohen & Davi-
dovitch, 2020). It is apparent that many students, after registering for distance
learning courses, remain behind and do not progress during the course, allowing
themselves to lag (Song et al., 2004). Therefore, when planning any course
involving distance teaching it is important to provide learners with constant
encouragement and reinforcement in order to raise their motivation for this
form of learning (Salmon, 2019). Moreover, students must be encouraged to
find the motivation to follow the course consistently and to equip themselves
well for future challenges. Only a positive attitude and motivation will help
them overcome the challenges of E-Learning. Despite the difficulties involved
in practicing the transition, students must understand that it is essential in order
to reap the benefits of E-Learning in the future (Palloff & Pratt, 2007).
In summary, the COVID-19 crisis is a great opportunity to accelerate pedagogical
processes in academia. A real window of opportunity has opened for change
and for transitioning from a system based on teaching and imparting information
to one based on learning. In fact, this is an opportunity for rapid assimilation
of technologies, innovative pedagogy, and advanced management mechanisms
(Altbach & De Wit, 2020). The COVID-19 crisis has proven that the academic
system in general and the faculty and students in particular have the resilience
and ability to function in times of crisis. Distance learning took a quantum leap
while teaching, learning, and evaluation took center stage in academic work.
In a short span of time the classroom walls came down, digital spheres were
established, contents were developed, and collegiality and collaboration among
academic institutions intensified. The use of E-Learning accelerated, making room
for independence and creativity. These independence and initiatives must be further
encouraged and enhanced. At the same time, it was clarified that E-Learning cannot
function by copy-pasting frontal learning and requires a different pedagogy. It is
necessary to rethink contents, pedagogy, learning methods, as well as of means of
evaluation (DePietro, 2020). There is a large discrepancy between the learning skills
acquired in the traditional classroom and in the virtual classroom, and this must be
compensated for.
Side by side with the difficulties that emerge in the shadow of crisis, distance
learning was found to have advantages as well, and the challenges that await us
must be further treated. We are at a point in time that constitutes an opening for
many studies on the topic of E-Learning and a vehicle for changes and actions
in academic instruction. An opportunity has been formed for developing creative
learning models. Hence, the current study examines the effect of E-Learning in times
of emergency in Israel’s system of higher education, including the perception of its
advantages and disadvantages by students and lecturers.
15.4 Research Method
This study is a quantitative study based on an attitudes survey conducted among

students and lecturers at academic institutions in Israel: universities and colleges.
The purpose of the study was to examine the impact of E-Learning in times of
emergency in Israel’s system of higher education, including its perceived advantages
and disadvantages among students and lecturers. The questionnaire presented the
respondents with several claims concerning the impact of E-Learning on the quality
of learning, to identify its advantages and disadvantages. The respondents were
asked to rank their replies on a scale of 1–5 (where 1 means not at all and 5 means
very much). The respondents replied, “very much” (5) or “not at all” (1) to several
statements related to the different effects of E-Learning on the quality of teaching
and learning.
The study was carried out via a wide survey among students at universities
and colleges in Israel, of whom 57.7% were students at public colleges, 26.6% at
teachers’ colleges, 12.8% at universities, and 3% at private colleges.
15.4.1 Research Questions
1. What are the personal, family-occupational, and academic background charac-

teristics of the students?
2. What are the personal, occupational, and academic background characteristics of
the lecturers?
3. What is the personal preference of students and lecturers for E-Learning in
general and for different types of classes in particular?
4. What is the personal preference of students for E-Learning by an academic

institution (teachers’ college, university, private college, public college)?
5. What is the personal preference of students for E-Learning by department (social
sciences and humanities, exact sciences)?
6. What type of E-Learning do students most prefer (synchronous, videotaped,
combined)?
7. What are students versus lecturers’ perceptions regarding the advantages and
disadvantages of E-Learning?
8. What difficulties arise in E-Learning as perceived by lecturers?
15.4.2 Characteristics of the Sample
The research population included students and lecturers.
15.4.2.1 Student Characteristics
We shall present the personal, family-occupational, and academic characteristics of

the 2015 students examined (N = 2015). Of all students, 46.8% were male and
53.2% female. In addition, 87.7% were studying for a bachelor’s degree, 39.0%
were not employed, 67.6% were single, and 31.1% were married. Also, 89.3% were
Jewish, 8.4% had a high socioeconomic status, 68.0% a medium status, and 23.6% a
low status. Nevertheless, the large majority (90%) noted that they have the necessary
resources and tools for E-Learning. Tables 15.1, 15.2, and 15.3 show the background
characteristics of the students who participated in the study.
15.4.2.2 Characteristics of the Lecturers
We shall present the personal, family-occupational, and academic background char-

acteristics of the 223 lecturers. These included 51.0% men and 49.0% women. Also,
62.2% had the rank of lecturer, 19.6% teacher, and 18.2% professor. Regarding
discipline, 64.7% were teaching at faculties of social sciences and the humanities
and 35.3% exact sciences (Table 15.4).
Table 15.1 Personal background characteristics of students in the sample (N = 2015)

Age Range 16–67
Mean 27.81
Median 26
Standard deviation 7.22
Gender N (%) Male 933 (46.8%)
Female 1061 (53.2%)
Total 100.0%
Jewish (89.3%)
Religion N (%) Muslim 126 (6.3%)
Christian 28 (1.4%)
Other 60 (3.0%)
Total 100.0%
Has special needs N (%) Yes 98 (5.0%)
No 1878 (95.0%)
Total 100.0%
Has language difficulties N (%) Yes 81 (4.1%)
No 1882 (95.9%)
Total 100.0%
Table 15.2 Family-occupational background characteristics of students in the sample (N = 2015)

Occupational status N (%) Employee 1116 (56.4%)
Not working/unemployed 772 (39.0%)
Self-employed 89 (4.4%)
Total 100.0%
Marital status N (%) Single 1354 (67.7%)
Married 621 (31.1%)
Divorced 23 (1.2%)
Widowed 2 (0.1%)
Total 100.0%
Financial status N (%) High 166 (8.4%)
Medium 1347 (68.0%)
Low 467 (23.6%)
Total 100.0%
Residential status N (%) Rental 805 (40.5%)
Parents’ home 703 (35.3%)
Dorms 132 (6.6%)
Other 349 (17.5%)
Total 100.0%
Table 15.3 Academic background characteristics of students in the sample (N = 2015)

Studying for degree Bachelor’s 1715 (87.7%)
Master’s 238 (12.2%)
PhD 3 (0.2%)
Total 100.0%
Type of academic institution N (%) University 146 (12.8%)
Teachers’ college 304 (26.6%)
Public college 660 (57.7%)
Private college 34 (3.0%)
Total 100.0%
Discipline N (%) Social sciences and humanities 1112 (64.5%)
Exact sciences 611 (35.5%)
Total 100.0%
Table 15.4 Background characteristics of lecturers in the sample (N = 223)

Age Range 29–83
Mean 52.30
Median 51
Standard deviation 10.66
Gender N (%) Male 107 (51.0%)
Female 103 (49.0%)
Total 100.0%
Academic rank N (%) Teacher 41 (19.6%)
Lecturer 130 (62.2%)
Professor 52 (18.2%)
Total 100.0%
Type of academic institution N (%) University 34 (23.1%)
Teachers’ college 43 (29.3%)
Public college 56 (38.1%)
Private college 14 (9.5%)
Total 100.0%
Field of teaching N (%) Social sciences and humanities 119 (64.7%)
Exact sciences 65 (35.3%)
Total 100.0%
15.5 Research Findings
15.5.1 Preferences Regarding E-Learning
15.5.1.1 Level of Preference
Table 15.5 lists the degree of personal preference expressed by students and lecturers
for E-Learning.
Table 15.5 Personal preference for E-Learning

Level of preferencea Standard
Low preference Medium preference High preference Total Mean deviation
Students N (%) 623 (31.1%) 461 (23.0%) 918 (45.9%) 100% 3.30 1.14
Lecturers N (%) 57 (25.6%) 87 (39.0%) 79 (35.4%) 100% 3.18 0.96
a The variables were aggregated such that values of 1–2.5 were categorized as indicating a low
preference for E-Learning, values of 2.51–3.5 a medium preference, and values higher than 3.51
a high preference for E-Learning. All the distributions of the research variables presented below
were aggregated in this method
Table 15.6 Mean of Mean SD n

personal preference for
E-Learning by academic Teachers’ college 3.48 1.11 303
institution University 3.37 1.18 146
Public college 3.36 0.80 34
Academic college 3.09 1.14 658
Both students and lecturers expressed high readiness to study/teach theoretical

classes via E-Learning, but their degree of readiness to study practical classes or
workshops online was low. The rate of preference for learning theoretical classes
online was high (59%) among students and (69%) among lecturers. The degree of
readiness to teach theoretical classes online was higher among lecturers than among
students (t = 4.05, p < 0.01).
The rates of preference for studying an exercise class online were lower than the
rates of preference for studying a theoretical class online, 47% among students and
42% among lecturers. No differences were found between lecturers and students
in readiness to study an exercise in a theoretical class online (t = 0.50, p = NS).
The rates of preference for studying a practical class online were very low (18%
among students and 14% among lecturers) and on this topic as well no differences
were found in the preferences of the two groups (t = 1.12, p = NS). Finally,
low preference rates were found also for studying workshops online (27% among
students and 19% among lecturers). Students displayed a higher preference than
lecturers for studying workshops online (t = 2.49, p < 0.05).
15.5.1.2 Preference for E-Learning by Features of Students’ Studies
Table 15.6 lists the degree of preference for E-Learning by students’ type of
academic institution.
Significant differences were found in degree of preference by type of institution
(F = 9.17, p < 0.01).
Table 15.7 presents the mean for degree of preference for E-Learning by
features of the students’ department (exact sciences versus social sciences and the
humanities).
Table 15.7 Mean for degree Mean SD n

of personal preference for
E-Learning by features of the Social sciences and humanities 3.31 1.15 1108
department Exact sciences 3.30 1.14 608
Table 15.8 Type of preferred online class

Preferred type of online class
Synchronous online class Videotaped lecture Combination Total
Students 54.3% (n = 1053) 10.5% (n = 203) 35.2% (n = 682) 100%
No significant differences were found in the preference for E-Learning by this

feature (t = 0.01, p = NS).
15.5.1.3 Type of Online Class Preferred by Students
Another aspect of preferences for E-Learning examined in the study is the type of
online class preferred by students: synchronous online class, videotaped lecture, or
a combination. Table 15.8 lists students’ preferences on this issue.
Evidently, slightly more than half the students preferred a synchronous online
class and slightly more than one-third preferred a combination of a synchronous
online class and videotaped lectures. Only about one-tenth of the students preferred
E-Learning solely by online lectures.
15.5.1.4 Perception of E-Learning
Table 15.9 lists students’ different perceptions of E-Learning, as evident from their
replies.
Evidently, there is a high degree of agreement among students regarding two
perceptions of E-Learning that are unrelated to the teaching process itself: a very
high proportion of the students are of the opinion that E-Learning saves resources
(such as petrol for traveling to school, hours spent in traffic, time waiting between
classes at the university), however, a high proportion of the students regret that E-
Learning prevents interpersonal interaction with students and lecturers, to which
they are accustomed in the regular form of learning at the university.
Students’ perceptions of the contribution of E-Learning to improving teaching
are usually negative: about 40% of students are of the opinion that E-Learning
improves study abilities versus 32% of students who are of the opinion that E-
Learning has a negative impact on study abilities. Regarding the dimensions of
lecturer availability, order and organization of studies, interest in studies, and
clarity of studies—more students were found to hold the opinion that E-Learning
is detrimental to the study experience than those who thought that E-Learning helps
improve the academic process.
Table 15.9 Perceived E-Learning by students
Distribution of scores
Low Medium High Total Mean SD Median
Saving resources 22.6% (n = 437) 12.4% (n = 240) 65.0% (n = 1259) 100% 3.74 1.37 4.33
Lack of interpersonal interaction 23.5% (n = 450) 17.3% (n = 332) 59.2% (n = 1136) 100% 3.60 1.27 4.00
Improving learning abilities in E-Learning 32.3% (n = 647) 28.6% (n = 575) 39.2% (n = 789) 100% 3.15 1.00 3.16
Lecturer availability 44.2% (n = 881) 28.3% (n = 563) 27.5% (n = 547) 100% 2.84 1.18 2.75
Improving teaching—order and organization 45.8% (n = 907) 23.2% (n = 460) 30.9% (n = 612) 100% 2.76 1.29 2.80
Improving teaching—interest 51.1% (n = 1015) 20.6% (n = 409) 28.3% (n = 563) 100% 2.67 1.27 2.50
Improving teaching—clarity 51.4% (n = 1012) 21.4% (n = 421) 27.3% (n = 537) 100% 2.58 1.29 2.33
15 E-Learning in Times of Crisis: An Incidental or Facilitative Event?
467
Table 15.10 presents lecturers’ perceptions of E-Learning.

It is evident from Table 15.10 that lecturers have negative opinions on this type
of teaching, even more so than students: Two-thirds of the lecturers were unhappy
that E-Learning reduces their interpersonal interactions with the students and among
themselves. The proportion of lecturers who missed the social interaction was even
higher than the proportion of students with this attitude (t = 14.68, p < 0.01).
In addition, only about one-fifth of the lecturers were of the opinion that E-
Learning is of a higher quality than regular teaching, and here too it seems that
lecturers’ perception of E-Learning is more negative than students. Only one-sixth
of the lecturers were of the opinion that E-Learning is beneficial for students with
regard to students’ ability to deal with the studies and with the study material or to
benefit from the classes.
15.5.1.5 Difficulties with Implementing E-Learning
Table 15.11 presents the findings of an open question in which students were asked
to note the resources they lack in order to implement E-Learning.
The replies given show that 62% of the interviewees who have difficulties imple-
menting E-Learning noted that the difficulty is related to technological infrastructure
(lack of a computer, Internet infrastructure, camera, microphone, and so on), 13.7%
noted that the difficulty is related to a lack of proper studying conditions (place
to study, quiet necessary to study), and 13.2% noted a difficulty with adjusting
to E-Learning (lack of concentration, motivation, or time, preference for frontal
teaching, lack of technological knowledge necessary for E-Learning). Ten percent
of the students noted that they do not have the necessary resources to implement
E-Learning.
Analysis of the demographic features of interviewees who noted that they lack
resources for online studies indicates the following findings:
• More men than women noted that they lack resources for E-Learning (11.2% vs.
8.4%, respectively; χ = 4.48, p < 0.05).
• Those who noted having difficulties with E-Learning were slightly older than
those who did not note such difficulties (mean age of 28.0 vs. 26.2; t = 10.52,
p < 0.01).
• Those who were not working noted that they lack resources more than those
who were employed on a salary basis (12.1% vs. 7.5%, respectively; χ = 11.74,
p < 0.01).
• Those who had language difficulties reported much more difficulties with
E-Learning than those who had no language difficulties (25.0% vs. 9.1%,
respectively; χ = 22.72, p < 0.01).
• Students of exact sciences reported more difficulties than students of social
sciences and the humanities (13.1% vs. 8.2%, respectively; χ = 10.43, p < 0.01).
Table 15.10 Perception of E-Learning by lecturers
Distribution of scores
Low Medium High Total Mean SD Median
Lack of interpersonal interaction 13.4% (n = 29) 20.7% (n = 45) 65.9% (n = 143) 100% 4.36 0.63 4.00
Interest, order, organization, and clarity in teaching 42.6% (n = 92) 37.5% (n = 81) 19.9% (n = 43) 100% 2.80 0.89 2.81
Benefit for the students 45.0% (n = 100) 39.2% (n = 87) 15.8% (n = 35) 100% 2.71 0.77 2.63
15 E-Learning in Times of Crisis: An Incidental or Facilitative Event?
469
Table 15.11 Missing N %

resources for E-Learning
(among students who noted a Lack of computer 69 36.5%
lack)a Lack of internet infrastructure 47 24.9%
Lack of place for studying 21 11.1%
Lack of camera/microphone 15 7.9%
Lack of quiet place for studying 10 5.3%
Lack of concentration/motivation/time 10 5.3%
Prefers frontal teaching 10 5.3%
Lack of printer 8 4.2%
Lack of technological knowledge 5 2.6%
Other 9 4.8%
Did not reply 15 7.9%
Total 189 115.80%
a The report included replies noted by at least 5 students.
Each student could mark more than one answer, so the

percentages do not total one hundred
• No differences were found in the proportion of those who reported that friends
are resources for learning between those with versus those without special needs.
No differences were found by academic institutions (university vs. college) as
well.
15.5.1.6 Factors Affecting Personal Preference for E-Learning Among

Students
The variables predicting personal preferences for E-Learning among students

were examined by constructing a linear regression model: Model 1, in which the
dependent variable is the student’s preference for E-Learning and the independent
variables are all the other research measures (saving resources, lack of interpersonal
interaction, improving learning abilities in E-Learning, lecturer availability, and the
three measures of teaching improvement).
The regression model in Model 1 is significant (F(7,1888) = 1363.2, p < 0.001)
and explains 83% of the variance between students in preferences for E-Learning
(r2 = 0.835). The regression coefficients are listed in Table 15.12. Although all the
independent variables were found to be significant predictors of personal preference
for E-Learning among students, the most conspicuous predictor is improving
learning abilities in E-Learning. Other variables that are significant predictors
of personal preference for E-Learning (although the strength of their prediction
is about half that of improving learning abilities) are desire for interpersonal
interaction, interest in teaching, and saving resources.
The second model examined is a hierarchic regression model, which in the first
stage included background variables of the students (gender, occupational status,
studying in a department of exact sciences or social sciences and the humanities,
Table 15.12 Linear regression coefficients for students: Model 1—predicted only by measures of
perceived E-Learning
B β t
Constant 1.52 17.35∗∗
Improving learning abilities in E-Learning 0.48 0.43 19.16∗∗
Lack of interpersonal interaction −0.20 −0.23 16.15∗∗
Improving teaching—interest 0.19 0.21 8.91∗∗
Saving resources 0.15 0.18 15.15∗∗
Improving teaching—order and organization 0.07 0.08 3.88∗∗
Lecturer availability −0.07 −0.07 −5.55∗∗
Improving teaching—clarity −0.04 −0.06 −2.44∗
**p < 0.01, *p < 0.05
Table 15.13 Features of the model among students: Model 2—predicted by background variables
and measures of perceived E-Learning
F r2
Sub-model A—Predictors: background variables 111.1* 0.62
Sub-model B—Predictors: background variables and perception of online 232.0* 0.85
environment
*p < 0.01
special needs, language difficulties, significance of grades for the student and the
level of grades attained, difficulties with online learning) and in the second stage
also the research measures as in the first model. Table 15.13 presents the features of
the regression model in the first and second stages of the overall model.
It is evident that both sub-models are significant. The background variables
explain a considerable proportion (62%) of the differences between students in their
preference for E-Learning, but the measures of perceived E-Learning also add to the
ability to explain the variance in the overall model, which reached 85%.
The following is the key for the dichotomous variables in the tables:
Gender: 0 = female, 1 = male; occupational status: 1 = employee, 2 = other
Special needs: 1 = yes; 0 = no; language difficulties: 1 = yes, 0 = no
Department: 1 = exact sciences; 0 = other; marks: 1 = higher than 80, 0 = lower
than 80
Type of institution: 1 = university, 0 = other
Table 15.14 presents the regression coefficients in the first model that includes
predictors of preference for E-Learning, which are background variables.
At this stage, 8 background variables were found to explain the preference for
E-Learning, but the most conspicuous was the perception of E-Learning on grades:
the more important it is for the student to achieve high grades in his or her academic
studies, the lower the readiness to study online. Other variables found to predict the
preference for E-Learning but their impact on preference is one-fifth or less than that
of the grade significance variable are lack of resources for E-Learning, studying in
Table 15.14 Linear regression coefficients for students: Model 2 sub-model A—predicted by
background variables
B β t
Constant 6.03 20.74∗∗
Significance of grades −0.52 −0.73 −31.55∗∗
Has resources for E-Learning −0.55 −0.15 −6.66∗∗
Type of department (exact sciences/social) 0.27 0.11 4.54∗∗
Occupational status 0.20 0.09 3.95∗∗
Type of institution (university/college) −0.14 −0.09 −2.22∗
Special needs −0.21 −0.04 1.86
Age −0.01 −0.03 −1.56
Personal evaluation by the student −0.01 −0.01 0.27
Gender −0.04 −0.02 0.86
Language difficulties 0.15 0.02 1.07
Student’s grades −0.06 −0.02 −1.13
**p < 0.01, *p < 0.05
a department that belongs to the exact sciences, employee occupational status, and
studying in a college.
Table 15.15 presents the regression coefficient in the second model that includes
predictors of the preference for E-Learning, which are background variables and
measures of perceived E-Learning.
Improving learning abilities via E-Learning is the strongest predictor of prefer-
ence for E-Learning, as found in Model 1. Other conspicuous variables predicting
preference are saving resources, which increases the preference, and desire for
interpersonal interaction and high significance of grades for the student, which
decrease the preference rates.
15.5.1.7 Factors that Affect One’s Personal Preference for E-Learning

among Lecturers
As among students, two models were calculated to try and predict faculty members’
personal preference for E-Learning. The first model included only variables related
to teaching per se (namely, the perceived benefits of E-Learning for students;
interest, order, organization, and clarity of teaching; and perceived lack of personal
interaction). The following is the first linear regression model (Table 15.16).
The first regression model is significant (F(3,212) = 111.05, p < 0.01) and
explains 61% of the differences between faculty members in the preference for
E-Learning (r2 = 0.611). The findings indicate that all three variables related to
perceived E-Learning are significant predictors of the preference for E-Learning,
but the variable of perceived interest, order, organization, and clarity in teaching—
has greater prediction strength than the two other variables.
Table 15.15 Linear regression coefficients for students: Model 2 sub-model B (full model)—
predicted by background variables and measures of perceived E-Learning
B β t
Constant 2.75 11.67∗∗
Improving study abilities in E-Learning 0.46 0.41 11.46∗∗
Saving resources 0.15 0.19 10.30∗∗
Lack of interpersonal interaction −0.18 −0.19 −8.79∗∗
Significance of grades −0.10 −0.15 −6.17∗∗
Improving teaching—interest 0.08 0.09 2.45∗
Improving teaching—order and organization 0.07 0.09 2.63∗∗
Has resources for E-Learning −0.27 −0.07 −5.05∗∗
Type of institution (university/college) −0.08 −0.06 −2.09∗
Improving teaching—clarity −0.05 −0.06 −1.84
Type of department (exact sciences/social) 0.12 0.05 3.07∗∗
Lecturer availability −0.04 −0.04 −1.86
Age −0.01 −0.03 −2.21∗
Occupational status 0.05 0.02 1.46
Special needs −0.14 −0.02 −1.98∗
Language difficulties −0.04 −0.01 −0.46
Student’s grades −0.02 −0.01 −0.57
Personal evaluation of the student −0.01 −0.01 −0.66
Gender 0.01 0.00 0.28
**p < 0.01, *p < 0.05
Table 15.16 Linear regression coefficients among lecturers: Model 1—predicted only by mea-
sures of perceived E-Learning
B β t
Constant 2.14 5.94∗
Interest, order, organization, and clarity in teaching 0.39 0.37 4.99∗
Lack of interpersonal interaction −0.24 −0.25 −4.59∗
Benefits for students 0.32 0.25 3.18∗
*p < 0.01
The second regression model calculated includes in sub-model A an examination

of the effect of background variables on the preference for E-Learning (including the
significance of teacher evaluations, present score in teacher evaluations, academic
institution—university versus college, teaching exact sciences or social sciences and
the humanities, self-evaluation as a lecturer, age, gender, academic rank). Sub-model
B includes all the above variables as well as the measures of perceived E-Learning.
Table 15.17 presents the characteristics of the regression model in the first and
second stages of the overall model.
It is evident that both sub-models are significant, and that the sub-model that
includes only background variables explains about one-third of the differences
Table 15.17 Characteristics of the model among lecturers: Model 2—predicted by background
variables and measures of perceived E-Learning
F r2
Sub-model A—Predictors: Background variables 6.10* 0.32
Sub-model B—Predictors: Background variables and perception of the online 18.18* 0.67
environment
*p < 0.01
Table 15.18 Linear regression coefficients among lecturers: Model 2 sub-model A—predicted by
background variables
B β t
Constant 5.74 7.45∗
Significance of evaluations received by the faculty member −0.38 −0.53 −6.52∗
Age −0.01 −0.13 −1.53
Gender −0.22 −0.12 −1.35
Mean teacher evaluations of the faculty member −0.20 −0.10 −1.06
Self-evaluation as a lecturer −0.09 −0.06 −0.65
Academic rank 0.17 0.05 0.64
Teaches in department −0.10 −0.05 −0.61
Academic institution 0.10 0.04 0.53
*p < 0.01
among faculty members in the preference for E-Learning, while the full model
explains about two-thirds of the differences in preference (Table 15.18).
The findings indicate that the only background variable that explains the prefer-
ence for E-Learning among lecturers is their concern that their teacher evaluation
scores will drop following the transition to E-Learning.
Table 15.19 presents the regression coefficients for the entire second model,
which includes predictors of preference for E-Learning in the form of background
variables and measures of perceived E-Learning. The perceived lack of interpersonal
interaction is the most influential variable (negatively) on lecturers’ degree of
preference for E-Learning, and two variables related to learning highly affect
readiness: benefits of E-Learning for students, and interest, order, organization, and
clarity of teaching.
Two background variables that have less of an effect on preference, but their
impact is significant are also worthy of note: age (the preference for E-Learning
drops with the lecturer’s age) and the concern that a transition to E-Learning will
negatively affect the evaluations received by the faculty member.
Table 15.19 Linear regression coefficients among lecturers: Model 2 sub-model B (full
model)—predicted by background variables and measures of perceived E-Learning
B β t
Constant 4.33 6.28∗∗
Lack of interpersonal interaction −0.28 −0.30 −4.12∗∗
Benefits for students 0.33 0.27 2.50∗
Interest, order, organization, and clarity in teaching 0.27 0.26 2.47∗
Age −0.01 −0.17 −2.83∗∗
Significance of the evaluations received by the faculty member −0.11 −0.15 −2.26∗
Self-evaluation as a lecturer −0.14 −0.09 −1.34
Gender −0.16 −0.08 −1.38
Academic institution 0.11 0.05 0.79
Department 0.06 0.03 0.53
Mean evaluation score of the faculty member 0.05 0.02 0.38
Academic rank −0.03 −0.01 −0.19
**p < 0.01, *p < 0.05
15.6 Summary of the Findings and Discussion
The current study examined the opinions of students and lecturers regarding the
advantages and disadvantages of E-Learning from various aspects, in a systemic
multi-institutional perspective. The study included students at various academic
institutions: universities, teachers’ colleges, public colleges, and private colleges.
The research findings indicate that the degree of preference for E-Learning is
not high: Less than half the students and about one-third of the lecturers expressed
a preference for studying in this format. Both groups noted the lack of personal
interaction with students and lecturers as one of the main shortcomings of E-
Learning. Another perceived shortcoming of E-Learning that emerged in both
groups is the concern of harm to students’ grade average or to lecturers’ mean
evaluations by students following the transition to E-Learning.
Saving resources is one of the strengths of E-Learning as perceived by the
students. Nevertheless, most do not perceive E-Learning as advantageous for them
with regard to the quality of learning per se. Notably, improving the ability to study
online is the measure that most influenced students’ preference for E-Learning, but
only 39% of the students were of the opinion that E-Learning indeed offers them
such an advantage.
The perceived benefit of E-Learning for students and the degree of interest, order,
organization, and clarity in teaching are elements that have considerable influence
on the preference for E-Learning among lecturers, but less than half the lecturers
were of the opinion that E-Learning encompasses these features. Notably, older
lecturers showed less preference for E-Learning versus younger lecturers, perhaps
because they feel stronger adjustment difficulties to teaching in this method.
Both the students and the lecturers were of the opinion that E-Learning is not
equally suited for all types of classes. Both groups mostly think that it is possible to
teach a theory class via E-Learning, but few think that a workshop or practical class
can be taught in this way.
Students prefer E-Learning via a synchronous online class over E-Learning via
videotaped lectures, and some think that the best way to learn online is by a
combination of these methods.
About one-tenth of the students reported that there is a problem that prevents
them from studying online. The main difficulty reported by the students was a lack
of technology that facilitates implementation of E-Learning (computer, microphone,
camera, etc.), and other difficulties were a lack of proper study conditions and a
difficulty with adjustment to online studies. The large majority (90%) noted that
they have the necessary resources and tools for E-Learning.
According to students’ evaluations, use of technology was not found to be
thoughtful or to affect the quality of teaching. It appears that faculty members
use technology technically more than pedagogically. A possible explanation for
the criticism voiced by students is the considerable difference between emergency
E-Learning and routine E-Learning, which requires replanning courses, including
extensive pedagogical thinking. Therefore, those who experience distance learning
in times of emergency are not truly exposed to high-quality distance learning and
thus may formulate a negative opinion about distance learning.
In times of emergency, E-Learning constitutes a solution to the option of closing
down the entire school system (Altbach & De Wit, 2020). The measures of teaching
quality and success in times of emergency and in times of routine differ. In times
of emergency, there is less of a demand that lecturers maintain the study pace,
in the understanding that these times will end shortly and it will be possible to
close the gaps, which does not happen in times of routine. In this case as well the
students demanded more professionalism of lecturers in E-Learning and preparation
of high standard study material, as well as advanced interactions facilitated by online
systems, such as the interaction between students and lecturers, discussions, group
work in private rooms, collaborative environments, and more.
As stated, the students and the lecturers noted the lack of personal, social,
and emotional interaction (SEL—social and emotional learning) with students and
lecturers as one of the main shortcomings of E-Learning/teaching. These findings
correspond with Salmon’s (2019) model, which claims that at the beginning of
every online course, after overcoming the technical difficulties, it is necessary
to develop socialization among the learners, encourage them to text each other
and establish their identity on the Internet. This stage is essential for establishing
efficient teaching/learning further on.
The study illuminates the role of the lecturer in the digital era, in teaching
and particularly—the role of the professional elements responsible for teaching
and learning in academic institutions, with its pedagogical aspects. The new
pedagogic world must set itself renewed goals and reformulate the image of the new
teacher, i.e., the digital pedagogue. The digital pedagogue must understand the new
opportunities afforded by the technological world to create a learning environment
that will deal with more plentiful, varied, and value-laden tasks than ever before
(Benade, 2017).
Most of the institutional studies on ICT (Information and Communications
Technology) focus on the number of courses taught, the number of students
participating, and the success of E-Learning. They scarcely refer to how ICT-based
teaching can be utilized to upgrade teaching and learning and how online aspects
can be used to improve various pedagogical issues such as interaction between the
faculty and the students or improving course contents. In the recognition that the
COVID-19 era facilitates thinking that encompasses a paradigmatic change, there is
a need for discovery and assimilation of new pedagogies that are uniquely suited to
the new technologies found. In order to improve lecturers’ teaching, it is necessary
to boost the pedagogical aspects of the new technological tools. A program should
be offered that assimilates the new tools as an inseparable part of the practice of
teaching rather than as an external aide to the teaching and learning process. We can
assume that the big challenge of online learning is the day after the crisis. Is there a
way back?
It will be interesting to repeat the study a year after the crisis and compare the
results.
References
Allen, I. E., & Seaman, J. (2007). Online nation: Five years of growth in online learning. Sloan
Consortium.
Altbach, G. P. G., & De Wit, H. (2020). Are we at a transformative moment for online learn-
ing? https://www.forbes.com/sites/andrewdepietro/2020/04/30/impact-coronavirus-covid-10-
collegesuniversities/#1867f57d61a6
Anderson, T. (Ed.). (2008). The theory and practice of online learning. Athabasca University Press.
Ashburn, F. A., & Floden, E. R. (2006). Meaningful learning using technology: What educators
need to know and do. Teachers College Press.
Barak, M., Dori, Y. J., & Hussein-Farraj, R. (2012). Lifelong learning at the Technion: Students’
perceptions of and experiences in distance learning. In Y. Eshet-Alkalai, A. Caspi, S. Eden,
N. Geri, Y. Yair, & I. Kalman (Eds.), Conference proceedings: Chais conference of learning
technologies 2012: The learning individual in the age of technology. Open University. [Hebrew]
Bates, R., & Khasawneh, S. (2007). Self-efficacy and college students’ perceptions and use of
online learning systems. Computers in Human Behavior, 23(1), 175–191.
Benade, L. (2017). The impact on teachers’ work: Practitioner attitudes and reflective transitions.
In Being a teacher in 21st century (pp. 163–176). Springer.
Berger-Tikochinski, T., Cohen, E., Hadad, N., & Manny-Ikan, E. (2020). Essential skills for
distance learning (Vol. 1, pp. 1–16). Henrietta Szold Institute [Hebrew]
Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and
Development, 42, 21–29.
Cohen, A. (1999). Moderated teaching and distance learning via the internet. Mahshevim Behinuch,
49, 8–16. [Hebrew]
Cohen, E., & Davidovitch, N. (2020). The development of online learning in Israeli higher
education. Journal of Education and Learning, 9(5), 15–26. Published by Canadian Center
of Science and Education, Canada.
Dabbagh, N., & Bannan-Ritland, B. (2005). Online learning: Concepts, strategies, and application
(pp. 68–107). Pearson/Merrill/Prentice Hall.
Davidovitch, N. (2014). Learning-focused teaching and backward course design—from trans-
ferring knowledge to imparting skills. In N. Notzer (Ed.), To excel in academic teaching:
LECTURER handbook of updated strategies and competencies (pp. 63–74). College for
Academic Studies. ISBN: 978-965-916628-2-0
Davidovitch, N., & Eckhaus, E. (2020). Exploring the true motivation of faculty members to
promote technological innovation in their courses. In S. Nazir, T. Ahram, & W. Karwowski
(Eds.), Advances in human factors in training, education, and learning sciences. AHFE 2020.
Advances in intelligent systems and computing (p. 1211). Springer.
Davidovitch, N., & Yavich, R. (2015). Technology-related involvement: The effect of the Mashov
system on parent involvement in Israeli junior highs. Journal of International Education
Research, 11(4), 243–252.
Davidovitch, N., & Yavich, R. (2016). WhatsApp messaging: Achievements and success in
academia. International Journal of Higher Education (IJHE), 5(4), 255–261.
Davidovitch, N., & Yavich, R.. (2018). Usage of mobile phone applications and its impact on
teaching and learning. International Journal of Higher Education (IJHE), 7(1), 1–9. ISSN:
1927-6044
DePietro, A. (2020). Here’s a look at the impact of Coronavirus (COVID-19) on colleges
and universities in the U.S. Forbes. https://www.forbes.com/sites/andrewdepietro/2020/04/30/
impact-coronavirus-covid-19-colleges-universities/?sh=3ec5a84e61a6
Ertmer, P. A., & Ottenbreit-Leftwich, A. T. (2010). Teacher technology change: How knowledge,
confidence, beliefs, and culture intersect. Journal of Research on Technology in Education,
42(3), 255–284.
Eshet-Alkalai, Y., & Geri, N. (2007). Does the medium affect the message? The influence of text
representation format on critical thinking. Human Systems Management, 26(4), 269–279.
Goldschmidt, R. (2013). Online academic learning and its recognition. Jerusalem: Knesset—
Research and Information Center. [Hebrew]
Hershkowitz, A., & Kaberman, T. (2009). Training and professional development of teachers via
distance learning as a way of coping with the shortage of teachers. Technion, Israel Institute of
Technology, Department of Education in Technology and Science. [Hebrew]
Joanna, N. P., & Jason, A. S. (2016). Making hybrids work: An institutional framework for
blending online and face-to-face instruction in higher education. National Council of Teachers
of English.
Kovoor, S. M. (2020). Crisis management: Resilience and change. SAGE Publications, Inc.
Miller, N., & Forkosh-Baruch, A.. (2018). Use of online teaching in linguistics among high
school students taking the matriculation exam in the eleventh grade. In Y. Eshet-Alkalai, et al.
(Eds.), Conference proceedings: 13th Chais conference of learning technologies: The learning
individual in the age of technology. Open University. [Hebrew]
Mioduser, D., Nachmias, R., & Forkosh-Baruch, A. (2010). New literacies for the knowledge
society. Eureka.
Nir-Gal, O. (2002). Distance learning: The role of the teacher in a virtual learning environment.
Maof Umaaseh, 6, 103–124. [Hebrew]
Palloff, R. M., & Pratt, K. (2007). Building online learning communities: Effective strategies for
the virtual classroom. John Wiley & Sons.
Phelps, R. (2018). Crisis Management: How to develop a powerful program. Chandi Media.
Salmon G. (2019) E-moderating: The key to online teaching and learning. .
Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: Student
perceptions of useful and challenging characteristics. The Internet and Higher Education, 7(1),
59–70.
Wadmany, R. (2017). Digital pedagogy—from theory to practice. Mofet, Kibbutzim College.
[Hebrew]
Wadmany, R. (2018). Digital pedagogy—Opportunities for different learning. Mofet, Kibbutzim

College. [Hebrew]
Wagner, E. D., & McCombs, B. L. (1995). Lerner centered psychological principles in practice:
Designs for distance education. Educational Technology, 35, 32–35.
Weissblei, E. (2020). Emergency distance learning while education institutions are closed due to
the coronavirus outbreak. Knesset—Research and Information Center. [Hebrew]
Chapter 16
The Relative Efficiencies of Higher
Education in OECD Countries
Zilla Sinuany-Stern and Arthur Hirsh
Abstract Studies of productivity of systems of Higher Education (HE) on the

national level are of interest for two main reasons: education is an important factor
for productivity growth for the macro-economy, and the efficiency of spending
public resources on HE is of key interest in the context of accountability specifically
relative efficiency compared with other developed countries. The objective of this
study is to evaluate the relative efficiency of HE in OECD countries from the public
viewpoint; how well OECD countries utilize their public resources to achieve their
outputs relative to each other. For this study, two inputs are chosen reflecting the
public investment in HE. Six outputs are chosen reflecting the main outcomes of
HE in terms of: accessibility of tertiary education, employment level, earnings level
relative to secondary education, net financial returns from HE, internal rate of return,
and research articles level. The data is taken, mostly, from the OECD report on
education in 2019. Out of 37 OECD countries 29 are considered in this study. Due
to missing data 8 countries are not included. The stress on efficiency from the public
viewpoint is a strength of this study in relation to previous OECD efficiency studies.
The original Data Envelopment Analyses (DEA) basic models are, which provide
dichotomy of the countries into two groups: efficient and inefficient. Moreover,
several efficiency rank-scaling methods based on DEA, and several multivariate
statistic methods are utilized here. The use of a variety of efficiency rank-scaling
methods, while choosing the robust one, is another strength of this research. The
results indicate that the robust method is cross efficiency, as it is significantly
correlated with each of the other efficiency methods, and it has the highest average
correlation with other efficient methods. From the 29 studied OECD countries, the
USA is found to be the most efficient in HE. However, when we use only the first
Z. Sinuany-Stern ()
Beer Sheva, Israel
A. Hirsh
Webster Vienna Private University, Vienna, Austria

482 Z. Sinuany-Stern and A. Hirsh
input versus the six above outputs, Italy became the most efficient country. The USA
is ranked third in this case, while Italy is ranked fourth in the original case.
Keywords Efficiency · OECD · Higher education · Data envelopment analysis ·

Multivariate statistics · Ranking · Rank-scaling
16.1 Introduction
Since the establishment of the world’s first university in the eleventh century,
universities have traditionally functioned as ivory towers, often tied with religion,
permitting entry to a chosen few. Over centuries, universities retained their elitist
status and helped perpetuate social inequalities. After 900 years, this situation,
in which an “elite” has hegemony over higher education (HE) came to an end.
The end of WWII marks the transition of HE Institutions (HEIs) from elitist to
democratic and open in nature. In most Western countries, the accelerated pace of
this transformation in the twentieth century led to the massification of academic
studies with up to 200 million HE students in 22,000 HEIs institutions by the end of
the century (Altbach et al., 2017). In the 1950s, undergraduate students accounted
for a mere 3%–5% of their relevant age group (cohorts) in Europe. At the end
of the century, these figures ranged from 36% to 53%. Today, in most European
countries and the USA, over 60% of cohorts are students in academic institutions
(Davidovitch et al., 2012; Lindberg, 2007). In the modern world, universities have
two main purposes: to equip students with advanced skills useful in the workplace,
and to further human knowledge and understanding of the world. Over the years, the
accountability of higher education, and its efficiency have become important issues
for public constituencies of HE.
The Organization for Economic Co-operation and Development (OECD) is an
international organization of those developed countries that accept the principles of
representative democracy and a free market economy. The objective of this study is
to evaluate the relative efficiency of HE in OECD countries from a public viewpoint.
How well do OECD countries utilize their public resources to achieve their outputs
relative to each other country?
Studies of productivity of systems of HE are of interest for two main reasons:
education is an important factor for productivity growth for the macro-economy,
and the effectiveness of spending the resources is of key interest in the context of
accountability (specifically relative efficiency, and benchmark analysis). As shown
in Table 16.1 (output Y5) the rate of return on HE public investment among the 29
OECD countries included in this study ranges from 3.1% to 14.3%. Three countries
have a rate less than 5%, and eight countries have a rate more than 10%.
There are additional inputs and outputs to be considered in measuring the
efficiency of a complex system such as HE. Thus, the basic approach taken here
to measure the relative efficiency of decision-making units (DMUs—countries in
this case) is data envelopment analysis (DEA), as it can handle multiple inputs
16 The Relative Efficiencies of Higher Education in OECD Countries 483
Table 16.1 OECD countries input and output data

Inputs Outputs
Country X1 X2 Y1 Y2 Y3 Y4 Y5 Y6
Australia 3.626 39.523 45.727 84 131 117,686 0.121 914
Austria 3.555 93.575 32.711 86 147 149,454 0.060 620
Belgium 2.747 82.357 40.638 86 130 207,637 0.111 748
Canada 3.831 53.263 57.888 83 146 70,747 0.070 1102
Chile 5.358 35.842 25.168 84 238 30,983 0.099 349
Czech Republic 1.782 72.506 24.262 87 169 66,202 0.064 427
Denmark 3.708 89.793 38.055 88 128 67,739 0.039 705
Finland 3.274 92.479 45.188 87 140 79,832 0.054 609
France 2.161 77.130 36.897 85 157 103,930 0.069 1094
Germany 2.843 83.013 29.065 89 169 177,514 0.076 1203
Hungary 1.619 62.806 25.101 86 179 105,700 0.092 419
Ireland 2.626 71.699 46.935 85 174 247,211 0.143 488
Israel 2.297 55.956 50.916 87 156 104,724 0.122 665
Italy 1.493 60.971 19.324 81 139 127,733 0.078 953
Korea 2.912 37.577 49.008 78 141 25,636 0.056 624
Latvia 2.047 65.301 33.939 89 146 52,052 0.090 151
Luxembourg 1.136 91.859 43.887 86 148 288,310 0.070 173
Netherlands 4.014 67.485 38.344 90 150 284,446 0.105 957
New Zealand 3.691 50.597 39.293 88 133 64,265 0.074 495
Norway 4.199 93.060 43.576 89 126 48,696 0.034 580
Poland 2.578 78.690 30.920 89 156 73,547 0.074 519
Portugal 1.811 61.390 24.950 88 169 127,847 0.094 457
Slovak Republic 2.005 69.502 24.583 83 163 51,472 0.058 263
Slovenia 2.092 81.623 32.455 89 164.4 146,528 0.084 278
Spain 2.173 65.585 37.253 82 157 61,566 0.065 830
Switzerland 3.885 84.240 43.744 89 153 32,517 0.031 919
Turkey 4.634 73.831 20.780 74 164 123,837 0.115 402
UK 3.269 28.260 45.782 86 142 96,621 0.110 1373
USA 3.086 34.571 47.431 82 172 199,758 0.113 2222
and multiple outputs, with various types of measurement units, when their weights
(prices) are unknown. DEA provides dichotomy of the countries into two groups:
efficient and inefficient (Banker et al., 1984; Charnes et al., 1978). Moreover, several
efficiency rank-scaling methods based on DEA, and several multivariate statistic
methods were utilized here (Adler et al., 2002). At the first stage, all the efficiency
rank-scaling methods were screened for feasibility in our case study. At a second
stage, from the set of feasible efficiency rank-scaling methods, the robust method
is found and selected to represent the final robust efficiency scale-rank of OECD
countries based on the maximal average correlation of each efficiency method with
the other methods (based on Sinuany-Stern & Friedman, 2021).
For this study, two inputs are chosen reflecting the public investment in HE: as
a percent of the total government budget, and as a percent of the HEIs total budget.
Six outputs are chosen reflecting the main outcomes of HE from public perspective
in terms of: accessibility of tertiary education, employment level, earnings level
relative to secondary education, net financial returns, internal rate of return, and
research articles’ level. Out of 37 OECD countries 29 were considered in this study.
Due to missing data 8 countries are not included.
There are various applications of DEA to measure the efficiency of OECD
countries (e.g., Agasisti, 2011; Aristovnik & Obadic, 2011; Saljoughian et al.,
2013). However, as shown in Sect. 16.2.2, each application was limited to a specific
DEA version, concentrating on DEA versions which provide a mere dichotomy of
the countries into efficient and inefficient—not a full rank scale. None of them deals
with a robust solution, and leaves the reader perplexed as to why one method is
used and not another. Moreover, some articles concentrate on specific aspects of HE
system: research or teaching. While our study considers a variety of multivariate
statistical methods and DEA versions dichotomy and full-scale ranks of OECD
countries, pointing at the robust solution. Furthermore, both education and research
aspects of HE in OECD countries are considered.
Section 16.2 discusses the variables affecting HE systems efficiency;
inputs/outputs chosen by previous studies to measure OECD countries’ HE
efficiency; and the methodologies they used versus the methodology used in
this study. Section 16.3 lays out the inputs and outputs chosen for our OECD
study considering Sect. 16.2. Section 16.4 presents the results of two basic DEA
models for grouping the countries into two groups efficient and inefficient. CRS
(constant return version) is presented showing the weights of inputs and outputs
for each country, the improvements each country needs for each output, and some
benchmark analysis. Also, VRS (variable returns to scale) efficient and inefficient
groups are presented. Following are the results of several full rank-scaling methods
DEA based, followed by multivariate statistical efficiencies in the DEA context.
Section 16.5 deals with another version of the data, where only 1 input and the six
outputs are used. Section 16.6 selects the robust efficiency method. Section 16.7
provides other external factors that may be correlated with the efficiency rank-scale.
Section 16.8 highlights the risks of massification in HE, which is the implication
of the two first outputs inefficient countries tend to give high weights for (percent
of adults attained HE, and percent of employed adults with HE), and its possible
ramification, mainly on reducing the economic value of HE. Finally, summary and
conclusions are given.
16.2 Variables Affecting National HE Systems
Modern society has developed new expectations of HEIs and academic faculty
responsibility: expectations of involvement in social issues and relevant research
and studies; expectations of a system that seeks to satisfy national needs; and
expectations of professional training to help national advancement—providing

necessary professional, technological, and intellectual foundations for promoting
efficiency, economy, and administrative capabilities.
16.2.1 The Economic Gains
The economic returns for HE graduates are the highest in the entire educational
system (including elementary and secondary education). According to Altbach et
al. (2017), today there are around 200 million HE students in the world (it was 89
million in 1998). This is critical because, according to a World Bank Group (WBG)
report, a student with a HE degree may earn more than twice as much as a student
with only a high school diploma. As expected, there exist differences in the value of
HE by country (ibid). Much of this can be understood through each country-specific
demand and supply factors.
The Financial Times rankings give overriding importance to graduate’s salary
criterion (40%) [https://www.universityrankings.ch/methodology/financial_times_
rankings, accessed on Feb 27, 2019].
Within the same country, there may be differences among HEIs regarding
economic returns on investment. For example, comparing two specific universities
in the USA: Stanford graduates—with an average 4-year cost of HE $224,500 and
a graduation rate of 96%—expect $809,700 Return on Investment (ROI). Thus, the
ratio between returns and investment is 3.6. However, Georgia Tech graduates—
with a 4-year cost of $86,700 and graduation rate of 82%—expect ROI of $796,300.
Thus, the ratio between the returns and the investment is 9.1. The difference between
3.6 and 9.1 between the two universities’ ratios is tremendous! (data source, https://
ourworldindata.org/returns-to-education, accessed on Feb. 25, 2019).
16.2.2 Inputs and Output Chosen by Other OECD Efficiency

Studies
Agasisti (2011) used DEA (CRS and VRS) to measure HE efficiency on the
national level among OECD countries (18 countries) using the following inputs and
outputs (OECD in a Glance 2002–2006). They used the following input and output
variables.
Inputs:
• Accessibility of the system, which could be measured by means of entry rates.
• The availability of financial resources (private and public investments, in absolute
terms and as a proportion of GDP).
• Human resources (number and quality of students and teachers).
Outputs:
• Proportion of population that has attained tertiary education, measured in terms
of graduation rates and percentage of educated population.
• Employability of graduates, which also proxies the private returns of HE.
• Attractiveness of the systems, measured through the flows of students from one
country to another.
Agasisti (2011) did not use any research outputs.
Aristovnik and Obadic (2011) used two outputs to utilize DEA efficiency of
EU and OECD countries (37 in total). DEA technical efficiency was used (VRS).
The data was an average over the years 1999–2007. The highlight of the paper is
a comparison of Croatia and Slovenia efficiency in relation to the other countries
using 3 combinations of the following inputs and outputs.
The inputs were:
• Expenditure per student, tertiary (% of GDP per capita)
• School enrollment, tertiary (% gross)
The outputs were:
• School enrollment, tertiary (% gross)
• Labor force with a tertiary education (% of total)
• Unemployed with a tertiary education (% of total unemployment).
They did not use any research outputs (ibid).
Saljoughian et al. (2013) used science and technology indicators of 28 OECD
countries of 2009 data (from World Bank) applying DEA (VRS and Super Effi-
ciency). They rank the countries based on the DEA efficiency results (Super
Efficiency), though the weights of inputs and outputs vary from country to country.
The inputs are:
• Research and development expenditure
• Researchers in R&D
The outputs are:
• High-tech exports (current US$)
• Patent applications, residents
• Scientific and technical journal articles
• Trademark applications, direct resident
They used only research inputs and outputs, but they did not use any enrollment
data (ibid).
The following references use DEA for evaluating HE in some aspects of OECD
countries or subgroup, thus we did not list their inputs/outputs. For example,
Ceccobelli et al. (2012) used DEA (Bootstrap) for the evaluation of 14 OECD
countries studying the impact of their Information Communication Technology
(ICT) inputs on their labor productivity growth during 1995–2005. Wolszczak-
Derlacz and Parteka (2011) used DEA (VRS) bootstrap approach, on 259 HEIs from
7 European countries. At a second stage, the efficiency was regressed as a function

of explanatory variables. However, there was no analysis by country. Similarly,
Veiderpass and McKelvey (2016) evaluated DEA (VRS) efficiency of 944 HEIs
in 17 European countries, by institution, using 11 inputs (expenditures and various
types of staff), and 5 outputs (headcounts of graduates at different levels). They
calculate the average efficiency of every country. Afonso and Kazemi (2017) used
DEA (CRS and VRS) for evaluating the efficiency of 20 OECD countries using 9
indicators—education is only one of the indicators.
16.2.3 Methodology—Robust Efficiency of Rank Scales

in DEA Context
Our study utilizes the robust efficiency approach where several efficiency rank-
scaling methods are used and the best, robust, method is verified (Sinuany-Stern &
Friedman, 2021). Several DEA-based methodologies and the statistical multivariate
efficiency methods are used here for measuring countries efficiencies also taken
from Sinuany-Stern and Friedman (2021) and Adler et al. (2002). Thus, they will not
be detailed mathematically, but rather intuitively in the context of efficient countries
in HE.
The potential of deriving rank-scaling efficiencies based on Super Efficiency (SE)
and Cross Efficiency (CE) based on CRS and VRS are presented.
The multivariate statistical methods (in the context of DEA) considered here
are Discriminant analysis efficiency (Disc), Canonical Correlation Analysis (CCA)
efficiency, and multiple regression efficiency (REG).
16.2.3.1 Constant Returns to Scale Efficiency and Inefficiency: Dichotomy
CRS assumes Constant Return to Scale. CRS is the original version of DEA
efficiency (Charnes et al., 1978). CRS efficiency is defined as the ratio between
the weighted sum of the output and the weighted sum of the inputs, where the
weights are unknown (or not agreed upon). In our study, CRS finds for each OECD
country the ideal weights of CRS efficiency ratio, subject to positive efficiency
ratio of all OECD countries bounded by 1. A country is CRS efficient if its CRS
efficiency is 1, while a country is inefficient if its CRS efficiency is less than 1. The
advantage of DEA is the ability to reformulate each of its optimization problems
(one for each OECD country in our case) into linear programming formulations,
which its dual solution provides information on the improvements each country
needs to do (the slacks of the dual problem). DEA also provides benchmark analyses
pointing at efficient countries that caused an inefficient country not to achieve the
maximal CRS efficiency of 1 (countries whose dual variable was positive in the
solution of the inefficient country). Consequently, the weights of inputs and outputs
vary greatly from one country to another; though all weights are constrained to be
non-negative. Due to the variability of the weights from one country to another,
CRS efficiencies cannot be comparable to each other and cannot be used for rank-
scaling. Nevertheless, CRS efficiencies provide dichotomy of the countries into two
groups:
1. CRS efficient (when CRS efficiency = 1 consisting the efficient frontier)
2. CRS inefficient (when CRS efficiency < 1)
16.2.3.2 Variable Returns to Scale Efficiency and Inefficiency: Dichotomy
VRS assumes Variable Returns to Scale (Banker et al., 1984). VRS is very similar
to CRS only a constant is added to the output or input (depends on the orientation
of the application), while for CRS the constant is defined as zero. Again, VRS
divides the countries into two groups: efficient (VRS efficiency = 1) and inefficient
(efficiency < 1). Any country which is CRS efficient, will always be VRS efficient.
VRS usually will add efficient countries since it allows both: constant returns to
scale and variable returns to scale. Obviously, with more efficient countries more
countries will not need improvements. Weights of inputs and outputs of VRS will
differ in relation to CRS. Also, benchmark analysis will differ as the number of
inefficient countries decrease for VRS in relation to CRS.
16.2.3.3 Super Efficiency Based on CRS and Cross Efficiency

Rank-scaling Based on CRS Method
SE-CRS, Super Efficiency rank scale is based on CRS efficiency method, which
allows for each country’s CRS to run an efficiency value more than 1 by allowing its
objective function to be unconstrained by the value1 (Andersen & Petersen, 1993).
Therefore, any efficient CRS country may get an efficiency value of more than 1.
Thus, a full rank-scale occurs also for the efficient countries.
CE-CRS, Cross Efficiency rank-scale is based on CRS efficiency. CE-CRS is
derived directly from the original weights of CRS, where each country is evaluated
by the weights across all the other countries. The resulting average over all cross
efficiencies is the specific country’s CE-CRS efficiency rank-scale (Sexton et al.,
1986).
16.2.3.4 SE-VRS and CE-VRS Efficiency Based on VRS Rank-scaling

Method
SE-VRS Super Efficiency VRS efficiency rank-scale, and CE-VRS Cross Efficiency
VRS efficiency rank-scale, are derived from VRS efficiency in procedures parallel
to SE-CRS and CE-CRS. It is not always the case that SE-VRS, or CE-VRS is
feasible since a constant value is involved in constructing the efficiencies which

can be negative or positive in calculating SE-VRS or CE-VRS. While SE-CRS and
CE-CRS are always feasible since the constant of CRS is defined as zero.
16.2.3.5 Disc/CRS and Disc/VRS Efficiency Rank-scaling Methods
Disc-CRS is Discriminant analysis based on the CRS dichotomy, into efficient

and inefficient groups. The discriminant function is a linear combination of the
inputs and outputs with common weights, each country gets a discriminating value.
Higher discriminating value indicates a higher efficiency value. For feasibility, the
coefficients of Disc analysis must be negative for the inputs and positive for the
outputs.
Disc-VRS is Discriminant analysis based on the VRS dichotomy into efficient
and inefficient countries, the rest of the procedure is like Disc-CRS.
16.2.3.6 Canonical Correlation Efficiency Rank-scaling Method
CCA is Canonical Correlation efficiency rank-scale in the DEA context in

general—only in using multiple inputs and multiple outputs (Adler et al., 2002).
The CCA is, somewhat, a generalization of regression analysis, finding common
weights for a linear combination of the multiple outputs as a linear combination of
the inputs. The objective of CCA is to maximize the correlation between the two
linear combinations. CCA efficiency is calculated as the ratio between the weighted
outputs and the weighted inputs. In our context of inputs and outputs of HE in
OECD countries, we require that a feasible CCA model must have positive weights,
since each output must contribute positively to the overall output, and every input
must contribute positively to the overall input in increasing the overall output. If
any of the coefficients are negative, then CCA is not feasible.
16.2.3.7 Regression Efficiency Rank-scale
REG is regression analysis rank-scale efficiency. REG is applicable only for one
input versus multiple outputs or one output versus multiple inputs. The rank-scale
is derived from the error terms of the regression for each country. For one input
versus multiple outputs, the larger is the error the smaller is the efficiency—positive
error means that the actual input is larger than expected (above the regression line);
thus, inefficiency and vice versa. Similarly, to CCA, also for REG, the regression
coefficients must be positive for feasibility of the rank-scale.
16.2.3.8 Robust Efficiency
At the first stage, all these considered methods were screened for feasibility in
our case study. At a second stage from the set of feasible efficiency rank-scaling
methods, of the first stage, the robust method is found and selected to represent
the final robust efficiency scale-rank of OECD countries, based on the maximal
average correlation of each efficiency method with the other methods (Sinuany-
Stern & Friedman, 2021).
The next section presents the inputs and outputs chosen for measuring the relative
efficiency of OECD countries from the public view including educational outputs
and research output.
16.3 The Data on HE Inputs and Outputs of OECD

Countries
The objective of this study is to evaluate the efficiency of HE in OECD countries

from a public viewpoint in 2019. How well OECD countries utilize their public
resources to achieve their outputs relative to each other country. The choice of
inputs and outputs is very important. Moreover, the number of variables is limited by
the number of observations (Edvardsen et al., 2017). Taking all these into account
and considering the inputs and outputs in the literature as shown above, for this
study two inputs were chosen reflecting the public investment in HE (see sources
in Appendix 1). Six outputs were chosen reflecting the main outcomes of HE in
terms of accessibility of HE, employment level, earnings level relative to secondary
education, net financial returns, internal rate of return, and research articles level.
The variables chosen, reflect the efficiency of public investment in HE. The exact
definitions of the inputs and outputs are as follows.
Two inputs are used:
X1—Percentage of Government expenditure allocated for HE.
X2—Percentage of public sources from total HE expenditure.
Six outputs are used:
Y1—Percentage of adults with a HE (any level of HE) the highest level attained.
Y2—Percentage of employed 25–64 year old among all 25–64 year old who attained
HE (all levels of HE).
Y3—Relative earning of workers who attained HE relative to upper secondary
education ( = 100).
Y4—Net financial returns of a person attaining HE as compared with a person
attaining upper secondary education, in equivalent USD converted using PPPs
for GDP (future values are discounted at a rate of 2%).
Y5—Internal rate of return (of Y4).
Y6—h index: countries number of articles (h) that have received at least h
citations—a measure which encompasses quality and quantity simultaneously.
From the public viewpoint, the two inputs reflect the public expenditures in HE
as a percentage of total public expenditures (X1), and from the point of view of the
HE system—what percentage of their expenditures comes from the public sector
(X2). Other expenditures of HE are imbedded in the net public financial returns of
a person attaining HE (Y4); namely, Y4 is the difference between the total benefit
and the total expenditure. Since the number of OEDC countries with full data is
limited, the number of variables (2 + 6 = 8) is quite large as is. The six outputs
selected to cover a large array of outputs from the public viewpoint: percentage of
adults with HE (Y1) reflecting HE accessibility, employment rate (Y2), graduates’
earnings (Y3), net financial returns (Y4), investment internal rate of return (Y5),
and level of scientific publications (Y6). We would have added another variable—
the ratio of students to teaching staff, as a proxy for the quality of HE. However, this
data was missing for 12 countries. Thus, this ratio was not included. In relation to the
studies on OECD countries’ efficiency cited above, our study has more variables (8),
including more aspects: accessibility, employability, economic value, and research.
Treatment of missing data is discussed in Appendix 1. Due to missing too much
data we included only 29 OECD countries. A transformation was done for each
variable, so that the maximal value of each input/output variable will be 100. Thus,
the weights of variables (Vi, Ui) are comparable and reflect the relative intensity of
each variable and its weight regarding its contribution to the overall efficiency.
16.4 Results
This section presents the results of the two main DEA efficiency methods, CRS and
VRS, with their weights improvement and benchmark analyses. Then, the feasible
SE and CE for CRS rank-scale efficiency results are presented. Finally, the feasible
multivariate statistical rank-scaling efficiency results are presented, as applied to our
OECD study.
16.4.1 CRS Efficient and Inefficient Countries
As shown in the first column of Appendix 2, Table 16.10, CRS efficiency results
indicate that 10 countries are efficient among OECD countries: Chile, Hungary,
Ireland Israel, Italy, Korea, Luxembourg, Portugal, the United Kingdom, and the
USA. The other 19 countries are inefficient. CRS efficiencies range here in the
interval [0.570, 1]. For CRS efficient countries the efficiency value is 1, while for
CRS inefficient countries the efficiency value is less than 1.
Weights: The ideal weights of the inputs and outputs vary greatly from one
country to another as presented in Appendix 2, Table 16.8 (For example, Australia
has an ideal weight of 0.0108 for output 5 and zero for all other outputs, while the
USA has an ideal weight of 0.01 for output 6 and zero to all other outputs.). Most
of the inefficient countries had positive weights for outputs Y1 and Y2 (12 and 14
inefficient countries, respectively)—reflecting their relative strength in percent of
adults attaining HE (Y1) and percent of adults employed (Y2), which are two of the
main objectives of countries from public HE (as shown in Sect. 16.8).
There is a strong relationship between the weights of the inputs/outputs and the
improvements. Whenever a country has a positive weight for a certain variable,
the improvement this variable needs is zero, and whenever a country needs an
improvement in a certain variable the weight of this variable is zero (this is the
complementary slackness rule).
Improvements: The inefficient countries need improvements in some of the
outputs in which they are weak. As shown in Appendix 2, Table 16.10 most of
the 19 inefficient countries needed improvements in outputs Y3, Y4, Y5, Y6—the
number of countries which needed improvements in these variables are 14,13,16,
and 12 countries, respectively. This indicates that accessibility has been reached
(Y1, and Y2), while the economic gain is not necessarily achieved (Y3, Y4, Y5).
(Nor is the research excellence achieved (Y6—country h index).) Economic gain
is where most inefficient countries need improvements (Appendix 2, Table 16.10).
This is the reason for our discussion in Sect. 16.8 on accessibility and employment
(Fig. 16.1).
Most of the efficient countries were strong in at least one of these variables.
Moreover, out of the 10 efficient countries, 7 countries became efficient because
they had a relative advantage in only one of these 4 specific outputs (Y3, Y4, Y5,
Y6), as reflected by the positive weights they received to one of those outputs. For
example, the USA has a positive weight for output Y6 (0.01 weight) and zero to all
other outputs (Appendix 2, Table 16.8).
Benchmark analyses: Appendix 2, Table 16.12 presents for each inefficient
country the efficient countries that caused it to be relatively inefficient, those who
have positive λ value (dual variables). For example, Slovak Republic is inefficient
because of two countries: Hungary (λ = 0.895) and United Kingdom (λ = 0.07).
16.4.2 VRS Efficient and Inefficient Countries
As shown in the second column of Table 16.2 VRS (variable return to scale)
efficiency shows additional 7 efficient countries: Australia, Canada, Germany,
Latvia, Netherlands, Slovenia, and Switzerland, in addition to the previous 10
countries. Altogether 17 countries are VRS efficient—VRS takes into account return
to scale of the countries, while the remaining 12 countries are VRS inefficient.
Appendix 2, Table 16.9 presents the variability of VRS weights of the input and
output variables from one country to another, by presenting their ranges (from zero
Table 16.2 Efficiencies of OECD countries via several methods

Method: Case of 2 inputs Case of 1 inputs
Country: SE-CRSa VRSb CE-CRS SE-CRSa VRSb CE-CRS Disc. VRS
Australia 0.918 1 0.7508 0.592 0.670 0.4879 −0.649
Austria 0.594 0.594 0.4797 0.392 0.475 0.3426 −0.913
Belgium 0.801 0.856 0.6706 0.705 0.856 0.5851 −0.094
Canada 0.881 1 0.6120 0.599 1 0.4446 −0.325
Chile 1.321 1 0.5153 0.348 1 0.2349 1.758
Czech Repub. 0.901 0.936 0.7038 0.797 0.936 0.5934 0.178
Denmark 0.602 0.667 0.4437 0.420 0.615 0.3091 −1.901
Finland 0.642 0.646 0.4953 0.478 0.621 0.3730 −0.849
France 0.878 0.985 0.6942 0.878 0.985 0.7395 0.455
Germany 0.739 1 0.6290 0.678 1 0.5993 1.540
Hungary 1.138 1 0.8633 0.954 1 0.7558 1.100
Ireland 1.137 1 0.8093 0.900 1 0.6476 2.501
Israel 1.175 1 0.8926 0.908 1 0.7250 1.514
Italy 1.232 1 0.8644 1.232 1 0.9594 −1.380
Korea 1.036 1 0.7114 0.571 0.705 0.4257 −2.086
Latvia 0.941 1 0.7401 0.715 1 0.5055 −0.298
Luxembourg 2.697 1 0.8374 2.697 1 0.9545 0.964
Netherlands 0.886 1 0.6443 0.498 1 0.4374 1.305
New Zealand 0.774 0.956 0.5978 0.374 0.576 0.3250 −1.377
Norway 0.570 0.910 0.4061 0.363 0.910 0.2540 −2.051
Poland 0.758 0.917 0.6050 0.535 0.887 0.4650 0.047
Portugal 1.007 1 0.8552 0.879 1 0.6942 1.006
Slovak Repub. 0.837 0.854 0.6424 0.643 0.683 0.4527 −1.069
Slovenia 0.831 1 0.6911 0.663 1 0.5420 0.957
Spain 0.884 0.902 0.7018 0.766 0.797 0.6370 −0.462
Switzerland 0.626 1 0.4528 0.474 1 0.3348 −0.461
Turkey 0.586 0.589 0.4449 0.411 0.446 0.2773 −2.548
UK 1.283 1 0.8912 0.723 1 0.6191 0.487
USA 1.742 1 0.9743 1.144 1 0.8684 2.649
a SE-CRS column also presents CRS efficiency—all efficiencies which are more than 1 have
efficiency 1 for CRS

b Some of the countries have infeasible efficiency in SE-VRS column thus only VRS is presented
to the maximal value over all the countries) for CRS too. Appendix 2, Table 16.13
presents some benchmark analysis for VRS and CRS—the number of countries for
which each efficient country served as a benchmark. We used the VRS dichotomy
of efficient and inefficient (column 2 of Table 16.2) for the Discriminant analysis
efficiency scaling.
Table 16.3 Coefficients of Canonical and Discriminant analysis for CRS and VRS efficiencies
Coefficient of: X1 X2 Y1 Y2 Y3 Y4 Y5 Y6
Case of 2 inputs
CCA 0.012 −0.046 0.013 −0.044 0.022 −0.029 0.038 0.021
Disc-CRS −0.648 −0.068 0.039 0.014 0.030 0.000 −91.372 2.190
Disc-VRS −0.331 −0.079 0.056 0.135 0.026 0.000 −54.772 0.898
Case of 1 input
CCA 0.395 0.500 −0.295 0.231 −0.539 −0.019 0.455
Disc-CRS −0.036 −0.003 −0.173 0.021 0.041 −0.035 0.035
Disc-VRS −0.022 0.024 0.139 0.113 0.013 0.017 0.018
16.4.3 SE-CRS and SE-VRS Rank-scaling Efficiency Methods
As shown in the first column of Table 16.2, the Super Efficiency of CRS (SE-CRS)
gets efficiency values of 1 and beyond up to 2.6967 in our study. The country with
the minimal relative CRS efficiency is Norway (0.57); while the country with the
maximal SE-CRS efficiency is Luxembourg (2.697).
SE-VRS was not feasible in this OECD case study, since for 10 countries SE-
CRS values were infeasible. Thus SE-VRS is not considered in our case study.
16.4.4 CE-CRS and CE-VRS Rank-scaling Efficiency Methods
The Cross Efficiency (CE) is given in the third column of Table 16.2, CE-CRS. The
values of CE range here are in the interval [0.4061, 0.9743], the USA received the
maximal value, and Norway received the minimal CE-CRS. Due to some negative
values of the constant of VRS, CE-VRS is not feasible here.
16.4.5 Multivariate Efficiency Results
As shown in Table 16.3, Canonical Correlations Analysis (CCA) is infeasible here,

since some of its coefficients are negative, implying that X2 contributes negatively
to the sum of weighted inputs, and outputs Y2 and Y4 contribute negatively to the
sum of weighted outputs.
Disc-CRS Discriminant (Disc) analysis based on CRS dichotomy (efficient and
inefficient), is infeasible rank-scale efficiency method due to the negative coefficient
of Y5, as shown in Table 16.3.
For the same reason, Disc-VRS efficiency method, based on VRS dichotomy, is
also infeasible as shown in Table 16.3.
16.5 The Case of 1 Input
One can argue that X2, “Percentage of public sources from total HE expenditure,”
should not be an input, since when X2 decreases, it implies that X1 composes
a smaller percentage of the total income of HE, implying that “percentage of
Government expenditure allocated for HE” has a higher percentage of HE income—
from the private sector, which further benefits HE, beyond the public amount
allocated to it. For example, if in a certain country X1 = 3%, which is a public
allocation to HE of 10 billion dollars and it composes 40% = X2 of the total
expenditure of HE in that country, then the total allocation to HE is 25 billion dollars.
However, if for the above X1 = 3%, the 10 billion dollars are only 25% = X2 of
the total allocation to HE, then the total budget of HE is 40 billion dollars. Since we
analyze the efficiency from the viewpoint of the public, the resolution is to drop X2
altogether. Consequently, we also considered the case of only 1 input, X1, with the
above six outputs.
For the case of 1 input, the results in Table 16.2 for the SE-CRS column,
also represent the CRS division to efficient and inefficient. Wherever a country
has SE-CRS > 1 it means that its CRS efficiency is 1, thus it is an efficient
country. Consequently, SE-CRS shows that only 3 countries are CRS efficient:
Italy, Luxemburg, and the USA. The country with the maximal SE-CRS rank-scale
efficiency is Luxembourg with SE-CRS efficiency of 2.697.
As shown in Table 16.2 for the case of 1 input, the maximal value for CE-
CRS efficiency rank-scale is 0.9594 in this case, for Italy—the leading country,
Luxembourg has the second highest efficiency rank-scale (0.9545), and the USA
has the third CE-CRS efficiency rank-scale (0.8684), etc.
As expected, in relation to CRS, more countries are VRS efficient—12 additional
countries: Canada, Chile, Germany, Hungary, Ireland, Israel, Latvia, Netherlands,
Portugal, Slovenia, Switzerland, and the United Kingdom. However, 10 countries
had infeasible SE-VRS value, thus we could not use the SE-VRS efficiency method
in this OECD application. Also, CE-VRS is infeasible in this case. Nevertheless,
we were able to utilize the VRS dichotomy of efficient and inefficient for the
Discriminant analysis efficiency scaling.
Variability of the weights is shown in Appendix 2, Table 16.9, for CRS and VRS
inputs and output, which vary greatly from country to country for both cases: with 2
inputs and with 1 output. It is hard to point strongly at the dominant output, Y2 has
the maximal number of positive weights for the case of 2 inputs (18 for CRS and 17
for VRS), but Y6 has the maximal number of positive weights for the case with 1
input—27 for CRS and 22 for VRS. Y1, reflecting high accessibility of HE (percent
of adults attained HE) has a relatively high number of positive weights over all the
models (CRS and VRS) for the two cases (2 inputs and 1 input)—between 12 and
15 countries had positive weights. This finding led to devote Sect. 16.8 to the danger
that oversupply of HE may reduce the economic value of HE.
Improvements for the case of 1 input concentrate mainly on Y3, Y4, Y5, for
CRS model and VRS model. In common with the case of 2 inputs, these outputs
represent the economic gains from HE for the public (see Appendix 2, Table 16.11).
The improvements of other outputs vary greatly among the various models.
Benchmark analysis, as summarized in Appendix 2, Table 16.14, shows that for
both CRS and VRS Luxembourg serves most often as peer to inefficient countries
(27 countries in CRS, and for 12 countries in VRS), for the case of 1 input. For the
case of 2 inputs, as summarized in Appendix 2, Table 16.13 other efficient countries
are most often peers of inefficient countries (Hungary serves a peer for 15 countries
in CRS, and Israel serves as a peer for only 7 countries in VRS).
Multivariate analysis for the case of 1 input: As shown in Table 16.3, CCA
is infeasible, due to negative weights for Y2, Y4, and Y5. As shown in Appendix
3, we tried to use regression analysis (REG), but the weights of output Y2 and
output Y4 are negative. Thus, REG efficiency method is infeasible here. Disc-CRS
is infeasible since the weights of Y1, Y2, and Y5 are negative. However, Disc-VRS
is feasible since all the outputs have positive weights; while the 1 input has negative
weight as required. In summary, from all multivariate statistical efficiency methods,
only Disc-VRS efficiency for the case of 1 input is feasible, as reported in the last
column of Table 16.2. The USA has the highest Disc-VRS efficiency (2.649), while
Turkey has the lowest Disc-VRS efficiency (−2.548).
16.6 Robust Solution
In summary, the 7 methods listed in Table 16.4 were considered, whenever relevant
and feasible.
For the case of 2 inputs, two efficiency rank-scale methods were feasible for
OECD countries: SE-CRS and CE-CRS.
For the case of 1 input, three efficiency rank-scale methods were feasible for
OECD countries: SE-CRS, CE-CRS, and Disc-VRS. The nominal efficiency values
are detailed in Table 16.2.
All together 5 efficiency rank-scales were feasible. The robust efficiency method
is the one with the highest average correlation with all other feasible efficiency
methods (Sinuany-Stern & Friedman, 2021).
Appendix 5 presents the Pearson and Spearman rank correlations between all
pairs of the five feasible efficiency scale rank methods for our OECD efficiency
study (the ranks of the OECD countries for each of the 5 feasible efficiency methods
are given in Appendix 4). Based on these correlations the average correlation of
each method with the other four methods was calculated (see Table 16.5). The
highest average correlation points at the robust method for our study. In our OECD
study, CE-CRS is the robust efficiency method with average correlation of 0.727 (the
average of Pearson and Spearman correlations) for the case of 2 inputs. Also, for the
case of 1 input CE-CRS is the robust method with average correlation of 0.735. The
use of the average between Pearson and Spearman correlation is necessary when the
robust method is not identical in both correlations, as happens here for both cases.
Table 16.4 Rank scale efficiency methods considered and their feasibilitya
Efficiency
method SE-CRS CE-CRS SE-VRS Canonical Disc-CRS Disc-VRS Regression
The case Y YY N N N N NN
of 2 inputs
The case Y YY N N N Y N
of 1 input
a N—considered
method but not feasible, NN—not considered method at all, Y—yes feasible
method, YY—Robust feasible method
Table 16.5 Average correlations of each efficiency rank scale with all other efficiency models
The case of 2 inputs The case of 1 input
Correlation typea SE-CRS CE-CRS SE-CRS CE-CRS Disc-VRS
Pearson 0.678b 0.677 0.667 0.716b 0.525
Spearman 0.688 0.777b 0.761b 0.755 0.520
Average correlation 0.683 0.727b 0.714 0.735b 0.523
a The calculations are based on Appendix 5—each entry is an average of 4 correlations in the
Appendix
b The highest average correlations
16.7 Other Explanatory Factors Affecting the Efficiency
After finding the robust solution CE-CRS method, an attempt was made to find
external explanatory factors which may be correlated with CE-CRS.
The seven explanatory factors are:
Z1—Percentage of foreign and international students (Source: OECD, 2019, Table
B6.1 Col. 6)
Z2—Gross National Income (GNI) (Source: UN, 2019, last col. Of table)
Z3—Human Development Index (HDI, source as Z2, first column ibid)
Z4—Cost–Benefit ratio (Source: OECD, 2019, Tables A5.2, col. 6/col. 3)
Z5—1 for English speaking country, 0 otherwise
Z6—European country 1, non-European 0
Z7—Z1 × Z5
Z8—Global competitiveness index 4.0 (Schwab, 2019)
As shown in the above list, external factors, Zi, were taken from various sources.
Our basic hypothesis was that all 8 factors, Zi, will be positively correlated with
the efficiency CE-CRS. In fact, as shown in Table 16.6, the correlations between
the robust efficiency method CE-CRS and each of the Zi factors reveals that only
Z4 (benefit/cost), and Z5 (English speaking countries) are positive significantly
(R4 = 0.509 with P-value < 0.01 and R5 = 0.318 with P-value < 0.05) for the
case of 2 inputs. However, for the case of 1 input, only the correlation of CE-CRS
with Z4 (benefit/cost) is positive significantly (R = 0.338 with P-value < 0.05). All
the correlations with the other Zi are not significant. Appendix 6 provides the data
of all the factors.
Table 16.6 Explanatory variable Zi Pearson correlation with CE-CRS (the robust method)
Z1 Z2 Z3 Z4 Z5 Z6 Z7 Z8
CE-CRS 0.131 −0.100 −0.098 0.509b 0.318c −0.043 0.202 −0.025
CE-CRSa 0.246d 0.085 −0.058 0.338c 0.072 0.199 −0.062 0.005
a The case with 1 input
bP value less than 0.01
c P value less than 0.05
d P value less than 0.1
16.8 Possible Risks of Over Access of HE
Higher Education (HE) is an indication of the human development of a country.

Thus, there is an urge to increase HE enrollments, as shown in the previous
paragraphs. However, the quest for increasing the number of students and graduates,
to increase their future income, may backfire when there is an excess of enrollments
in HE, as shown below.
Consider a simplified supply and demand model for HE, on the national level,
where demand for HE is a function of tuition (Sinuany-Stern, 1991). As shown in
Fig. 16.1a, in the HE market with supply and demand curves, the equilibrium point
in a free market will be at the point (Q1’, P1’) where Q1’ is the number of enrolled
students and P1’ is the tuition. Figure 16.1b shows the market of graduates as a
function of wage. The associated supply and demand curves show the corresponding
equilibrium at the point (Q2’, P2’) with Q2’ graduates employed getting a wage of
P2’.
However, if the government subsidizes HE, as shown in the lower dotted supply
curve in Fig. 16.1a, then the new enrollment level at the equilibrium point will grow
to Q1”, and the new tuition will be reduced to P1”. This rise in enrollment level will
eventually increase the number of graduates who will enter the workforce to Q2”.
More enrollment yields more graduates (see Fig. 16.1b, the new graduates’ supply
curve in the dotted line). This results in a decrease of the wage to P2”. Lower wages,
in return, reduce the value of HE. Thus, in the long run, it may affect enrollment and
reduce it to Q1”’ (in extreme cases, even, less than its original equilibrium Q1’).
Moreover, the wage will reduce now to P2”; thus, reducing the economic value of
HE.
Yet another scenario could be, that, if the government forces wages to the higher
original level of P2’ (greater than the new equilibrium level, P2”), there will be
unemployment of Q2”- Q2’, unless other actions are taken.
In summary “credentialism” is the result of the quest for excessive increase of
HE. Credentialism (the value of a degree) was defined by Dore (1976) as emphasis
placed by students or employers, on the piece of paper certifying a student, rather
than on mastery of a subject or a profession. The propensity to appreciate the
significance of a certificate—more than the skills themselves—creates a kind of
vicious cycle in which increasingly advanced degrees are required in the attempt to
compensate for the diminishing value of the academic certificate (Davidovitch et al.,
Fig. 16.1 Supply and demand for HE enrollment (a) and graduates (b) (Sinuany-Stern, 1991)
2012). Similarly, Altbach et al. (2017) wrote about the massification of HE and its
implications—toward differentiation in HE. For example, Pietrzak (2021) describes
a similar process about the massification of HE in Poland, where the college age
group grew from 13% in 1988 to its peak of 54% in 2010 and decreased to 46% by
2019.
This process occurs when employers believe that education “improves”
employee quality. Thus, when they hire an individual with a degree—more advanced
than typically required—they are getting more for their money. This belief led
employers to demand a bachelor’s and a master’s degree from employees for the
same previous salary (see Davidovitch et al., 2013; Lindberg, 2007). This indicates
that accessibility has been reached, while the economic gain is not necessarily so.
Schwab and Davis (2018) point at the job generators industries needed for the
fourth industrial revolution. However, sometimes there are discrepancies between
the demand of the industry for HE graduates and the supply of HE graduates in
specific professions, which may result in local over supply of graduates in some
professions and shortage in other professions.
In this chapter, the efficiency of higher education (HE) in 29 OECD countries was
studied from the public viewpoint. From 7 efficiency methods considered for the
case of 2 inputs and 6 outputs only two methods were feasible: SE-CRS (super
efficiency of constant return to scale efficiency) and CE-CRS (cross efficiency of
constant return to scale efficiency). The robust method, in this case, is CE-CRS.
The leading country is the USA.
For the case of 1 input and the same 6 outputs, three efficiency methods were
feasible: SE-CRS, CE-CRS, and Discriminant-VRS. Again CE-CRS turned to
be the robust efficiency method. The leading country is Italy, while the USA is
ranked in third place (in the previous case Italy is ranked fourth). Four multivariate
statistical efficiency methods were considered, almost all of them were not feasible
in all cases. Only discriminant-VRS efficiency in the case of 1 input was feasible.
As a second stage, to explain the robust efficiency, other 8 possible external
explanatory factors were tested for positive correlation (Hypothesis). Only two
factors were correlated significantly positively with the robust efficiency rank-
scale, CE-CRS: 1. benefit/cost of HE of the country and 2. whether the country
is English speaking. Other six factors had insignificant very small correlations with
CE-CRS: 1. percent of international students, 2. Global competitiveness index, 3.
Gross National Income, 4. Human Development Index, and 5. Multiplication of
percent of international students with English speaking country.
Besides being the robust efficiency method based on the average correlation
criterion, CE-CRS efficiency method has the advantage that each country is
evaluated with the input and output weights of all the other countries, which is a
fair comparable comparison. While SE-CRS uses for each country only its own
ideal input and output weights. Thus, the countries are not compared with the same
weights. Although the correlation between the two efficiency methods (SE-CRS and
CE-CRS) is statistically significant.
We found that the first output, reflecting high accessibility of HE (percent of
adults attained HE) has a relatively high number of positive weights over all the
models used here (CRS and VRS) for the two cases (2 inputs and 1 input). Therefore,
we show by supply and demand curves of the HE market and the job market that,
increasing enrollments on the national level (e.g., by offering significant grants) may
cause oversupply of graduates in the job market, which may cause unemployment of
graduates or wage decline, which may reduce the economic value of HE, and may
reduce enrollments in the long run. Indeed, we found that, in all the models (CRS
and VRS) and in the two cases of 2 inputs and 1 input, the improvements that the
inefficient countries need to perform concentrate mainly on the three outputs that
represent economic gains from HE: earnings level relative to secondary education,
net financial returns from HE and internal rate of return from HE.
For future research, other types of efficiency models and variables can be applied
(e.g., De Witte & Lopez-Torres, 2017), including efficiency over time, such as the
Malmquist productivity index. Moreover, other outputs of government expenditure
for HE could be considered, such as foreign students as an export market for HE, and
patents as another research output; and other inputs such as public investment per
student. While our study evaluated the efficiency of OECD countries from the public
viewpoint, there is a need to perform a parallel study from the private viewpoint.
We are yet to see the long-run influence of the current crisis of the COVID-
19 pandemic on OECD countries in general and on various aspects of HE on
the national level such as governments investment in HE, enrollments level,
unemployment of graduates, globalization, and international students’ exchange,
and online learning. Such changes may change the efficiency of HE significantly in
2020 and in future years.
Appendix 1: OECD data sources
All inputs and outputs data except Y6 were taken from the OECD education 2019
report, taken from: http://meyda.education.gov.il/files/MinhalCalcala/EAG2019.pdf
(accessed on May 12, 2020)
Inputs sources:
X1—Table C4.1 column 11 (2016 data)
X2—Table C3.1 column 6 (2016 data)
Outputs sources:
Y1—Table A1.1 as the sum of columns 8,9,10,11 Ed. Attainment in % of age 25–64
(2018 data)
Y2—Table A3.1 column 9 Employment rate (%) from age 25–64 years (2018 data)
Y3—Table A4.1 column 6 Relative earning of workers by educational attainment
(2017 data)
Y4 and Y5 were calculated from columns 7 and 8, respectively of Tables A5.2a (for
man), and A5.2b (for woman) weighted average where men and women rates by
country were taken from Figure A1.4 (2016 data)
Y6—Cimcago h index by country (https://www.scimagojr.com/countryrank.php),
accessed on May 2020)
Note that a transformation was done for each variable so that the maximal value
of each input/output variable will be 100.
Treatment of missing data:
Overall, 8 countries were deleted due to problems with data—mainly missing data.
In the first stage, countries with more than 2 missing values were deleted: Columbia,
Greece, Iceland, Lithuania, Mexico, Japan, and Sweden.
At the second stage, Estonia was deleted due to a negative value of Y4.
At the third stage, there were 3 countries with at most 2-missing data—altogether 4
numbers were missing, as follows: data from previous years was taken wherever
possible, in addition, Mice package was used to estimate the missing value using
Table 16.7 Summary of missing values treatment and new values

Past year valuea Mice missing values procedure New value for 2019b
Denmarkb X1 = 4.2 (2017) Table B4.1 3.2 3.708
X2 = 95 (2017) Table B3.1 84.5 89.793
Sloveniab Y3 = 171 (2017) Table A6.1 157.8 164.4
Switzerland X2 is Missing for 2014–2019 84.24 84.24
a Previous year report where data is available
b The value is the average between the first two columns
the given complete data of 2019. The final value was an average of the value from
the Mice package and the old value found prior to 2019. At this stage:
X1 was missing for Denmark and Switzerland
X2 was missing for Switzerland
Y3 was missing for Slovenia
Note that, for all the calculations, a transformation was done for each variable
so that the maximal value of each input/output variable will be 100. Thus, the
calculated weights are comparable for the various inputs and outputs.
Appendix 2
Table 16.8 CRS weights of inputs (Vi)/outputs (Ur) for each country, for the case of 2 inputs
V1 V2 U1 U2 U3 U4 U5 U6
Australiab 0.0093 0.0087 0 0 0 0 0.0108 0
Austria 0.0073 0.0051 0.0010 0.0051 0 0.0010 0 0
Belgium 0.0106 0.0052 0.0001 0.0033 0 0.0020 0.0042 0.0002
Canada 0.0087 0.0066 0.0086 0 0 0 0 0.0004
Chilea 0.0062 0.0099 0 0 0.0100 0 0 0
Czech Repub. 0.0180 0.0052 0 0.0093 0.0001 0 0 0
Denmark 0.0074 0.0051 0.0011 0.0054 0 0 0 0
Finland 0.0094 0.0043 0.0017 0.0051 0 0 0 0.0006
France 0.0248 0 0.0035 0 0.0031 0 0 0.0092
Germanyb 0.0086 0.0061 0 0.0051 0.0014 0.0021 0 0.0001
Hungarya 0.0100 0.0104 0 0 0.0133 0 0 0
Irelanda 0.0111 0.0060 0 0.0038 0 0.0022 0.0045 0
Israela 0.0183 0.0036 0 0 0 0 0.0117 0
Italya 0.0150 0.0089 0.0023 0.0097 0 0 0 0.0012
Koreaa 0.0118 0.0089 0.0118 0 0 0 0 0
Latviab 0.0116 0.0080 0.0018 0.0085 0 0 0 0
Luxembourga 0.0064 0.0088 0 0 0 0.0100 0 0
Netherlandsb 0.0057 0.0079 0 0 0 0.0090 0 0
New Zealand 0.0088 0.0073 0 0.0079 0 0 0 0
Norway 0.0068 0.0047 0.0010 0.0050 0 0 0 0
Poland 0.0095 0.0065 0.0014 0.0069 0 0 0 0
Portugala 0.0122 0.0089 0 0.0071 0 0.0011 0.0039 0
Slovak Rep. 0.0101 0.0084 0 0.0091 0 0 0 0
Sloveniab 0.0101 0.0069 0.0016 0.0069 0 0.0012 0 0
Spain 0.0144 0.0060 0.0028 0.0060 0.0014 0 0 0.0017
Switzerlandb 0.0079 0.0048 0.0012 0.0052 0.0000 0 0 0.0005
Turkey 0.0072 0.0047 0 0 0.0035 0 0.0043 0
UKa 0.0127 0.0076 0.0020 0.0082 0 0 0 0.0010
USAa 0.0174 0 0 0 0 0 0 0.0100
a CRS and VRS efficient
b Only VRS efficient
Note that, for all the calculations, a transformation was done for each variable
so that the maximal value of each input/output variable will be 100. Thus, the
calculated weights are comparable for the various inputs and outputs
Table 16.9 Weights’ maximal valuea for CRS and VRS and number of countries with positive
weight, for the case of 2 inputs and for the case of 1 input
V1 V2 U1 U2 U3 U4 U5 U6
Case of 2 inputs
CRS max. Weight 0.025 0.0103 0.012 0.009 0.013 0.01 0.012 0.01
# posit. weightsb 29 27 15 18 7 9 6 10
VRS max. weight 0.030 0.0155 0.050 0.451 0.013 0.014 0.028 0.022
# posit. weightsb 29 23 14 17 9 6 6 13
Case of 1 input
CRS max. weight 0.047 0.005 0.002 0.010 0.01 0.014 0.014
# posit. weightsb 29 12 7 7 3 11 27
VRS max. weight 0.047 0.071 0.451 0.033 0.034 0.013 0.045
# posit. weightsb 29 15 16 7 4 9 22
a Note that a transformation was done for each variable so that the maximal value of each
input/output variable will be 100
b Number of countries with positive weights, the rest have zero weight
Table 16.10 Improvementa for input/output by country for CRS efficiency: the case of 2 inputs
CRS SX1 SX2 SY1 SY2 SY3 SY4 SY5 SY6
Australia 0.918 0 0 8.344 9.527 11.903 0 0 24.161
Austria 0.594 0 0 0 0 11.155 0 26.755 15.466
Belgium 0.801 0 0 0 0 16.975 0 0 0
Canada 0.881 0 0 0 12.910 12.664 4.216 14.493 0
Chile 1 0 0 0 0 0 0 0 0
Czech Republic 0.901 0 0 0.507 0 0 20.225 17.546 8.633
Denmark 0.602 0 0 0 0 16.459 13.068 47.608 0.673
Finland 0.642 0 0 0 0 8.364 12.325 41.215 0
France 0.878 0 1.728 0 1.102 0 36.616 13.604 0
Germany 0.739 0 0 8.889 0 0 0 14.565 0
Hungary 1 0 0 0 0 0 0 0 0
Ireland 1 0 0 0 0 0 0 0 0
Israel 1 0 0 0 0 0 0 0 0
Italy 1 0 0 0 0 0 0 0 0
Korea 1 0 0 0 0 0 0 0 0
Latvia 0.941 0 0 0 0 12.913 19.544 9.568 17.508
Luxembourg 1 0 0 0 0 0 0 0 0
Netherlands 0.886 0 0 39.729 20.672 29.258 0 22.987 64.885
New Zealand 0.774 0 0 2.465 0 9.775 12.935 23.225 28.231
Norway 0.570 0 0 0 0 15.970 19.926 55.407 8.963
Poland 0.758 0 0 0 0 9.293 11.939 17.930 2.903
Portugal 1 0 0 0 0 0 0 0 0
Slovak Republic 0.837 0 0 1.897 0 3.004 17.308 21.784 9.386
Slovenia 0.831 0 0 0 0 5.838 0 7.5165 11.082
Spain 0.884 0 0 0 0 0 31.412 21.3059 0
Switzerland 0.626 0 0 0 0 1.432 27.548 57.5707 0
Turkey 0.586 0 0 46.934 10.262 0 11.909 0 51.517
UK 1 0 0 0 0 0 0 0 0
USA 1 0 0 0 0 0 0 0 0
Max. slack 0 1.73 46.93 20.67 29.26 36.62 57.57 64.88
Slacks no.b 0 1 7 5 14 13 16 12
a The improvement (slack) required for each input/output
b The number of countries that need positive improvement (slack) for an input/output
The DEA version used here is output oriented, which means that most of the
improvements concentrate on the outputs, thus most of the inputs have positive
weights as presented in Appendix 2, Table 16.9 for CRS and VRS versions of DEA.
The improvements are in percentages
Table 16.11 Number of The case of 2 inputs The case of 1 input

countries that need
CRS VRS CRS VRS
improvements for CRS/VRS
in case 1 and case 2 X1 0 0 0 0
X2 1 4 NA NA
Y1 7 6 15 6
Y2 5 3 20 5
Y3 14 7 19 10
Y4 13 11 24 13
Y5 16 10 15 11
Y6 12 5 1 2
Table 16.12 Benchmarka of CRS efficiency: the case of 2 inputs

Eff. Dept./
Ineff. Dept. Chile Hungary Ireland Israel Italy Korea Luxem-bourg Portugal UK USA
Australia 0 0 0 0.183 0 0 0 0 0.782 0.115
Austria 0 0.345 0 0 0 0 0.043 0.319 0 0.300
Belgium 0 0.152 0.340 0.016 0 0 0.177 0.137 0 0.186
Canada 0 0 0. 0.173 0 0.785 0 0 0 0.224
Czech 0 0.627 0 0 0.367 0 0.040 0 0 0
Repub.
Denmark 0 0.491 0 0.298 0 0 0 0 0.230 0
Finland 0 0.178 0 0.744 0.032 0 0.050 0 0 0
France 0 0 0 0 0.393 0 0.347 0 0 0.297
Germany 0 0.162 0 0 0.393 0 0.133 0.064 0 0.318
Latvia 0 0.712 0 0.279 0 0 0 0 0.041 0
Netherlands 0 0 0 0 0 0 0.252 0 0 1.06
New 0 0.296 0 0 0 0 0 0 0.727 0
Zealand
Norway 0 0.295 0 0.498 0 0 0 0 0.236 0
Poland 0 0.814 0 0.083 0 0 0 0 0.136 0
Slovak 0 0.895 0 0 0 0 0 0 0.070 0
Republic
Slovenia 0 0.733 0 0.070 0 0 0.170 0 0 0.063
Spain 0 0.270 0 0.238 0.099 0 0.162 0 0 0.196
Switzerland 0 0.049 0 0.529 0.213 0 0 0 0.250 0
Turkey 0 0.018 0 0.399 0 0 0 0 0 0.573
a In the table, the λ values are the weights given by efficient country to inefficient country. The 10
countries at the top titles of the columns are the efficient countries, while the first column presents
the inefficient countries
Table 16.13 Benchmarka analysis for the case of 2 inputs for CRS and VRS
No. of times as Max. valuea of No. of times as Maximal valuea
Efficient country j peera for CRS λ for CRS peera for VRS of λ for VRS
Chile 0 0 0
Hungary 15 0.895 6 0.520
Ireland 1 0.340 1 0.378
Israel 12 0.744 7 0.702
Italy 6 0.393 2 0.383
Korea 1 0.785 0 NA
Luxembourg 9 0.347 6 0.471
Portugal 3 0.319 6 0.527
UK 8 0.782 3 0.484
USA 10 0.573 8 0.497
Australia 0 NA
Canada 0 NA
Germany 4 0.283
Latvia 3 0.501
Netherlands 3 0.452
Slovenia 1 0.116
Switzerland 2 0.861
a Two parameters are used for benchmark: the number of appearances of an efficient country j as
peer of other inefficient countries (namely λj is none zero), and the maximal value of λj
Table 16.14 Benchmarka analysis for CRS and VRS—the case of 1 input
CRS VRS
Number of times a Number of times a
Efficient country is Range of country is Range of
country benchmark of others λ > 0 for CRS benchmark of others λ > 0 for VRS
Italy 24 0.936 4 0.179
Luxembourg 27 0.888 12 0.695
USA 14 0.429 9 0.369
Canada 0
Chile 0
Germany 8 0.362
Hungary 5 0.484
Ireland 4 0.611
Israel 8 0.703
Latvia 0
Netherlands 1 0.093
Portugal 3 0.527
Slovenia 4 0.739
Switzerland 1 0.861
UK 0
a CRS is constant return to scale, VRS is variable return to scale, λ is a dual variable representing
the maximal influence that an efficient country has on another inefficient country
Appendix 3: Regression of X1 with the 6 outputs
Coefficients: Estimate Std. Error t value Pr(>|t|)

(Intercept) 4.336e + 1 1.020e + 2 0.425 0.675
Y1 4.781e−1 4.484e−1 1.066 0.298
Y2 −3.548e−1 1.102e + 0 −0.322 0.750
Y3 1.670e−1 2.040e−1 0.819 0.422
Y4 −6.901e−5 6.324e−5 −1.091 0.287
Y5 1.576e + 1 1.746e + 2 0.090 0.929
Y6 6.263e−3 9.763e−3 0.642 0.528
Residual standard error: 19.81 on 22 degrees of freedom
Multiple R-squared: 0.1557, Adjusted R-squared: −0.07456
F-statistic: 0.6762 on 6 and 22 DF, p-value: 0.6702
Appendix 4: Efficiencya ranks of OECD countries
Case of 2 inputs Case of 1 input

Country SE-CRS CE-CRS SE-CRS CE-CRS Disc.-VRS
Australia 12 9 18 16 20
Austria 27 25 26 23 22
Belgium 20 16 13 13 15
Canada 16 20 17 19 17
Chile 3 23 29 29 3
Czech Republic 13 12 9 12 13
Denmark 26 28 24 26 26
Finland 24 24 22 22 21
France 17 14 8 5 12
Germany 23 19 14 11 4
Hungary 7 5 4 4 7
Ireland 8 8 6 8 2
Israel 6 2 5 6 5
Italy 5 4 2 1 25
Korea 9 11 19 21 28
Latvia 11 10 12 15 16
Luxembourg 1 7 1 2 9
Netherlands 14 17 21 20 6
New Zealand 21 22 27 25 24
(continued)
Case of 2 inputs Case of 1 input

Country SE-CRS CE-CRS SE-CRS CE-CRS Disc.-VRS
Norway 29 29 28 28 27
Poland 22 21 20 17 14
Portugal 10 6 7 7 8
Slovak Republic 18 18 16 18 23
Slovenia 19 15 15 14 10
Spain 15 13 10 9 19
Switzerland 25 26 23 24 18
Turkey 28 27 25 27 29
UK 4 3 11 10 11
USA 2 1 3 3 1
a In each column rank 1 represents the highest efficiency, . . . , and rank 29 represents the lowest
efficiency. SE-CRS is super efficiency of constant return to scale efficiency, CE-CRS is cross
efficiency of constant return to scale efficiency, Disc-VRS is discriminant analysis of variable
return to scale efficiency
Appendix 5: Correlations between efficiency models—Pearson

and Spearman(rank)
Pearsonb \Spearman SE-CRS CE-CRS SE-CRSa CE-CRSa LDA-VRSa

SE-CRS 0.6507 0.8880 0.6804 0.4934
CE-CRS 0.8551 0.6129 0.8674 0.5763
SE-CRSa 0.6847 0.8802 0.8079 0.3607
CE-CRSa 0.6472 0.8674 0.9768 0.5072
Disc-VRSa 0.5655 0.5068 0.5014 0.5280
a The case with 1 input
b Pearson is above the diagonal
Appendix 6: Data of other explanatory variablesa Zi by OECD

country
Z1 Internal Students Z2 GNI Z3 HDI Z4 Benefit/cost Z5 Eng Z6 Europe Z7 Z1 × Z5 Z8 Compet.
510
Australia 21.478 44,097 0.938 4.9062 1 0 21.478 78.7

Austria 17.186 46,231 0.914 2.7312 0 1 0 76.6
Belgium 8.653 43,821 0.919 4.6744 0 1 0 76.4
Canada 12.917 43,602 0.922 2.4952 1 0 12.917 79.6
Chile 0.3800 21,972 0.847 4.1961 0 0 0 70.5
Czech Republic 12.543 31,597 0.891 2.5068 0 1 0 70.9
Denmark 10.757 48,836 0.930 1.6981 0 1 0 81.2
Finland 8.178 41,779 0.925 1.9563 0 1 0 80.2
France 10.195 40,511 0.891 2.6793 0 1 0 78.8
Germany 8.373 46,946 0.939 3.0183 0 1 0 81.8
Hungary 9.974 27,144 0.845 3.4657 0 1 0 65.1
Ireland 8.880 55,660 0.942 7.2928 0 1 0 75.1
Israel 2.883 33,650 0.906 5.3324 0 0 0 76.7
Italy 5.311 36,141 0.883 4.4484 0 1 0 71.5
Korea 2.257 36,757 0.906 2.1445 0 0 0 79.6
Latvia 7.393 26,301 0.854 3.0048 0 1 0 67.0
Luxembourg 46.727 65,543 0.909 2.5756 0 1 0 77.0
Netherlands 10.9987 50,013 0.933 4.8476 0 1 0 82.4
New Zealand 19.609 35,108 0.921 2.7436 1 0 19.609 76.7
Norway 3.1527 68,059 0.954 1.5062 0 1 0 78.1
Poland 4.124 27,626 0.872 2.7375 0 1 0 68.9
Portugal 6.397 27,935 0.850 4.5595 0 1 0 70.4
Slovak Republic 6.898 30,672 0.857 2.2580 0 1 0 66.8
Slovenia 3.884 32,143 0.902 3.7700 0 1 0 70.2
Spain 3.230 35,041 0.893 2.6821 0 1 0 75.3
Switzerland 17.753 59,375 0.946 1.2984 0 1 0 82.3
Turkey 1.501 24,905 0.806 5.1289 0 0 0 62.1
UK 17.917 39,507 0.920 3.9118 1 1 17.917 81.2
USA 5.180 56,140 0.920 4.6347 1 0 5.1797 83.7
Z. Sinuany-Stern and A. Hirsh
a Z5—1 for English native speaking, and 0 otherwise, Z6—1 for a European country, Z8—Global Competitiveness Index 4.0
References
Adler, N., Sinuany-Stern, Z., & Friedman, L. (2002). Review of ranking methods in the data
envelopment analysis context. European Journal of Operation Research, 140(2), 249–265.
Afonso, A., & Kazemi, M. (2017). Assessing public spending efficiency in 20 OECD countries.
In B. Bökemeier & A. Greiner (Eds.), Inequality and finance in macro dynamics (Dynamic
Modeling and Econometrics in Economics and Finance book series) (Vol. 23, pp. 7–42).
Springer.
Agasisti, T. (2011). Performances and spending efficiency in higher education: a European
comparison through non-parametric approaches. Education Economics, 19(2), 199–224.
Altbach, P. G., Reisberg, L., & de Wit, H. (2017). Responding to massification: Differentiation in
post-secondary education worldwide. Sense Publishers.
Andersen, P., & Petersen, N. C. (1993). A procedure for ranking efficient units in data envelopment
analysis. Management Science, 39(10), 1261–1264.
Aristovnik, A., & Obadić, A. (2011) The funding and efficiency of higher education in Croatia and
Slovenia: A non-parametric comparison with EU and OECD countries. The William Davidson
Institute, at the University of Michigan
Ceccobelli, M., Gitto, S., & Mancuso, P. (2012). ICT capital and labour productivity growth: A
non-parametric analysis of 14 OECD countries. Telecommunications Policy, 36(4), 282–292.
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision-making
Cimcago Journal and Country Rank. (2019). Retrieved May 2020, from https://
www.scimagojr.com/countryrank.php
Davidovitch, N., Sinuany-Stern, Z., & Iram, Y. (2012). The price of success: Some consequences of
increased access to higher education in Israel. Cross-Cultural Communication, 8(2), 101–111.
Davidovitch, N., Sinuany-Stern, Z., & Iram, Y. (2013). Paradoxes in higher education: Universities
versus academic colleges. Special journal issue: community colleges and two-year post-
secondary institutions worldwide. Research in Comparative and International Education, 8(2),
132–148.
De Witte, K., & Lopez-Torres, L. (2017). Efficiency in education. A review of literature and a way
forward. Journal of the Operational Research Society, 68(4), 339–363.
Dore, R. (1976). The diploma disease: Education, qualification and development. Comparative
Education Review 22(2) https://doi.org/10.1086/445991
Edvardsen, D. F., Førsund, F. R., & Kittelsen, S. A. C. (2017). Productivity development of
Norwegian institutions of higher education 2004–2013. Journal of the Operational Research
Society, 68(4), 399–415.
Lindberg, M. (2007). At the frontier of graduate surveys: Assessing participation and employability
of graduates with master’s degree in nine European countries. Higher Education: The
International Journal of Higher Education and Educational Planning, 53, 623–644.
OECD. (2019). Education at a glance: OECD Indicators. Retrieved May 2020, from
www.oecd.org/about/publishing/corrigenda.htm
Pietrzak, M. (2021). The application of a balanced scorecard in higher education institutions—
a case study of WULS. In Z. Sinuany-Stern (Ed.), Handbook of operations research &
management science in higher education. Springer, NY (forthcoming)
Saljoughian, M., Ghandehari, M., Shirouyehzad, H., & Dabestani, R. (2013). Performance
evaluation of OECD countries by data envelopment analysis based on science and technology
factors. Journal of Applied Science and Engineering Management, 1(1), 24–35.
Schwab, K., & Davis, N. (2018). Shaping the future of the fourth industrial revolution: A guide to
building a better world. World Economic Forum.
Schwab, K. (2019). The global competitiveness report 2019. World Economic Forum.
Sexton, T. R., Silkman, R. H., & Hogan, A. J. (1986). Data envelopment analysis: Critique and
extensions. In R. H. Silkman (Ed.), Measuring efficiency: An assessment of data envelopment
analysis (pp. 73–105). Jossey-Bass.
Sinuany-Stern, Z. (1991). The need for institutional research. Higher Education Policy, 4(4), 57–
60.
Sinuany-Stern, Z., & Friedman, L. (2021) Robust efficiency via average correlation: the case
of academic departments. In Z. Sinuany- Stern (Ed.), Handbook of operations research &
management science in higher education. Springer, NY (forthcoming)
UN. (2019). United Nations Development Program. Retrieved February 24, 2019, from http://
hdr.undp.org/en/countries
Veiderpass, A., & McKelvey, M. (2016). Evaluating the performance of higher education institu-
tions in Europe: a non-parametric efficiency analysis of 944 institutions. Applied Economics,
48(16), 1504–1514.
Wolszczak-Derlacz, J., & Parteka, A. (2011). Efficiency of European public higher education
institutions: A two-stage multicountry approach. Scientometrics, 89(3), 887–917.
Index
A regression analysis, 141

Academia–industry collaboration, 253–255 multiple regression model, 136
Academic departments simple regression model, 136
CRS weights, inputs and outputs, 381–382 simulation models, 142
descriptive statistics for inputs and outputs agent-based simulation or model, 137
of, 380 discrete event simulation, 137
efficiencies, 384 system dynamics, 137
rankings of, 383 Active learning, 457
Academic governance, 445 Activity Based Costing (ABC), 232
Academic management, 118, 119, 121, 127 Adaptive large neighbourhood search (ALNS),
Academic Ranking of World Universities 326–327
(ARWU), 160, 161, 168, 174, 176, Advertising
177, 189–191, 199 articles on, 58–61
Academic units, 305–307, 309–315, 319 marketing communications, 43, 58–61
Accelerated TA metaheuristic, for examination Agent-based simulation (ABS), 89–91, 137
timetabling problem, 335–340 Aggregated university long-run budget
cooling schedule, effect of, 335–339 planning model, 287–289
FastTA algorithm, 339–340 Airway management education, 99
Accreditation procedures, 127–128 Aleph, 122
Active forecasting methods Alumni, articles on, 72–73
data mining, 142 Analytical hierarchy process (AHP), 306, 307,
association rule mining and pattern 316, 317, 319
mining, 138 weighting process, 174–177
decision tree, 139 Analytic simulations models, 83–85, 91, 106
educational data mining, 138, 139 advantages of, 108
genetic programming, 139 agent-based simulation, 89–91
grey model, 139–140 combining analytic with OR and other
neural networks, 139 methods, 92–93
random forests, 139 deterministic and analytic simulation
SPSS Clementine, 138 models, 91
statistics and visualization, 138 discrete-event-simulation and Monte Carlo,
text mining, 138 88–89
questionnaires and surveys, 140, 142 system dynamic, 85–88

& Management Science 309, https://doi.org/10.1007/978-3-030-74051-1
514 Index
Annual reports, 126–127 internal organization perspective, 223

Arrow theorem, 204 learning and growth perspective, 223
Artificial intelligence (AI), 242, 252–255 publications, 220–221
Arts and Humanities Citation Index, 162 placement manager, 217
Assessment of Higher Education Learning publications on, 225–228
Outcomes (ANHELO), 165 quality, 215
Assessment portfolios, 127–128 quality assurance, 232
Atira A/S, 122 questionnaires and statistical methods, 230,
Auto Regressive Integrated Moving Average 231
(ARIMA) methods, 134 radar chart, 231
raising ranking, 217
rate of growth in HEIs, 234
B shareholders/constituencies and customers,
Balanced Scorecard (BSC) 218
activity based costing, 231 strategic objectives, 218
advantages, 233 strategy map, 231
articles in sustainable BSC, 231
academic department level, 229 SWOT analysis, 231
college level, 225 University of California system, 219
multinational level, 229 WULS, 419–420
national level, 229 authority distribution, changes in, 445–446
school/faculty level, 225, 229 benchmark analysis, 438–443
service department level, 229 challenges, 443–447
several universities, 229 context of, 424–427
university level, 225 general approach, choosing, 426–430
country of BSC applications, 230 mission, SWOT and vision, 430–434
data envelopment analysis, 231 operationalization and development,
DEMANTEL, 231 437–438
disadvantages, 233 plan, a ploy, a pattern, a position, and a
educational costs and benefits, 234 perspective, 447
effectiveness of, 235 professional bureaucracies, 446, 447
external and internal audiences in HE, project, defining, 430
223–224 strategy map, 434–437
financial measures, 214 Warsaw University of Life Sciences,
future goals, 214 421–423
fuzzy AHP, 231 workable strategic management, 445
fuzzy ANP, 231 Bargaining model for budget division, 312–313
fuzzy delphi and ANP, 231 Bayh–Dol Act, 243, 245, 256–258
goals and measures in HE, 216, 222–223 Benchmarking
implementation of BSC, 218 definition, 165
internal processes, 217 drawbacks of OR/MS techniques, 201,
limitations and hurdles, 233–234 204–205
managerialism and commercialism of HE, HEIBIT (see Higher Education Institution
220 Benchmarking Interactive Tool
not-for-profit organizations, 215 (HEIBIT))
number of articles on BSC versus all BSC See also Ranking
articles over time, 224 Berlin Principles, 163
organization strategy, 217 Bibliometrics, 121
performance management, 234 Binary (Boolean) decision variables, 332
performance measurement, 214, 230, 233, Borda method, 182
234 Bourdieusian approach, 45
perspectives, 216, 218, 219 Box-Jenkins (ARIMA), 132, 142
customer perspective, 222 Box trainer, 96–97
financial perspective, 222
Index 515
Branding and licensing activities, 245

articles on, 62–67 and technology transfer, 249
branding research, 46 See also Technology commercialization
image and reputation, 47–48 Competition-based learning, 308
personality, 53 Computerized simulation games, 101
researchers, 46 Consistency index (CI), 176
strategies, 46 Constant return to scale (CRS), 188, 369–370,
of universities, 45 487, 488
Budget allocation benchmark analysis, 492, 493, 496, 506,
optimization models, higher education, 507
279–280 CCA, 494
chance constraint, 287 CE-CRS, 488, 494, 495, 497, 498
linear model with one period and one disc-CRS, 489, 494
hierarchical level, 280–281 efficiency results, 491, 493
linear multilevel hierarchical linear network feasibility, 496, 497
budgeting model, 285–287 improvements of inputs/outputs by country,
linear two-level hierarchical budgeting 492, 503
model over time, 284–285 SE-CRS, 488, 494, 495
quadratic model with one period and one weights of inputs/outputs for each country,
hierarchical level, 281–284 492, 503, 504
Budgeting systems Constraint programming, 326
formula budgeting, 267 Corporate balanced scorecard, 216
incremental budgeting, 266 Correlations, 230, 231
planning–programming–budgeting system, Cost simulation model, 277
266–267 Council for the Evaluation, Accreditation
responsibility-centered budgeting, 267–268 and Quality Assurance of Higher
zero-based budgeting, 266 Education of Ecuador (CEAACES),
Budget planning 166
determining versus predicting in, 273–275 Course evaluation, 409
simulation/optimization scheme of, Course Experience Questionnaire (CEQ),
277–279 390–393, 406, 409
Business management, simulations, 100–101 original and adapted items, 411–413
Business marketing models, 36 COVID-19 pandemic, 24, 454, 455, 458, 460,
477
Cross-efficiency (CE), 371–373, 376–379
C CSS themes, 167
Canonical correlation (CCA) efficiency Current Research Information Systems (CRIS)
rank-scale, 374, 489 program
Catchball, 428 accreditation procedures, 127–128
Causal explanatory methods, 142–143 annual reports, 126–127
CBG (Cyber@Ben-Gurion University of the assessment portfolios, 127–128
Negev), 255 challenges, 129
CCR model, 187, 188 evolution of
Check Point Institute for Information Security Garfield’s studies, 121
(CPIIS), 254–255 H-Index, 121
City planning education, 102 lower operational costs and workload,
Clarivate analytics, 122, 162 122
Cobb-Douglas production function, 309 ORCID, 122
Cohort-survival method, 135, 143 PURE, 122, 125
Commercialization Science Citation Index, 122
comprehensive metrics approach, 251 promotion procedures, 124–126
distribution of income, 245, 246 recruitment of new faculty, 123–124
evolution, 245 Curve-fitting techniques, 142
IP commercialization, 247 CWTS Leiden Ranking, 160, 162, 163
516 Index
D E
Database management, 52 Editorial manager, 125
Data envelopment analysis (DEA), 167, 231, Educational data mining (EDM), 138, 139
308, 366 Educational simulations, 83, 106
Data mining, 142 Efficiency method, 366
association rule mining and pattern mining, canonical correlation efficiency method,
138 374
decision tree, 139 CE-CRS cross-efficiency method, 372–373
educational data mining, 138, 139 CRS method, 369–370
genetic programming, 139 disc-CRS efficiency method based on
grey model, 139–140 discriminant analysis, 373
neural networks, 139 disc-VRS efficiency method based on
random forests, 139 discriminant analysis, 373
SPSS Clementine, 138 evaluation of, 374–375
statistics and visualization, 138 regression SFA, 374
text mining, 138 SE-CRS method, 370, 371
Data science (DS), 7, 242, 252–255 SE-VRS method, 372
Decision Making Trial and Evaluation VRS efficiency method, 370, 372
Laboratory (DEMANTEL), 231 Efficiency, OECD
Decision-making units (DMUs), 163–165, applications of DEA, 484
366, 370 CCA, 489, 494
Decision tree, 139 CRS (see Constant return to scale (CRS))
Descriptive statistics, 230 economic gains, 485
Deterministic simulation models, 91 inputs variables, 485, 486, 490
Digital pedagogue, 476 outputs variables, 486, 490, 491
Disaggregation analysis, information REG, 489, 496
communication technology course regression of X1 with 6 outputs, 508
CEQ, 391 risks of over access of HE, 498–499
criteria weights and variability, 400 robust efficiency, 490
global satisfaction function, 401–402 supply and demand for HE enrollment,
methodology, 393–395 498, 499
MUSA method, 395–400 VRS (see Variable returns to scale (VRS))
sample, 395 E-Learning
partial satisfaction functions, 402 active learning, 457
relative action diagram, 402–404 advantages of, 457
relative improvement diagram, 404–405 benefit for students, 475
satisfaction and demanding indices, 401 challenges of transition to
secondary and tertiary education students’ change and adjustment, 459
course evaluation, 406–408 equality, 460
Disc-CRS efficiency method based on “flexible” time and place, 459
discriminant analysis, 373 lack of digital literacy, 459
Discrete-event simulation (DES), 88–89, 137 lecturer skills, 459–460
Discrete-timed stochastic simulation model, 90 motivation, 460
Discriminant analysis operating technology, 459
disc-CRS, 489 quarantine, 460
disc-VRS, 489 time management and learning, 459
Disc-VRS efficiency method based on willpower and self-demand, 460
discriminant analysis, 373 changes in teaching techniques, 454
Dissimilarity index, 199, 205 COVID-19 crisis, 454, 455, 458, 460, 477
Distance learning, 454–460, 476 difficulty faced by students, 476
See also E-Learning digital pedagogue, 476
Down-up (top-down/bottom-up) approach, emergency vs. in times of routine, 458
427–428 features, 456
Index 517
information and communication constraint programming and simulated

technology-based teaching, 455–456 annealing, 326
information literacy skills, 458 experiments, 343–344
lecturers characteristics, 462, 464 compared TA variants, 344–345
preferences FastTA and TA on the Toronto and ITC
characteristics of regression model in 2007 Sets, 346–350
lecturers, 474 ITC 2007 dataset, state-of-the-art
difficulties with implementation, 468, approaches, 354–358
470 parameter settings, 345–346
features of regression model in students, Toronto dataset, state-of-the-art
471 approaches, 350–353
lecturers’ perceptions, 468, 469 fast threshold acceptance, 327
lecturers personal preferences, 472–475 greedy randomised adaptive search
level of preference, 464–465 procedure, 326
linear regression coefficients for ITC 2007 benchmark set
students, 471–473 hard constraints, 330–331
linear regression coefficients in initial feasible timetables, construction
lecturers, 473–475 of, 342–343
online class types, 466 mathematical formulation, 331–334
students’ perceptions, 466–467 neighbourhood operators, 343
student’s personal preferences, 470–472 set of soft constraints, 331
questionnaire to students and lecturers, 461 solution representation and fitness
research questions, 461–462 function, 342
saving resources, 475 mathematical programming approaches,
student characteristics, 462 325
academic background characteristics, methodologies, characteristics of, 359
464 notations and abbreviations, 327–328
family-occupational background operations research and metaheuristic
characteristics, 463 approaches, 326
personal background characteristics, simulated annealing, 325
463 threshold acceptance metaheuristic,
synchronous online class, 466, 476 334–335
technology-integrated learning, 458 Toronto benchmark set, application to,
Elsevier, 122 340–342
English language, 38 uncapacitated toronto dataset, 328–329
Entrepreneurial university, 426 Ex-Libris, 122
and academic capitalism, 22, 264 Exponential smoothing, 134
Humboldtian model, 426 External audiences in HE, 223
and third mission, 242–243
Esploro, 122
Euroleague for Life Sciences (ELLS), 421 F
European Foundation for Quality Management Facebook, 121
(EFQM), 232 Factor analysis, 231
European Higher Education Area (EHEA), Fairness index, 307–308
165, 166 FastTA80, 344, 347–349
European Network for Quality Assurance in FastTA100, 344, 347–349
Higher Education (ENQA), 165 FastTA algorithm, 339–340
Examination timetabling problem (ETP), 324 Fast threshold acceptance (FastTA), 327
accelerated TA metaheuristic for Financial perspective, 216
cooling schedule, effect of, 335–339 Financial Times rankings, 485
FastTA algorithm, 339–340 Forecasting
adaptive large neighbourhood search, accuracy, 144
326–327
518 Index
active methods (see Active forecasting Swedish reform of HE, 309

methods) weights of criteria, 317
areas of application, 146–147 Gaussian preference function, 181
Box-Jenkins (ARIMA), 132, 142 Genetic programming, 139
counting number of articles Globalization, 36
on each forecasting method in HE, 145 Global Research University Profiles (GRUP),
main areas of application, 146 161
enrollment predictions, 143–144 Global satisfaction function, 401–402
future research., 147 Global sensitivity analysis
judgmental methods, 132, 142 applications of uncertainty and sensitivity
managerial and planning aspects, 143–144 analysis methods to HEI, 199,
passive methods (see Passive forecasting 202–203
methods) correlation coefficients, 201, 205
qualitative methods, 132, 142 dissimilarity index, 199, 205
quantitative methods, 132, 142 Friedman test, 201, 205
regression and econometrics, 132, 142 graphical comparison of classifications,
smoothing and decomposition, 132, 142 204
tuition income income models, 143–144 Google Scholar, 38
Formula budgeting, 267 Graduate Tracking System (GTS), 442
Free disposability hull (FDH), 188 Greedy randomised adaptive search procedure
Friedman test, 201, 205 (GRASP), 326
Frontal teaching, 454, 456 Grey model, 139–140
Fuzzy AHP, 231
Fuzzy ANP, 231
Fuzzy Delphi, 231
Fuzzy time series, 134 H
Health disciplines, 97–100
Hebrew University of Jerusalem, 122
G HEI budget income, 272
Games, 95–96 Higher Education Act, 424
Game theory approach Higher education (HE), 264
AHP analysis, 317 aggregated university long-run budget
article equivalent research outputs of two planning model, 287–289
departments, 318 BSC (see Balanced Scorecard (BSC))
AUs with identical abilities, 314–316 budget allocation, optimization models for,
bargaining model for budget division, 279–280, 304, 307
312–313 chance constraint, 287
budget allocation problem, 307 linear model with one period and one
budget partition in HE, 307–308 hierarchical level, 280–281
coefficient of marginal research capability, linear multilevel hierarchical linear
316 network budgeting model, 285–287
competition-based learning, 308 linear two-level hierarchical budgeting
contest versus budget division, 313–314 model over time, 284–285
efficiency, 308 quadratic model with one period and
equilibrium model, 309 one hierarchical level, 281–284
fairness index, 307–308 budgeting systems, types of
future research, 319 formula budgeting, 267
optimal budget of bargaining prize, 318 incremental budgeting, 266
pairwise comparison matrix, 317 planning–programming–budgeting
performance-based funding policies, 309 system, 266–267
price and non-price, 309 responsibility-centered budgeting,
proposed contest model, 309–312 267–268
research outputs of two departments, 317 zero-based budgeting, 266
challenges, 303–304
Index 519
efficiency, 308 consistency index, 176

HEI budgets, expenses and income in, correlations between components and
268–269 indicators, 175
budget expense structure, 270–272 correlations between indicators, 168, 173,
budget planning, determining versus 174
predicting in, 273–275 dynamic efficiency analysis via Malmquist
budget planning, index, 190–191, 195–201
simulation/optimization scheme of, efficiency analysis via DEA
277–279 applications of DEA methods to HEI,
HEI budget income, 272 190, 192–194
HE via budgeting, incentives for BCC models, 188
maximal performance in, 275–276 CCR model, 187, 188
HE via budgeting, planning, 276–277 DEA (VRS) scores of ARWU, 189
HEIF, 252 DMUs, 183, 186, 187
HESA, 252 efficiency scores, 189, 191
linear and quadratic budget allocation indicators, 183
modes, 294–296 input and output orientation, 186
managerial practices, 5 institution’s endowments 2018, 190
marketing communications production frontier endowment, 189,
advertising, 43 190
social media marketing, 45 future developments, 206
marketization of, 39, 41–42 global sensitivity analysis
in OR journals, 7–10 applications of uncertainty and
OR reviews, 6–7 sensitivity analysis methods to HEI,
Planning and Budgeting Committee, 304, 199, 202–203
305 correlation coefficients, 201, 205
planning and budgeting, literature on, dissimilarity index, 199, 205
290–294 Friedman test, 201, 205
risks of over access, 498–499 graphical comparison of classifications,
Higher Education Innovation Funding (HEIF), 204
252 indicators description and weights, 168,
Higher Education Institutions (HEIs), 4, 215, 172
264 normalization method, 177
branding, 45–48 Gaussian preference function, 181
expenses and income in, 268–269 level preference function, 181
budget expense structure, 270–272 linear preference function, 181
budget income, 272 U-shape preference function, 180
budget planning, determining versus usual preference function, 180
predicting in, 273–275 V-shape preference function, 180
budget planning, principal components analysis, 168–171,
simulation/optimization scheme of, 175
277–279 PROMETHEE, 180–186
HE via budgeting, incentives for R package, 167
maximal performance in, 275–276 sidebar and flowchart, 168
HE via budgeting, planning, 276–277 value of indicators, 168, 173
Higher Education Institution Benchmarking Higher Education Statistics Agency (HESA),
Interactive Tool (HEIBIT) 252
AHP in weighting process H-Index, 121
applications of AHP to HEI, 176, htmlwidgets, 167
178–179 Human capital, 242
example, 176, 177 Human Development Index (HDI), 4
numerical rating, 176
principle, 174–175
520 Index
I Knowledge dissemination, 243, 244

Improvement diagrams, 399 Knowledge transfer
Incremental budgeting (IB), 266 academia–industry collaboration, 253–254
Independence of irrelevant alternatives (IIA), AI, 252
201, 204 data, 253
Induced Course Load Matrix (ICLM), 274 DS, 252
Information and communication technology- industry benefit, 253
based teaching, 455–456
Information communication technology (ICT)
course L
CEQ, 391 Learning approaches, 407
criteria weights and variability, 400 Lecturer skills, 459–460
global satisfaction function, 401–402 Level preference function, 181
methodology Library databases, 126, 127
MUSA method, 395–400 Linear budget allocation (LBA) model, 307
sample, 395 Linear innovation model, 242
partial satisfaction functions, 402 Linear multilevel hierarchical linear network
relative action diagram, 402–404 budgeting model, 285–287
relative improvement diagram, 404–405 Linear preference function, 181
satisfaction and demanding indices, 401 Linear regression coefficients
secondary and tertiary education students’ for lecturers, 473–475
course evaluation, 406–408 for students, 471–473
Information literacy skills, 458 Linking enrollment predictions, 143
Innovation process, 242 Local search approaches, 325, 335
Internal audiences in HE, 224
Internationalization, 36
International Ranking Expert Group (IREG), M
163 Malcolm Baldrige National Quality Award,
IP protection policies, 255, 257–258 232
Israel Science Foundation (ISF), 274 Malmquist index, 190–191, 195–201
ITC 2007 benchmark set Management information systems (MIS), 4
examination timetabling problem Management Successful Programs (MSPs® ),
hard constraints, 330–331 274–275
initial feasible timetables, construction Marketing and Operations Research, 390
of, 342–343 Marketing communications, 37, 39, 52
mathematical formulation, 331–334 advertising, 43
neighbourhood operators, 343 articles on, 58–61
set of soft constraints, 331 social media marketing, 45
solution representation and fitness Marketing mix, articles on, 68–71
function, 342 Marketing strategy, 39
ITC 2007 dataset, state-of-the-art articles on, 68–71
approaches, 354–358 marketing mix, 50
market orientation, 49
relationship marketing strategy, 50–51
J segmentation, targeting, and positioning
JavaScript actions, 167 (STP), 49–50
Journal Citation Reports, 122 utilization and effectiveness, 48
Journal of Educational Data Mining, 138 Marketization
Judgmental methods, 132, 142 articles on, 55–57
research, 53
Market orientation, articles on, 68–71
K Markov analysis, 135–136
Kalai–Smorodinsky solution, 313 Mathematical models, 4
Key performance indicators (KPI), 220 Mathematical programming approaches, 325
Index 521
Midwifery education, 99 history of, 4

Mixed-integer linear programming (MILP) interdisciplinary nature, 5
model, 273 management information systems (MIS), 4
Modeling, 82 mathematical models, 4
Monotonicity, 396 military decisions, 4
Monte Carlo Simulation, 88–89 OR/MS publications, 5–10
Moodle system, 456 specific functional areas, 16–18
Moving average techniques of order N, 134 ORCID, 122
MUlti-criteria Satisfaction Analysis (MUSA), Organization for Economic Co-operation and
390, 391 Development (OECD) countries
Multi-criteria User Satisfaction Analysis, 400 applications of DEA, 484
Multiple regression model, 136 data of other explanatory variables Zi by,
MUSA method, 395–400 510
data sources, 501–502
efficiency ranks of, 508–509
N input and output data, 483, 485–486
Nash bargaining solution, 313 Pearson and Spearman correlations, 509
National Accreditation Council of Colombia
(CAN), 166
National Agency for Quality Assessment and P
Accreditation of Spain (ANECA), Partial satisfaction functions, 402, 413–415
166 Passive forecasting methods
Neo-institutionalism, 309 ratio methods, 141
Neural networks, 139 cohort-survival method, 135
New Public Management (NPM), 420 Markov analysis, 135–136
Normalization method time series methods, 141
Gaussian preference function, 181 ARIMA, 134
level preference function, 181 exponential smoothing, 134
linear preference function, 181 fuzzy time series, 134
U-shape preference function, 180 moving average techniques of order N,
usual preference function, 180 134
V-shape preference function, 180 Pearson and Spearman rank correlations, 496,
Not-for-profit organizations, 213, 215 509
Nursing education, 98–99 Pedagogical aspects, 476, 477
Performance-based funding policies, 309
Performance measurement, 220, 233, 234, 419,
O 420, 446
Operations research (OR) Perron-Frobenius theorem, 176
group of functional areas, 15–16 Personal preference, E-Learning, 465
in higher education Pharmacy education, 100
articles frequency, 10–12 Physical Therapy (PT), 274
EduTech and Fourth Industrial Placement manager, 217
Revolution, 21–22 Plan, a ploy, a pattern, 912 a position, and a
groups of OR methods, 13 perspective (5 Ps)., 443–447
internationalization in, 20 Planning and Budgeting Committee (PBC),
pandemic and EduTech, 22–23 304, 305
parameters of industry, 18 Planning–programming–budgeting system
planning in, 23–24 (PPBS), 266–267
published articles, 26–32 Portable Ergo-Lap Simulator, 96
quasi-commercial service industry, 18 Predicting model, 143
regression analysis, 13 Preference Ranking Organization Method
simulation, 13 for Enrichment Evaluations
student debt, 20 (PROMETHEE), 180–186
university–industry relations, 19 Principal-agent theory, 309
522 Index
Promotion, 124–126 Regression analysis, 13, 141

Proposed contest model multiple regression model, 136
AUs, 309–311 simple regression model, 136
expected utility of principal, 310, 311 Regression and econometrics, 132, 142
Tullock contest, 311 Regression efficiency (REG), 374, 378–379,
ProQuest databases, 122, 127 385, 489, 496
PURE, CRIS program, 122, 125 Regression SFA, 374
Relationship marketing, articles on, 68–71
Relative action diagram, 391, 402–404, 409
Q Relative improvement diagram, 404–405
Quacquarelli Symonds World University Research
Rankings (QS), 160, 162 in branding, 53
Quadratic budget allocation model (QBA), 307 funding, 305, 311, 316
Qualitative methods, 132, 142 quantitative and qualitative research
Quality assurance, 232 designs, 53
Quantitative methods, 132, 142 Resource dependence theory, 309
Questionnaire Responsibility-centered budgeting (RCB),
E-Learning, 460, 461 267–268
and surveys, 140, 142 Robust efficiency, 367–368, 490
choosing method, 376–377
ranks of efficiencies, robustness for,
R 377–378
Radar chart, 231 R package, 167
Radiation therapy education, 100
Random forests, 139
Rankings S
drawbacks of OR/MS techniques, 201, Saturation degree (SD) graph, 341
204–205 Science Citation Index (SCI), 122, 162
international rankings of universities Science, technology, engineering, and
Academic Ranking of World mathematics (STEM) disciplines,
Universities, 160, 161, 168, 174, 101–102
176, 177, 189–191, 199 Scientific management, 13
Berlin principles on ranking of HE, 163 Scientometrics, 121
CWTS Leiden Ranking, 160, 162, 163 SE-CRS method, 370, 371
decision-making unit, 163–165 Self-regulation learning (SRL), 458
Quacquarelli Symonds World SE-VRS method, 372
University Rankings, 160, 162 Shanghai Ranking Consultancy, 161
Times Higher EducationWorld Shiny app, 167
University Rankings, 160, 162 Simple regression model, 136
U-Multirank World University Simulated annealing, 325, 326
Rankings, 160, 162, 164, 165, 177 Simulation, 82
US News Best Global Universities advantages of, 107
rankings, 160, 161 analytic simulations, for managerial
See also Benchmarking planning, 84–85
Rapid massification, 424 advantages of, 108
Ratio forecasting methods, 141 agent-based simulation, 89–91
cohort-survival method, 135 combining analytic with OR and other
Markov analysis, 135–136 methods, 92–93
Recruitment deterministic and analytic simulation
articles on, 72–73 models, 91
strategies, 51 discrete-event-simulation and Monte
Recruitment of new faculty, 123–124 Carlo, 88–89
Rector’s Commission of Development (RCD), system dynamic, 85–88
426, 430
Index 523
application areas, number of articles on Synchronous online class, 466, 476

simulation methods by, 104–107 System dynamics (SD), 85–88, 137
box trainer, 96–97
in business management, 100–101
classification, 83 T
games, 95–96 Technion, 455
health disciplines, 97–100 Technology-assisted learning, 458
manikin, 97 Technology commercialization
method, areas of application, 108–109 development, TTOs, 245
methods, number of articles on, 103–104 distribution in income, 246
simulator, 95 evolution, 245
taxonomy of, 109–111 IP commercialization, 247
Simulation-based medical education (SBME), mechanism, 245
98 policies, 255
Simulation models, 142 public research organizations, 243, 244
agent-based simulation or model, 137 vs. technology transfer
discrete event simulation, 137 Bayh–Dole Act, 257–258
system dynamics, 137 income generation vs. technology
Simulator, 95 diffusion, 255–256
Skills Development attribute, 407 IP protection policies, 257–258
Smart robotics, 242 new policies and missions for
Smoothing and decomposition, 132, 142 TTO/TLO, 256–257
Social and emotional learning (SEL), 476 Technology-integrated learning, 458
Social media management, 52 Technology Licensing Offices (TLO), 243,
Social media marketing, 45, 54 245, 248, 255–259
articles on, 58–61 Technology transfer
Social Sciences Citation Index, 162 IP protection policies, 255, 257–258
Soft period penalty, 333 mechanisms, 247–248
Soft room penalty, 333 missions, 242
Software IT project management, public research organizations, 243, 244
computerized simulation games for, quantification mechanisms
101 indices, 250–252
SPSS Clementine, 138 industrial-supported research, 250
State-level budgeting, 273 OECD data-base, 249
State-of-the-art approaches statistical analysis, 251
examination timetabling problem, 350–353 total research funding, 249
ITC 2007 dataset, 354–358 UK approach, 252
Strategic management, 420, 426, 427, 429, 431 vs. technology commercialization
Strategy map, 231, 434–437 Bayh–Dole Act, 257–258
Strengths, Weaknesses, Opportunities, and income generation vs. technology
Threats (SWOT) analysis, 231, 431, diffusion, 255–256
432 IP protection policies, 257–258
Structural equations, 231 new policies and missions for
Student experience evaluation, 390 TTO/TLO, 256–257
Super efficiency rank scale Thematic analysis, 38
CRS efficiency 3D midwifery visualization resources
CE-CRS, 488 (3DMVR), 99
SE-CRS, 488 Threshold acceptance (TA), 327
VRS efficiency Time series methods, 141
CE-VRS, 488, 489, 495 ARIMA, 134
SE-VRS, 488, 489 exponential smoothing, 134
Supply-side marketing, 37 fuzzy time series, 134
Sustainable BSC (SBSC), 231 moving average techniques of order N, 134
Swedish reform of HE, 309
524 Index
Times Higher EducationWorld University Virtual Laboratory Practical Class (VLPC),

Rankings (THE), 160, 162 100
Top-down approach, 427 Virtual learning, 93, 95
Toronto benchmark set, 328, 340–342 Virtual reality (VR), 93
Toronto dataset, 350–353 VIVO, 122
Transfer Offices (TTO), 243, 245, 248, VRS efficiency method, 370, 372
255–259 V-shape preference function, 180
t-test, 230
Tuition income models, 143
Tullock model, 306, 311, 315, 319 W
Twitter, 121 Warsaw University of Life Sciences (WULS),
BSE, 419–423, 430, 433, 440–443
authority distribution, changes in, 445–446
U benchmark analysis, 438–443
U-Multirank World University Rankings challenges, 443–447
(U-Multirank), 160, 162, 164, 165, context of, 424–427
177 general approach, choosing, 426–430
University of Warsaw (UW), 439 mission, SWOT and vision, 430–434
U-shape preference function, 180 operationalization and development,
US News Best Global Universities rankings 437–438
(USNEWS), 160, 161 plan, a ploy, a pattern, a position, and a
Usual preference function, 180 perspective, 447
professional bureaucracies, 446, 447
V project, defining, 430
Variable returns to scale (VRS), 188 strategy map, 434–437
benchmark analysis, 493, 496, 507 workable strategic management, 445
CCA, 494 Web of science (WoS), 5
CE-VRS, 488, 489, 494, 495 World Bank Group (WBG), 485
disc-VRS, 489, 494
efficiency results, 488, 492, 493
feasibility, 496, 497 Z
SE-VRS, 488, 489, 494 Zero-based budgeting (ZBB), 266
weights of the input output variables, 492,
504

1 Handbook of Operations

Uploaded by

Copyright:

Available Formats

1 Handbook of Operations

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

1 Handbook of Operations

Uploaded by

Copyright:

Available Formats

International Series in

Operations Research & Management Science

Zilla Sinuany-Stern Editor

ISSN 0884-8289 ISSN 2214-7934 (electronic)

© Springer Nature Switzerland AG 2021

Part I includes 9 chapters (Chaps. 1, 2, 3, 4, 5, 6, 7, 8, and 9) of reviews and

Chapter 8, by Bentur, Getz, and Shacham, provides a systems analysis of Tech-

This book is aimed at HE administrators, business administration graduates,

ence, ICT—information and communications technology, PCA—principal component analysis,

in contributing to the enhancement of the HE system. Each of these individuals

Beer-Sheva, Israel Zilla Sinuany-Stern

I express my thanks to the authors of the various chapters of this handbook, to

Part II New Methodologies

Part III Applications

About the Editor

Zilla Sinuany-Stern is a Professor Emerita in the Department of Industrial

Arnon Bentur Samuel Neaman Institute, Haifa, Israel

Yossi Hadad Department of Industrial Engineering and Management, SCE—

Zilla Sinuany-Stern Department of Industrial Engineering & Management, Ben

Abstract Operations Research (OR) is an interdisciplinary discipline that evolved

Keywords Higher education · Operations research · Management science ·

Operations Research (OR) and Management Science (MS) aid decision-makers

© Springer Nature Switzerland AG 2021 3

that prevents broad access to affordable and high-quality educational services.

1.2 OR/MS Publications

In academia, sometimes OR courses are given in various departments/schools such

Various OR journals have different subareas (“departments”) listed for paper

1.3 OR Methods and Functional Areas in HE: Articles in the

1.3.1 OR Methods in HE: Articles Frequency

Table 1.2 (continued)

1.3.1.1 Groups of OR Methods

1.3.1.2 Specific OR Methods

1.3.2 Functional Areas in HE: Articles Frequency

1.3.2.1 Groups of Functional Areas

1.3.2.2 Specific Functional Areas

1.3.3 Group of OR Methods by Group of Functional Areas

1.3.4 Specific OR Methods by Specific Functional Areas

Table 1.4 provides a deeper look at 12 selected specific OR methods in HE (out

business. Moreover, DEA is composed of many linear programming (LP) problems,

1.4 Issues in the HE Sector

This section presents examples of major issues in HE where OR methods can be

There is considerable debate in the HE literature about whether students are

commodification of HE is here, and OR/MS methods are applicable, as shown in

1.4.2 University–Industry Relations

STEM (Science, Technology, Engineering, and Mathematics) graduates/students

1.4.3 Student Debt

De Wit and Altbach (2020) review the development of internationalization in HE

1.4.5 EduTech and the Fourth Industrial Revolution

HE follows the development of technology, economy, political, and social processes.

Table 1.5 Evolvement of the four Industrial Revolution (IR) and HE

mobility, development of lifelong learning programs, E-learning, and working from

1.4.6 The Pandemic and EduTech

The years 2020–2021 will be remembered because of the COVID-19 pandemic,

Marshall (2018) wrote about using technology to catalyze change in university

1.5 Summary and Conclusions

methods to measure the efficiency of academic departments at a university. Cooper

Number of articles published in Number of articles published in

Appendix 2: The number of articles published in WoS

Appendix 3: The number of articles published in WoS

Izhar Oplatka and Jane Hemsley-Brown