Case Studies for Software Engineers

Dewayne E. Perry Susan Elliott Sim Steve M. Easterbrook

The University of Texas at Austin University of California, Irvine University of Toronto
perry@ece.uexas.edu ses@ics.uci.edu sme@cs.toronto.edu

Abstract Unfortunately, there is a great deal of confusion

regarding the term case study within software
The topic of this full-day tutorial was the correct use engineering. Some of these misuses of the term are
and interpretation of case studies as an empirical understandable because it has different meanings in
research method. Using an equal blend of lecture and different settings or disciplines. For the remainder of this
discussion, it gave attendees a foundation for conducting, Section, we will clarify what case studies are not.
reviewing, and reading case studies. There were lessons A case study is not an exemplar or case history. The
for software engineers as researchers who conduct and term case study is frequently used in medicine and law.
report case studies, reviewers who evaluate papers, and Patients or clients are referred to as cases, so a study of
practitioners who are attempting to apply results from interesting instances of these are sometimes called case
papers. The main resource for the course was the book studies [1]. However, empirical studies conducted using a
Case Study Research: Design and Methods by Robert K. case study method are very different from the interesting
Yin. This text was supplemented with positive and examples that practitioner-researchers encounter. In
negative examples from the literature. addition, a report on something interesting that was
attempted by researchers on a toy problem is not a case
1. Introduction study.
Case studies are a powerful and flexible empirical A case study is not an experience report. The latter
method. They are used for primarily for exploratory is a retrospective report on an experience that was
investigations, both prospectively and retrospectively, particularly illuminating and best examples of these
that attempt to understand and explain phenomenon or include lessons learned. However, even exploratory case
construct a theory. They are generally observational or studies need to start out with a research question and
descriptive in nature, though they can be relational as systematically collect and analyze data to answer the
well. They can also be used in the validation of research initial question. This confusion is very common as the
results. Due to this dexterity, they have become popular Experience Reports track of ICSE 2003 had a session on
in software engineering and are frequently used in papers Case Studies.
to understand, to understand explain or to demonstrate the A case study is not a quasi-experimental design
capabilities of a new technique, method, tool, process, with n=1. While some quasi-experimental studies are
technology or organizational structure. Unfortunately, conducted in the field, they still retain control over some
they are usually not used to their full potential, and often independent variables, so that time series designs, non-
not used correctly. The aim of this full-day tutorial was to equivalent before-after designs, and ex post facto designs
teach software engineering researchers and professionals can be brought to bear on the research question [2].
how to effectively design, conduct, evaluate and read case Finally, a case study is equivalent in scope to a single
studies. experiment, and both need a series of studies to fully
understand a phenomenon and produce results that
2. Characteristics of Case Studies generalize.
A case study is an empirical method. By this we mean
a defined, scientific, method for posing research 3. Goals of the Tutorial
questions, collecting data, analyzing the data, and The purpose of this tutorial was to help software
presenting the results. Each of these steps is planned from engineers understand and avoid and identify common
the outset of the study and do not come about through mistakes with case studies by giving them a solid
serendipity. Case studies are well-suited to how and grounding in the fundamentals and principles of case
why questions in settings where the researcher does not studies as a research method. For researchers, our goal
have control over variables and there is a focus on was to provide them a starting point for learning how to
contemporary events. conduct case studies. When they return to their home

Proceedings of the 26th International Conference on Software Engineering (ICSE04)

0270-5257/04 $20.00 2004 IEEE
 institutions, they would be able to find, assess, and apply
appropriate resources in designing their studies. For The following papers, in our opinion, are exemplary
reviewers, our goal was to provide them with guidance on research case studies:
how to judge the quality and validity of reported case Matthias M. Mller and Walter F. Tichy, Case Study:
studies. Reviewers They would be able to use the criteria Extreme Programming in a University Environment,
presented in this tutorial to assess whether research presented at Twenty-third International Conference on
papers based on case studies are suitable for publication, Software Engineering, Toronto, Canada, pp. 537-544,
allowing them to raise the quality of publications and give 12-19 May 2001.
appropriate feedback to authors. For practitioners, our Carolyn B. Seaman and Victor R. Basili, An Empirical
goal was to provide a better awareness of how to interpret Study of Communication in Code Inspections,
the claims made by researchers about new software presented at Nineteenth International Conference on
engineering methods and tools. We also aimed to offer Software Engineering, Boston, MA, pp. 96-106, 17-23
practitioners deeper insights into the roles they can play May 1997.
in designing and conducting case studies in collaborative D.N. Card, V.E. Church, and W.W. Agresti, An
research projects, and the ability to read case studies more Empirical Study of Software Design Practices, IEEE
effectively and be better able to identify results suitable Transactions on Software Engineering, vol. 12, no. 2,
for use in their workplace. pp. 264-271, 1986.
4. Format and Curriculum Sallie M. Henry and Dennis G. Kafura, Software
During this full-day tutorial, time was divided evenly Structure Metric Based on Information Flow, IEEE
between lecture and discussion. The lectures drew on our Transactions on Software Engineering, vol. 7, no. 5,
experience with empirical studies, research methodology pp. 545-522, September, 1981.
texts, and papers from the software engineering literature.
The tutorial covered a range of topics on the design During the break-out sessions, theThe tutorial was
and implementation of case studies. It started with basic divided into three discussion groups, each led by one of
issues in common to all empirical studies, moved on to the instructors. These smaller groups increased the
issues ones particular to case studies, and concluded with amount of interaction and allowed the material to be
an examination of practical issues. Students will gain tailored to the students. [At time of writing, we were
experience designing and evaluating case studies. planning to have tracks for investigators, reviewers, and
The curriculum included the following topics. practitioners, however, this may change depending on the
Research Methodology demographics of the tutorial attendees].
o Strategies for Software Engineering 5. Conclusion
Research Case studies are an empirical method in their own right,
o Approaches for Empirical Studies with their own established internal logic, and design
Case Study Fundamentals principles. Even for studies that are properly called case
o Exploratory Questions studies, there are often problems with selecting a unit of
o Validation analysis, validity of results, data observation and
Designing Case Studies collection. This tutorial sought to address these issues,
o Research Context because case study is a method that is well-suited to
o Validity software engineering. It is particularly appropriate when
o Ethical Issues we seek to understand how and why technology is used or
o Data Gathering and Analysis not used, functions or does not function in contemporary
Publishing Case Studies settings, and where we have little or no control over the
o Preparing Evidence variables. Our discipline can only be improved by the
o Elements of the Report addition of high-quality, published case studies that
Reviewing Case Studies employ solid methods and produce innovative results.
o Replication 6. References
The primary text text used for the course tutorial was [1] Blanche Geer, Everett C. Hughes, Anselm L. Strauss,
Case Study Methods 3/e, by Robert K. Yin [3]. This book and Howard Saul Becker, Boys in White: Student
is a respected resource on case studies and is widely cited Culture in Medical School: Transaction Publications,
both inside and outside software engineering. 1991.
The lessons were reinforced by small group sessions [2] William J. Ray, Methods Toward a Science of
where participants examined and discussed case studies Behavior and Experience, Fourth Edition. Pacific
that have been published in software engineering Grove, CA: Brooks/Cole Publishing Company, 1993.
conferences and journals.

Proceedings of the 26th International Conference on Software Engineering (ICSE04)

0270-5257/04 $20.00 2004 IEEE
 [3] Robert K. Yin, Case Study Research: Design and 2002.
Methods, 3/e. Thousand Oaks, CA: Sage Publications,

Proceedings of the 26th International Conference on Software Engineering (ICSE04)

0270-5257/04 $20.00 2004 IEEE