Research Methods

Instructor: Solomon (Ph.D)

Chapter 2: Research Methods in Computer
Science
 Overview of Research Methods in CS
• In CS research, a variety of methods are used to explore,
explain, and test hypotheses or phenomena.
• An overview of the common research methods in CS are:
1) Analytical vs. Empirical Methods
▪ Analytical Methods:
− Involve theoretical reasoning, mathematical modeling, or logical
analysis to solve problems, prove concepts, or validate systems.
− Example: Analyzing the quicksort algorithm with 𝑂(𝑛log𝑛) average-
case complexity and 𝑂(𝑛2) worst-case complexity.
▪ Empirical Methods:
− Involve experimentation, observation, or simulation to collect data
and validate hypotheses.
− Example: Comparing the execution time of bubble sort and merge
sort on randomly generated datasets.
 Overview of Research Methods in CS …
2) Surveys
− Collect data through questionnaires, interviews, or online forms to study
trends, opinions, or user behaviors.
− Proper sampling, questionnaire design, and statistical analysis are critical
for reliability.
− Example: Identify the most popular programming languages among
developers in 2024.
▪ Method: Distribute an online questionnaire asking respondents about their
preferred languages, usage contexts, and satisfaction levels.
3) Case Studies
− Are in-depth analyses of a particular instance, system, or organization to
explore phenomena or derive insights. These studies provide detailed
insights but can be challenging to generalize.
− Example: Investigate how agile methodologies affect software quality and
team collaboration.
▪ Method: Interviews with team members, observations of daily stand-ups and
sprint reviews, & analysis of software defect rates before and after agile
adoption.
 Overview of Research Methods in CS …
4) Controlled Experiments
− Involve manipulating one or more independent variables to observe their
effect on dependent variables while keeping other factors constant.
▪ Example: To compare the accuracy of two classification models.
✓ Independent Variable: Model type (e.g., Random Forest vs. Neural Network).
✓ Dependent Variable: Accuracy, F1 score, or recall.
✓ Control: Using the same training and testing data.

5) Ethnography and Action Research

▪ Ethnography:
− Is a qualitative research method where researchers immerse themselves in a
specific environment or community to understand cultural practices, social
dynamics, and interactions. In CS, it is often used to study user behaviors,
practices, and interactions with technology in real-world contexts.
− Example: A researcher studies how hospital staff use an electronic health record
(EHR) system.
▪ Action Research:
− Is a participatory and iterative research methodology where researchers actively
collaborate with stakeholders to solve practical problems while simultaneously
contributing to theoretical knowledge.
− Example: A researcher collaborates with teachers to create a digital tool for math
 Overview of Research Methods in CS …
6) Quantitative, Qualitative, and Mixed Methods
▪ Quantitative Methods:
− Rely on numerical data, statistical analysis, and mathematical models
to measure variables and test hypotheses.
− Example: Collect numerical data about how often users interact with
different app features.
▪ Qualitative Methods:
− Focus on understanding phenomena through non-numerical data,
such as text, interviews, and observations.
− Example: Analyzing developers’ motivations in contributing to open-
source projects.
▪ Mixed Methods:
− Combine both approaches to provide a comprehensive
understanding of the research problem.
− Example: Evaluating the effectiveness of a software tool by analyzing
user logs (quantitative) and conducting interviews (qualitative).
 Choosing Research Methods
• Choosing the right research method in Computer Science
depends on the research goals, available data, and the
specific problem being studied.
• The key factors to consider when selecting the research
method are:
1) Research goals
▪ Theoretical Exploration:
− If your goal is to develop new theories or algorithms, analytical methods (e.g.,
mathematical proofs, logical reasoning) are often best suited.
− These methods allow for precise and rigorous conclusions about the
properties of a system or algorithm.
− Example: Proving the computational complexity of a new algorithm.
▪ Practical Problem Solving:
− If you aim to solve a practical, real-world issue (e.g., improving system
performance), empirical methods like controlled experiments or case studies
may be more appropriate.
− They involve testing hypotheses or observing real-world phenomena.
 Choosing Research Methods …
2) Data Availability
▪ Quantitative (Numerical) Data:
− If you have access to measurable, numerical data (e.g., log files, system
performance metrics), quantitative methods are often ideal.
− Example: You could evaluate the model’s accuracy, precision, recall, and
F1 score based on a dataset of labeled data.
▪ Qualitative (Descriptive) Data:
− If your data is more descriptive (e.g., interviews, user feedback),
qualitative methods will help in understanding deeper insights,
experiences, and patterns.
− Example: Conducting interviews with users or observing them using the
system, and recording their feedback in a descriptive form.
▪ Mixed Data (Both Numerical and Descriptive):
− In cases where both types of data are available (e.g., user feedback with
usage statistics), mixed methods allow you to combine both to get a
comprehensive understanding of the research question.
− Example: Assess the usability of a web application in terms of speed,
error rates, and user satisfaction.
 Choosing Research Methods …
3) Research Scope
▪ Broad Generalizations:
− For research that aims to generalize findings across a wide range of
contexts (e.g., user preferences or technology adoption trends),
surveys or controlled experiments can gather large amounts of data
and provide statistical significance.
− Example: A survey to assess the adoption of machine learning
frameworks across the tech industry.
▪ In-Depth Analysis:
− If the goal is to explore a specific case or instance in great detail, case
studies or ethnography can provide rich, contextual insights.
− Example: Studying the development process of a specific open-
source project like the Linux kernel.
 Choosing Research Methods …
4) Iteration and Adaptation
− Continuous Improvement: If your research involves ongoing
improvement or adjustment based on real-time feedback, action
research is a suitable method. It involves cycles of planning,
acting, observing, and reflecting to address specific problems.
▪ Example: Iteratively refining a software development methodology
based on feedback from developers during each sprint.
5) Research Environment
− Controlled Setting: If you want to test a hypothesis in a controlled
environment where you can manipulate variables, controlled
experiments are appropriate. These experiments ensure that the
results are due to the manipulated variables and not external
factors.
▪ Example: Testing the effect of algorithmic optimizations on the
performance of a software system under controlled conditions.
 Validity Threats
• Validity threats refer to factors that may undermine the
trustworthiness or accuracy of a research study’s
conclusions.
– These threats are particularly significant in empirical research
methods such as experiments, surveys, and case studies.
– They can affect the internal, external, construct, and conclusion
validity of the study.
• The most common validity threats are:
1) Internal Validity Threats
▪ Confounding Variables:
− An outside factor that affects both the independent and dependent variables,
potentially distorting or biasing the results of an experiment or study. These
variables can lead to incorrect conclusions.
− Example: Measure user satisfaction with different user interface (UI) designs.
▪ Independent Variable: Type of UI design (e.g., minimalistic, complex, modern)
▪ Dependent Variable: User satisfaction (measured on a scale from 1 to 10)
▪ Potential Confounding Variable: The users' prior experience with similar software
 Validity Threats …
▪ Selection Bias:
− Occurs when there is a systematic difference between the groups being compared,
leading to inaccurate conclusions.
− Example: If a study on software usability includes participants who are already
familiar with similar software, it may not accurately reflect the general user
population's experience.
▪ History Effect:
− Arises when external events occurring during the study affect participants’
behavior or responses, thereby influencing the results.
− Example: A study examines the effectiveness of a new online learning platform in
improving students’ test scores over six months.
✓ Threat: During the study, a new policy mandates additional tutoring for all students, influencing
their learning outcomes independently of the platform.
✓ Outcome: The observed improvement might be partially due to the tutoring policy, not just the
platform.

2) External Validity Threats

▪ Sampling Bias:
− If the sample is not representative of the broader population, the findings cannot
be generalized.
− Example: A survey on programming language preferences conducted only in a
specific company may not be applicable to the global software industry.
 Validity Threats …
▪ Ecological Validity:
− Refers to the extent to which the study’s conditions match real-world
settings. If the research is conducted in an artificial environment, it may not
reflect real-world behavior.
− Example: Testing a new software tool in a laboratory setting might not reflect
how users will interact with it in their daily work environment.

3) Construct Validity Threats

▪ Measurement Errors:
− This occurs when the tools or instruments used to measure variables do not
accurately measure what they are intended to.
− Example: If a usability test measures how quickly a user completes a task but
doesn’t consider the complexity of the task, it might not truly reflect user
experience.
▪ Operationalization Issues:
− When theoretical concepts are defined too loosely or are difficult to measure,
leading to ambiguous or unreliable results.
− Example: Defining "user satisfaction" purely by the number of errors made
during a task, without considering other factors like ease of use or design
aesthetics.
 Validity Threats …
4) Conclusion Validity Threats
▪ Type I and Type II Errors:
− Type I Error (False Positive) is rejecting the null hypothesis when it is
actually true, while Type II Error (False Negative) is failing to reject
the null hypothesis when it is actually false.
− Example:
✓ A spam filter flags a legitimate email as spam (Type I) or
✓ A malware detection system fails to recognize a genuine malware attack
(Type II).
▪ Low Statistical Power:
− A study with a small sample size may not have enough power to
detect a real effect, leading to inaccurate conclusions.
− Example: A researcher tests a novel algorithm on a limited dataset
to compare its accuracy with an existing model.
✓ Problem: The dataset includes only 50 samples, leading to high
variability in results.
✓ Impact: The algorithm might appear no better than the baseline, even
though it could perform better on a larger dataset.
 Validity Threats …
• Mitigating Validity Threats
▪ Randomization: Helps control for selection bias and confounding
variables, ensuring that results are not skewed by external
factors.
▪ Control Groups: Comparing the experimental group with a
control group can help isolate the effect of the independent
variable.
▪ Longitudinal Studies: Following participants over time can help
mitigate the history effect and provide a more accurate measure
of changes.
▪ Clear Operational Definitions: Ensuring that concepts are clearly
defined and measured consistently enhances construct validity.
• Hence, by understanding and addressing these threats,
researchers can improve the quality and reliability of
their studies.
Thank You!