Research Methodology

Dr Purnendu Kumar Patra

Unit-4
Measurement and Scaling
Measurement Techniques
Concept of Measurement
Metrology is the name given to the science of pure
measurement.
Engineering Metrology is restricted to measurements
of length & angle.
Measurement is defined as the process of numerical
evaluation of a dimension or the process of
comparison with standard measuring instruments
Importance or Need of measurement
 Research conclusions are only as good as the data on which they
are based
 Observations must be quantifiable in order to subject them to
statistical analysis
 The dependent variable(s) must be measured in any quantitative
study.
 The more precise, sensitive the method of measurement, the better.
 Establish Standard Interchangeability.
 To check Customer Satisfaction
 To Validate the design
 Physical parameter into meaningful number.
 True dimension Evaluate the Performance.
 Direct measures
 Physiological measures
 Heart rate, blood pressure, galvanic skin response,
eye movement, magnetic resonance imaging, etc.
 Behavioral measures
 In a naturalistic setting.
 Example: videotaping leave-taking behavior (how
people say goodbye) at an airport.
 In a laboratory setting
 Example: videotaping married couples’
interactions in a simulated environment
Indirect measures
 Relying on observers’ estimates or perceptions
Indirect questioning
 Example: Asking executives at advertising firms if they think their
competitors use subliminal messages.
 Example: Asking subordinates, rather than managers, what
managerial style they perceive their supervisors employ.
 Unobtrusive measures.
Measures of accretion, erosion, etc.
 Example: Studying discarded trash for clues about lifestyles, eating
habits, consumer purchases, etc.
 Levels of data
Ratio

Nominal
Ordinal Interval
Interval (Scale in SPSS)
Ratio (Scale in SPSS)
Ordinal

Nominal
Qualitative (Categorical) data types
Nominal Data
Ordinal Data
Quantitative (Numerical) Data Types
Interval Data
Ratio Data
Nominal Data
Nominal data is a type of data that represents discrete
units which is why it cannot be ordered and measured.
They are used to label variables without providing any
quantitative value.
Also, they have no meaningful zero.
Some examples of nominal data include
Gender ( Male, Female)
Hair color ( Black, Brown, Gray, etc)
Nationality (Indian, American, Chinese, etc)
The only logical operation that you can apply to them
is equality or inequality which you can also use to
group them.
The descriptive statistics you can do with nominal data
include
frequencies,
proportions,
Percentages
And, to visualize nominal data, you can use a pie chart
or a bar chart.
 Nominal data
Limited possibilities for Nominal categories aren’t
statistical analysis hierarchical, one category
Categories, isn’t “better” or “higher”
classifications, or than another
groupings Assignment of numbers
Merely measures the to the categories has no
presence or absence of mathematical meaning.
something Nominal categories should
Gender: male or female be mutually exclusive.
Zip codes, 90210, 92634,
91784.
Nominal data-continued
Nominal data is usually
represented
“descriptively”
Graphic representations
include tables, bar
graphs, pie charts.
There are limited statistical
tests that can be performed
on nominal data
Ordinal Data
Ordinal values represent discrete as well as ordered units.
Unlike nominal, here the ordering matters.
However, there is no consistency in the relative distance
between the adjacent categories.
And, similar to nominal data, ordinal data also don't have
a meaningful zero.
Examples of ordinal data
Opinion (agree, mostly agree, neutral, mostly disagree,
disagree)
Socioeconomic status (low income, middle income, high
income)
The descriptive statistics that you can do with
ordinal data include
frequencies,
proportions,
percentages,
percentiles,
median,
Mode
Here the visualization methods that can be used
are the same as nominal data
Ordinal data
 Examples:
More sensitive than nominal data, but still lacking in
precision
1st, 2nd, 3rd places finishes in a horse race
 Exists in a rank order, hierarchy, or sequence
Top 10 movie box office successes of 2006
Highest to lowest, best to worst, first to last
Bestselling books (#1, #2, #3 bestseller, etc.)
 Allows for comparisons along some dimension
Example: Mona is fairer than Fifi, Rex is taller than
Niles

1st 2nd 3rd

Note that the difference between Elementary and High
School is different than the difference between High
School and College.
This is the main limitation of ordinal data, the
differences between the values is not really known.
Because of that, ordinal scales are usually used to
measure non-numeric features like happiness,
customer satisfaction and so on.
It is nearly the same as nominal data, except that its
ordering matters.
Interval data (scale data)
 Represents a more sensitive type of data or
sophisticated form of measurement.
 Assumption of “equidistance” applies to data or
numbers gathered.
gradations, increments, or units of measure are
uniform, constant.
 Examples:
Scale data: Likert scales,
Most standardized scales or diagnostic
instruments yield numerical scores
More about Interval data
Scores can be compared to one another, but in relative,
rather than absolute terms.
Example: if Fred is rated a “6” on attractiveness, and Barney
a “3,” it doesn’t mean fred is twice as attractive as Barney
No true zero point (a complete absence of the
phenomenon being measured)
Example: A person can’t have zero intelligence or zero self
esteem, or zero weight
Scale data is usually aggregated or converted to
averages.
Amenable to advanced statistical analysis.
Interval values represent ordered units that have the
same difference.
Therefore we speak of interval data when we have a
variable that contains numeric values that are ordered
and where we know the exact differences between the
values.
An example would be a feature that contains
temperature of a given place
The problem with interval values data is that
they don’t have a “true zero”.
That means in regards to our example, that there is no
such thing as no temperature.
With interval data, we can add and subtract, but we
cannot multiply, divide or calculate ratios.
Because there is no true zero, a lot of descriptive and
inferential statistics can’t be applied
Ratio data
The most sensitive, powerful type of data
Ratio measures contain the most precise
information about each observation that is
made
Ratio values are also ordered units
that have the same difference.
Ratio values are the same as interval
values, with the difference that they
do have an absolute zero.
Examples:
Time as a unit of measure
Distance as a unit of measure (setting an
odometer to zero before beginning a trip)
Weight and height as units of measure
More about Ratio data
More prevalent in the natural
sciences, less common in social
science research
Includes a true zero point (complete
absence of the phenomenon being
measured)
Allows for absolute comparisons
If fred can lift 200 lbs and barney can
lift 100 lbs, fred can lift twice as much
as barney, e.G., A 2:1 ratio
VALIDITY AND RELIABILITY
Sound measurement must meet the tests of validity,
reliability and practicality.
In fact, these are the three major considerations one
should use in evaluating a measurement tool.
Validity refers to the extent to which a test
measures what we actually wish to measure.
Reliability has to do with the accuracy and
precision of a measurement procedure ...
Practicality is concerned with a wide range of
factors of economy, convenience, and
interpretability ...”
 Commonly used terms…

“She has a valid point”

“My car is unreliable”

…in science…
“The conclusion of the study was not valid”

“The findings of the study were not reliable”.

 Some definitions…
Validity

“The soundness or appropriateness of a test or

instrument in measuring what it is designed to
measure”
(Vincent 1999)
 Some definitions…
Validity

“Degree to which a test or instrument measures what

it purports to measure”

(Thomas & Nelson 1996)

 Some definitions…
Reliability

“…the degree to which a test or measure produces the

same scores when applied in the same
circumstances…”

(Nelson 1997)
 Types of Experimental Validity
Internal.

Is the experimenter measuring the effect of the independent

variable on the dependent variable?

External.

Can the results be generalised to the wider population?

Reliability
The test of reliability is another important test of sound
measurement. A measuring instrument is reliable if it
provides consistent results.
Reliable measuring instrument does contribute to
validity, but a reliable instrument need not be a
valid instrument.
For instance, a scale that consistently overweighs
objects by five kgs., is a reliable scale, but it does
not give a valid measure of weight. But the other
way is not true i.e., a valid instrument is always
reliable.
Reliability
Two aspects of reliability viz., stability and equivalence
deserve special mention.
The stability aspect is concerned with securing consistent
results with repeated measurements of the same person and
with the same instrument. We usually determine the degree
of stability by comparing the results of repeated
measurements.
The equivalence aspect considers how much error may get
introduced by different investigators or different samples
of the items being studied. A good way to test for the
equivalence of measurements by two investigators is to
compare their observations of the same events.
Attitude
An attitude is viewed as an enduring disposition to
respond consistently in a given manner to various
aspect of the world, including persons, events and
objects.
Attitude cannot be measured directly
Attitude is derived from the perceptions
Attitude are indirectly observed
Attitude
Attitude

Cognitive Affective Intention or Action

Attitude
Cognitive component: Represents and individual’s
information and knowledge about an object. Example
of remembering about Tupperware..
Affective Component: Summarizes a person’s overall
feeling or emotions towards the object. Example of
tasty food cooked in pressure cooker
Intention or Action component: It also reflects a
person’s expectation future behavior towards an object.
Example: Purchase intention to buy things in future
Scaling Techniques
Scaling
Scaling describes the procedures of assigning
numbers to various degrees of opinion, attitude and
other concepts.
It may be stated here that a scale is a continuum,
consisting of the highest point (in terms of some
characteristic e.g., preference, favourableness, etc.)
and the lowest point along with several intermediate
points between these two extreme points.
 Classification of Scales

Comparative vs Non Comparative

Scale.
Classification of Scales
Comparative vs Scaling
Non Techniques
Comparative Scale
Comparative Non Comparative
Graphic
Paired Comparison
Rating Scale

Constant Sum

Rank Order

Q-Sort
Paired-Comparison Scale
In comparative scales it is assumed that respondents
make use of a standard frame of reference before
answering the question,
Example: How do you rate barista in comparison to
cafe coffee Day on Quality of Beverages?
Please rate Domino's in comparison to Pizza Hut on
basis of your satisfaction level on the 11-point scale ,
based on the following parameters: 1-Extremely poor,
6-Average, 11-Extremely Good.
 Comparative Scale – Rank Order Scale
 In Rank order scaling, Soft Rank
respondents are presented with Drink
several objects simultaneously and
asked to order or rank them Coke
according to some criterion. Pepsi
 The rank order scale is a type of
Limca
ordinal scale because it orders the
attributes from the most preferred Sprite
to the least preferred but does not Mirinda
have a specific distance between
the attributes. Seven up
 Consider, for example the Fanta
following question:
Comparative Scale – Constant Sum
Rating Scale
In this the respondents
School Points
are asked to allocate a
DPS
total of 100 points
Kendriya Vidyalaya
between various objects
Mount Litera
and brands.
DAV Public School
The respondents
Sai International
distribute the points to
Vikas
the various objects in
ODM
the order of his
St Xavier
preference.
Total 100
Comparative Scale – Q-Sort
 Q-Sort scale is a type of measurement scale that uses a rank order
scaling technique to sort similar objects with respect to some
criterion
 The respondents sort the number of statements or attitudes into piles,
usually of 11.
 The Q-Sort Scaling helps in assigning ranks to different objects
within the same group, and the differences among the groups (piles)
are clearly visible
 It is a fast way of facilitating discrimination among a relatively large
set of attributes.
 For example, a new restaurant that is just preparing its menu may
want to collect some information about what potential customers
like:
The document provided contains a list of 50 meals. Please choose 10
meals you like, 30 meals you are neutral about (neither like nor dislike)
Non Comparative Scales
In this the respondents Non Comparative Scales
do not make use of any Graphic Rating Scales Itemised Rating Scale
frame of reference
Likert Rating
before answering the Scale

questions.
The resulting data is Semantic differential
Rating Scale

generally assumed to
be interval or ratio
Stapel Rating
scale. Scale
 Graphic Rating Scale
This continuous scale, also called graphic rating Scale. In
the graphic rating scale the respondent is asked to tick
mark on the following question:

Least Most
Proffered Preferred
 Itemized Rating Scale
The itemized rating scale is a type of ordinal scale that
assigns numbers each attribute.
Respondents are usually asked to select an attribute that
best describes their feelings regarding a predefined
criterion.
Likert Scale-
A Likert scale is an ordinal scale with 3/5/7 response
categories, which is used to order a list of attributes from
the best to the least.
This scale uses adverbs of degree like very strongly,
highly, etc. to indicate the different levels.
Stapel Scale-
This a scale with 10 categories, usually ranging from -5 to
5 with no zero point.
It is a vertical scale with 3 columns, where the attributes
are placed in the middle and the least (-5) and highest (5)
is in the 1st and 3rd columns respectively.
Semantic Differential Scale
This is a five/seven-point rating scale with endpoints
associated with bipolar labels (e.g. good or bad, happy,
etc.).
It can be used for marketing, advertising and in different
stages of product development.
Questionnaire Design
 A questionnaire is a list of questions or items used to gather
data from respondents about their attitudes, experiences, or
opinions.
 Questionnaires can be used to
collect quantitative and/or qualitative information.
 Questionnaires are commonly used in scientific research as
well as in the social and health sciences.
For example,
 A company may ask for feedback about a recent customer
service experience, or
 Psychology researchers may investigate health risk
perceptions using questionnaires.
Questionnaire methods
Questionnaires can be self-
administered or researcher-administered.
Self-administered questionnaires are more common
because they are easy to implement and inexpensive,
but researcher-administered questionnaires allow
deeper insights.
Self-administered questionnaires
Self-administered questionnaires can be delivered online
or in paper-and-pen formats, in person or through mail.
All questions are standardized so that all respondents
receive the same questions with identical wording.
Self-administered questionnaires can be:
cost-effective
easy to administer for small and large groups
anonymous and suitable for sensitive topics
self-paced
But they may also be:
unsuitable for people with limited literacy or verbal skills
susceptible to a nonresponse bias (most people invited
may not complete the questionnaire)
biased towards people who volunteer because impersonal
survey requests often go ignored.
Researcher-administered questionnaires
Researcher-administered questionnaires are interviews
that take place by phone, in-person, or online between
researchers and respondents.
Researcher-administered questionnaires can:
help you ensure the respondents are representative of
your target audience
allow clarifications of ambiguous or unclear questions
and answers
have high response rates because it’s harder to refuse an
interview when personal attention is given to respondents
But researcher-administered questionnaires can be
limiting in terms of resources. They are:
costly and time-consuming to perform
more difficult to analyze if you have qualitative
responses
likely to contain experimenter bias or demand
characteristics
Open-ended Vs. Closed-ended Questions
Your questionnaire can include open-ended or closed-
ended questions or a combination of both.
Using closed-ended questions limits your responses,
while open-ended questions enable a broad range of
answers.
You’ll need to balance these considerations with your
available time and resources.
Closed-ended questions
Closed-ended, or restricted-choice, questions offer
respondents a fixed set of choices to select from.
Closed-ended questions are best for collecting both
categorical and quantitative variable data.
Categorical variables can be nominal or ordinal.
Quantitative variables can be interval or ratio.
Understanding the type of variable and level of
measurement means you can perform
appropriate statistical analyses for generalizable
results.
Which type of data?
What is your race?
 White
 African American
 Hispanic
 Asian
Are you satisfied with the current work-from-home
policies?
 Yes
 No
Nominal
Which type of data?
How satisfied or dissatisfied are you with your online
shopping experience today?
 Very dissatisfied
 Somewhat dissatisfied
 Neither satisfied nor dissatisfied
 Somewhat satisfied
 Very satisfied
Ordinal
Pros and cons of closed-ended questions
 Well-designed closed-ended questions are easy to understand
and can be answered quickly.
 However, you might still miss important answers that are
relevant to respondents.
 An incomplete set of response items may force some
respondents to pick the closest alternative to their true answer.
 These types of questions may also miss out on valuable detail.

To solve these problems, you can make questions partially closed-

ended, and include an open-ended option where respondents can
fill in their own answer.
Open-ended questions
Open-ended, or long-form, questions allow
respondents to give answers in their own words.
Because there are no restrictions on their choices,
respondents can answer in ways that researchers may
not have otherwise considered.
For example, respondents may want to answer
“multiracial” for the question on race rather than
selecting from a restricted list.
Example: Open-ended questions
How do you feel about open science?
How would you describe your personality?
In your opinion, what is the biggest obstacle for
productivity in remote work?
Problems with Open-ended Questions
They require more time and effort from respondents,
which may deter them from completing the
questionnaire.
For researchers, understanding and summarizing
responses to these questions can take a lot of time and
resources.
You’ll need to develop a systematic coding scheme to
categorize answers, and you may also need to involve
other researchers in data analysis for high reliability.
Cost of Qualitative Data Analysis software is higher
than that of Quantitative Data
Question wording
 Question wording can influence your respondents’ answers,
especially if the language is unclear, ambiguous, or biased.
 Good questions need to be understood by all respondents in the
same way (reliable) and measure exactly what you’re interested
in (valid).
Use clear language
 You should design questions with your target audience in mind.
 Consider their familiarity with your questionnaire topics and
language and tailor your questions to them.
 For readability and clarity, avoid jargon or complex language.
 Don’t use double negatives because they can be harder to
understand.
Use balanced framing
Respondents often answer in different ways depending
on the question framing.
Positive frames are interpreted as more neutral than
negative frames and may encourage more socially
desirable answers
Example: Positive vs negative frames
Positive frame Negative frame

Should protests of pandemic- Should protests of pandemic-

related restrictions be related restrictions be
allowed? forbidden?
Use a mix of both positive and negative frames to
avoid research bias, and ensure that your question
wording is balanced wherever possible.
Unbalanced questions focus on only one side of an
argument.
Respondents may be less likely to oppose the question
if it is framed in a particular direction.
It’s best practice to provide a counter argument within
the question
Example: as well. vs balanced frames
Unbalanced
Unbalanced Balanced
Do you favor…? Do you favor or oppose…?
Do you agree that…? Do you agree or disagree
that…?
Avoid leading questions
Leading questions guide respondents towards
answering in specific ways, even if that’s not how they
truly feel, by explicitly or implicitly providing them
with extra information.
It’s best to keep your questions short and specific to
your topic of interest.
Example: Leading questions
The average daily work commute in the US takes 54.2
minutes and costs $29 per day. Since 2020, working
from home has saved many employees time and
money. Do you favor flexible work-from-home
policies even after it’s safe to return to offices?
Experts agree that a well-balanced diet provides
sufficient vitamins and minerals, and multivitamins
and supplements are not necessary or effective. Do you
agree or disagree that multivitamins are helpful for
balanced nutrition?
 Keep your questions focused
 Ask about only one idea at a time and avoid double-barreled questions.
 Double-barreled questions ask about more than one item at a time,
which can confuse respondents.
 Example: Double-barreled question
Do you agree or disagree that the government should be responsible
for providing clean drinking water and high-speed internet to
everyone?
 Strongly Agree
 Agree
 Undecided
 Disagree
 Strongly Disagree
This question could be difficult to answer for
respondents who feel strongly about the right to clean
drinking water but not high-speed internet.
They might only answer about the topic they feel
passionate about or provide a neutral answer instead –
but neither of these options capture their true answers.
Instead, you should ask two separate questions to
gauge respondents’ opinions.
Example: Ask about one topic at a time
 Do you agree or disagree that the government should be responsible
for providing clean drinking water to everyone?
 Strongly Agree
 Agree
 Undecided
 Disagree
 Strongly Disagree
Do you agree or disagree that the government should be responsible
for providing high-speed internet to everyone?
 Strongly Agree
 Agree
 Undecided
 Disagree
 Strongly Disagree
Question order
You can organize the questions logically, with a
clear progression from simple to complex.
Alternatively, you can randomize the question
order between respondents.
Logical flow
Using a logical flow to your question order means starting
with simple questions, such as behavioral or opinion
questions, and ending with more complex, sensitive, or
controversial questions.
The question order that you use can significantly affect
the responses by priming them in specific directions.
Question order effects, or context effects, occur when
earlier questions influence the responses to later
questions, reducing the validity of your questionnaire.
While demographic questions are usually unaffected by
order effects, questions about opinions and attitudes are
more susceptible to them.
Example: Order effects
 Prime Ministerial approval ratings are often influenced by any
related previous questions or references to political situations.
 These ratings are lower if they follow relevant questions.
How knowledgeable are you about PM Narendra Modi’s executive
orders in his first 100 days?
Are you satisfied or dissatisfied with the way PM Narendra Modi is
managing the economy?
Do you approve or disapprove of the way PM Narendra Modi is
handling his job?
 For this reason, prime ministerial approval ratings questions are
asked at the start of surveys to measure opinions more
accurately.
Randomization
Randomization involves presenting individual
respondents with the same questionnaire but with
different question orders.
When you use randomization, order effects will be
minimized in your dataset.
But a randomized order may also make it harder for
respondents to process your questionnaire.
Some questions may need more cognitive effort, while
others are easier to answer, so a random order could
require more time or mental capacity for respondents
to switch between questions.
Questionnaire Design Guide
Step 1: Define your goals and objectives
Step 2: Use questions that are suitable for your
sample
Step 3: Decide on your questionnaire length and
question order
Step 4: Pretest your questionnaire
Step 1: Define your goals and objectives
 The first step of designing a questionnaire is determining your
aims.
 What topics or experiences are you studying?
 What specifically do you want to find out?
 Is a self-report questionnaire an appropriate tool for investigating
this topic?
 Once you’ve specified your research aims, you
can operationalize your variables of interest into questionnaire
items.
 Operationalizing concepts means turning them from abstract
ideas into concrete measurements.
 Every question needs to address a defined need and have a clear
purpose.
Step 2: Use questions that are suitable
for your sample
 Create appropriate questions by taking the perspective of your
respondents.
 Consider their language proficiency and available time and energy when
designing your questionnaire.
 Are the respondents familiar with the language and terms used in your
questions?
 Would any of the questions insult, confuse, or embarrass them?
 Do the response items for any closed-ended questions capture all
possible answers?
 Are the response items mutually exclusive?
 Do the respondents have time to respond to open-ended questions?
 Consider all possible options for responses to closed-ended questions.
 From a respondent’s perspective, a lack of response options reflecting
their point of view or true answer may make them feel alienated or
excluded.
 In turn, they’ll become disengaged or inattentive to the rest of the
Step 3: Decide on your questionnaire
length and question order
Once you have your questions, make sure that the length
and order of your questions are appropriate for your sample.
If respondents are not being incentivized or compensated,
keep your questionnaire short and easy to answer.
Otherwise, your sample may be biased with only highly
motivated respondents completing the questionnaire.
Decide on your question order based on your aims and
resources.
Use a logical flow if your respondents have limited time or
if you cannot randomize questions.
Randomizing questions helps you avoid bias, but it can take
more complex statistical analysis to interpret your data.
Step 4: Pretest your questionnaire
When you have a complete list of questions, you’ll
need to pretest it to make sure what you’re asking is
always clear and unambiguous.
Pretesting helps you catch any errors or points of
confusion before performing your study.
Ask friends, classmates, or members of your target
audience to complete your questionnaire using the
same method you’ll use for your research.
Find out if any questions were particularly difficult to
answer or if the directions were unclear or inconsistent,
and make changes as necessary.
If you have the resources, running a pilot study will
help you test the validity and reliability of your
questionnaire.
A pilot study is a practice run of the full study, and it
includes sampling, data collection, and analysis.
You can find out whether your procedures are
unfeasible or susceptible to bias and make changes in
time, but you can’t test a hypothesis with this type of
study because it’s usually statistically underpowered.
Sample Hypothesis-Item Correlation
https://
www.researchgate.net/figure/5-Survey-Questionnaire-I
tems-Relations-with-the-Hypotheses-and-Variables_tbl
4_49402955
Link for more info on Questionnaire
Design
https://blog.hubspot.com/service/questionnaire
Thank You!