Factor Analysis

Factor means variables and analysis means examination of the variables. So factor analysis is a
technique in research that attempts to identify underlying variables, or factors, that explain the
pattern of correlations within a set of observed variables.

Factor analysis is a statistical data reduction and analysis technique that strives to explain
correlations among multiple outcomes as the result of one or more underlying explanations, or
factors. The technique involves data reduction, as it attempts to represent a set of variables by a
smaller number.

Meaning of underlying variables:

Here underlying variables means variables which are not observable. Suppose there are variables
y1, y2, y3, ………………..yn, which could be related to factors which are unobservable f1, f2,
and f3. The fact that the factors are not observable disqualifies regression and other methods.

Factor analysis and data reduction

Factor analysis is often used in data reduction to identify a small number of factors that explain
most of the variance that is observed in a much larger number of manifest variables. Factor
analysis can also be used to generate hypotheses regarding causal mechanisms or to screen
variables for subsequent analysis (for example, to identify collinearity prior to performing a
linear regression analysis).

Factor Analysis and Correlation

Because the purpose of factor analysis is to uncover underlying factors that explain correlations
among multiple outcomes, it is important that the variables studied be at least somewhat
correlated; otherwise, factor analysis is not an appropriate analytical technique.

Characteristics of Factor Analysis

 Simplification
Factor analysis allows its user to input a large number of variables and yield an output of only a
few variables (now called "factors"). In essence, factor analysis is similar to principal
components analysis in that it reduces a large amount of data to a clear set of factors that can be
used to describe certain aspects in the data. In addition, the user of factor analysis has a moderate
degree of freedom in choosing the number of factors in a solution, allowing the user to plan how
much to simplify the data.
  Correlation Elucidation
Unlike principal components analysis, factor analysis has the ability to describe correlations
hidden in the data. In fact, one of the main benefits of using factor analysis is that you can
instantaneously know the correlations between a large set of variables; this would be difficult to
do in a meaningful way using other statistical methods. Ultimately, factor analysis uses these
correlations to determine which variables are similar enough to each other to be interpreted as a
single factor.
 Interpretability
One of the main reasons factor analysis is so popular in the sciences is its ability to take abstract
data and convert it into interpretable, concrete factors. This characteristic of interpretability does
not exist in principle components analysis. The interpretability of factor analysis is clear enough
that even a novice user can directly use the output of a factor analysis to categorize the resulting
factors.
 Multiple Solutions
Another interesting characteristic that factor analysis has is its multiplicity of solutions. If the
user of factor analysis does not accept or cannot interpret the output of the analysis, the user can
rotate the solution until an acceptable solution appears. This characteristic gives factor analysis
versatility that is impossible with principal components analysis.

Assumptions for Running Factor Analysis

Assumptions of factor analysis are as follows:

Assumption #1: You have multiple variables that should be measured at the continuous level
(although ordinal variables are very frequently used).

Assumption #2: There needs to be a linear relationship between all variables. The reason for this
assumption is that a PCA is based on Pearson correlation coefficients, and as such, there needs to
be a linear relationship between the variables. In practice, this assumption is somewhat relaxed
(even if it shouldn't be) with the use of ordinal data for variables.

Assumption #3: You should have sampling adequacy, which simply means that for PCA to
produce a reliable result, large enough sample sizes are required. Many different rules-of-thumb
have been proposed. These mainly differ depending on whether an absolute sample size is
proposed or if a multiple of the number of variables in your sample are used. Generally speaking,
a mimimum of 150 cases, or 5 to 10 cases per variable, has been recommended as a minimum
sample size. There are a few methods to detect sampling adequacy: (1) the Kaiser-Meyer-Olkin
(KMO) Measure of Sampling Adequacy for the overall data set; (2) the KMO measure for each
individual variable; and (3) Bartlett's test of sphericity.
Assumption #4: Your data should be suitable for data reduction. Effectively, you need to have
adequate correlations between the variables in order for variables to be reduced to a smaller
number of components. The method used by SPSS to detect this is Bartlett's test of sphericity.

Assumption #5: There should be no significant outliers. Outliers are important because these can
have a disproportionate influence on your results. SPSS recommends determining outliers as
component scores greater than 3 standard deviations away from the mean

Type of factor analysis

a) Exploratory factor analysis (EFA)

This is used to identify complex interrelationships among items and group items that are part of
unified concepts. The researcher makes no "a priori" assumptions about relationships among
factors.

b) Confirmatory factor analysis (CFA)

It is a more complex approach that tests the hypothesis that the items are associated with specific
factors. Hypothesized models are tested against actual data, and the analysis would demonstrate
loadings of observed variables on the latent variables (factors), as well as the correlation between
the latent variables.

Types of factoring
There are many methods of factor analysis namely Principal Factor Analysis, Canonical factor
analysis, Alpha factoring, Image Factoring etc. Most commonly used method is Principal Factor
Analysis.

Principal components analysis (PCA, for short) is a variable-reduction technique that shares
many similarities to exploratory factor analysis. Its aim is to reduce a larger set of variables into
a smaller set of 'articifial' variables, called 'principal components', which account for most of the
variance in the original variables.

Terminology of Factor Analysis

 Factor loadings:
The factor loadings, also called component loadings in PCA, are the correlation coefficients
between the variables (rows) and factors (columns). Similar to Pearson's r, the squared factor
loading is the percent of variance in that indicator variable explained by the factor.
 Interpreting factor loadings:
By one rule of thumb in confirmatory factor analysis, loadings should be .7 or higher to confirm
that independent variables identified a priori are represented by a particular factor, on the
rationale that the .7 level corresponds to about half of the variance in the indicator being
explained by the factor. However, the .7 standard is a high one and real-life data may well not
meet this criterion, which is why some researchers, particularly for exploratory purposes, will
use a lower level such as .4 for the central factor
 Communality:
The communality measures the percent of variance in a given variable explained by all the
factors jointly and may be interpreted as the reliability of the indicator.
 Eigen values:/Characteristic roots:
The Eigen value for a given factor measures the variance in all the variables which is accounted
for by that factor. The ratio of Eigen values is the ratio of explanatory importance of the factors
with respect to the variables. If a factor has a low Eigen value, then it is contributing little to the
explanation of variances in the variables and may be ignored as redundant with more important
factors. Eigen values measure the amount of variation in the total sample accounted for by each
factor.
 Factor scores (also called component scores in PCA):

These are the scores of each case (row) on each factor (column). Computing factor scores allows
one to look for factor outliers. Also, factor scores may be used as variables in subsequent
modeling.

Criteria for determining the number of factors

This is an important decision for the researcher to decide about the number of factors to be
considered. Methods may not agree. For instance, the Kaiser criterion may suggest five factors
and the scree test may suggest two, so the researcher may request 3-, 4-, and 5-factor solutions.
The criteria for number of factors to be retained as are follows:

 Comprehensibility: A purely subjective criterion would be to retain those factors whose

meaning is comprehensible to the researcher.
 Kaiser criterion: The Kaiser rule is to drop all components with Eigen values under 1.0.
The Kaiser criterion is the default in SPSS and most statistical software.
 Variance explained criteria: Some researchers simply use the rule of keeping enough
factors to account for 90% (sometimes 80%) of the variation. Where the researcher's goal
emphasizes parsimony (explaining variance with as few factors as possible), the criterion
could be as low as 50%
 Scree plot: The Cattell scree test plots the components as the X axis and the
corresponding Eigen values as the Y-axis. As one moves to the right, toward later
components, the Eigen values drop. When the drop ceases and the curve makes an elbow
toward less steep decline, Cattell's scree test says to drop all further components after the
one starting the elbow.
Advantages of Factor Analysis
 Both objective and subjective attributes can be used, provided the subjective
attributes can be converted into scores.
 Factor analysis can identify latent dimensions or constructs that direct analysis
may not.
 It is easy and inexpensive.

Disadvantages of Factor Analysis

 Usefulness depends on the researchers' ability to collect a sufficient set of product
attributes. If important attributes are excluded or neglected, the value of the
procedure is reduced.
 If sets of observed variables are highly similar to each other and distinct from
other items, factor analysis will assign a single factor to them. This may obscure
factors that represent more interesting relationships
 Naming factors may require knowledge of theory because seemingly dissimilar
attributes can correlate strongly for unknown reasons.
Discriminant Analysis
Discriminant Function Analysis (DA) undertakes the same task as multiple linear regression by
predicting an outcome. However, multiple linear regression is limited to cases where the
dependent variable on the Y axis is an interval variable so that the combination of predictors
will, through the regression equation, produces estimated mean population numerical Y values
for given values of weighted combinations of X values. But many interesting variables are
categorical, such as political party voting intention, migrant/non-migrant status, making a profit
or not, holding a particular credit card, owning, renting or paying a mortgage for a house,
employed/unemployed, satisfied versus dissatisfied employees, which customers are likely to buy
a product or not buy, what distinguishes whether a person has a credit risk or not, etc.

DA is used when:

 The dependent is categorical with the predictor IV’s at interval level such as age, income,
attitudes, perceptions, and years of education, although dummy variables can be used as
predictors as in multiple regression. Logistic regression IV’s can be of any level of
measurement.
 There are more than two DV categories, unlike logistic regression, which is limited to a
dichotomous dependent variable.

Discriminant analysis linear equation

DA involves the determination of a linear equation like regression that will predict which group
the case belongs to. The form of the equation or function is:

D = v1 X1+ v2 X2+ v3 X3 ........vi Xi+ a

Where D = Discriminate function

v = the discriminant coefficient or weight for that variable

X = respondent’s score for that variable

a = a constant

i = the number of predictor variables

This function is similar to a regression equation or function. The v’s are unstandardized
discriminant coefficients analogous to the b’s in the regression equation. These v’s maximize the
distance between the means of the criterion (dependent) variable. Standardized discriminant
coefficients can also be used like beta weight in regression. Good predictors tend to have large
weights. What you want this function to do is maximize the distance between the categories, i.e.
come up with an equation that has strong discriminatory power between groups. After using an
existing set of data to calculate the discriminant function and classify cases, any new cases can
then be classified. The number of discriminant functions is one less the number of groups. There
is only one function for the basic two group discriminant analysis.

A discriminant score- This is a weighted linear combination (sum) of the discriminating

variables.

Assumptions of discriminant analysis

The major underlying assumptions of DA are:

 the observations are a random sample;

 each predictor variable is normally distributed;
 each of the allocations for the dependent categories in the initial classification are
correctly classified;
 there must be at least two groups or categories, with each case belonging to only one
group so that the groups are mutually exclusive and collectively exhaustive (all cases can
be placed in a group);
 each group or category must be well defined, clearly differentiated from any other
group(s) and natural. Putting a median split on an attitude scale is not a natural way to
form groups. Partitioning quantitative variables is only justifiable if there are easily
identifiable gaps at the points of division; for instance, three groups taking three available
levels of amounts of housing loan;
 the groups or categories should be defined before collecting the data;
 the attribute(s) used to separate the groups should discriminate quite clearly between the
groups so that group or category overlap is clearly non-existent or minimal;
 Group sizes of the dependent should not be grossly different and should be at least five
times the number of independent variables.

Purpose of DA
There are several purposes of DA:
  To investigate differences between groups on the basis of the attributes of the cases,
indicating which attributes contribute most to group separation. The descriptive technique
successively identifies the linear combination of attributes known as canonical
discriminant functions (equations) which contribute maximally to group separation.
 Predictive DA addresses the question of how to assign new cases to groups. The DA
function uses a person’s scores on the predictor variables to predict the category to which
the individual belongs.
 To determine the most parsimonious way to distinguish between groups.
 To classify cases into groups. Statistical significance tests using chi square enable you to
see how well the function separates the groups.
 To test theory whether cases are classified as predicted.

Discriminant analysis – creates an equation which will minimize the possibility of

misclassifying cases into their respective groups or categories.