RESEARCH DESIGN AND METHODOLOGY I N EVALUATING THE

RESULTS O F PSYCHOTHERAPY
ROBERT I. WATSON
Washington University School of Medicine

Before dealing critically with the present status of research in this field we may
consider briefly certain of the difficulties facing a research worker in this area. The
most obvious source of difficulty is the presence of various systematic approaches to
psychotherapy-psychoanalytic, nondirective and so on. Each of these points of
view may become proliferated by defections from the ranks. In addition, there are
the minor unorthodoxies and individual idiosyncracies within the fold which still
further complicate the issue. One cannot speak of the “effects of psychotherapy”
in a bald unqualified fashion, but research must be framed in terms of a particular
approach to psychotherapy with attention to the particular individual nuances given
by the specific practioners concerned.
Another difficulty, which is intimately related to that just mentioned, is the
relative lack, both qualitative and quantitative, of research in this field. One reason
for this is that many psychotherapists do relatively little research. This is not un-
usual in the practice of a clinical art as witness the fact that approximately 90 per-
cent or more of physicians do no research. The practice of psychotherapy is, after
all, not intended as a scientific experiment. Its aim is prevention and cure by any
available method, empiric or otherwise. But this immersion in practice is only a
part of the picture. Whether verbalized or not, many psychotherapists act as if
psychotherapy were beyond the ken of scientific research. The psychodynamic ebb
and flow, the cross currents, and the storms and the calms seem to occur on such a
vast scale as to make them withdraw from charting their deep mysteries even though
they swim in their waters. Sensitivity to the nuances of a psychodynamic relation-
ship is not necessarily correlated either in terms of training or of temperament with
ability to carry on research on this topic. Then, too, the characteristics of a psycho-
therapist, the deep identification, the critical empathy, the desire to succor the per-
son in difficulties are not necessarily characteristics which serve to further research
aims. I might even say that a certain obtuseness in these regards is sometimes needed
for research planning and production in this field. Fools with a research bent rush
in where therapeutically sensitive angels fear to tread! It might be well to add that
these last remarks are not intended as a condemnation of either experience in or a
talent for psychotherapy on the part of research workers in the field. Certainly
some modicum of both are necessary. Rather, a flair for psychotherapy is no guar-
antee of the presence of research talent and sometimes works against it.
Another difficulty complicating research in this field stems from the fact that our
knowledge of the etiology of the conditions treated by psychotherapy is meager and
confused. Clear definition of causative factors is, as yet, impossible. This is a statle
of affairs so clear to the present audience that there is no reason to labor the point.
To add to the difficulties sketched above it is not enough in my opinion, as some
of our critics seem to feel, to use a statistical approach alone in an attempt to under-
stand the effects of psychotherapy. The uniqueness of the individual and the desire
on the part of the therapist t o help him find his own way of solving his problems mil-
itate against research in which cases are thrown into a statistical hopper, there to
be mingled with others. Something must be done to preserve intraindividuality in
research on the effects of psychotherapy.
In the short time at my disposal I can review no specific studies. Instead, I shall
summarize certain trends in research, trends regarding which I am sure specific in-
stances can be supplied from the experience of members of the audience. Studies
dealing with the effectiveness of psychotherapy have been appearing for many years
in the psychiatric literature. More lately the social worker and the psychologist
have concerned themselves with this problem.
30 ROBERT I. WATSOK

Probably the most common technique of evaluation is in terms of anecdotal

imprrssioiis without attempt a t quantification. Either a complete presentation of a
singltx case or a series of shorter presentations, later summarized for their common
elcnic~ntsand unique features, are used as a basis for these discussions. In the same
general category are those reports utilizing the results obtained from one or more
psychological tests. These reports cite change after therapy in individual cases and
usuvlly offer as verification only selected bits of clinical impression which arc con-
gruent with test findings. Very often some of the data for the research Comes from
the test material secured and then intermingled with other information in the usual
fashion in clinical practice. For research purposes, this is, of course, a process of
contamination of data. Although indispensable in teaching and practice, and in-
deed in illustrating any research study, thesc studies when presented in this guisc
alone (unsupplemented by other approarhes to evaluation) reflect the naturalistic,
not the scientific phase of evaluation. It is perhaps unnecessary to remind this
audicnce that this phase is a necessary prelude to scientific research. Such studies,
even if not entirely scientifically “respectable,” pay tribute to the uniqueness and
drams of the individual and contain murh th at is valuable.
,Icoiisiderable number of early studies concerned with the effectiveness of
therapy are found to be summary studies of treatment applied to one or more noso-
logical groups in which the results are evaluated in terms of general global judg-
ments. These reports characteristically tell less about the iiidividual patient than
do tlic anecdotal studies just considered, arid lump together 100 psychoneurotics
or 500 scbizophrenics, and so on. I n so doing they gain relatively little and l o ~ e
considerably more in failure to capture intraindividual nuances. These studies
also suffer from the difficulties created by the unsolved problems of etiological and
nosological classification. The patients in these investigations are submitted to
some form or forms of treatment and, after a lapse of time, judgments are made
conceriiirig the effects. The reports using this approach fail, in almost every in-
stance, t o more than mention thc various forms of treatment-somatic, individual
psychotherapy, occupational therapy, the therapeutic effects of institutional life
and so on-which the patients received. Moreover, most of these reports concern
all sorts of treatment without any attempt at differential weighting. Most important
of all, the criteria of effectiveness th at are used are stated in such general terms as
“cured,” “improved,” “same,” and ‘worse,” often with no further identification of
the bases of judgment. A psychologist, long on experience with research design and
statistics and short on actual experience with psychotherapy, would view these
studics askance. I n effect he would say, ‘‘HOW naive one must be to use categories
of recovered, improved and so on, which categories are only vaguely or not a t all
defined as t o the nature of the success in treatment.” Then he would jump to the
conclusion that this naivete must be due to the lack of research training on the part
of the investigators. Although this may be so in some instances, it is not necessarily
the case. The sheer complexity of the problem also creates this counting and global
sorting approach. Nevertheless, there are many grounds for rejecting this way of
stating the criteria of success-the subjectivity of the judgments, the lack of pre-
cision of the definition, the dependence on the skill of the judge, the tendency to use
one or at least very few categories of success when usually “success,” whatever it
may be, is a matter of degree involving gradations.
Although understandable a t this present stage of research development, the
approach just described must inevitably be superceded by more precise methods,
with the important qualification, however, that judgment by rating as a criteria is
not necessarily dismissed simply because it has been misused. All the aforemen-
tioned objections concerning the use of ratings can be met b y procedures already
available. Judgments can be made on less global and more precise characteristics,
steps of the rating scales may be defined; judge reliability may be investigated and
validity determined b y correlation of judgments with independent criteria.
It is iiow appropriate to turn to more adequately forinulated studies of evalua-
tion of effectiveness. Here the criticisms are more in the nature of statements of
 RESEARCH DESIGN AND METHODOLOGY 31

what are considered to be errors of omission. The published studies falling in this
group are, for the most part, adequate so far as they go, but by failing to take into
consideration certain factors they are to some degree at least misleading and in-
complete.
So far as I am aware, no published research on the evaluation of effectiveness
of psychotherapy has:
(1) Considered, even by specification, all the major variables pertinent trothis
field of research.
(2) Applied uniform, objective measures of effectiveness at onset, close and
at follow-up.
(3) Made provision for study of spontaneous remission by use of control
groups.
(4) Reported simultaneously both the objective and the individual psycho-
dynamic findings.
(5) Arrived at the criteria of effectiveness by other than an arbitrary, though
plausible, selection from those criteria potentially available.
Each of these vital but neglected facets of research on evaluation will be con-
sidered in turn. What kinds of data are needed? “Information is needed about the
patient himself, his environmental background, the therapeutic situation, and the
therapist. I n other words, data are to be obtained on variables(1) intrinsic to the
individual (patient variables), (2) related both currently and historirally to his life
(situational variables), (3) descriptive of the therapy (therapeutic variables) , and
(4) descriptive of the therapist (therapist variables). The same study may devote
some thought to more than one of these variables, but no study has simultaneously
considered all of them within the same group of patients(3).” This is not to say that
research studies in this field must simultaneously investigate factors involved in all
these variables. Rather, there should be a precise specification of the nature of these
variables in each instance with some discussion at least of their probable influence
upon the results in order to allow for their adequate interpretation. This, present
studies have failed to do. A more extended discussion of this problem is offercd in a
recent paper ( 3 ) .
In order to properly evaluate effcctiveness, no matter the measures used to do
so, it appears to me that uniform objective techniques must be applied a t the time
of initial contact with the patient, a t the close of therapy, and at the time of follow-
up at a subsequent date. In published studies to date there is no objective informa-
tion available at these three crucial points from which change in psychological
status can be measured. Some studies have reported evaluations of t,he patient a t
more than one point in the course of therapy but with the criteria for evaluation
different at the various points in treatment. Several investigators have used psy-
chological tests at the beginning and again at the close of trcatment, but without a
later follow-up. Other studics begin with the patient at time of discharge from
treatment and relate the observations made at this point to situational variablrs.
Still other studies although involving the three temporal points were of the mass
scale summary judgment type described earlier.
What crudely and somcwhat inexactly may be referred to as ‘(spontaneous”
recovery does take place, and without some attention to the implications of this
phenomenon research in this field is weakened. Change for the better, interpreted as
effectiveness of the psychotherapy, may to some as yet unidentified degree be due to
general, uncontrolled, extratherapeutic effects. The most obvious solution is, of
course, the quantification of its effect through the presence of a non-treated control
group. There is, of course, nothing to prevent giving this group treatmerit after a
delay period, in which case they become their own controls. To the charge that such
deliberate delay in treatment is socially harmful, one need merely call attention
to the long delays experienced in busy outpatient clinics in any casc. The use of a
32 ROBERT I. WATSOK’

delay control group merely makes a virtue of a necessity. This technique of a non-
treated control group, the use of which is not as yet reported in the literature, is
now in the process of being used in a t least two research programs to be mentioned
later.
Both for the sake of acceptance by the practicing clinicians, and for the sake of
completeness, i t is necessary to simultaneously consider both objective and psycho-
dynamic findings. The dynamic character of the material cannot be completely
sacrificed to objective ends. It appears to me that the case study approach sketched
and judged incomplete earlier is, nevertheless, necessary. Without it, the intrain-
dividual aspect is lost. Attention both to uniqueness and commonality is necessary
for a complete and therefore accurate picture. The difficulty is th a t so far no re-
searcher has found it possible or decided it was useful to present detailed informa-
tion concerning both aspects on the same group of patients. To be sure, illustrations
on a case or two may be offered by some workers but this is not enough. The sug-
gestion is offered th at parallel and interrelated objective group and intraindividual-
dynamic studies should be done simultaneously with the same patients.
The last neglected pertinent area in research to be discussed is the issue of the
criterion or, more strictly speaking, the criteria of effectiveness. At present we are
in the unhappy state of not knowing what are the criteria of effectiveness of psycho-
therapy. The criteria used are arrived a t not by study of this as a research problem,
but by selection. Research has not yet isolated criteria on which there has been
any sort of general agreement concerning their value as indices of improvement.
Paradoxically, it is quite easy t o decide in advance that a certain specific character-
istic, e.g., degree of insight or reorganization of basic personality structure is indica-
tivc and then make it the criterion. But have we the right to make such assump-
tions? I, for one, do not think so.
I n connection with some research on this topic we have taken the position that
instead of deciding upon a limited number of arbitrarily selected general criteria of
effectiveness we would formulate as many criteria as possible and secure evidence
through rating forms, psychological tests, and narrative case history material.
The results in these criteria studies when interrelated b y techniques of cluster
or factor analysis should tell us something about the criteria to be used in future
research even though they too emerge from a process of selection but one which
provides a broader sample of potential criteria. The therapeutic goals in any one
individual case must, of course, be specified (and these would inevitably w r y ) but
they in turn could be related to the more general criteria.
Since this report is primarily addressed to a n audience of psychologists, it might
be well t o single out the work of certain members of our profession for special con-
sideration. It might be well to emphasize th at most studies of psychotherapy by
psychologists emphatically are not studies of effectiveness. The studies of Fiedler,
for example, are concerned with the therapist variable, as such, and the techniques
as used by him will presumably be quite valuable when applied in evaluating effect-
iveness. This is a task for the future.
The studics of Hunt and his associates with the movement scale and DRQ,
are primarily concerned with the therapeutic variable as a measure of effectiveness
but only incidentally with the patient, background, or therapist variable. They
might be even more useful in a broader sphere of research on effectiveness when
used in a research setting implied b y the earlier general criticisms.
It is t o the work of Rogers more than any other man that we, as psychologists,
arc indebted for stimulating our interest in research on psychotherapy. Many rc-
search studies of psychotherapy by Rogers and his colleagues and students, it may
be remembered, arc concerned with the process as such and contribute only very in-
directly t o the study of its effectiveness. They, therefore, cannot be criticized for
failing to do something they did not start out to do. So far as they are coiicerned with
effccts they expose to the reader the process taking place between onset, and close of
therapy (as do those of Hunt) and leave one to judge its effectiveness on this basis.
 RESEARCH DESIGN AND METHODOLOGY 33

The reported studies b y Rogers and his associates are often astonishingly incomplete
in the information they give concerning the therapist, the background of the patient
or even the characteristics of the patient. They are especially prone to ignore the
patient’s characteristics-we know very little about his background, how sick he
was, the nature of the symptoms he exhibits, pathological processes presumed to be
operative, the chronicity, the background factors affecting prognosis, etc. It may
be th a t from their point of view their investigation and specification is irrelevant but
this is hardly the case with therapists of othcr orientations.
A qualification concerning the foregoing criticisms must be offered. One must
note that I refer t o “published” studies. The criticisms so baldly stated are in some
measure, although pcrhaps differently defined, a part of the research climate of the
times, and no claim is made that they are necessarily original with the speaker. At
least two research programs now in progress may meet some or all of these object>ions.
The present research program of Rogers and his associates (familiar to me only
through the unpublished first and second interim reports) may eventually be in a
position t o meet a t least some of these errors of omission.
It might be mentioned in closing that a research program on evaluation of the
effects of psychotherapy in part specifically designed to meet these difficulties and
objections t o previous research is underway a t the Washington University Medical
School. Preliminary reports are given in a series of papers(l9 2 . 3 . 4 * 5 , 6 ) . By no
means, have all of these difficulties been overcome even to our own satisfaction, b u t
a t least they are recognized for what they are, and it is hoped that the project wilI
supply partial answers, incomplete and preliminary though they may be.
BIBLIOGRAPHY
1. MENSH,I. K., and WATSOX, R. I. Psychiatric opinions on personality factors in psychotherapy.
J . clin. Psychol., 1950, 6, 237-242.
2. MENSIr, I. N., and GOLDEN, J. M. Factors in psychotherapeutic success. J . Mo. Xt. Med. ASSOC.,
1951, 48, 180-184.
3. WATSOK,R. I. Measuring the effectiveness of psychotherapy: problems for investigation. (To
be published.)
4. WATSON, R. I.,and MENSH,I. N. The evaluation of the effects of psychotherapy: I. Sources of
material. J. Psychol., 1951, 32, 259-273.
5. WATSON, R. I., and MENSH,I. N. The evaluation of the effects of psychotherapy: 11. A case
study. J . Psychol., 1951, 32, 275-291.
6. WATSON,R. I., MENSH,I. N., GILDEA,E. F. The evaluation of the effects of psychotherapy:
111.Research design. J . Psychol., 1951, 32, 293-308.