Synthesizing Qualitative and Quantitative Evidence
Synthesizing Qualitative and Quantitative Evidence
Synthesizing Qualitative and Quantitative Evidence
net/publication/8063434
CITATIONS READS
1,263 24,855
5 authors, including:
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Mary Dixon-Woods on 04 June 2014.
Mary Dixon-Woods, Shona Agarwal, David Jones, Bridget Young1, Alex Sutton
Department of Health Sciences, University of Leicester, Leicester; 1Division of Clinical Psychology, University of Liverpool, Liverpool, UK
Background: The limitations of traditional forms of systematic review in making optimal use of all forms of
evidence are increasingly evident, especially for policy-makers and practitioners. There is an urgent need for robust
ways of incorporating qualitative evidence into systematic reviews.
Objectives: In this paper we provide a brief overview and critique of a selection of strategies for synthesising
qualitative and quantitative evidence, ranging from techniques that are largely qualitative and interpretive through
to techniques that are largely quantitative and integrative.
Results: A range of methods is available for synthesising diverse forms of evidence. These include narrative
summary, thematic analysis, grounded theory, meta-ethnography, meta-study, realist synthesis, Miles and
Huberman’s data analysis techniques, content analysis, case survey, qualitative comparative analysis and Bayesian
meta-analysis. Methods vary in their strengths and weaknesses, ability to deal with qualitative and quantitative forms
of evidence, and type of question for which they are most suitable.
Conclusions: We identify a number of procedural, conceptual and theoretical issues that need to be addressed in
moving forward with this area, and emphasise the need for existing techniques to be evaluated and modified, rather
than inventing new approaches.
Journal of Health Services Research & Policy Vol 10 No 1, 2005: 45–53 # The Royal Society of Medicine Press Ltd 2005
For the sake of clarity, we begin by distinguishing synthesis and would effectively form the categories
synthesis of primary studies from the secondary analysis under which the data extracted from any empirical
of qualitative data. Some qualitative researchers use studies are to be summarised. This summary may be
‘secondary analysis of qualitative research’ to refer to the achieved through pooling of the data, perhaps through
re-analysis of datasets originally collected for a different meta-analysis, or less formally, perhaps by providing a
purpose.7,8 In this paper, we will discuss techniques for descriptive account of the data. It is important not to
synthesising evidence from primary study reports, not for exaggerate how secure such categories are (for example,
re-analysis of the original datasets. how to define ‘older people’ might be debated) but the
Some of the approaches described below were primary focus of an integrative synthesis is not on the
originally developed for analysis of primary data, and development of concepts or their specification. This
would require evaluation and possible adaptation to does not prevent integrative syntheses from fulfilling
serve the purposes of synthesis. Other approaches have theoretical or interpretive functions: all types of
been developed specifically for synthesis, but few have syntheses involve interpretation. The kinds of theory
been extensively used or comprehensively evaluated. that integrative syntheses may be especially likely to
Many approaches were originally developed to deal produce will often be theories of causality, and may also
solely with one form of evidence (e.g. quantitative), and include claims about generalisability. For example, an
while potentially adaptable to include other forms of integrative synthesis may be able to show which types of
evidence (e.g. qualitative), there are few empirical intervention appear most likely to improve rates of
examples. The methods vary in their degree of formality: breast-feeding in which groups of women.
some, such as the qualitative comparative method, Second, we suggest that the defining characteristic of
involve highly specified procedural techniques; others, an interpretive synthesis is its concern with the devel-
such as narrative summary, are more informal. Some are opment of concepts, and with the development and
much better suited to some types of question than specification of theories that integrate those concepts.
others. This review is necessarily selective and partial, An interpretive synthesis will therefore avoid specifying
and, because of the limitations of space, only the most concepts in advance of the synthesis. In contrast with an
striking features of each approach will be selected for integrative synthesis, it will not be concerned to fix the
comment. meaning of those concepts at an early stage in order to
facilitate the summary of empirical data relating to those
concepts. The interpretive analysis that yields the
Types of syntheses
synthesis is conceptual in process and output, and the
Noblit and Hare introduce a useful distinction between main product is not aggregations of data, but theory.
integrative and interpretive reviews.9 Integrative synthesis is Again it is important not to caricature an interpretive
concerned with combining or amalgamating data. It synthesis as therefore floating free of any empirical
involves techniques, such as meta-analysis, that are anchor: an interpretive synthesis of primary studies must
concerned with assembling and pooling data, and be grounded in the data reported in those studies. An
require a basic comparability between phenomena interpretive synthesis may be able to address questions
studied so that the data can be aggregated for analysis. that are difficult to address through integrative synth-
Noblit and Hare argue that interpretive reviews, by eses, and might be concerned with the generation of
contrast, see the essential tasks of synthesis as involving middle-range theories – explanations which apply in a
both induction and interpretation. Interpretive reviews specified domain, such as seeking to explain why people
achieve synthesis through subsuming the concepts defer help-seeking for some types of symptoms. We
identified in the primary studies into a higher-order argue that interpretive syntheses can be carried out on all
theoretical structure. Noblit and Hare appear to suggest types of evidence, both qualitative and quantitative.
that integrative reviews are primarily suitable for We do not see interpretive and integrative forms of
synthesising quantitative studies, while interpretive synthesis as being completely distinct. There is consider-
reviews are suitable for synthesising interpretive studies. able overlap. Whilst most forms of synthesis can be
We wish to elaborate on Noblit and Hare’s original characterised as being either primarily interpretive or
conceptualisation and to propose a new way of thinking primarily integrative in form and process, every inte-
about these different forms of synthesis, rather than grative synthesis will include elements of interpretation,
different forms of review. First, we want to be careful not and every interpretive synthesis will include elements of
to identify integrative with positivist or with quantitative. aggregation of data. The choice of the form of synthesis
Instead, we suggest that integrative syntheses are those is likely to be crucially related to the form and nature of
where the focus is on summarising data, and where the the research question being asked.
concepts (or variables) under which data are to be We present below a selection of synthesis methods,
summarised are assumed to be largely secure and well together with an informal commentary on issues in their
specified. For example, in an integrative synthesis of the use. The different methods (summarised briefly in the
impact of educational interventions on uptake of Table) can be broadly grouped in terms of their
influenza immunisation in older people, the key epistemological and ontological foundations and
concepts (educational intervention, uptake, older whether the aim of synthesis is primarily interpretative
people) would be defined at an early stage in the or integrative. Clustering towards the interpretive end of
the spectrum are the methods of narrative summary, have adopted this kind of approach. Garcia et al, for
grounded theory, meta-ethnography, meta-synthesis, example, report a thematic analysis of women’s views of
meta-study, realist synthesis and Miles and Huberman’s ultrasound in pregnancy, involving tabulation of papers
data analysis techniques, while lying at the more and counts of papers contributing data on each
integrative end of the spectrum are content analysis, theme.14
case survey, qualitative comparative analysis and Thematic analysis allows clear identification of promi-
Bayesian meta-analysis. Within these clusters or groups, nent themes, and organised and structured ways of
elements of the methods show some overlap. At the dealing with the literature under these themes. It is
interpretive end, all involve some form of creative flexible, allowing considerable latitude to reviewers and
process where new constructs are fashioned by identi- a means of integrating qualitative and quantitative
fying related concepts in the original studies, and then evidence. However, it suffers from several important
reworked and reformulated to extend theory. At the problems. Thematic analysis can be either data driven –
integrative end, all approaches involve the quantifica- driven by the themes identified in the literature itself –
tion and systematic integration of data. In practice, or theory driven – oriented to evaluation of particular
many approaches involve elements both of interpreta- themes through interrogation of the literature. The
tion and integration. failure of much writing on thematic analysis to
distinguish adequately between these two approaches
has resulted in a lack of transparency. More generally,
Table
there is lack of clarity about exactly what thematic
The Table is included only in the online version of analysis involves and the processes by which it can be
the journal, and may be viewed free of charge at: achieved; for example, there is a lack of explicitness
http://www.ingentaselect.com/rpsv/cw/rsm/13558196/ about procedures and aims, including the extent to
v10n1/s10/ p45. which thematic analyses should be descriptive or
interpretive. It is unclear whether the structure of the
analysis should reflect the frequency with which
Narrative summary
particular themes are reported, or whether the analysis
Narrative summary typically involves the selection, should be weighted towards themes that appear to have
chronicling, and ordering of evidence to produce an a high level of explanatory value. If thematic analysis is
account of the evidence. Its form may vary from the limited to summarising themes reported in primary
simple recounting and description of findings through studies, it offers little by way of theoretical structure
to more interpretive and explicitly reflexive accounts within which to develop higher order thematic cate-
that include commentary and higher levels of abstrac- gories beyond those identified from the literature.
tion. Narratives of the latter type can account for
complex dynamic processes, offering explanations that
Grounded theory
emphasise the sequential and contingent character of
phenomena.10 Narrative summary is often used in Grounded theory is a primary research approach that
systematic reviews alongside systematic searching and has been hugely influential in the development of
appraisal techniques, as, for example, in a systematic qualitative methods in health for several decades.
review of interventions to promote uptake of breast- Originally formulated by Glaser and Strauss, the
feeding.11 approach describes methods for qualitative sampling,
Narrative summary can ‘integrate’ qualitative and data collection and data analysis.15 It sees the over-riding
quantitative evidence through narrative juxtaposition – concern of qualitative research as the generation of
discussing diverse forms of evidence side by side – but, as theory (generalisable explanations for social
a currently largely informal approach, is likely always to phenomena). In principle, grounded theory offers a
be subject to criticism of its lack of transparency. potentially suitable approach to the synthesis of primary
However, under the UK ESRC Methods Programme,12 studies. The constant comparative method, the most
methodological guidance on the conduct of narrative widely used element of grounded theory, has the most
summaries is being developed, which will inform future obvious potential for application, in part (especially in
good practice in this area. later formulations) because it offers a set of procedures
by which data may be analysed.16
There are as yet only a few examples of the use of
Thematic analysis
grounded theory for synthesis. One of the most robust
Thematic analysis, clearly sharing some overlaps with and theoretically sophisticated is Kearney’s grounded
narrative summary and content analysis, involves the theory analysis of 15 qualitative papers on women’s
identification of prominent or recurrent themes in the experience of domestic violence.17 Studies were selected
literature, and summarising the findings of different for inclusion based on Kearney’s judgement of their
studies under thematic headings. Summary tables, methodological rigour and their theoretical contribu-
providing descriptions of the key points, can then be tion. Data were extracted and assembled onto a grid to
produced.13 Several recent attempts at providing struc- facilitate cross-case comparison. The constant compara-
tured or systematic overviews of diverse areas of evidence tive method as described by Strauss and Corbin16 was
then used to cluster concepts that were identified across findings of the separate studies. Some analogies can
studies into new categories, followed by axial coding, in be drawn between LOA and the constant comparative
which the nature of the categories was more fully method.
specified and the relationships between categories were
tested. Theoretical sampling was undertaken by Britten et al offer a well-documented demonstration of
returning repeatedly to the study reports, though it is meta-ethnography to synthesise four papers on the
not clear exactly what this entailed. Kearney describes meanings of medicines.19 A grid was created in which
using memos with specific links to the source texts to the rows were labelled with the relevant concepts from
document the analysis. each paper. The last row of the grid represented the
Kearney’s study shows how grounded theory can deal main theory arising from each paper. In developing the
with sampling issues, and allow a synthesis of studies by grid, Britten and colleagues used Schutz’s notion of first
treating study reports as a form of data on which analysis and second order constructs.20 ‘First order constructs’
can be conducted using the constant comparative refer to the everyday understandings of ordinary people,
method. The generation of higher order themes as a whereas ‘second order constructs’ refer to the constructs
means of synthesis encourages reflexivity on the part of of the social sciences. The grid reported these second
the reviewer while preserving the interpretive properties order constructs. Britten and colleagues built on these
of the underlying data. Grounded theory, in the notions explanations and interpretations to develop what they
of theoretical saturation and theoretical sampling, also call ‘third order interpretations’, which were consistent
offers means of limiting the number of papers that need with the original results but extended beyond them.
be reviewed, especially where the emphasis is on They argue that these third order interpretations justify
conceptual robustness rather than on completeness of claims that meta-ethnography achieves more than
data. The approach could potentially deal with quanti- traditional literature review.
tative data by converting quantitative data to qualitative Meta-ethnography represents one of the few areas in
form, for example through a narrative descriptive which there is an active programme of funded metho-
process, though there are currently no examples of this. dological research for qualitative synthesis.21 It is thus of
Grounded theory does, however, have several disad- considerable potential and interest. It offers several
vantages as a method for review. Even in its more advantages, including its systematic approach combined
proceduralised forms, as an interpretive method it with the potential for preserving the interpretive
inherently lacks the transparency important to the properties of the primary data. Like grounded theory,
systematic review community. It also offers no advice it can potentially deal with quantitative data, though
on how to appraise studies for inclusion in a review. again there are no empirical examples of this.
There are several important epistemological issues to be It is clear, however, that several issues need to be
resolved, including the status of the accounts offered in resolved if meta-ethnography is to develop in ways that
the studies and how to deal with the varying credibility are helpful and useful to reviewers. Meta-ethnography,
of these accounts. Moreover, the methodological at least in its original form, offers no guidance on
anarchy that characterises the area, with ‘grounded sampling or appraisal, and is solely a means of synthesis.
theory’ being used to label many different types of It is demanding and laborious, and might benefit from
analysis, should not be underestimated as a barrier to the development of suitable software. Britten et al note
the development of this approach as a means of that, like most interpretive methodologies, the process
synthesising primary studies.18 of qualitative synthesis cannot be reduced to a set of
mechanistic tasks, and meta-ethnography thus runs into
the usual problems of transparency. Campbell et al
Meta-ethnography point to the problem of determining in which order the
Meta-ethnography is a set of techniques specifically papers should be synthesised for reciprocal translational
developed for synthesising qualitative studies. First analysis.22
proposed by Noblit and Hare, it involves three major The distinction between hermeneutic and dialectic
strategies:9 aspects of synthesis, originally proposed by Guba and
Lincoln in relation to primary studies, is crucial.23 For
1. Reciprocal translational analysis (RTA). The key meta- purposes of synthesis, the hermeneutic aspects refer to
phors, themes, or concepts in each study are the need to portray the findings of the original papers
identified. An attempt is then made to translate accurately, and might be said to be the function of
these into each other. Some analogies can be drawn reciprocal translational analysis (RTA). The dialectic
between RTA and content analysis. aspects refer to comparing and contrasting these
2. Refutational synthesis. Key metaphors, themes or original findings, to generate new interpretations, and
concepts in each study are identified, and contra- might be said to be the function of the lines-of-
dictions between the reports are characterised. argument synthesis (LOA). Further, reciprocal transla-
Possible ‘refutations’ are examined and an attempt tional analysis is procedurally akin to content analysis,
made to explain them. and therefore is primarily integrative, yet requires an
3. Lines of argument synthesis (LOA). This involves interpretive process to identify the concept that best
building a general interpretation grounded in the ‘fits’ concepts identified from across the study.
primary research, but offers a means of synthesising synthesise case studies from areas outside health care in
study reports by allowing a systematic way of categorising addressing questions relating to organisation or policy.
and counting themes. Evans and Fitzgerald describe The approach also has a number of important
using content analysis in a systematic review of reasons limitations. It relies on having a sufficient number of
for physically restraining patients and residents.31 They cases to make quantitative analysis worthwhile. Many
identify ‘safety’ as one of the categories, and show that qualitative researchers would probably reject the
this is cited in 92% of reports as a reason for using description of their studies as ‘cases’. The case survey
physical restraint. method would have difficulty in coping with the
Content analysis is a well-developed and widely used interpretive properties of qualitative data; and contex-
method in the social sciences, with an established tual factors that might be important in explaining the
pedigree. It is fairly transparent in its processes and features of particular cases may be ‘stripped out’ as the
easily auditable. Software packages for undertaking the data are reduced to quantitative form. Yin and Heald
analysis are available. The data or synthesised evidence themselves acknowledge that the case survey method is
resulting from content analysis lend themselves to likely to be more suited to studies of outcomes than
tabulation. Content analysis converts qualitative data processes, and that it should be used in carefully
into quantitative form, making it easier to manipulate selected situations.
within quantitative frameworks should this be appro-
priate. Though not a criticism of the method, the term
‘content analysis’ is often mis-used to refer to thematic
Qualitative comparative analysis method
analysis.
There are disadvantages of content analysis (derived The qualitative comparative analysis (QCA) method,
from its limitations as a primary research method) that originally proposed by Ragin, was based on the view that
limit its usefulness as a means of synthesis. It is the same outcome may be achieved in different
inherently reductive and tends to diminish complexity combinations of conditions, and that causation must
and context. It may be unlikely to preserve the be understood in terms of necessary and sufficient
interpretive properties of underlying qualitative conditions.36 Complex causal connections are analysed
evidence. Frequency-counting may fail to reflect the using Boolean logic to explain pathways to a particular
structure or importance of the underlying phenomenon; outcome.
in the example above, ‘safety’ may have been the most Qualitative comparative analysis requires the construc-
cited reason, but not necessarily the most important tion of a ‘truth table’, showing all logically possible
reason. A related problem is the danger that absence of combinations of the presence and absence of indepen-
evidence (non-reporting) could be treated as evidence dent variables and the corresponding outcome variable.
of absence (not important). The results may be over- Actual cases in the data that match each possible
simplified and count what is easy to classify and count, combination of independent variables and the
rather than what is truly important. There is a need for outcome are sought. A Boolean minimisation process
caution over analyst perspective. Content analysis has then eliminates all logically inconsistent variables. From
traditionally had a role in media analysis where it has many potential explanations, only a small number of
been involved, for example, in exposing a range of combinations of variables to account for particular
biases in reporting.32 Clearly a different orientation may outcomes is left. This yields a parsimonious and logically
be required where a synthesis is the objective. consistent model of the combination of variables
associated with the outcome under study.37 Recent
refinements of the method have included the introduc-
Case survey
tion of ‘fuzzy’ logic, so that it is not necessary to
The case survey method proposed by Yin and Heald is a dichotomise variables so precisely.38
formal process for systematically coding relevant data Cress and Snow’s review used QCA to look at
from a large number of qualitative cases for quantitative outcomes of mobilisation by the homeless, examining
analysis.33 A set of highly structured closed questions is 15 case studies.39 Their ‘truth table’ allowed them to
used to extract data from individual case studies. These show under what circumstances social movements could
data are converted to quantitative form subsequently lead to a range of outcomes, including rights. Effectively
used for statistical analysis. Developments of the this analysis showed which conditions must be satisfied
approach use multiple coders to score the cases.34 in order to achieve specific outcomes.
Jensen and Rodgers advocate the use of meta-analytic Qualitative comparative analysis has the advantage
schedules with rows being case studies and columns that it may not require as many cases as the case survey
being variable-related findings or other study attri- method. It can be used with previously conducted
butes.35 studies as well as with new studies, and thus encourages
Chief among the strengths of the method is, perhaps, an evolutionary and integrative approach to knowledge
its ability to synthesise both qualitative and quantitative creation. It allows easy integration of both qualitative
evidence. It converts qualitative evidence into a quantita- and quantitative forms of evidence, and is transparent
tive form, making it easy to manipulate within quantitative and systematic. Complex and multiple patterns of
frameworks. The approach may make it possible to causation may be explored.
However, as with any approach that relies on may lack appeal to some sections of the qualitative
converting qualitative evidence into quantitative form, community. The techniques for achieving this form of
qualitative researchers are likely to argue about the analysis are still under development, and many method-
ontological and epistemological assumptions of QCA. It ological issues remain to be resolved, including those
is mainly appropriate when a causal pathway is sought, relating to elicitation of the priors and the impact of
and may be ill-suited to the more usual concerns of different methods of qualitative synthesis.
qualitative research, including the meanings that people
give to their experiences. Qualitative comparative
Issues in synthesis
analysis appears to be designed primarily to deal with
case studies, and may not cope well with the more usual We have presented a brief selective overview of strategies
form of qualitative study reports. Qualitative compara- for synthesising qualitative and quantitative evidence.
tive analysis also suffers the problem that absence of Questions can be asked about whether apparent
evidence is treated as evidence of absence, and the data differences between the strategies reflect superficial
may not reflect the underlying structures of the differences in terminology and the degree to which
phenomenon. methods have been specified. At least some differences
may reflect how the approaches have developed in
isolation, rather than more fundamental points of
Bayesian meta-analysis
divergence. Work that compares the results of applying
Meta-analysis is the quantitative synthesis of data, in the different methods of synthesis will be useful in
which evidence is pooled using statistical techniques. distinguishing trivial from non-trivial divergences
Bayesian forms of meta-analysis offer flexibility in between the methods. Few to date have been extensively
handling data from diverse study types. Bayesian applied and evaluated in reviews of the health literature.
analysis begins with beliefs that are temporally or In the longer term, methods that incorporate the most
logically prior to the main data, formally expressed as useful elements of these approaches may be developed;
a probability distribution, and updates these beliefs it is important that future work builds on the existing
using the main data to produce an assessment of current approaches. Extending, adapting and evaluating these
evidence. methods would seem to provide greatest opportunities
It allows the integration of qualitative and quantitative for capitalising on what has already been done, rather
forms of evidence, and explicitly allows qualitative than invention of ‘new’ approaches. For the area to
evidence to contribute to meta-analysis by identifying move forward at all, however, it is important that a
variables to be included and providing evidence about number of key common questions be addressed.
effect sizes. It therefore reflects important precedents
from primary research, where qualitative research is
Is it acceptable to synthesise studies?
often used to identify the variables of interest before
conducting a quantitative study. It may therefore help to The acceptability of many strategies for synthesis to the
ensure that meta-analyses more properly reflect the various research communities has been alluded to
diversity of evidence at primary level, perhaps especially throughout this paper, and clearly will have major
to show where quantitative data relevant to people’s implications for their uptake. There are arguments
concerns might be absent. about whether it is feasible or acceptable to conduct
The method proposed by Roberts et al for synthe- syntheses of qualitative evidence at all,41 and whether it
sising data from 11 qualitative and 32 quantitative is acceptable to synthesise qualitative studies derived
primary studies about uptake of childhood immunisa- from different traditions. The distinctions, tensions and
tion used the qualitative studies as a source of external conflicts between these have been vividly described.42
evidence to identify the relevant variables to include in Some have argued that studies to be synthesised should
the synthesis and to contribute to informing initial share a similar methodology, suggesting that even when
judgments about the likely effects of these variables.40 similar themes can be identified across all studies, the
The prior distribution of variables likely to influence mixing of methods leads to difficulties in developing
uptake of immunisation was then combined with theory because of differences in their epistemological
evidence from quantitative studies to produce an foundations.23,43 Others have taken a more pragmatic
overall synthesis, the results of which were expressed approach, and Paterson et al deal with the problem of
as posterior probabilities. These posterior probabilities, different approaches in part through their techniques of
reflecting both quantitative and qualitative evidence, meta-method and meta-theory.24
considerably altered estimates of the importance of Perhaps even more likely to generate controversy are
those factors compared with the qualitative evidence attempts to synthesise qualitative and quantitative
alone. This synthesis showed that exclusion of either the evidence. It is evident from the discussion above that
quantitative or the qualitative studies from the meta- synthesis of diverse forms of evidence will generally
analysis would have resulted in only a partial synthesis of involve conversion of qualitative data into quantitative
the available evidence. form or vice versa.44 While it is clear in principle that
However, Bayesian meta-analysis, though conceptually many of the approaches that involve ‘quantitising’
straightforward, is not necessarily easy to implement and qualitative data lend themselves easily to subsequent
analysis, few approaches that have dealt with qualitative might result in synthesisers discounting important
synthesis have dealt directly or explicitly with how to studies for the sake of ‘surface mistakes’, which are
incorporate quantitative evidence. Issues of how to distinguished from fatal mistakes that invalidate the
‘qualitise’ quantitative data and how to deal with it findings.23,50
within a qualitative synthesis, also require further Published examples include reviews that have chosen
development. Questions of which body of evidence not to appraise the papers,14 as well as those which have
should be synthesised first (e.g. qualitative or quantita- opted to appraise the papers using a formalised
tive) or whether they should be synthesised at the same approach.22 If the argument prevails that some quality
time have barely been addressed. appraisal is necessary, the problem then arises as to how
this should be undertaken.
guidance for those carrying out or commissioning reviews. 29. Bryman A. Social research methods. Oxford: Oxford
CRD Report Number 4 (2nd edn). York: NHS CRD, 2001 University Press, 2001
6. Dixon-Woods M, Fitzpatrick R. Qualitative research in 30. Silverman D. Interpreting qualitative data: methods for
systematic reviews has established a place for itself. BMJ analysing test, talk and interaction. London: Sage, 2001
2001; 323: 765–766 31. Evans D, Fitzgerald M. Reasons for physically restraining
7. Heaton, J. Secondary analysis of qualitative data. Social patients and residents: a systematic review and content
Research Update 1998; 22 http://www.soc.surrey.ac.uk/ analysis. International Journal of Nursing Studies 2002; 39:
sru/SRU22.html 735–743
8. Thorne S. Secondary analysis: issues and implications in: 32. Henderson L, Kitzinger J, Green J. Representing infant
JM Morse Issues in Qualitative Research Methods. feeding: content analysis of British media portrayals of
London: Sage, 1994 bottle feeding and breast feeding. BMJ 2000; 321: 1196–1198
9. Noblit GW, Hare RD. Meta-ethnography: synthesising 33. Yin RK, Heald KA. Using the case survey method to analyse
qualitative studies. Newbury Park, California: Sage, 1988 policy studies. Administrative Science Quarterly 1975; 20:
10. Abbott A. Conceptions of time and events in social science 371–381
methods: causal and narrative approaches. Historical 34. Larsson R. Case survey methodology: quantitative analysis
Methods 1990; 23: 140–150 of patterns across case studies. Academy of Management
11. Fairbank L, O’Meara S, Renfrew MJ, Woodridge M,
Journal 1993; 36: 1515–1546
Sowden AJ, Lister-Sharp D. A systematic review to
35. Jensen JL, Rodgers R. Cumulating the intellectual gold of
evaluate the effectiveness of interventions to promote the
case study research. Public Administration Review 2001;
initiation of breastfeeding. Health Technology Assessment
2000; 4: 1–171 61: 236–246
12. Economic & Social Research Council Research Methods 36. Ragin CC. The comparative method: moving beyond
Programme. http://www.ccsr.ac.uk/methods/ [accessed 5 qualitative and quantitative strategies. Berkeley, California:
June 2003] University of California Press, 1987
13. Mays N, Roberts E, Popay J. Synthesising research 37. Haggerty TR. Unravelling patterns of multiple conjunc-
evidence. In: Fulop N, Allen P, Clarke A, Black N (eds). tural causation in comparative research: Ragin’s qualitative
Studying the organization and delivery of health services. comparative method. Journal of Comparative Physical
London: Routledge, 2001 Education and Sport 1992; 14: 19–27
14. Garcia J, Bricker L, Henderson J, Martin M, Mugford M, 38. Ragin CC. Fuzzy set social science. Chicago: University of
Nielson J et al. Women’s views of pregnancy ultrasound: a Chicago Press, 2000
systematic review. Birth 2002; 29: 225–250 39. Cress D, Snow D. The outcomes of homeless mobilization:
15. Glaser BG, Strauss AL. The discovery of grounded theory: the influence of organization, disruption, political media-
strategies for qualitative research. New York: Aldine de tion, and framing. American Journal of Sociology 2000;
Gruyter, 1967 105: 1063–1104
16. Strauss A, Corbin J. Basics of qualitative research: 40. Roberts KA, Dixon-Woods M, Fitzpatrick R, Abrams KR,
techniques and procedures for developing grounded Jones DR. Factors affecting the uptake of childhood
theory, 2nd edn. Thousand Oaks, California: Sage, 1998 immunisation: a Bayesian synthesis of qualitative and
17. Kearney M. Enduring love: a grounded formal theory of quantitative evidence. Lancet 2002; 360: 1596–1599
women’s experience of domestic violence. Research in 41. Sandelowski M, Docherty S, Emden C. Focus on qualitative
Nursing and Health 2001; 24: 270–282 methods. Qualitative meta-synthesis: issues and techni-
18. Dixon-Woods M, Shaw RL, Agarwal S. The problem of ques. Research in Nursing and Health 1997; 20: 365–371
quality in qualitative research. Quality and Safety in 42. Barbour R. Mixing methods: quality assurance or qualitative
Healthcare 2004; 13: 223–225 quagmire? Qualitative Health Research 1998; 8: 352–361
19. Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill 43. Estabrooks CA, Field PA, Morse JM. Aggregating qualita-
R. Using meta-ethnography to synthesise qualitative tive findings: an approach to theory development.
research: a worked example. Journal of Health Services Qualitative Health Research 1994; 4: 503–511
Research & Policy 2002; 7: 209–215 44. Tashakkori A, Teddli C. Mixed methodology: combining
20. Schutz A. Collected papers, vol 1. The Hague: Martinus qualitative and quantitative approaches. Thousand Oaks,
Nijhoff, 1962 California: Sage, 1998
21. University of Birmingham, National Co-ordinating Centre 45. Strauss AL, Corbin J. Basics of qualitative research.
for Research Methodology: ongoing research. http://
London: Sage, 1990
pcpoh.bham.ac.uk/publichealth/nccrm/ongoing_research.
46. Schreiber R, Crooks D, Stern PN. Qualitative meta-analysis.
htm [accessed July 2004]
In: Morse JM, ed. Completing a qualitative project: details
22. Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M
and dialogue. Thousand Oaks, California: Sage, 1997
et al. Evaluating meta-ethnography: a synthesis of qualita-
tive research on lay experiences of diabetes and diabetes 47. Paterson BL, Thorne S, Dewis M. Adapting to and
care. Social Science and Medicine 2003; 56: 671–684 managing diabetes image. Image: The Journal of
23. Jensen LA, Allen MN. Meta-synthesis of qualitative Nursing Scholarship 1998; 30: 57–62
findings. Qualitative Health Research 1996; 6: 553–560 48. Booth A. Cochrane or cock-eyed? How should we
24. Paterson BL, Thorne SE, Canam C, Jillings C. Meta-study conduct systematic reviews of qualitative research? Paper
of qualitative health research. Thousand Oaks, California: presented at the Qualitative Evidence-based Practice
Sage, 2001 Conference, Taking a Critical Stance. Coventry Univer-
25. Pawson R. Evidence-based policy: the promise of ‘realist sity, May 14–16 2001. http://www.leeds.ac.uk/educol/
synthesis’. Evaluation 2002; 8: 340–358 documents/00001724.doc
26. Pawson R. Evidence and policy and naming and shaming. 49. Sherwood G. Meta-synthesis: merging qualitative studies to
Policy Studies 2002; 23: 211–230 develop nursing knowledge. International Journal for
27. Miles MB, Huberman M. Qualitative data analysis: an Human Caring 1999; 3: 37–42
expanded sourcebook. London: Sage, 1994 50. Sandelowski M, Docherty S, Emden C. Focus on qualitative
28. McNaughton DB. A synthesis of qualitative home visiting methods. Qualitative meta-synthesis: issues and techni-
research. Public Health Nursing 2000; 17: 405–414 ques. Research in Nursing and Health 1997; 20: 365–371
Narrative summary Narrative description and ordering of . May be theory-led . Lack of transparency . Can cope with large evidence base,
primary evidence (perhaps selected) with . May include triangulation of different . Many variants and absence of comprising diverse evidence types
commentary and interpretation evidence types procedures/standards . Flexibility
. May be dependent on skills/prejudices of . Could be used for theory-building
reviewer
Thematic analysis Identification of major/recurrent themes in May be theory- or data-driven . Lack of transparency regarding process . Flexible procedures for reviewers
literature; summary of findings of primary decisions . Copes well with diverse evidence types
studies under these thematic headings . Largely descriptive basis to groupings, not . Could be used for theory-building
necessarily developing theory, accounting
for contradictions
Grounded theory Constant comparative method (CCM) Many researchers use only part of approach, . Possible criticism of lack of transparency . Seeks generalised explanations/theories
identifies patterns and interrelations in e.g. just CCM . Variants are rife . Suggests means of sampling papers for
primary data. Sampling responds to analysis, inclusion in review
until theoretical saturation reached. . Sampling to theoretical saturation limits
number of papers to review
. Can potentially deal with diverse evidence
types
Meta-ethnography Reciprocal translational analysis identifies key Sometimes restricted to primary studies . Does not guide sampling . Seeks higher order (generalised) theories
themes in each study, then seeks to translate from a single paradigm . Lack of transparency regarding selection . Can potentially deal with diverse evidence
these into context of each other study. Themes of primary studies types
. May be order-of-synthesis effects
J Health Serv Res Policy Vol 10 No 1 January 2005
Realist synthesis Theory-driven synthesis based on purposive Lacks transparency regarding choice of . Can deal with very diverse evidence types
sampling of primary studies. Context data evidence to synthesise . Explicit orientation towards testing of theory
retained. Basic theory is refined concerning
applicability in context.
Meta-study General framework for and guidance on Subdivisions: meta-data synthesis, drawing . Currently proposed only for qualitative . Can potentially deal with diverse
technique, including question formulation and on meta-ethnographic approach; meta- research evidence types
selection of primary studies method synthesis, comparing impact of . Depends on the rigour of the underlying . Usefully draws attention to different
methods on findings; meta-theory synthesis, methods purposes and outcomes of synthesis
exploring theoretical frameworks . Explicitly oriented towards production of
mid-range theory
Miles and Huberman Evidence from each primary study first No guidance on sampling of primary studies . Highly systematic method
summarised and coded under broad thematic . Potentially allows inclusion of diverse
headings. Evidence then aggregated and evidence types
summarised within theme across studies, . Could be used for theory-building
Review article
Case survey Each primary study treated as a ‘case’. Study . Applicable to outcomes, but less . Can incorporate diverse evidence types
findings and attributes extracted using adequate for process . Can cope with large numbers of primary
closed-form questions, for reliability. Survey . Lacks sensitivity to interpretive aspects studies
analysis methods used on extracted data. of evidence . Could be used for theory-building
Qualitative Boolean analysis of necessary and sufficient Focused on causality determination, not . Transparent
comparative analysis conditions for particular outcomes to be interpretive aspects of qualitative data . Systematic
observed, based on presence/absence of . Can incorporate diverse forms of
independent variables and outcomes in each evidence
primary study . Allows competing explanations to be
explored and retained and permits theories
about causality
. Does not require as many cases as case
survey method
Bayesian meta- Quantified beliefs about effects of variables Reversal of roles of qualitative and quantitative Conceptually simple, but may be technically . Impact of analysts’ prior beliefs can be
analysis from qualitative studies formally combined evidence possible in principle complex to implement (and thus lose explicitly explored
(through Bayes paradigm) with evidence from transparency) . Can incorporate diverse forms of evidence
quantitative studies . Could be used for theory-building
J Health Serv Res Policy Vol 10 No 1 January 2005
Review article
53b