Supporting literature reviews:

Materials for supervisors to use with students

This tool is based on research and scholarship by:
Dr Christine Bruce, Associate Professor and Director, Teaching and Learning,
Faculty of Information Technology, Queensland University of Technology
Dr Peggy Nightingale assisted in converting the published materials into an online resource.

Contents
Helping students understand how literature reviews are judged
Handout 1: Marking Guidelines for Literature Reviews
Handout 2: Characteristics of Strong and Weak Literature Reviews
Starting with an annotated bibliography
Handout 3: Annotated Bibliographies the basics
Handout 4: Annotated Bibliographies being critical
Introducing a reflective model for reviewing the literature
Handout 5: Reflective Model for Reviewing the Literature
Helping students shape the literature review
Handout 6: Finding form and structure for the literature review

Helping students understand how literature reviews are judged

The questions in Handout 1, Marking Guidelines for Literature Reviews, were framed by
Christine Bruce to assist in giving feedback to research candidates on their draft literature reviews
early in their candidacy. These questions may help students understand the criteria that are
commonly applied to judging the quality of literature review chapters of theses.
You should take some time with your students to discuss the question: what is a literature review?
The answer in the handout gives you a place to start, but each discipline and, indeed, each project
will interpret and expand the general statements in particular ways. This is where your students
need you, your experience in your field and your understanding of their project.
Handout 2 lists some of the characteristics of good and poor literature reviews. Again you should
discuss these qualities with your students, stressing those matters which are especially important
in your field and their work.

Handout 1: Marking Guidelines for Literature Reviews

What is a literature review?
A literature review is a well-organised discussion of published (and sometimes unpublished)
research and scholarship relevant to an area of study. The literature review demonstrates that you
have acquired a full professional grasp of your research area. A good literature review also puts
forward an argument for the research you intend to do; it points the way towards your own work.
Your literature review will continue to develop as you progress in your research. Eventually it will
probably form a chapter in your thesis. Look at examples of finished theses in your field and study
how they present and deal with the literature.

Format

Are the aims/goals of the literature review included in the introduction? Is the research
focus made clear?

How well is the literature review structured? Are the headings/sub-headings used
appropriately to help the reader navigate the subject area?

Is there a conclusion to the review? Are main points summarised and future research
directions suggested?

Content

Is the nature of the material reviewed appropriate?

Are important authors/research papers included in the review?

Is the coverage of the literature sufficient for the moment? Are there any key items/areas
overlooked?

Have you synthesised the work providing coherent representation of the field? Or is the
review pointing to previous work in a disconnected manner?

Do you take a critical approach to the literature (identifying important contributions,

comparisons, strengths and weaknesses)?

Does the review show appropriate depth of thinking for the degree being undertaken?

Presentation and Expression

Is the quality of English expression and punctuation of an appropriate standard?

Has your chosen referencing system been used correctly?

Handout 2: Characteristics of Strong and Weak Literature

Reviews

Does

your

literature review

demonstrate any

of

the

following

deficiencies?
Areas in which literature reviews are commonly deficient (Afolabi, 1992) include:

exclusion of landmark studies

emphasis on outdated material

adopting a parochial perspective

not being critical

not discriminating between relevant and irrelevant material

lacking synthesis

Hansford and Maxwell (1993) cite the literature review as the second most frequent chapter of the
thesis to be criticised by examiners. They cite these common criticisms:

failure to use recent literature

lack of critical assessment

not relating the literature review to the research questions or hypotheses

incorrectly interpreting sources

What improvements can you make to ensure the following criteria are
met?

includes current literature as well as material of historical interest

aims of the review are clearly stated

the range of resources from which literature has been gathered are identified

a breadth of knowledge of the area is demonstrated

a strong argument is developed justifying the nature of the recommended line of research

organisation and structure of the review are made clear to reader

the review should be interesting to read

References
Afolabi, M. 1992. The review of related literature in research. International Journal of Information and Library Research 4(2), 5966.
Hansford, B. and Maxwell, T. 1993. A masters degree program: structural components and examiners comments. Higher
Education Research and Development, 12 (2), 171-88.

Starting with an annotated bibliography

Rather than just sending a new student to the library to "do your literature review" a task which
many students find overwhelming, you might find it productive to ask your student to begin by
preparing an annotated bibliography. You can set weekly or fortnightly targets and monitor
progress, helping your students build the bibliography they need for their projects.
This task will require them to think seriously about what sort of resources they currently need
most and where to find them. And it requires them to engage in critical reading. But it does not
require them to deal with problems such as finding a way to shape and structure the literature
review. Thus, they are starting with a task that has somewhat more manageable dimensions and
they are less likely to procrastinate about getting started.
Handout 3 will help you make sure students understand what you are asking of them. You should
also discuss the following questions:

What do you know already/what materials do you have that will help you?

What else do you need to know? What other materials do you need?

How can you find out about/obtain the above?

What can you do this week to get started?

Most students will be using software such as ProCite or EndNote which will help them manage
their bibliography. You should check to be sure that they are familiar with appropriate software or,
if for some reason they cannot use a technological solution, you should help them develop a card
file system.
Preparing an annotated bibliography requires students to engage in critical thinking. Handout 4
contains prompts which should help your students approach materials systematically and
thoughtfully. Again we encourage supervisors to discuss these handouts with students. For
instance, if you begin to discuss how you determine the reliability of a source, you might explain
the reviewing practices common in major journals in your field which help ensure (but cannot
guarantee) that material published in these journals is reliable. Then questions about materials on
the internet might lead to discussion of the reviewing practices of key sites.

Handout 3: Annotated Bibliographies the basics

The difference between a collection of abstracts and an annotated
bibliography
An abstract is an item that stands alone to represent some document. It does not refer to other
documents, that is, it does not contain references or refer to other abstracts.
An annotated bibliography is a collection of abstracts that do refer to each other or interconnect
in various ways. The annotated bibliography represents an interrelated body of work and states
what the relationships are. The individual annotations may not stand alone, but the bibliography as
a whole will stand alone.

Referencing
An annotated bibliography should show consistency in the referencing style used. You will find it
useful to begin now to use the referencing style you will use for your thesis/project report. Check
with your research supervisor if you are not sure which style to use; and obtain a copy of the
relevant chapter from the appropriate style manual (e.g. APA, AGPS) or journal notes for
contributors.

Suggestions for what to include in the annotation for each paper:

1. The purpose of the paper and its findings including any hypotheses
2. The theoretical position of the author may be explicit or implicit [e.g. world
views/theoretical position]
3. The type of research [e.g. scientific, naturalistic, interpretive]
4. The research method [e.g. experiment, survey, etc.]
5. The usefulness/validity of the results
6. What further research areas are suggested
7. Contribution to your field of research
8. Relevance to your specific research
9. Relationship to other papers in the annotated bibliography
In summary, the annotations should summarise the content of the papers, indicate the relevance to
your research, and interconnect between each other. That is, if one paper builds on the work done
in another, or confirms or refutes the results of some other research you should say so.

Handout 4: Annotated Bibliographies being critical

What questions should I ask when critiquing an article?

Who is the author?

What is the motive for writing/doing the research?
To what audience is the author writing?
Does the author have a bias?
What research approach or data-gathering method was used?
What conclusions does the author arrive at?
Does the author satisfactorily justify the conclusions?
How does the study compare with similar studies?
(Engeldinger, 1988)

What does thinking critically mean? Consider how you would do each of the following:

Distinguishing between verifiable facts and value claims

Determining the reliability of a source
Determining the factual accuracy of a statement
Distinguishing relevant from irrelevant information, claims or reasons
Detecting bias
Identifying unstated assumptions
Identifying ambiguous claims or arguments
Recognising logical inconsistencies
Distinguishing between warranted and unwarranted claims
Determining the strengths of an argument
(Beyer, 1985)

How can I approach internet materials critically?

Is your resource an example of vanity publishing or has it been through a rigorous review process?
What evidence is there to suggest that the resource is of high quality?
Who are the authors? What are their credentials?
How current is the resource? Can you establish when it was last updated?
How complete is the content of the resource? How unique is the content?
How easily accessible is the resource?
Is the resource stable? Is it likely to remain stable?
How well is the resource regarded? Do other people refer to it regularly? Can you identify how often it has been
used?
Is the resource organised in some way? Is there a contents page, an index?
Is there an abstract or other summary to communicate the nature of the
document?
(Tillman, 1996)

References
Beyer, B. 1985. Critical thinking: what is it? Social Education, 49(4), 270-76.
Engeldinger, E.A. 1988. Bibliographic instruction and critical thinking: the contribution of the annotated bibliography. RQ 28(2) 195202.
Tillman, Hope. 1996. Evaluating quality on the net. URL: http://www.tiac.net/users/hope/findqual.html

Introducing a reflective model for reviewing the literature

Handout 5 introduces a reflective model for reviewing the literature. You will need to explain this
model to your students and discuss with them how they may use it to enhance all aspects of
working on their literature review. The following, taken from Christine Bruces article "From
Neophyte to Expert: Counting on Reflection to Facilitate Complex Conceptions of the Literature
Review," explains the model.
The model is cyclical and is made up of two parts: the reflective cycle of planning, acting,
recording and reflecting, and the characteristics of the person, which influence the way in which
he or she engages in the cycle.
These characteristics, the person's information style and conceptions of the literature review
process, will also develop as he or she works with the cycle. Each part of the model is explained
below.

Characteristics of the person

Christine Bruce has described six ways in which beginning researchers conceive of a literature
review. The six ways students see literature reviews are:
1. As a list
2. As a search
3. As a survey
4. As a vehicle for learning
5. As a research facilitator
6. As a report
Each of these conceptions may be useful at different stages of developing a literature review.
What is not useful is for a student to hold only a limited conception of a literature review, for
instance as a list, and to fail to recognise that the literature review may help them identify or
clarify their research question, and that the review is itself a report needing shape, structure and
sophisticated organisation.
It is important to note here that we should not think of individual students as having particular
conceptions. Rather they use one or more of the conceptions of which they are aware at a
particular time. It is the role of the reflective model to help them expand their repertoire of
conceptions and encourage them to think about their literature review as something other than an
exercise in literature searching.
The way in which a student will engage in the reflective process will also be influenced by his or
her information style. Judith Palmer (1991) describes five different ways in which scientists seek
information; the differences in style lead her to label the different types of people as information
overlords, information entrepreneurs, information hunters, information plodders and information
derelicts. New researchers can be asked to consider their own information style by answering the
following questions (adapted from Judith Palmer [1991] and included in Handout 5):

How active are you in your efforts to find information?

Are you generally relaxed or anxious about the undersupply or oversupply of your
information needs?

To what extent do you feel the need to be in control of your information environment?

How broadly do you search electronic- and print-based information access tools?

How extensive are your formal information networks, i.e. personal contacts, conference
attendance etc.?

The cycle of planning, acting, recording and reflecting

Because people do have different information styles, or heuristics, they will enter the cycle at
different points. Whether they enter the cycle by devising an elegant plan for a literature search,
browsing the library shelves or surfing the Internet, they should continue to systematically work
through the cycles as many times as necessary until the literature review is complete.
Planning the literature review will involve determining the goals of the literature review, deciding
what questions the literature review will answer, as well as determining what literature searching
should be done to gather the database of journal articles, reports, conference papers, books and
other material required. Writing goals for the literature review and specifying questions that the
review will answer will guide the researcher when he or she is evaluating material found. Due to
the cyclical nature of the process, however, these goals and questions will change during the life of
the review. Quite early in the process it would also be useful to create a concept map for the
review chapter or create a list of possible headings and subheadings. Thinking about the overall
purpose of the literature review in this way will also supply guidelines for literature searching.
The student should begin to identify key discipline areas, important key words, authors,
organisations and seminal works in the field, and plan to create a system that will keep notes and
citations organised. In later cycles, planning may involve restructuring the literature review as a
result of reflection.
Acting on plans made is the second stage of the process. This means that the student should do
whatever is required as a result of the planning stage. This might involve consulting a librarian,
conducting a search of the citation indexes to trace a seminal reference, or collecting and analysing
papers about the relevant research methodology. Acting on plans includes reading and writing
about the literature collected on the basis of the preliminary structure of the literature review set up
in the previous stage. It is often easy for students to spend much of their early `activity' in relation
to the literature review chasing papers, leaving the reading and writing until a pile of documents
have accumulated. Writing in the early cycles will help form their thinking about the content and
structure of the literature review, making it possible to influence directions set for subsequent
literature searching.
Designing a system for maintaining records of bibliographic references and notes is a critical
activity which can require the students to think about the overall structure of his or her literature
review. Keywords or a system of notation can be designed to indicate that part of the thesis to
which particular papers or notes will contribute.
Recording, or maintaining a diary, is a laborious but critical part of this cycle. It is really a part of
the process that should happen in conjunction with making plans, taking action and reflecting.
Because we so often do things spontaneously, particularly in the `action' phase, it is useful to
regularly ask whether details of what has occurred have been recorded. For example, when
browsing indexes for material about teaching art criticism, the student might encounter useful
references about the problem of validity in qualitative research. This should be recorded, if only as
an aide-de-memoir for the future. Similarly, ideas about what points should be highlighted in the
literature review and similar insights that occur as part of the reflection process should be recorded.
Students often find it difficult to get in the habit of finding time to keep such records/diaries of
their thinking. If they do develop the habit, they invariably find it extremely useful.

Reflecting is that part of the cycle where the student reviews what he or she has done in relation to
the literature review and analyses the strengths and weaknesses of that work. It is a precursor to
generating ideas for further work that is creating a new set of plans. The conceptions of a literature
review discussed earlier provide a framework within which students could reflect about their
progress. Each of the following questions (included in Handout 5) has been designed to focus
thinking around one of the conceptions. Use of the questions (Bruce, 1994) from the earliest cycles
will encourage students to think about the literature review in the full range of ways; it is likely
however that more emphasis will be placed on certain questions at different stages of the research
process:

What is the present state of my list of references? Is it up to date in my areas of present

interest? Is it adequate?

What literature searching have I done this fortnight? Are there any new areas that I have
become interested in which I may need to search on?

What have I read recently? Have I found time to read recently?

What have I learned from the literature this fortnight? Have I changed, in any way, my
understanding of the area in which I am working?

Is what I have read going to influence my research in any way? Has it given me any ideas
that I need to consider and incorporate?

Have I been writing about what I have read? Do I need to reconsider how what I have been
reading fits into my research?

Handout 5: Reflective Model for Reviewing the Literature

(Bruce, C.S. 1994. Research students' early experiences of the dissertation literature review. Studies in
Higher Education, 19(2), 217-229.)

What is your information style?

From Palmer, J. 1991. Scientists and information: using cluster analysis to identify information style.
Journal of Documentation, 47 (2), 105-129.

How active are you in your efforts to find information?

Are you generally relaxed or anxious about the undersupply or oversupply of your information
needs?

To what extent do you feel the need to be in control of your information environment?

How broadly do you search electronic- and print-based information access tools?

How extensive are your formal information networks, i.e. personal contacts, conference
attendance. etc?

Consider these questions frequently as you work on your literature

review:

What is the present state of my list of references? Is it up to date in my areas of present

interest? Is it adequate?

What literature searching have I done this fortnight? Are there any new areas that I have
become interested in which I may need to search on?

What have I read recently? Have I found time to read recently?

What have I learned from the literature this fortnight? Have I changed, in any way, my
understanding of the area in which I am working?

Is what I have read going to influence my research in any way? Has it given me any ideas
that I need to consider and incorporate?

Have I been writing about what I have read? Do I need to reconsider how what I have been
reading fits into my research?

The literature review reflective cycle

Planning

What is a literature review?

Why do we do a literature review?

What problems are faced by research students doing literature reviews?

What are the different ways in which research students conceive of literature reviews?

What are the varying roles of thinking, searching, reading and writing in the process?

Action

How can you start the writing process?

(Focus on one or two strategies, work on them and receive peer feedback.)

How do you decide what to include in the literature review?

Recording

How can I use a journal to help me with writing my literature review?

Reflection

What are the features of a good literature review?

What are the weaknesses associated with poor literature reviews?

How does your literature review shape up against Coopers taxonomy of literature reviews?
(See Handout 6)

How does your literature review need to be modified to meet the above criteria?

Planning

What steps will you take to progress your literature review in the next weeks?

Helping students shape the literature review

We have seen that the criteria for judging literature reviews include the quality of good
organisation. Examiners are, after all, readers like anyone else, and value writing that guides
them logically through complex ideas. Handout 6 includes some prompts for your discussions
with your students about how to find an appropriate form for the literature review and how to
design its structure.

Handout 6: Finding form and structure for the literature review

What decisions do I need to make about the form of my literature
review?
Use Coopers taxonomy of literature reviews to help you make decisions about the content and
arrangement of your review.

Focus What will be the focal elements of your review? These may include research
methods, outcomes, theories, etc.

Goals What will be the goals of your review? There may be more than one goal such as
synthesising previous research, identifying key issues, and critically analysing previous
research.

Perspective What perspective will you adopt? Will you assume the role of honest broker
or that of an advocate arguing for a particular position?

Coverage What extent of the literature will you cover? Pivotal works only, works
representing particular groupings of the literature, or will you make an attempt at
exhaustive coverage?

Organisation Will you arrange your review conceptually, historically or

methodologically? Perhaps you will need to use more than one organisational technique.

How can I design a structure for my literature review?

Use some of the following prompts to help you create a structure for your literature review:

Important questions that my literature review will address are:

Important contrasts I wish to demonstrate are:

My literature review will argue the following main points:

The main goal of my literature review is to:

Some headings and sub-headings for my review could be:

References
Cooper, Harris M. 1989. Integrating Research: A Guide for Literature Reviews. Newbury Park: Sage.
Christine Bruces Developing Students Library Research Skills (HERDSA Green Guide No 13, 1992) is a useful resource for
dealing with matters such as these.