Quality Guide en

Download as doc, pdf, or txt
Download as doc, pdf, or txt
You are on page 1of 106

A GUIDE FOR THE EVALUATION AND DESIGN OF QUALITY

LANGUAGE LEARNING AND TEACHING PROGRAMMES AND


MATERIALS

A project related to the


WHITE PAPER
”Teaching and learning. Towards the learning society”
Objective 4, 1st support measure

Guide21
Produced by
Jean-Claude Lasnier, A.C.F.C.I./A.G.E.R.C.E.L., France. agercel@porte-oceane.com
Petra Morfeld, International Certificate Conference, Germany. ICC_europe@compuserve.com
Brian North, Eurocentres Foundation, Great Britain. bnorth@eurocentres.com
Carlo Serra Borneto, University of Rome, Italy. serra@uniroma3.it
Preben Späth, PS Sprog & Edb, Denmark. p.spaeth@mail1.stofanet.dk

10/09/99

________________________________________________________________________________

Co-funded by DG XXII, EUROPEAN COMMISSION, Brussels


2

Contents

A Aims of the Guide

B Quality in Modern Language Teaching and Learning


 Introduction
 Development of a concept of quality
 The principles
 The stages
 The monitoring cycle
 Putting quality into practice
 Profiles
 Printouts

C The Principles

D The Stages

E Putting Quality into Practice


1. Questions and Examples for Stage 1 (Design)
2. Questions and Examples for Stage 2 (Implementation)
3. Questions and Examples for Stage 3 (Outcomes)

F References
3

Preface
The idea to develop a guide that helps different target groups to identify aspects of quality with
regard to modern language teaching and learning is directly connected with one of the objectives
of the ”White Paper” Teaching and learning. Towards the learning society
(Brussels/Luxembourg 1996). The fourth general objective in the White Paper is ”Proficiency in
three Community languages”, in relation to which the European Commission proposes the
”introduction of assessment systems (including elaboration of quality indicators) and quality
guarantee systems, covering the methods and materials used to teach Community
languages”(page 68).
This guide is divided into five sections. The aims of the guide are introduced in Section A, while
Section B presents the background of our approach to quality and the structure underlying the
guide. Section C then introduces a set of general principles of quality. A presentation of the
three stages of quality management in the learning/teaching process is undertaken in Section
D. In the main body of the Guide, Section E, the general principles of quality are applied to an
analysis of these three stages. Section E1 deals with Design, Section E2 with Implementation
and Section E3 with Outcomes. Each of these stages is analysed into components (called
"sections") and sub-components (called "elements"). The general principles of quality are used to
formulate and cross-reference indicators relevant to the components of each stage of the
teaching/learning process.
A version of this guide is published as a CD-ROM. This CD-ROM is an interactive tool which
tracks the answers given to the questions in Section E and so enables the user to profile his/her
language learning provision at the three stages of the learning process: before (Design), during
(Implementation), and after (Outcomes). These profiles graphically portray strengths and
weaknesses and offer feed back in quality planning and quality control, which is available as a
print out. An abstract profile in relation to the general principles of quality introduced in Section
C is also offered.

A. Aims of the guide


Three main aims have been identified for the realisation of this guide:

1. A general aim to develop a concept of quality in relation to modern language learning


and teaching, thereby proposing quality as a key criterion for consideration in the design and
evaluation of programmes and materials.

By defining general principles of quality on the basis of different approaches to quality an


attempt is made, firstly, to make concepts explicit which have influenced the quality debate in
the area of modern language learning and teaching for some time, and secondly, to demonstrate
how quality can be related to all aspects of the teaching and learning process.
2.The realisation of the guide as an ”Exemplary instrument”, conceived as a reference or
stimulus for publishing companies, materials and/or programme designers, teachers and trainers,
course decision-makers etc. when planning their work.

For instance, in Section E1 (Design) the main aspects which have to be taken into account when
starting to write a course book or to set up a new language programme are described on the basis
of quality principles. Sets of questions are formulated which can be used as a checklist to make
sure that none of the relevant aspects have been overlooked or to find out to which extent the
different principles of quality have been taken into account.
4

3.A practical aim to provide an evaluation tool for teachers/educationalists and project managers to use
in assessing programmes and materials and increase motivation in giving a focus to efforts to increase
quality in the learning/teaching process.

The guide can also be employed as an instrument for the evaluation of existing programmes or materials,
e.g. by teachers who have to decide if a course book or a CD-Rom meets certain standards or by
programme designers or educationalists who want to know if a course has been successful in terms of
results and satisfaction on the part of their clients or stakeholders.

B. Quality in modern language teaching and learning

Introduction
In recent years the concept of and the concern for quality has shifted from the production branch more and
more into service and education, requiring differentiation with respect to the sectors involved. For instance,
quality in service is characterised by limited measurability, context sensitivity, heterogeneity, relevance of
the human factor, the affective dimension, strong relevance of the client’s needs etc. (cf. Negro 1996). The
concept of quality in modern language teaching and learning [MLTL] cannot rely on a consistent
tradition except for the more specific issue of course evaluation (cf. Alderson/Beretta 1992, Lynch 1996).
Only in recent years has it been applied in practical learning contexts, for example in EAQUALS (1995)
and O.E.C.D. projects (OECD 1992). EAQUALS [European Association for Quality Language Services] is
an attempt to develop a widely recognised set of standards with regard to the services necessary to organise
and run language courses. Comparable schemes are EXCELLANGUES or initiatives from other providers
of language training such as Eurocentres, International House, Bell School etc. Quality is also advocated in
a number of works produced for textbook writers in order to help them to design better materials (e.g.
Goethe-Institut 1990; Kast/Neuner 1994; Hopkins 1996).

Generally speaking, quality in modern language teaching and learning has to face a very specific issue: the
”cognitive factor”. Language learning is a process which affects and is affected by the cultural-cognitive
structure of the subjects involved. This has several implications on the concept of quality: psychological
and affective components should be considered a major issue. Such variables - for example needs,
motivation, social conditions, cultural barriers, social learning environment, cognitive styles and
preferences, subjective reactions etc. - are by their nature context bound. The traditional educational view
is that these variables cannot be assessed - or can be assessed only in a very limited sense - using
quantitative measurements. In this field, the ratio of ”soft”, qualitative (as opposed to ”hard” quantitative)
factors and data is higher than in any other kind of quality assessment context; higher even than in relation
to general education, which affects the cognitive structure in a differentiated, often less dramatic way.

The development of the concept


As mentioned above, the concept of quality was in some sense "imported" into the educational sector from
the production branch. Originally, quality evoked mainly the idea of exceeding the normal size of an offer
("the more the better"), being equated with abundance and even luxury. Quality was not associated with
ordinary goods. In a second phase, quality was considered part of the product, i.e. the average product
should correspond to specific standards. These standards were externally defined by special boards (like
ISO in Vienna), which set codified requirements to be satisfied in order to achieve a recognised level of
quality. The issue of quality was also internally promoted by the producers themselves, in that they
determined their own minimum standards for the goods to meet. On the basis of objective criteria the
extent to which a product/service met these standards became the issue of a process of inspection and
evaluation.
In the following phase, the focus shifted from the product/service as such to the functions it fulfils for
different users in relation to a particular context: "Quality is not the best in absolute terms, it is the better
one under certain conditions dictated by the client" (Feigenbaum 1983). The needs of the client have to be
5

identified and taken into account before quality can be achieved. The extent to which these needs are met
serves as a measure of the satisfaction of the client. Accordingly, the product/service will be constantly
adjusted and improved in order to reach an optimal matching of the users’ needs and the attributes of the
product/service itself.
The issue of needs orientation and constant improvement was taken up and further developed into the
"Total Quality Management" (TQM) approach. The TQM philosophy implies that the satisfaction of the
clients’ needs is the main concern of all parties involved in the production or delivery of a product/service.
Moreover, a fundamental feature of TQM consists in carefully monitoring every step in the process of
generating goods (production cycle) with relation to defined quality indicators. Such indicators
operationalise general principles of quality such as efficiency or transparency in the context of the
subprocesses (stages) concerned.
The challenge in transferring such a notion to MLTL or any other educational field is to apply principles
which have been successful in the original contexts in a differentiated rather than mechanistic manner in
order to put humanistic, interpersonal, cognitive, and affective factors at the centre of concern. On the
other hand, educational and - more specifically - MLTL theories have developed a range of general
propositions and methodological suggestions which themselves can be interpreted as principles of quality.
Therefore, in order to develop an independent concept of quality for MLTL, the findings of both
approaches have been taken into consideration and adjusted to form the theoretical background of the
guide.

The principles
The theoretical concept of the guide is based upon nine general principles of quality and their relationship
to the language learning and teaching process. The principles are broad superordinate notions which have
been developed by relating widely used concepts of quality to current MLTL theories and adjusting them
for our purposes. Each principle is divided into several sub-principles which represent its different
dimensions. The principle of Transparency, for example, consists of the four sub-principles Clarity of
Aims, Clarity about Achievement, Clarity of Presentation, and Clarity of Rationale. The demand for
comprehensive and comprehensible information about the objectives of a programme/material, about the
progress one has made and about the methods which have been applied to achieve these objectives has not
only become an objective in quality programmes of language schools etc. but is also implicit in educational
concepts like learner autonomy or reflective learning. Definitions for the nine principles are given in the
next chapter.

The stages
As is the case in comparable quality assurance schemes (e.g. ISO norms, TQM etc.) this guide
refers to three distinct stages of the process of provision: Design (before), Implementation (during),
Outcomes (afterwards). The characteristics and functions of these three stages can be summarised
as follows:

 Design
The process of analysing and planning to provide for needs in context and of presenting the
resultant content and activities in an attractive and effective manner;
 Implementation
The process of providing a positive learning environment and of adapting planned provision
flexibly and appropriately to circumstances (translating design into action);
 Outcomes
The process of assessing gains in context in relation to resources and of ensuring satisfaction on
the part of the actors involved.

Each stage can be divided into smaller sections covering the different aspects of the teaching and learning
process. These sections are then themselves sub-divided further into "elements". This conceptual division
of the process is reflected in Section E which is divided into three parts reflecting the three stages.
6

The monitoring cycle


In general, to achieve quality means meeting and/or exceeding the needs and expectations of
clients. Good teachers naturally consider learner needs in an ongoing process. However, a target
needs analysis used to inform all aspects of Design is a prerequisite for a quality programme. In all
three stages, quality of provision is monitored and evaluated in relation to the general principles of
quality by using questions and indicators. The product of the monitoring process is a Profile of
Quality Achievement. This profile may provide useful suggestions which can be fed back into the
Design stage in order to further improve the quality of the language learning opportunities.

Needs analysis

Design Implementation Outcomes

Monitoring by the Profile of quality


principles of quality achievement

Figure 1: The Quality Monitoring Cycle

Putting Quality into Practice


While Sections A to D explain the objectives and concept of the guide, Section E constitutes the diagnostic
tool which is operationalised on the CD ROM. It helps the user to develop a concept of quality with regard
to MLTL and to determine how far quality is represented in particular programme/materials. For these
purposes, hierarchical sets of questions have been developed relating the quality principles to each aspect
of the learning and teaching process. At the top level, lead questions ask if certain quality principles have
been taken into account when planning, implementing or evaluating the programmes/materials. At the
second level, the questions have the function of quality indicators asking for the measures which were
used to implement these quality principles. The quality indicators are supplemented by a number of
suggestions demonstrating how the quality indicators can be put into practice. These suggestions are not
intended to be ready-made solutions but to support the users in gathering ideas and developing the most
suitable way of putting a particular quality principle into practice in their particular context.
Lead question: What has been done to ensure that the approach to vocabulary work is clear and valid?
Indicator:  Is vocabulary practised or activated in a systematic and varied way?
Suggestion:  Vocabulary exercises consistently connected with the other activities
 Vocabulary for specific purposes explained or introduced in a transparent way
 Regular recycling of vocabulary

Table 1. Hierarchy of questions

Profiles
Every indicator is connected to one or more related sub-principles. By ticking off the indicators the user
scores points with regard to the principle(s) in question. The final score for each principle is then calculated
7

with regard to the number of questions referring to it and the responses given by the users. The resulting
percentage is shown for each principle and gives an overview of the extent to which the principles are
represented in the particular programme/materials. The profile in relation to principles of quality offered by
the CD ROM is illustrated in Figure 2.

Figure 2: A Principles Profile


Principles are represented in the form of a bar chart. The length and colour of any particular bar
reflects the extent to which the principle or sub-principle concerned is evaluated positively in the
assessment. The length of the bar reflects the extent to which the principle is relevant to the
questions which have been answered thus far. The colour of the bar indicates the degree
(percentage score) to which the principle has been successfully addressed according to the
ratings given. A short white bar indicates that the principle was not relevant to the areas
addressed in the session concerned.
In addition to the bar chart profiling principles, which is built up cumulatively during an
evaluation session, the CD ROM also offers a profile of ratings per stage in the form of a pie
chart of concentric circles. The stage profiles (one for each stage) do not show the score in
relation to the principles and sub-principles as in the bar chart. They offer a transparent record of
the way in which different aspects related to the three stages have been rated. Again, percentage
scores for each segment of the chart (each section and element) are expressed by differences in
colour. This enables the user to identify at a glance those areas of the program which were rated
highly, and those which might benefit from closer inspection. Such a stage profile is illustrated
in Figure 3.
8

Figure 3: A Stage Profile

Print outs
The user is able to print the colour profiles and is also offered the option of two kinds of print
out. One prints to screen or printer a record of the low scoring indicators associated with
principles and sub-principles which were rated relatively low (less than 50%) in the evaluation.
The other print out reports low scoring indicators by segments (sections and elements) of the
stage profiles.
These print outs offer the user a diagnostic checklist of points which may be worth investigating
in order to improve the overall quality and effectiveness of the programme or materials.

Summary
Figure 4 below summarises the structure of the guide as discussed so far.
9

Figure 4. Structure of the guide


10

C. General principles of quality

C1. Rationale
”Definitions of quality vary and, to some extent, reflect different perspectives of individual and society”
(Green: 17).
Given this caveat, we have tried to develop a concept of quality based on a set of general principles,
which should subsume ”different perspectives” and at the same time reflect common tendencies of our time
both in economic and in educational (especially Modern Language Teaching and Learning = MLTL)
theory.
In our approach, general principles of quality are very broad superordinate notions which represent a
distillation of the experience of the authors as well as of concepts commonly used in economics and
MLTL. They can be applied to any step of the teaching/learning process and provide an opportunity for
reflection and re-thinking the process itself in order to provide an improved design, as described above.
They have been revised and elaborated in the light of feedback during the consultation stage of the project.
A list of nine general principles of quality is presented below. They will be defined together with their
corresponding sub-principles which more specifically illustrate the different dimensions implied.
General principles belong to the ’upper’ theoretical layer of the model illustrated in Figure 4. Accordingly,
on the one hand they reflect theoretical premises (see below). On the other hand, they are conceptually
linked to the next layers in so far as the corresponding sub-principles are related to each aspect of the
learning and teaching process and to the quality indicators. Therefore, the sub-principles are very important
to ensure the consistency and coherence of the whole model.
The complete list of principles identified is the following:
1. Relevance
2. Transparency
3. Reliability
4. Attractiveness
5. Flexibility
6. Generativeness
7. Participation
8. Efficiency
9. Socialisation

These principles are discussed and defined in the next section.

C2. Definitions of principles and sub-principles


In this section each principle is defined by a brief description of its characteristics and of the characteristics
of the corresponding sub-principles. This is followed by a short illustration of the relevant theoretical
background.
11

C2.1 Relevance
Relevance refers to the appropriateness of an issue in terms of its accordance with the needs,
objectives and purposes of the users (learners, teachers, stakeholders etc.). Relevance can apply
to content as well as methodology.
The principle of relevance comprises the following sub-principles:
 Learner Centredness
Programmes and materials take account of the personal characteristics of the learners and/or
their professional objectives (referring to: content, situations, language, skills). This requires a
good knowledge of the learners and of their needs and differentiation in the selection and
organisation of programmes and materials to meet them.
 accountability
Programmes and materials meet social and political needs (referring to general educational aims).
Accountability is also highly relevant in terms of materials and programmes for more specific target groups
such as commercial clients and international exchange programmes.
 appropriateness
Programmes and materials suit the learning/teaching context. Approaches, materials, examples and
activities take into account the social, cultural and educational characteristics of the learner (i.e. age, level,
customs).
Theoretical background: Relevance is a function of needs. The adjustment of a product or service to the
needs of the customers/clients is an essential concept of the TQM philosophy, especially in connection with
quality in service (Negro 1996). Needs can be implicit, i.e. so obvious that they may not even be
mentioned; they can be explicit (for example as a result of an audit); and finally they can be latent, i.e. the
client might not envisage particular aspects overtly, but would appreciate them if offered. The orientation
towards the needs of the learners in MLTL theory has a long tradition in E.U policy (see Richterich 1973;
Richterich/Chanterel 1977; cf. also Tarone /Yule 1989) and refers to individual as well as to cultural and
socio-political issues (cf. Reeves,/Wright 1996).

C2.2 Transparency
Transparency, as applied to programmes and materials, describes those features which
(cognitively) contribute to facilitating access and use.
The principle of transparency comprises the following sub-principles:
 clarity of aims
Programmes and materials clearly indicate the learning objectives. In other words the user is
given the opportunity to determine whether the material or programme or learning
environment actually suits his or her needs. Furthermore, the skills taught and the target level
should be clearly specified.
 clarity about achievement
Programmes and materials help learners become conscious of areas of success and of
weakness and inform them about progress made in relation to the learning objectives.
 clarity of presentation
Programmes and materials present a clear and logical structure with appropriate,
comprehensible instructions and layout. Some relevant issues are whether progression is
implicitly or explicitly stated, and whether tasks, exercises and simulations clearly indicate
what the learners should do.
 clarity of rationale
Programmes and materials provide the teacher with a methodological rationale, explaining
12

the language learning approach(es) used. For teachers it is important that the assumptions and
aims of the methodology are clearly explained.
Theoretical background: Transparency is a central issue in the client-led approach of TQM: the client
should be best informed about the characteristics of a product or service, in order to be able to make
choices, compare performances, and complain in case of unfulfilled promises. Similarly, transparency is a
key issue with regard to the organisation of language learning courses at private or non-private language
schools (cf. in particular EAQUALS). Transparency is also topical in approaches which aim at autonomous
learning (cf. Dickinson 1987; Holec 1988;) and in all forms of reflective teaching and learning (cf.
Richards/Lockhart 1994).

C2.3 Reliability
Reliability refers to the internal consistency of programmes and materials and the dependability of
contents and methods, thereby facilitating the implementation of the teaching/learning process.
The principle of reliability comprises the following sub-principles:
 consistency
Programmes and materials show consistency between stated objectives and actual realisation.
A course purporting to provide the development of spoken language should, for example,
provide extensive opportunities for free fluency practice, and material for practising listening
to native-speakers interacting.
 internal coherence
Programmes and materials are well constructed with a consistent inter-relationship between
their parts. Presentation, explanation, practice, exercise, feedback etc. should be coherently
inter-related.
 methodological integrity
Programmes and materials reflect validated methodological premises in the selection of
activities, exercises etc. If a functional approach is chosen, then the situations and exercises
offered should be in line with that methodology and not reflective of a grammar-translation
approach with new labelling.
 linguistic integrity
Programmes and materials present models of good, current language usage with correct and
socio-linguistically appropriate examples. The concept of authenticity is important here, as is
the need for idiomatically and phonetically correct language based on native speaker usage.
 textual integrity
Programmes and materials respect the integrity and authenticity of the relevant genre in the
creation, adoption, and adaptation of texts. This does not mean that only authentic and
unedited texts can be presented, rather it implies that texts should be written, edited or
simplified in a way which retains key linguistic features.
 factual integrity
Programmes and materials present accurate information, examples and statements and
authentic social behaviour.
 practicality
Exercises included in programmes and materials can be successfully completed by the
learners concerned. One has to be sure that the exercises actually work, and that their degree
of difficulty corresponds to the language level of the learners concerned.
Theoretical background: The issue of reliability is reminiscent of the same concept in quality control
procedures, which have been developed as a part of quality management. Quality management also
includes quality assessment procedures. The ISO-norms play a leading role in this respect (cf. AFNOR
1998). In our context, the notion of reliability is mainly related to the planning phase of the
teaching/learning interaction, i.e. it refers to the design of programmes and especially materials. Therefore,
this principle has been elicited from the analysis of works concerned with the development of materials (cf.
Tomlinson 1998) the analysis of textbooks (i.e. Engel 1977/79, Kast/Neuner 1994), as well as from
13

discussions of teaching and learning methods and approaches (for instance Stern 1983, Richards/Rodgers
1986, Neuner/Hunfeld 1993, Serra Borneto 1998).

C2.4 Attractiveness
The principle of attractiveness summarises all the features of programmes and materials which
are appealing to the user and therefore contribute to enhance his/her motivation.
The principle of attractiveness comprises the following sub-principles:
 user friendliness
Programmes and materials are easy to use (easily accessible, with a user-friendly interface
and appropriate illustrations). This issue is particularly relevant to multimedia programmes
and for autonomous learning.
 interactivity
Programmes and materials create a dialogue with the learner, in that they offer facilities and
feedback, stimulate curiosity and generate an atmosphere of play.
 variety
Programmes and materials are characterised by a range of activities, types of interaction and
working rhythms. Motivation in learning is enhanced by switches of emphasis.
 sensitivity
Programmes and materials take the affective dimension into consideration, creating an
attractive environment and motivating activities to help create a relaxing atmosphere.
Furthermore, it is important for any learning environment that the learners’ progress is
encouraged, that correction does not stultify the desire to learn and that there is attention and
patience for weaker learners.

Theoretical background: Attractiveness, as intended here, is a notion connected with the client-
led TQM approach and with marketing principles in general. TQM, especially if applied to
services (as opposed to products) has a strong user-oriented bias (Negro 1996). The MLTL
interpretation of this issue is influenced by motivational theories of learning (e.g.
Gardner/Lambert 1972, Gardner 1985), the autonomous learning movement as well as by the
methodological discussion of affective approaches, which has been partly taken over by
communicative or action-oriented approaches (cf. FRAMEWORK 1996 among others).

C2.5 Flexibility
Flexibility accounts for the individual modes (both cognitive and affective) in approaching the
teaching/learning process. It refers to features of programmes and materials which are sensitive
towards the individual specificity of the user, including group differentiation in classroom
environments.
The principle of flexibility comprises the following sub-principles:
 individualisation
Programmes and materials take account of learner characteristics like former learning
experience, learning styles, disposition for autonomous learning, etc. The material or
programme should allow different types of learners to use different activities or techniques to
approach a topic or problem.
 adaptability
Programmes and materials allow for expansion, reduction, easier and more challenging
adaptation and exploitation for different learning purposes. The material or programme
should be open and flexible enough to allow the individual learners to work more in depth
with particular items.
14

Theoretical background: Educational theories in general and studies on the psychology of FL learning
(especially individual characteristics of learning) account for the notion of individualisation (cf. Skehan
1989, Horwitz/Young 1991, Ehrman 1996). While individualisation refers to the type of activities and
exercises, the sub-principle of adaptability is related to the number and the scope of activities, allowing the
learners to determine their own pace (cf. Tomlinson 1998).

C2.6 Generativeness
Generativeness accounts for the open-ended character of programmes and materials which
facilitate cognitive development and the transfer of what has been learnt in one context to more
general tasks and/or other contexts.
The principle of generativeness comprises the following sub-principles:
 transferability
Programmes and materials encourage the transferability from controlled through guided to
free activities and the transferability of strategies, skills and contents to different contexts in
and outside the learning environment.
 integration
From a linguistic point of view, programmes and materials build onto previous knowledge
(progression) and help learners to relate concepts.
 language awareness
Programmes and materials help learners to develop awareness of the way language operates
and to pay attention to the way they use language and the way language is used around them.
 cognitive development
Programmes and materials provide opportunity for learning to learn (problem solving,
strategy training, etc.) and promote the awareness of these aspects in the learners. By
explaining clearly what is actually going on in the learning process, and by providing the
learners with opportunities for reflection materials and programmes contribute to the
awareness of what learning is about.
Theoretical background: Since Generativeness is mainly concerned with the elaboration and processing of
(linguistic) knowledge, the main theoretical source for this principle are cognitive theories of learning (see
for instance Anderson 1983; Skehan 1998) as applied to language teaching and learning, for example the
notions of awareness raising (Hawkins 1984) and strategy training (O’Malley/Chamot 1990)

C2.7 Participation
Participation is concerned with opportunities to make choices and to share responsibility in the
process of teaching/learning.
The principle of participation comprises the following sub-principles:
 involvement
Programmes and materials allow learners to be fully and actively involved in the learning
process. This requires that the activities - both in the cognitive and affective dimension - are
designed in such a way that the learner at any time has the knowledge and skills required for
taking part in the learning process.
 personal interest
Programmes and materials allow learners to bring their interests, opinions and experiences to
the learning process, thereby making it personally meaningful. The materials and
programmes may achieve this by challenging, provoking, reassuring and confirming attitudes
the learners have.
 partnership
Programmes and materials encourage learners to make choices, contribute to decisions, and
share responsibility for their learning. In some learning environments this may imply that
15

learners are consulted in the process of planning the next module. In others it may lead to the
definition of a learning contract - to do particular tasks, reach a particular level.
Theoretical background: Participation is one of the principles that seems self-evident. Everything that is
directly related to the learner and his/her interests, experiences or emotions is of personal relevance and
thus not only more motivating but also easier to remember. This is due to the fact that the affective
dimension seems to support the storage and retrieval of information (Goleman 1996) but also because
learning material that is related to the knowledge of the learners can be more easily processed than
something that is beyond their experience or imagination (O’Malley/Chamot 1990).
Partnership is the first step in direction of autonomy. The emancipation from the teacher is a central
demand among educationalists and foreign language teachers aiming to help individuals take on
responsibility for their own (learning) actions.

C2.8 Efficiency
Efficiency is an “economic variable“ relating resources to results.
The principle of efficiency comprises the following sub-principles:
 cost-effectiveness
Time and resources are carefully exploited in the realisation of programmes and materials.
This aspect implies that there is a good match between the investment (in terms of money,
energy and time) and the results.
 ergonomy
Programmes and materials demonstrate an optimal matching of requirements to needs in a
given context in order to generate an outcome.
Theoretical background: The concept of Efficiency has been adopted from the economic theory of
Management, Quality Control theory and TQM (Feigenbaum 1983). The way it is used here is strongly
reminiscent of its original use. It brings the economic factor into focus, the fact that language learning and
teaching has to be paid for and that prices have to cover the expenses of the provider of
programmes/materials on the one hand and yet be attractive for the potential client (e.g. learners) on the
one hand.

C2.9 Socialisation
Socialisation concerns “added value“ to a language programme which pursues the development
of additional skills beyond purely language competencies.
The principle of socialisation is fulfilled if programmes and materials pursue linguistic achievements
together with:
 social skills
Programmes and materials aim at promoting the social skills of the learners such as the
ability to co-operate with others or to develop empathy. In addition to that, the learners are
encouraged to develop new concepts and to look at things from a different perspective.
 intercultural awareness
Programmes and materials ask learners to reflect upon the knowledge and understanding of
their own culture as well as the cultures of other countries and their personal attitude towards
them. It is important to develop sensitivity towards the different and the “other“, and to
provide bilingual education where appropriate. Activities can include considering
“contrastive“ cultural patterns and discussing cross-cultural behaviour.
16

C3 Grouping of principles
The general principles listed above have been developed as a conceptual framework for quality in
language learning. It should be remarked that virtually no principle alone can assure quality
standards but that the notion of quality arises as the result of the convergence of as many principles
as possible in particular programmes and materials. Any conceptual framework in the context of
modern language teaching and learning has to include three main ”conceptual” dimensions of the
teaching/learning process which determine the conditions of its realisation:
a) the teacher/learner,
b) the context,
c) the means.
The teacher and the learner constitute the central dimension in the teaching/learning process, since
they represent the main actors involved. As far as the concept of quality is concerned, their
perspective can be summarised in following terms.
 from the point of view of the learner quality implies:
1) opportunity for choice with optimal matching of provision to needs;
2) satisfaction and achievement;
 from the point of view of the teacher (or: provider):
1) consideration for the needs and expectations of the users;
2) provision of a well-organised, efficient, user-friendly product or service.
The context represents the conditions under which the teaching/learning process takes place
determining the frame of reference and the departure point of any reflection on quality. The
variables influencing quality in this respect are (cf. FRAMEWORK 1996, 20-24):
 the domain in which the teaching/learning process takes place (personal/public/occupational/
educational);
 the situations which arise in a particular domain (institution, location, persons involved,
environment, events and operations);
 physical, social, time conditions and constraints;
 the user/learner’s mental context (cognitive factors, intentions, reflections, interests, needs and
motivations).
The means can be equated with the ”methods and materials ”as they are mentioned in the ”White
Paper”. They also constitute a major factor of quality in the realisation of the learning/teaching of
modern languages.
Starting from these basic reflections, the principles can be grouped according to the "conceptual
dimensions". From this point of view, the principle of Relevance becomes central, since it
applies to all three dimensions. The other principles are connected with one or two dimensions
and contribute in establishing an interface between them. Figure 5 illustrates this point, showing
the groupings of the principles centred on the principle of Relevance.
17

Materials and the


learning environment:
Reliability,
Flexibility,
Efficiency
Learner-material Personal and social
interface: aspects of the learning
Attractiveness, / teaching process:
Transparency Relevance Participation,
Generativeness,
Socialisation

Figure 5: Grouping of Principles

The box on the left contains those principles that concern the interface between the learner and the
material or programme in question. Clarity, User Friendliness and Motivation are some of the
aspects in this domain. The top box includes the principles of quality that concern the material or
programme ”per se” and the variables that are related to them, i.e. the learning environment and the
group of learners in question. The right hand box groups together those principles of quality that
deal with the personal and social aspects of the language learning/teaching process in general
(including cognitive and affective variables) and the way in which this is being taken account of
within the means used.
18

D. Stages of quality management in the in the learning /


teaching process
A Total Quality Management approach to the provision of language learning / teaching programmes and
materials can be seen as involving three distinct conscious stages of quality assurance: Before (Design);
during (Implementation) and afterwards (Evaluation of Outcomes).
Quality in the design stage will facilitate but cannot substitute for quality at the implementation stage.
Quality in implementation can partially compensate for bad design but is in effect hamstrung unless it can
exploit appropriate content adequately presented. Quality of the service provided by the design and
implementation of the programme can only be assured by evaluating whether, independent of subjective
impressions, the programme generates reasonable outcomes.

The three stages are summarised in Figure 6:

DESIGN IMPLEMENTATION
Quality in the design stage Quality in the implementation
relates to the specification of stage implies providing a
the learning content and positive affective atmosphere
activities on the basis of the and a supportive learning
analysed needs of the learners,
plus the presentation of these  environment in which the
realisation of the design is
content and activities in the monitored and adapted flexibly
programme or materials. to the evolving needs of the
learners.
Before During

OUTCOMES
The quality of the outcomes
can be evaluated in terms of the
gains in competence and
 awareness in relation to the
resources and conditions 
concerned and also in terms of
the satisfaction of the
stakeholders involved.
Afterwards

Figure 6: Stages in the learning / teaching process


19

Each stage of quality is divided into sections which represent the main aspects of the stage in
question; each section is further subdivided in elements which illustrate the specific contents of the
sections. Various types of questions, indicators, and examples are attached to each element (see
Section E). They are intended to serve both as an awareness raising tool and as a means to check
quality.

D2.1. DESIGN
Quality in the design stage relates to the specification of the learning content and activities on the
basis of the analysed needs of the learners including the presentation of these content and activities
in the programme or materials.
Design is divided into four sections (Content, Activities, Organisation, Presentation) and the
corresponding elements:

Content refers to the selection of the linguistic content, i.e. functions and notions, grammar, and
vocabulary, the kinds of texts and the themes to be covered.

Activities is used here to cover all types of actions carried out by the learners in order to learn a
new language. This comprises the skills, listening, reading, speaking, writing, interacting,
mediating (cf. Framework), as well as exercises and pedagogic tasks, "real life" tasks and
projects, and learning strategies.

Organisation at this stage refers to the inner structure of programmes/materials. It concerns the
design of the individual units as well as the curriculum design as a whole; the assessment
procedures used to evaluate the achievement of the objectives, and the way interaction
among the learners is organised.

Presentation refers to the packaging of contents and materials and the effect it has on the user. It
concerns both the aesthetic and the functional dimension of the material/programme and
includes primarily the following aspects: navigation, illustrations, layout, and colour &
music.

D2.2. IMPLEMENTATION
Quality in the implementation stage implies providing a positive affective atmosphere and a
supportive learning environment in which the realisation of the design is monitored and adapted
flexibly to the evolving needs of the learners.

Implementation is divided into three sections (Affective Management, Monitoring Activities,


Support Measures), and the corresponding elements:

Affective management includes steps to ensure that the learning process is an enjoyable,
stimulating and rewarding experience. Relevant concepts are framing & centring (e.g.
warming up, variable pacing, recuperating), matching (topics and activities to the specific
learners), stimulating (the curiosity and creativity of the learners), and encouraging (i.e.
challenging as well as praising and rewarding).

Monitoring activities include observation (e.g. of the performance of the individual learner or of
the progress of the teaching process), assessment (i.e. achievement in relation to the starting
point and/or the time and resources available etc.), feedback (to the learner on language
performance), and adjusting provision (e.g. of the presentation or the activities).

Support measures include people who can help, if things go wrong (apart from the teacher, i.e.
"tutors", service personnel), information tools (e.g. content overview, reference material),
20

awareness-raising & self-training tools (teachers’/learners’ guide, fast track facility to help
more independent learners etc.).

D2.3. OUTCOMES
Outcomes is concerned with the assessment of the effectiveness of the programme / material with
regard to the results obtained. The quality of the outcomes can be evaluated in terms of the
objective gains in competencies and awareness in relation to the resources and conditions given on
the one hand and in terms of the subjective satisfaction of the learners or the stakeholders involved
on the other.
One can differentiate between successful learning of the language concerned (Success), the
reactions of learners and stakeholders to the programme/material (Satisfaction), and the
achievement of broader educational aims (Educational Bonus).

Success from the learners’ point-of-view is evaluated by means of internal criteria (achievement of the
objectives stated, assessed in e.g. progress tests, teacher & self-assessments), external criteria
(measurable improvement in language proficiency established by comparing results on pre-course
proficiency tests to those on post-course tests or examinations), real life competence gains
(observable improvement in actual real life language use recorded in e.g. testimonials, self-
assessments and documentation), and depth of acquisition (long term acquisition and ability to cope
with new tasks).

Satisfaction can be considered from the point of view of both those receiving the service (learners,
etc.) and those involved in using the material or programme to provide the service (teachers
etc.). Satisfaction is measured by evaluating e.g. modalities (duration, sequencing, logistics),
facilities (materials, resources, support), activities (tasks, interaction, learning strategies),
and progress.

Educational Bonus includes such aspects as intercultural awareness, learning strategies,


communication strategies, and self-confidence & motivation in the learners.
21

E. Putting Quality into Practice


E1. Design

DESIGN
Quality in the design stage relates to the specification of the learning content and activities on the basis of the
analysed needs of the learners including the presentation of these content and activities in the programme or
materials.
Before

The following diagram gives an overview of the different aspects which have to be considered when planning
programmes / materials. The four main areas are
 content specification: a description of the linguistic and non-linguistic contents;
 activity specification: a description of different types of activities for practising and using the language;
 organization: how the different parts are put together to form a coherent and consistent product;
 presentation: the means used to meet both functional and aesthetic objectives.
Each of these areas is again subdivided into sub-categories, the content of which will be illustrated in detail in the
following section.
22

Quality in modern language teaching and learning implies in our view a learner-centred approach. Learner-
Centredness means here that the needs and expectations of the learners form the point of departure for the selection of
contents and activities and for the way these are organised and presented within the given context.

Content and methodology have to find a balance between the needs of the individual and the needs of society as a
whole. While the former refers to different learning styles and experiences, expectations, attitudes and reasons for
language learning, the latter concerns the needs of the economy and/or the stakeholders on the one hand and general
educational needs on the other. Educational needs cover competences such as social and intercultural competence.
This competence involves the willingness to interact with others and the ”ability to handle social situations” (van Ek
1976: 2) within one's own culture as well as in interaction with members of other cultures.

1.1 Context Description


Learners’ needs are the starting point for the selection of contents.
The satisfaction of these needs, however, is dependent on the
context in which learning and teaching takes place. Each context
has different conditions and constraints that determine what is
feasible as regards type and length of activities, materials used,
relevance appropriateness
etc. Teaching on a one-to-one basis or a CD for self-study, for
example, requires different types of activities than teaching in a
class. Thus, in order to ensure quality, the context in which
materials are used and programmes are realised has to be
adequately taken into account.

1.1.1 What has been done to ensure that the characteristics of the
target group have been adequately taken into account?

Have the following characteristics been identified:


1.1.1.1  Age group; relevance appropriateness
1.1.1.2  Educational sector, e.g. Secondary School; relevance appropriateness
1.1.1.3  Orientation, e.g. general ,subject-specific, job-specific; relevance appropriateness
1.1.1.4  Language level; relevance appropriateness
1.1.1.5  Mother tongue of the learners; relevance appropriateness
1.1.1.6  Cultural/social background of the learners? relevance appropriateness

1.1.2 What has been done to ensure that the learning environment
has been adequately taken into account?

1.1.2.1  Has the relevant learning environment been identified: relevance appropriateness

 One-to-one: - intensive (e.g. 15 hrs per week) relevance appropriateness


- short intensive (e.g. a weekend)
- extensive;

 Class: - intensive (e.g. 15 hours per week or more


relevance appropriateness
- short intensive (e.g. a weekend
- extensive;
 Open learning centre;
23

relevance appropriateness
 Distance course (”correspondence”);
relevance appropriateness
 Home study?
relevance appropriateness
1.2 Content Specification
Content as it is used here refers to the linguistic learning
objectives (functions, notions, grammar and vocabulary), the
themes and the texts addressed during the course.
Being able to understand and express oneself in a foreign language
is the ultimate goal of communicative language teaching and
learning. Programmes / materials should provide learners with the
linguistic and strategic means to use the foreign language linguistic
appropriately and successfully in the situational context that is integrity
reliability
relevant to the group of learners in question.
relevance learner-
Appropriately means that learners need to be able to say the right Centredness
thing at the right time. To do that, they have to be able to take the
cultural and the social background against which communication
intercultural
takes place into account. Learners therefore have to be equipped socialisation
awareness
with sufficient awareness of or knowledge about social
conventions, social rituals and politeness conventions to be able to
avoid or repair misunderstandings.
Successfully means that the learning environment and the time and
resources available need to be taken into account when efficiency economy
determining the learning load: the amount of content. In addition,
since people learn in different ways, success requires enough flexibility individualisatio
variety to allow learners and teachers to cater for different n adaptability
interests, learning styles and paces of learning. participation personal
interest
Relevant means that the content specification should be based relevance learner-
upon the present and future communicative needs of the group of Centredness
learners concerned.
1.2.1 What has been done to ensure that the choice of linguistic and
thematic content is relevant for the learners?

1.2.1.1  Has a formal analysis of needs been conducted for this learning relevance learner-
population as a whole? Centredness
 Questionnaire
 Interviews with a sample of respondents

 Have the needs and interests of this particular group been learner
1.2.1.2 relevance
investigated? Centredness
 Questionnaire participation personal
interest
 Interviews with a sample of respondents
 Discussion with the group
 Consultation of relevant resources such as Waystage Level,
Threshold Level etc.
24

1.2.2 What steps ensure that the choice of the linguistic and
thematic content is relevant to present or future vocational
requirements?

1.2.2.1  Have the vocational requirements of the target group been


taken into account?
 Questionnaire
relevance accountability
 Job descriptions, surveys and other relevant documents
learner-
 Interviews with employers / trainers Centredness
 Are the language needs during training of the target group
1.2.2.2
taken into account?
 Feedback sessions
accountability
relevance
 Interviews with trainers

1.2.3 What has been done to ensure that the language used in
programmes/materials represents correct, appropriate,
current usage?

1.2.3.1  Has language used in instructions and produced in exercises


been checked for correctness and appropriateness?
reliability linguistic
 Use of reference works integrity
 Consultation beforehand with native speaker teachers
 Feedback from trialling linguistic
reliability
integrity
 Are differences between spoken and written language and
1.2.3.2
between colloquial and more formal styles taken into account? textual integrity
 Use of reference works
reliability
 Consultation beforehand with native speaker teachers linguistic
integrity
 Feedback from trialling
factual
 As regards vocational courses, have the customs and/or
integrity
1.2.3.3 terminology of the profession in question been considered?
 Use of reference works to identify specialist vocabulary
(specialised lexicons, glossaries, handbooks etc.)
 Consultation with experts in the field

1.2.4 What is done to ensure that information included is accurate


and up-to-date?

1.2.4.1  Has any factual content been checked? reliability factual integrity
 Sources given and checked for reliability
 Facts and dates regularly updated
1.2.5 Themes
Language is normally used as a means for conducting interactions
25

and transactions which take place within a particular context. The


situational context determines the type of action and influences
the topics for communication. Both the situational context and the
topics for communication are here referred to as 'themes'.
Themes should engage the interest of the learner both by their relevance learner-
relevance to the learners' daily life and professional needs and by Centredness,
introducing topics of intrinsic interest which extend their
knowledge and field of interest. Information about the other generativeness transferability
culture concerned is an integral part of language learning.
socialisation intercultural
awareness

1.2.5.1. What has been done to ensure that the choice of themes is
interesting for the learners?

1.2.5.1.1 Does the selection of topics:


 Appear motivating and consistent with the interests of Attractiveness sensitivity
learners?
Participation personal
 Matched to fields of interest interest
 Novelty aspect - not just stereotyped themes involvement
1.2.5.1.2 participation
appropriateness
 Take into account the personal characteristics and everyday relevance
experience of the users?
 Appropriate to age group
 Related / contrasted to life style

1.2.5.2 What steps ensure that the choice of themes take educational
and / or intercultural aspects into account?
1.2.5.2.1  Raise awareness of cultural and gender stereotyping? socialisation social skill,
intercultural
 Comparing and contrasting conditions and patterns of
awareness
behaviour
 Pointing out instances of inter-cultural differences in
materials and scenarios used for teaching
1.2.5.2.2
 Offer opportunities for cross-curricula links appropriate to the transferability
generativeness
level?
cognitive
 Integrating information from other subjects (history, development
geography) into language lessons
 Teaching an aspect of another subject as a limited
immersion experience
 Learning another subject through the medium of the foreign
language
 Exploiting the internet
1.2.6 Texts
The term 'text' refers to spoken as well as to written language. generativeness
Texts play three main roles in foreign language learning and participation
teaching:
integration
- a source for comprehension practice and a means of getting
personal
26

accustomed to authentic language; socialisation interest,


involvement
- a source of speech patterns and written rhetorical patterns;
intercultural
- a source of information about the thinking and customs of awareness
another culture.

A wide range of text types and genres appropriate to the needs and attractiveness variety
expectations of the target group in question is necessary.
relevance learner-
Furthermore, text types and genres should also be in line with the
Centredness,
methodological approach adopted and be chosen in such a way
appropriateness
that the users are given the opportunity to practise different
reading or listening skills.
reliability
If texts are adapted for didactic reasons, the original features of method;
the genre need to be preserved in order to help learners recognise integrity
and, if appropriate, acquire such features. Language and current
reliability textual integrity
factual information in texts also needs to be correct and up-to-
date. linguistic
integrity
reliability
factual integrity
reliability

1.2.6.1 What steps have been taken to choose texts which will increase
motivation?

1.2.6.1.1  Do the texts meet the learners personal interests and participation personal
emotions? interest
 Fields of professional interest
 Music, culture
 Hobbies / personal interests
 Love / romance interest
 Current issues
 Is a variety of different kinds of texts and documents variety
1.2.6.1.2 attractiveness
provided?
personal
participation
 Short anecdotes interest
 Stories
 Information carrying text
 Texts from encyclopædias, travel guides etc.
 Newspaper articles
 Editorials / short essays
 Cartoons
 Comics
 Poems
 Songs
1.2.6.2 How are the texts used in the programme / materials matched
to didactic purpose?
27

1.2.6.2.1  Is the selection of texts consistent with the way texts are to be reliability methodological
exploited in the didactic methodology adopted? integrity
 Dialogues for a situational / audio visual approach
 Model presentation texts for a classic- inductive approach
(presentation – comprehension - pattern practice - transfer)
 Authentic source materials for a needs-based approach
 Amusing drawings, cartoons, short stories, drama scripts for
a "learning-centred" story-based approach
 Literary texts for a grammar-translation method

methodological
 Are appropriate texts used to present language forms in a reliability
integrity
meaningful context?
1.2.6.2.2
 Presentation texts which illustrate use of the target forms
 Texts for comprehension with occurrences of the target
forms
 Dialogues, prompt sets for fluency practice methodological
reliability integrity
 Texts for language awareness activities
1.2.6.2.3
 Are appropriate texts selected to practise specific reading
skills?
 Searching for specific information and/or matching
information to needs
(authentic notices, adverts, leaflets, brochures etc.)
 Reading for main ideas
(short articles and prose texts; longer prose texts)
 Reading for detailed comprehension
(short articles and prose texts)
 Reading to locate information
(longer factual and opinion-bearing texts; reference works)
 Reading for fluency (extensive reading)
(longer texts, short stories etc.)
28

1.2.6.2.4  Are appropriate texts selected to practise specific listening reliability methodological
skills? integrity
 Listening for specific information and/or matching
information to needs
(announcements, recorded messages, news broadcasts,
adverts, etc.)
 Listening for main ideas
(conversations, talks, presentations, broadcasts etc.)
 Listening for detailed comprehension
(conversations, extracts from short talks / presentations,
documentaries etc. )
 Listening for fluency (extensive listening)
(short stories, songs etc.)
involvement
1.2.6.2.5  Has the appropriateness of the difficulty and complexity of participation
the texts for the target group been checked? textual integrity
reliability
 Consultation with appropriate teachers practicality
reliability
 Comparison of vocabulary and structures with content
inventories (e.g. Threshold Level; examination specifications)
or linguistic inventories (corpora; frequency lists etc.)
 Analysis of cognitive complexity – information density,
rhetoric pattern, presence or absence of distracting
information- etc.)
 Trialling with representative classes.

1.2.6.3 What steps have been taken in order to ensure the integrity of
the different types of texts chosen?

1.2.6.3.1  Has each adapted text been checked to guarantee that the
specific textual conventions of the genre concerned (cohesion,
reliability textual integrity
coherence, structuring, signalling etc.) have been preserved?
 Conscious comparison by the materials developer
 Consultation with experts in field
 A discourse analysis / text linguistics analysis
1.2.6.3.2
 Has each custom written text been produced so as to comply
with the textual conventions of the genre concerned (cohesion,
coherence, structuring, signalling etc.)?
reliability textual integrity
 Conscious comparison by the materials developer
 Consultation with experts in field
 A discourse analysis / text linguistics analysis

1.2.7 Linguistic content


Functions, grammatical structures and vocabulary should cover
the ”most likely and urgent needs” of the target group (Threshold
Level 1990, p. 22) and should represent up-to-date, correct and
29

appropriate language. Learners should again be made aware of the


cultural implications language carries. The way we complain, for
relevance learner-
example, or ask for a favour, tells a lot about the values and the
Centredness,
politeness conventions of a society. Awareness of these features
accountability
contributes considerably to successful communication.
reliability linguistic integrity
Functions, grammar, vocabulary and pronunciation should be
introduced and practised in such a way that different learning intercultural
styles and learning paces are catered for and different learning socialisation awareness
experiences taken into consideration. language
generativeness awareness
The role of conscious grammar practice in foreign language
individualisation
learning ("focus on form") is somewhat disputed. It is generally
involvement
accepted that learners need to practise language for flexibility integration
communication as well as practising language as communication
(Widdowson), but the form this practice takes varies considerably participation
with different approaches. The fundamental distinction is between generativeness
clarity of aims
deductive approaches, in which rules are explained and then
practised, and inductive approaches, in which learners generalise
rules from examples. transparency

However, to help users develop into autonomous learners, learning


objectives should be described explicitly.
1.2.7.1 Functions and Notions
Functions describe what people do with language, for instance,
expressing an opinion, taking leave, or offering an apology.
Language functions are fulfilled with regard to particular concepts
or ‘notions’. With a remark like ”We are going to the cinema on
Saturday” we refer for instance to the concept of ‘time’. The
Threshold Level 1990 lists eight categories of general notions
which are: existential, spatial, temporal, quantitative, qualitative,
mental, relational, deistic.
1.2.7.1.1 What steps are taken to ensure that learners acquire a flexible,
functional repertoire?

1.2.71.1.1  Was a curriculum specification used to list language functions relevance learner-
relevant to the learners concerned and to link functions to centredness,
grammatical structures appropriate to the level concerned? appropriateness
reliability
 Collating relevant existing sources (course books, linguistic integrity
syllabuses from other projects etc.)
 Consultation of standard reference works (e.g. Threshold
Level)
 Derivation from own needs analysis
 Is language presented in a functional context?
1.2.71.1.2
 Real life situations appropriate to these learners
reliability
linguistic integrity
 Realistic purposes
methodological
 Identification of new language with relevant intentions and integrity,
appropriate moves in the situation. consistency
 Are learners presented with and encouraged to collect
1.2.71.1.3 alternative ways of expressing things?
transferability,
 Substitution table of alternative forms (e.g. Can you, Could cognitive
30

you ….) generativeness development,


lang. awareness
 Practice with parallel situational dialogues and / or different
routes through situations
 Identifying how intentions are expressed in listening and
reading materials
 Encouragement to note how intentions are expressed in real
life

1.2.71.1.4  Are notions (e.g. possibility, probability, time, shape etc. ) reliability methodological
developed systematically? integrity,
internal coherence
 Situations used as a vehicle for presentation
 Associated with related functions
 Recycled and extended at intervals
methodological
 Is the acquisition of language to express different notions
1.2.71.1.5 reliability integrity,
linked to the organisation of relevant linguistic forms (e.g.
internal coherence
time/tense; space/prepositions, shape/adjectives;
possibility/modals)?
 Notions presented with related structures
 Cross referencing provided in a chart or index
 Notions recycled and linked to new language as new
language is introduced
1.2.71.1.6
 Are learners encouraged to “unpack“ and analyse functional
language
phrases learnt as prefabricated chunks? generativeness
awareness
 Grammatical patterns occurring in such chunks pointed out
as they are encountered in the syllabus
 Language awareness-raising activities to identify underlying
grammatical rules
1.2.7.1.2 What has been done to ensure that pragmatic features are
adequately represented?

Have the following pragmatic features been taken into account?


1.2.7.1.2.1  Markers of social relations reliability textual+linguistic
integrity
 Politeness strategies
 Negotiations
 Turntaking
 Etc.
textual+linguistic
1.2.7.1.2.2  Register differences reliability
integrity
 Formal/informal/colloquial
 Written/spoken
 Etc.
textual+/linguistic
1.2.7.1.2.3  Schematic patterns of interaction reliability integrity
31

 Routines (verbal/non-verbal/paraverbal)
 Rituals (verbal/non-verbal/paraverbal)
 Gambits
 Etc.
1.2.7.1.2.4  Discourse signalling reliability textual+linguistic
integrity
 Cohesion markers
 Evidence of coherence
 Openings and closures
 Etc.
1.2.7.2 Vocabulary
Vocabulary refers to all lexical items including multi-word chunks relevance learner-
and idiomatic expressions. Vocabulary should be selected centredness,
carefully to focus on items that have a high priority for the target accountability
group in question. flexibility individualisation
Learners should be provided with sufficient comprehension generativeness cognitive
materials to acquire a wide passive vocabulary in addition to their development
learnt repertoire, and encouraged to note interesting new
language
expressions to build up a personalised wordlist. generativeness
awareness
In addition to this, learners need practice in relating words to each
other e.g. recognising and building derivatives from word roots;
associating similes and opposites. flexibility
individualisation
Finally, varied storage and retrieval techniques should be provided attractiveness variety
to cater for different learning styles and keep up motivation levels. generativeness
cognitive
development

1.2.7.2.1 What factors has been taken into account in selecting


vocabulary?

Have the following factors been taken into account:


1.2.7.2.1.1  The level of the target group relevance appropriateness
 Consultation with teachers
 Use of standard content inventories (e.g. Threshold Level)
 Use of word lists
1.2.7.2.1.2  Frequency and importance in the target context relevance learner-
centredness,
 Consultation with informants
accountability
 Use of linguistic inventories (corpora; frequency lists
etc.)
 Collocation (words which go together)
1.2.7.2.1.3 reliability linguistic integrity
 Use of dictionaries
generativeness lang. awareness
 Use of concordances printed from corpora
 Colligation (grammatical forms associated with words)
1.2.7.2.1.4 reliability linguistic integrity
32

 Use of dictionaries generativeness lang. awareness


 Use of concordances printed from corpora
linguistic integrity
1.2.7.2.1.5  "False friends" and concepts commonly confused reliability
lang. awareness
 Consultation with teachers generativeness
 Use of language-specific learner dictionaries
 Use of comparisons of the mother tongue and target
languages
 Expressions which will be needed in instructions practicality
1.2.7.2.1.6 reliability
 Consultation with teachers cognitive
generativeness
development
 Comparison with similar materials / programmes

1.2.7.2.2 What has been done to ensure that the approach to vocabulary
work is clear and effective?

methodological
1.2.7.2.2.1  Is vocabulary practised or activated in a systematic and varied reliability
integrity,
way?
internal coherence
 Vocabulary exercises consistently connected with the other practicality
attractiveness
activities variety
 A systematic progression in the treatment of vocabulary
 Regular recycling of vocabulary
variety
 Are learners introduced to a variety of techniques for individualisation
1.2.7.2.2.2 vocabulary association and memorisation in order to cater for
different learning styles? attractiveness
cognitive
flexibility
 Mind-maps development
 Lexical sets generativeness
 Flow charts
 Tree structures
 Contrasting lists
clarity of
 Etc. presentation
 Are vocabulary items listed?
1.2.7.2.2.3
 General list at the end of material transparency

 Unit by unit clarity of aims

 Are items distinguished with regard to their function?


1.1.3.2.2.4
 Basic/supplementary vocabulary transparency
 Basic vocabulary/vocabulary for specific purposes?
 Are activities provided to help learners develop their personal individualisation
1.2.7.2.2.5
repertoire?
cognitive
 Training in how to use dictionaries flexibility
development
 Encouragement to practice vocabulary outside the generativeness
classroom learner-
centredness
33

 Individual ”personal dictionaries” relevance

1.2.7.2.3 What has been done to promote intercultural awareness ?

 Has vocabulary been compared for cultural differences?


socialisation intercultural
1.2.7.2.3.1  Comparison of connotations
awareness
 Discussion of items from different cultural perspectives?
 Awareness raising of typically cultural association (for
instance about colours, numbers etc.)
 Comparison of proverbs and idioms
 Interpreting of metaphors

1.2.7.3 Grammar
Grammar describes the system of linguistic rules that underlies the
target language. Criteria are needed to decide how these rules are
selected, described and presented to the group of learners in
question. Equally important are the activities that are carried out
to ensure the consolidation and retention of new input.

1.2.7.3.1 How is grammar presented?

1.2.7.3.1.1  Is the form of language description chosen in line with the reliability consistency
overall approach taken in the programme/material? methodological
integrity
 The use of language rather than description of language is
the basis of categorisation
 Grammatical structures related to functions
 Appropriate reference works
 Do the forms chosen illustrate a wide functional range for a
1.2.7.3.1.2 variety of contexts? generativeness transferability
 Selection based on an analysis of functions necessary in the
contexts (ContextFunctions necessaryGrammatical
exponents
 Cross-referencing during planning of grammatical forms to
the different functions and contexts
 Provision in materials of tables matching forms to functions

1.2.7.3.2 What has been done to facilitate the comprehension and


acquisition of rules and structures?

1.2.7.3.2.1  Is there a systematic treatment of grammar appropriate to the reliability methodological.


level of the target group? efficiency integrity
ergonomy
 Number of exponents checked with regard to time available
34

 Core grammar identified and treatment checked


 Regular revision
1.2.7.3.2.2  Is the grammatical progression (linear or concentric)
transparent?
reliability internal coherence
 Connection between presentation text, grammatical transparency clarity of
explanations and exercises presentation
 Summary of grammar points
 Cross-references
methodological.
 Are grammar rules presented in a comprehensible form? integrity
reliability
1.2.7.3.2.3
 Terminology comprehensible for the target group transferability
generativeness
 Tables and diagrams clarity of
transparency
presentation
 Signals
 Are new grammatical phenomena introduced in context?
1.2.7.3.2.4
 Visuals used to support the introduction of new
reliability method. coherence
structures
 Language embedded in a situational context
1.2.7.3.2.5  Is the function and use of a structure demonstrated?
generativeness transferability
 Overview tables
 Variety of examples
1.2.7.3.2.6  Are ‘contrastive clues’ referring to learners’ mother tongue
generativeness
included in the presentation of grammar?
integration

1.2.7.3.2.7  Is an appropriate variety of grammar practice activities attractiveness variety


provided?
flexibility adaptability
 Exercises which demand creative production as well as individualisation
drills
 Grammar games
 Expansion and reduction exercises
appropriateness
1.2.7.3.2.8 relevance
 Are exercises well explained and easy to do?
user friendliness
attractiveness
 Exercises graded according to the proficiency level of the practicality
reliability
user clarity of
transparency
presentation
 Keys available
 Programme/material piloted
1.2.7.4 What steps are taken to improve pronunciation?

1.2.7.4.1  Is pronunciation and intonation a specific topic in the design of reliability consistency
the programme/material?
 Pronunciation drills (on cassette) for
 word stress
 sentence stress
 thought groups
 pitch
35

 voicing
 stops and continuants
 aspiration
 intonation
 Intonation markers (colour, arrows, contours, etc.) added to
written texts
1.2.7.4.2
 Is authentic material providing an appropriate variety used for
reliability linguistic integrity
pronunciation/intonation reference?
 Radio and TV example
 Different genres (News, drama, interviews, etc)

1.2.7.4.3
 Has the social, emotional and intercultural relevance of
pronunciation been dealt with?
 Examples of misunderstandings socialisation intercultural
awareness
 Examples of lack of social acceptance due to pronunciation generativeness
problems language
awareness
 Is the importance of rhythm and speed in fluency stressed?
1.2.7.4.4
 Practice in placing emphasis on key message by change of
stress and rhythm reliability
methodological
integrity
 Catenation practice (linking non-stressed sounds and words)

1.3 Activity specification


The expression ‘Activities’ is used here to cover all types of tasks
and exercises carried out by the learners in order to learn the
language. This includes comprehension as well as production of
language, practice activities such as stimulus-response patterns or
matching exercises as well as communicatively-oriented activities
such as role-plays or letter writing.
reliability consistency
The selection and combination of activities should be consistent
with the approach to language learning that underlies the
programme or material.
Even in programmes/materials focusing on specific language
points the ultimate goal of real language use should guide the relevance learner-
design and choice of activities. The activities chosen should be centredness,
appropriate to the learning objectives of the respective target accountability
group. Exercises and tasks should prepare the learners for the
situations they are likely to come across in real life.
The selection of activities should also take account of the reliability
classroom management experience of the teachers concerned, and practicality
the learning experiences, the learning habits and the cultural relevance
appropriateness
perspective of the target group. flexibility
participation individualisation
involvement
1.3.1 What is done to ensure that activities build on learners'
experiences?
36

1.3.1.1  Have the different types of activities been cross-checked with relevance Learner
the results of the needs analysis? Centredness;
accountability
 Has the social and cultural background of the learners been
1.3.1.2 relevance
taken into consideration in selecting or designing the appropriateness
activities?

1.3.2 What steps have been taken to ensure that the main
characteristics of activities are taken into consideration ?
1.3.2.1  Have the purpose and aims of the task been specified:
 Purpose: generating motivation, consolidating the weeks transparency clarity of aims
work; assessing learning reliability consistency
 Learning objective: skill, language point, strategies

1.3.3 What steps are taken to ensure that activities are success-
oriented?
1.3.3.1  Has account been taken of the circumstances in which the
activity will be used?
relevance appropriateness
 Materials, data, equipment, realia available reliability practicality
efficiency ergonomy
 Duration
flexibility cost-effectiveness
 Amount of freedom/constraint generativeness adaptability
flexibility internal coherence
 Availability of tools, supports and help facilities
attractiveness integration
participation user-friendliness
involvement,
personal interest
method. integrity
 Is the task adequately prepared and followed up so that
1.3.3.2
learners understand what the purpose of the task is? user friendliness
attractiveness
 Adequate pre-task preparation cog. development
generativeness
clarity of
 Guidance / notes for the learner transparency
presentation
 Post-task de-briefing and feedback reliability method. integrity
generativeness integration,
flexibility transferability,
participation cognitive
development
individualisation
involvement
1.3.3.3  Have exercises and tasks been tested for practicality? reliability practicality
 Clear, comprehensive instructions generativeness cog. development
participation involvement
 Check for "bugs" (illogicality, incomplete editing, incorrect
transparency clarity of
language)
presentation and
 Check that relevant learners could complete the task of aims
 Check that learners could work independently when
necessary
37

1.3.4 Skills
Skills have traditionally been considered in terms of the 1960s
“four skills” model. Skills refer to the use of language – language
proficiency traditionally being considered a combination of
knowledge and skill.
Skills are rarely encountered in isolation, but the distinction
between the four skills continues to have a psychological reality:
being a fluent speaker may be very different from being a fluent
writer. For this reason, this section adopts a classification
approach which is a compromise between the four skills and the
approach in the Common European Framework.
1.3.4.1. What has been done to take different learner characteristics
into account?

1.3.4.1.1  Have exercises and activities been designed in order to be relevance appropriateness
appropriate for the personal characteristics of the learners:
participation personal interest,
 Age involvement
 Level
flexibility
individualisation
 Interests
 Learning experience
 Learning style?
 Is the cultural background of the target group considered in the
organisation of skills practice?
1.3.4.1.2. relevance
 appropriateness

1.3.4.2. What provision is made for helping the learners to practise


skills autonomously?
!0 CORE INDICATORS

CORE INDICATORS
1.3.4.2.1.  Are learners introduced to relevant practice and performance
techniques for each skill? generativeness cognitive
development
 For spoken texts
 For written texts reliability
methodological.
 For watching a programme integrity
 For initiating a conversation
 For keeping up a conversation
 For overcoming a spoken communication problem
 For writing a letter
 Etc.

1.3.4.2.2  Do exercises and tasks promote the transfer of techniques? generativeness


 Transfer to freer practice activities transferability
38

 Transfer to homework tasks


 Transfer to real life conditions (e.g. writing to a tourist
information office to inquire about a holiday resort)
 Are learners encouraged to use the skills outside the
1.3.4.2.3 generativeness
classroom?
transferability,
 Homework tasks
cog. development
 Extensive reading / listening suggestions
 Media, internet suggestions
 An open learning centre with material classified by level
1.3.4.2.4
 Are learners allowed to influence practice methods?
participation
 To choose among activities
partnership
participation
 To suggest alternative activities
personal interest
 To bring in own material

1.3.4.3 What steps ensure that the skills training is efficient in relation
to time and means available?

1.3.4.3.1  Is the skills training integrated into the curriculum with links efficiency ergonomy
to other activities in the unit?
reliability internal coherence
 Listening and reading texts integrated round a theme
 Use of information from receptive practice in productive
practice
 Integration of information from different sources
1.3.4.3.2 relevance appropriateness

 Does the complexity of tasks correspond with the needs of the


learners and the conditions and constraints of the learning
environment?
 Progression for each skill
 Tasks appropriate to the level of the learners
1.3.4.3.3 Speaking
The development of speaking skills demands a careful balance of reliability methodological.
activities to achieve fluency and message precision whilst integrity
preserving sufficient accuracy and comprehensibility. internal coherence
Some speaking activities need to offer a chance to consolidate and
automatise language learnt and activate a passive repertoire in reliability
order to achieve fluency. Other activities need to provide learners consistency
generativeness
with an opportunity to combine all their linguistic and para-
transferability
linguistic resources to complete a task, to achieve communication.
To get his or her message across, the speaker needs to know when generativeness
and how to open, maintain and close a conversation, how to seek language
clarification or how to check whether s/he was understood or not. awareness
Speaking activities need also to offer opportunities for the
acquisition of such interaction strategies.
39

1.3.4.3.3.1 How does speaking practice help learners to develop a fluent


command of an integrated language repertoire?

1.3.4.3.3.1.1  Are activities provided which help the learner to achieve a reliability consistency,
fluent command of new language being learnt?
methodological.
 Warm-up activities such as description of situation, integrity
introduction of reference vocabulary, brain storming in
mother tongue etc
 Pair work with 4 line dialogues
 Prompt cards or flow charts outlining "moves" for
improvised dialogues
 Role plays
 Games
 Quizzes
 Etc.
 Are activities provided which help the learner to integrate new
1.3.4.3.3.1.2 generativeness
language into their overall repertoire?
transferability,
 Information gap exercises, integration
reliability
 Problem solving tasks consistency,
methodological.
 Consensus and discussion tasks
integrity
 Simulations and role-plays offering a variety of contexts and
(social) roles
 Are activities offered which develop learners’ sensitivity to the
appropriateness of certain items/patterns/registers in a specific generativeness
context?
 Activities which help learners to identify speech functions lang. awareness
1.3.4.3.3.1.3 and intention
 Activities exploiting photos, illustrations, videos etc. to
sensitise learners to non-linguistic contextual cues
 Comparison of politeness convention, discourse markers
etc. in the first language and in the target culture
 Language awareness activities on dependence of language
on context, role and status of speakers
 Are speaking activities connected with or embedded in other
activity types? generativeness
 Follow up or precursor to language practice reliability
1.3.4.3.3.1.4
 Use of listening / video / reading text as a stimulus for transferability,
speaking practice cognitive
development
 Integrated skills activities, e.g. "in tray" activities, case internal coherence
studies etc.
 Project work

1.3.4.3.3.2. What steps are taken to develop compensation strategies?


40

1.3.4.3.3.2.1  Do programmes / materials introduce learners to the concept of generativeness language


compensation strategies? awareness
 The use of compensation strategies in mother tongue
 The importance of compensation strategies to maintain
fluency in the target language

generativeness
 Do programmes / materials provide tasks which by providing a cognitive
1.3.4.3.3.2.2 communicative urgency encourage learners to apply these development
strategies? transferability
 Rephrasing, overgeneralising, substituting a similar word participation
 Using non-verbal signals such as gestures, mime, sounds
involvement
 Trying to invent foreign sounding versions of first language
words;
 Trying to create new words

1.3.4.3.4. Writing
Writing practice has two quite different uses in a language
learning programme:
transparency clarity of
(a) Consolidation for language development.
rationale,
(b) Development of writing skills.
clarity of aims
Writing skills can also be seen from two quite different
perspectives, in relation to:
relevance
(a) the acquisition of the rhetoric and socio-linguistic conventions Learner
associated with important, relevant genre (e.g. business Centredness,
letters) accountability
(b) the development of an ability to interact with others, to reflect
and express oneself in a creative and personal way
textual integrity
Writers have to organise their ideas in a systematic way and have reliability
ling. integrity
to be precise to make themselves understood because they cannot
take so many things as "givens" in the context. In addition to this,
spelling and grammar mistakes are more obvious in a written than reliability
Ling. integrity
in a spoken text. Writing in a foreign language thus requires textual integrity
planning at the beginning and constant reviewing of expressions,
content and organisation before the text is given its final form. relevance
appropriateness
The degree of assistance learners receive for content, organisation flexibility
and language depends on the needs and linguistic level of the individualisation
target group. Learners are often guided from controlled practice to
free production, but with some groups of learners this order of
activities might be reversed.

1.3.4.3.4.1 How does writing practice help learners to develop an ability


to express themselves precisely in well-structured language?
41

1.3.4.3.4.1.1  Do programmes / materials offer a variety of relevant genres as generativeness transferability


models? attractiveness variety
relevance accountability
 Letters
reliability textual integrity
 Notes
 Narratives
 Descriptions
 Texts presenting an argument
 etc.
language
generativeness
awareness
 Do programmes / materials include activities that help learners
1.3.4.3.4.1.2
identify relevant features of specific text types?
 Openings and closure of letters, essays etc.
 Linking: cause and effect; temporal; logical;
 Structuring description and expository texts: contrasting
highlighting key points
 Sequencing and developing points; integrating sub-points
and examples
etc.
1.3.4.3.4.1.3 Do the programmes / materials include activities to assist learners
to:
 Structure their ideas for a text?
reliability methodological.
 Brain-storming and mind-mapping integrity
generativeness
 Flow charts integration,
cog. development
 Tables
 Sets of questions
 Establish coherence and cohesion?
1.3.4.3.4.1.4
 Activities to help recognition of signposting and use of
reliability
cohesive devices methodological.
generativeness
integrity,
 Activities to practice selection of appropriate linking words
and cohesive devices (e.g. gap fill texts, multiple choice language
cloze etc.) awareness
 Activities to practise production of coherent, cohesive text
(e.g. sentence combination, paragraph writing, guided
composition from notes)
1.3.4.3.4.1.5  Formulate precisely and correctly what they want to say?
generativeness
 Activities such as looking up the meaning of new words
language
 Finding an appropriate synonym or opposite
awareness
 Finding an appropriate idiomatic expression
cognitive
reliability
 Checking grammar by using of a reference book development
generativeness
 Review and edit texts?
1.3.4.3.4.1.6  Activities to check logical structure and development of methodological.
42

ideas (e.g. flow chart) integrity,


 Activities to encourage monitoring grammar, spelling etc.
(e.g. checking for "favourite mistakes")

1.3.4.3.5 Listening
Understanding spoken language requires interaction between the
listener's knowledge of the world and expectations (schemata) and
generativeness integration
the information being transmitted by the speaker. The listener has
first to recognise sounds and identify the meaning of words and
sentences. On the basis of their existing knowledge and
experiences the listener then forms hypotheses of what is going to
be said next and adjusts these hypotheses constantly as new reliability methodological.
information comes in. Non-linguistic cues and prosodic elements, integrity
e.g. intonation, stress, rhythm, background noises and the
situational context help the listener in cases of ambiguity and
uncertainty.
1.3.4.3.5.1. How does listening practice help the learner to decode and
chunk speech under different conditions?
Do programmes / materials provide:
 Exercises which help the learners to automatise decoding
processes?
1.3.4.3.5.1.1. reliability methodological.
 Training in recognising sounds, minimal pairs etc. integrity
 Training to help learners "chunk" incoming text
 Training to compensate for interference, background noise,
poor fidelity
generativeness
integration
 Pre-listening activities to activate existing knowledge? reliability cog. Development
1.3.4.3.5.1.2. methodological.
 Activities to demonstrate the importance of inferring and
Integrity
anticipation
practicality
 Activities to identify the purpose of the listening and select
an appropriate "while listening" task
 Tasks to help learners to activate a "frame" for the situation
reliability
in order to form hypotheses
generativeness methodological.
Integrity
practicality
integration
cog. Development
1.3.4.3.5.2 How does listening practice help the learner to understand
messages from a variety of sources?
43

Do programmes / materials provide:


1.3.4.3.5.2.1  An appropriate range of authentic varieties?
 Formal genres Reliability ling. Integrity
 Colloquial chat Relevance Accountability
 Dialect variation
 Individual inflections
 Pre-listening activities to set a purpose for listening?
1.3.4.3.5.2.2 Method. integrity
 To catch specific information
reliability
 To form an overview of content
 To identify an argument, conclusion
 To form an impression of the speaker
 While-listening activities to check understanding?
1.3.4.3.5.2.3  Marking intonation cues to support inferring
 Information transfer / labelling tasks
reliability
practicality
 Re-ordering a picture story sequence
 Filling in a flow chart / labelling a diagram

 Post-listening activities to evaluate and/or discuss the content?


cog. Development,
1.3.4.3.5.2.4  Activities in which learners are asked to express their integration,
generativeness
opinion or feelings in reaction to a listening text transferability
participation personal interest
 Use of the listening text as the basis for the production of a
text of their own
 Use of the text as a basis for discussion

1.3.4.3.6 Reading
There are certain similarities in the processes involved in listening generativeness integration
and reading, both being types of reception. However, the fact that
generativeness cog. development
the whole text is available makes reading a more conscious
activity, with a wider variety of strategies with which to approach generativeness cog. development,
a text for different purposes. Given the attainment of a certain transferability
linguistic threshold, a good first language reader can be
encouraged to transfer such strategies.

1.3.4.3.6.1 How does reading practice help the learner connect what they
are reading to previous knowledge and activate appropriate
strategies to understand the text?
Do programmes / materials provide:
1.3.4.3.6.1.1  Pre-reading activities to activate previous knowledge? generativeness Integration
cog. development
 Tasks to encourage inferring and anticipation
reliability methodological.
 Tasks encouraging the learner to exploit cues in layout, integrity
titles, paragraphing, , source of text, etc. to form inferences
44

 Pre-reading activities to set a purpose for reading?


1.3.4.3.6.1.2.  To skim through the text to get a overview of its contents
 To read the abstract or conclusion to get an overview
 To scan the text to find specific information generativeness cog. development
methodological.
integrity
reliability
 While-reading activities to aid understanding?
practicality
1.3.4.3.6.1.3  Pointing out clues anticipating what is coming next, e.g.
after conjunctions such as but, although, nevertheless, etc.
 Underlining key words/phrases to help the learner monitor
and revise initial hypotheses
 Making notes of salient points
 While-reading activities to test comprehension generativeness
 Comprehension questions (open, true/false/not given) reliability integration,
1.3.4.3.6.1.4 cog. development
 Matching tasks, selecting a title for a paragraph
methodological.
 Information transfer tasks
integrity
practicality
 Post-reading activities to evaluate and/or discuss the content
reliability
1.3.4.3.6.1.5
 Use of the text as the basis for the production of a text of
generativeness
their own
methodological.
 Use of the text as a basis for discussion integrity
participation practicality
cog. development,
integration,
transferability
personal interest
1.3.4.3.7 Viewing
In viewing, the learner has to process an auditory and a visual
input at the same time, for example when watching a film.
”Viewing” (or ”audio-visual-reception”) also includes the ability
to interact with and use new technologies, such as multimedia,
CD-ROMs etc.
Video can be exploited autonomously, as additional material for a
general course, or as the focus of a language course which is
designed around the visual medium.
45

1.3.4.3.7.1 How is viewing exploited to add more than what is offered by


listening?
CORE INDICATORS
1.3.4.3.7.1.1  Does the use of video sequences or multimedia serve a clearly transparency Clarity of Aims
recognisable purpose within the design of the programme / reliability consistency
material? efficiency ergonomy
reliability internal coherence
 Video practice integrated into mainstream material /
textbook attractiveness user-friendly
sensitivity
 Video used as vehicle for the input of new language
practised in the mainstream material
 Video used to inject a piece of authentic life into the
materials
 Video used to stimulate interest and motivation

1.3.4.3.7.1.2 Do programmes / materials include:


integration,
generativeness
 Pre-viewing activities to activate previous knowledge? cog. development
methodological.
 Tasks to encourage inferring and anticipation reliability
integrity
 Tasks encouraging the learner to exploit paralinguistic clues
- e.g. video without sound

1.3.4.3.7.1.3  Post-viewing activities to evaluate and/or discuss the content?


reliability
methodological.
 Use of the video as a basis for discussion generativeness
integrity
 Use of the video as the basis for the production of a text integration,
transferability
participation
cog. development
personal interest
1.3.5 Pedagogic Tasks
Tasks can be thought of in terms of pedagogic tasks and real-life
tasks. This section concerns pedagogic tasks.
Pedagogic tasks: are tasks which have pedagogic rather than real
life objectives as their object.
Exercises are pedagogic tasks which serve the purpose of
exploring, consolidating, and automatising language learning.
Generally exercises are closely guided, and involve reproducing
language presented.
The traditional sequence of pedagogic tasks is from controlled to
free: (a) presentation, (b) repetition, (c) practice, (d) transfer, (e)
free practice in a more holistic activity . The alternative sequence
is: (a) undertake holistic activity, (b) feedback on performance,
with presentation of relevant language; (c) practice; (d) transfer;
(e) next holistic activity.

Tasks have to be practical and feasible within the given context relevance appropriateness
and have to be designed in such a way that instructions are clear reliability practicality
and precise. Institutional or other constraints such as length of
efficiency ergonomy
lessons, equipment, etc. also need to be taken into account.
46

1.3.5.1. What steps are taken to ensure that there is an appropriate


balance of controlled, guided and free tasks?
1.3.5.1.1  Are controlled practice exercises provided which are linked to reliability methodological.
language presentation. generativeness integrity
integration
 Matching exercises
 Sentences with gaps
 Open dialogues
 Short gapped texts
 Etc.

 Is there "fluency practice" to aid automaticity? reliability


1.3.5.1.2
methodological.
 Pair dialogues with prompts
integrity
 Grammar games
 Communicative drills
 Conventional repetition drills
 Etc.

 Are there transparent links between controlled, less controlled reliability


1.3.5.1.3.
and free practice to encourage transfer? generativeness
transparency methodological.
integrity
 Linked sequence of classroom tasks
transferability
 Links to current language focus in skill practice materials clarity of
presentation
 "Homework" personal tasks to encourage / check transfer
 Etc.

 Are there more holistic communicative activities in which


learners combine and use actively the language and skills reliability
1.3.5.1.4
which form the objectives for the unit? generativeness
 Information-gap, opinion-gap activities
methodological.
 Games integrity
transferability
 Consensus tasks
 Etc.

 Is it possible to vary the sequence in which the tasks are used?


 Presentation - controlled - guided -free flexibility
attractiveness
 "Deep-end": free - presentation controlled - guided
1.3.5.1.5
adaptability
variety
47

1.3.6 ” Real life ” Tasks and Projects


”Real life tasks” are a way of bridging the gap between the
learning context and the use of the foreign language in the real
world. Genuine or simulated ”real life tasks” demand of the
learners that they use the foreign language as they would do in generativeness transferability,
real life. cog. development
Projects offer the opportunity to work on a topic or a problem socialisation social skills
either inside or outside the classroom. Project work can help
overcome the gap between an artificial learning context and
reality, between traditional class work and practical use.
"Real life" tasks / projects can range from simulations and case
studies to "fieldwork" involving direct or virtual (Internet) contact
with native speakers.

1.3.6.1 What steps are taken to ensure that "real life" tasks and
projects help to bridge the gap between classroom practice and
real language use?

1.3.6.1.1  Do the tasks simulate language use in the real world? generativeness transferability
socialisation social skills
 Discussing how to approach a problem
 Gathering Information from written or recorded documents
 Gathering information from (native speaker) informants
 Presenting ideas and arguments in a meeting
 Presenting proposals and findings to an audience
 Reacting to ideas / proposals / findings etc.
 Etc.

 Are the learners involved in choosing, organising and participation partnership


1.3.6.1.2
evaluating the success of the real life tasks? transparency clarity about
achievement
 Options provided
 Information on logistical and organisational needs
provided
 Possible evaluation criteria provided for finalisation
with learners
 Self-assessment checklists

1.3.7 Learning Strategies


Successful learning strategies enhance the learning process, the
retention and the retrieval of the information learned. Strategy
training can
1.3.7.1 generativeness cog. development
 Help learners to become aware of the strategies they use;
1.3.7.2
48

1.3.7.3  Introduce them to a wider range of options; efficiency ergonomy


 Increase the effectiveness of their selection of a strategy transparency Clarity of Aims,
which works for them in relation to the task concerned. clarity of
presentation
The function and appropriate use of a new strategy should be
explained explicitly to facilitate transfer. Sufficient practice will
help the replacement of current less efficient learning habits with reliability
methodological.
more productive ones.
integrity

1.3.7.3.1 What has been done in order to stimulate the use and training
of strategies?
Do programmes / materials include
1.3.7.3.1.1.  Procedures to find out about existing strategies?
 Teacher observation checklists generativeness cog. development
 Questions to help learners reflect on current strategies transparency clarity of
presentation
 Questions to help learners make links to other skills and
forms of learning
 Learner diaries,
 Class discussions
 Etc
 Suggestions for using and/or training new strategies?
1.3.7.3.1.2.
cog. development
generativeness
 Reading differently for different purposes adaptability
flexibility
 Ways of using radio, TV, songs to help listening
 Ways of being active in the learning process
 Ways of memorising in a differentiated way to help
retrieval (key word techniques, imaging, associograms,
making diagrams, making tables, hierarchical diagrams)

In relation to new strategies, do programmes / materials transparency clarity about


achievement,
1.3.7.3.1.3  Demonstrate the possibilities and limitations of new strategies
clarity of
presented?
presentation
 Descriptions
 Examples of use
 Identification with particular purposes and contexts
 Identification with learning styles

 Give ample opportunity to encourage and practise new


1.3.7.3.1.4. reliability methodological.
strategies?
integrity,
practicality
generativeness
 Practice integrated in activities without being made explicit cog. development
 Overt presentation and practice
 Homework tasks (including exploiting media)
49

 Rewards for use of strategies

1.3.7.3.2 What has been done in order to help learners find out how,
when, where, how often they learn most efficiently?
CORE INDICATORS
Do programmes / materials: generativeness cog. development
participation partnership
1.3.7.3.2.1  Encourage learners to set themselves objectives?
 Presentation of ways of personalising content being learnt
 Provision of schemes, charts, indexes to help planning
 Suggestions on setting priorities
1.3.7.3.2.2  Introduce learners to self-assessment techniques?
generativeness cog. development
 Observation activities participation partnership
transparency clarity about
 Scales and checklists to rate performances
achievement
 Comparison of self-assessment with performance on an
actual activity
 Encourage learners to try out different learning techniques and
1.3.7.3.2.3
modes to find those that suit them best?
 Wide range of techniques presented industrialisation
flexibility
cog. development
 Linking of techniques to learning styles generativeness
partnership
participation
 Feed-back suggesting an appropriate pathway through the
programme / material
 Opportunities for discussion and reflection

1.4 Organisation
Organisation refers to the structure of programmes / materials. It
describes the principles according to which the content is
organised throughout the programme / material (Curriculum
design), and the organisation of contents within a unit (Unit
design).
Organisation concerns the co-ordination of Content and Activities
and is signalled through Presentation. Essentially, Organisation
involves:
- Chunking: grouping small bits into meaningful elements, and/
or breaking down large areas into elements
- Integrating: matching activities and content; mixing and
synthesising different strands of content and different types of
experience
- Sequencing: presenting things in a logical, flexible order
- Referencing: showing how parts relate to each other and to
the whole.
50

1.4.1 What steps have been taken to ensure that chunking and
integrating is successful?
1.4.1.1  Has the adequacy of connections and sequences been checked? Reliability Internal coherence
 Organisation of content and activity specifications into
hierarchical lists
 Classification of exercises and activities with categories
from the content and activity specifications
 Use of a database to classify elements in order to identify
potential links
 Use of a flowchart during authoring to plot connections
graphically
 Has the plausibility and acceptability of matches between
1.4.1.2 Reliability Methodological
content and activity or of media and skill been checked?
integrity
Relevance
 Workshops brain-storming matches and connections
Appropriateness
 Consultation with teachers/ learners
1.4.1.3
 Has the feasibility of the size of chunks and sophistication of
Practicality
the matching been checked in relation to cognitive load? Reliability
Appropriateness
 Consultation with teachers/ learners Relevance
 Observation of pilot use

1.4.2 What steps have been taken to ensure that important elements
have not been omitted?
1.4.2.1  Have exercises and activities been cross-referenced to Reliability Consistency
categories from the content specifications?
 Manual collation
 Use of database sorting, to identify inadequate coverage.
 Consultation with teachers/ learners
 Have vocabulary glossaries, grammar summaries, lists of
1.4.2.2. Reliability Consistency
functions, been checked to ensure that all exponents occurring
are included?
 Manual collation
 Use of a database sorting to identify inadequate coverage.
 Consultation with teachers / learners
1.4.2.3. Curriculum design
The term curriculum design is generally used to refer to the
definition of appropriate objectives at successive levels, the
organisation of those objectives into strands, the integration of the
strands into didactic units, the provision of suitable activities and relevance appropriateness
materials to practise and learn the knowledge and skills
flexibility individualisation
concerned, and the assessment of the extent to which the
adaptability
objectives are mastered.
appropriateness
The organisation of the curriculum is dependent on the conditions relevance
cost-effectiveness
and constraints of the learning context. Issues to be considered efficiency
51

include: cost-effectiveness
- the time and resources available for the course as a whole, for
each strand and for each lesson;
relevance
- the rhythm and intensity of the course (2 hours a week / 20
hours a week) transparency
Clarity of Aims,
- the roles of different learning modes (class work, homework, clarity of rationale
media-led self study, project work, etc).
reliability
To help learners to orient their learning, set priorities and made consistency
informed choices, the principles and objectives underlying the
curriculum should be made transparent and applied in a consistent
way.
1.4.2.3.1. What has been done to ensure that the curriculum design is
transparent and coherent?

1.4.2.3.1.1.  Does the curriculum present and describe an adequate set of Transparency Clarity of Aims
Levels/Stages?
Reliability Internal coherence
 Objectives for the different Levels/Stages developed from
Relevance Learner-
the actual achievement of the learners concerned
centredness
 Workshops undertaken to ensure that teachers/learners can
Appropriateness
see the difference between Levels or Stages
Accountability
 Progress traced to check the Stages/Levels represent a
development path
 Does the curriculum present and describe an adequate set of Clarity of Aims
1.4.2.3.1.2. Transparency
Content Strands?
Appropriateness
Relevance
 Theoretical basis to the selection and grouping
Method. Integrity
Reliability
 Consultation of models (e.g. the Common European
Framework)
 Feedback to check that teachers/learners can understand and
work with the categories

1.4.2.3.2. What has been done to ensure adequate and appropriate


progression in skills and content?
1.4.2.3.2.1.  Does the progression in language points have an adequate Reliability Consistency
basis?
Method. Integrity
 Reference to research on the order in which people learn
language points (Second Language Acquisition research)
Relevance Learner-
 Analysis of what language points learners need to use or
centredness
understand to complete tasks necessary at particular levels Transparency
Clarity about
 Analysis of student speech and / or written work to see if Relevance
Achievement
content presented is actually being learnt
Reliability
Appropriateness
 Consultation with teachers about the pace of progression
Transparency
Practicality
 Investigation of progress tests to check difficulty of
language points in relation to range of ability in the target Clarity about
1.4.2.3.2.2. group Reliability achievement
52

 Does the progression in skills have a adequate basis Relevance Consistency


Method. Integrity
 Logical development of cognitive complexity
Learner-
 Reference to relevant research into task difficulty Reliability
centredness
 Analysis of student speech and / or written work to see if the Appropriateness
tasks are practical in the context and at the level concerned
Practicality
 Consultation with teachers about the cognitive load of tasks,
1.4.2.3.2.3.
and the development from simpler to more complex tasks
 Is there appropriate recycling and extension? Is there a focus
on learning and expanding competences rather than racing Generativeness Integration
through a checklist of content? Transferability
 Systematic provision of consolidation points
 Spiralling development integrating and elaborating
grammatical exponents introduced at a lower stage
 Different strands woven together so that exponents, skills
and themes are consolidated in different combinations
 Extension opportunities offered to stronger learners at
recycling and consolidation points
1.4.2.3.3. What has been done to ensure that the curriculum design is
appropriate to the target group?

1.4.2.3.3.1.  Have steps been taken to check that the learning load and Reliability Practicality
number of exponents offered is realistic and challenging?
 Consultation with teachers
 Piloting on groups representative of the range of ability
 Investigation of workload in lesson phases, at home
 Teacher feedback
 Learner feedback
1.4.2.3.3.2.
 Have criteria such as frequency of use, a wide range of
Relevance Learner-
possibilities for application, etc. been taken into consideration
centredness
to restrict the learning load and set priorities? Transparency
Clarity of Aims
 Low priority exponents been cut out or relegated to a
reference section
 Core practice and extra practice separately identified
 Consolidation exercises on core content for work at home
provided
 Have exponents been explicitly marked as to their importance
Learner-
1.4.2.3.3.3. for reception, production or their usefulness for examinations? Relevance
centredness
 Systematic exposure to exponents learners should recognise Accountability
but not necessarily master actively
 Identification of aspects relevant for examination
 Is the language used to describe or explain rules and structures
1.4.2.3.3.4. Relevance Appropriateness
adjusted to the knowledge of the target group?
Reliability Practicality
 Consultation with teachers
53

 Consultation with learners


 Investigation of student work to identify cases in which
misunderstandings occur repeatedly
1.4.2.3.4. What has been done to ensure that the curriculum suits the relevance Appropriateness
pedagogic culture of the learning context?
Relevance Appropriateness
1.4.2.3.4.1.  Do the organising features consider language learning and
teaching traditions in the region or sector concerned?
 Teachers been involved in any adaptation
 Piloting in context
 Feedback from teachers and learners
 Does the approach chosen take the conditions and constraints
relevance Appropriateness
of the institutional context into account?
 Consultation with authorities
 Consultation with teachers
1.4.2.3.4.2.
 Piloting in context
 Feedback from teachers

1.4.2.3.5. What steps are taken to ensure that the relevance of an activity transparency Clarity of Aims
to the objectives is clear to the user?
clarity of
presentation
Transparency Clarity of Aims
1.4.2.3.5.1.  Are learners given an overview of the curriculum as well as the clarity of
information they need for the purpose at hand? presentation
 Explanation of each content strand
 Overview offered of each Level/Stage
 Overview of the Levels/Stages as a whole
 Is there an explanation of how the organisation of the course Clarity of Aims
1.4.2.3.5.2. Transparency
serves the achievement of the learning objectives in question?
clarity of
 Table of contents showing objectives presentation
 Introduction explaining content strands and objectives
 Summaries at key points to aid self-assessment

1.4.2.3.6. What steps are taken to ensure that the organising principles transparency Clarity of Aims
are clear to the teacher? clarity of
presentation

1.4.2.3.6.1.  Are teachers given an adequate overview of the curriculum as Transparency Clarity of Aims
well as the information they need for the purpose at hand?
clarity of
 Explanation of the content strand presentation
 Overview of the Levels/Stages as a whole
 Is there adequate documentation at differing levels of detail for
54

1.4.2.3.6.2. different purposes? Transparency Clarity of Aims


 Overview documents clarity of
presentation
 Complete curriculum
 Level / Stage summaries
 Is there an adequate "How to" Teachers' manual? Transparency
Clarity of Aims
1.4.2.3.6.3.  Explanation of the curriculum structure
Clarity of rational
 Explanation of processes,
 Notes on each teaching unit
 Suggestions for preparation / follow-up
 Suggestions for simplifying / elaborating
 Theoretical rationale
 Is there adequate teacher training? Reliability
Consistency
1.4.2.3.6.4.  Introduction course Transparency
Meteorological
 Ongoing liaison with other teachers
integrity
 Regular discussion of objectives and methodologies
Clarity of Aims
 Regular discussion on how to use and adapt additional
”external” material
 In-service workshops
 Opportunities to exchange experience with teachers in
similar programmes
1.4.2.4. Unit design
A unit is a self-standing, balanced component of a course.
Units help to group aims in a coherent way. These aims should
reflect the overall course design. In some courses, particularly for
extensive or distance teaching, a very consistent structure may be
preferred - with an identical pattern to the units. On the other
hand, different units may reflect or combine the strands of the
course in different ways and thus have a different focus.
Although activities, topics and exponents should be related to each
other to provide coherence, the design of a unit should be open
and flexible enough to allow teachers and/or learners to change
the plan of action spontaneously. There will be a need to introduce
extra language points or skip over other content to cater for the
interests of a particular target group.
In order to help learners to become autonomous they should be
trained to analyse their own progress and the efficiency of
methods and materials. Accordingly, they should be given
possibilities for self-assessment, i.e. comparing aims with their
learning results.

1.4.2.4.1. Which steps ensure that the organisation of the units meets the relevance Appropriateness
conditions and constraints of the learning context?

 Have the conditions of the specific learning context been


55

1.4.2.4.1.1 investigated? Relevance Appropriateness


 Consultation with authorities
 Elicitation of information with a questionnaire
 Consultation with teachers
 Observation of lessons
 Has the organisation of the units and lessons been tailored to
that context?
1.4.2.4.1.2. Relevance
 Frequency of lessons
Reliability Appropriateness
 Length of lessons
Practicality
 Availability of equipment
 Size of classes

1.4.2.4.2. What has been done to make the structure and the objectives transparency Clarity of Aims
of the units transparent to the learners? clarity of
presentation
Transparency Clarity of Aims
clarity of
1.4.2.4.2.1.  Are learning objectives stated explicitly for each unit?
presentation
 Appropriate title, headings and sub-headings
 Objectives listed in the table of contents
 Index / Grammar summary / Word list at the end
1.4.2.4.2.2.  Are units easy to distinguish from each other? Transparency
Clarity of Aims
 Beginning and end clearly marked clarity of
presentation
 Standardised presentation

1.4.2.4.3. What has been done to ensure that learners' differing level and flexibility individualisation
learning styles are catered for? adaptability

1.4.2.4.3.1.  Is the structure of the unit open enough to expand or reduce Flexibility Adaptability
content according to the needs of a particular target group?
 Optional exercises
 Core exercises identified
 Easier exercises first
 Suggestions for supplementary activities
 Are different learning modes catered for?
1.4.2.4 3.2. Flexibility individualisation
 Range of inputs (audio, print, illustration, video)
 Range of text types (story-based; expository; interactional)
 Range of activity types (accuracy-focus; fluency focus)
 Balance of conscious practice / games
56

1.4.2.4.4 What steps are taken in the unit design to promote


transferable learning?

1.4.2.4.4.1.  Are there structured steps between presentation - controlled Generativeness Integration
practice - free practice (and/ or the reverse) to foster
acquisition of new exponents?
 Controlled-Guided-Free practice
 Careful slow practice - Fast spontaneous use
 Focus on form through pre-activities and notification of
post-activities
1.4.2.4.4.2.  Are there suggestions for transfer activities, for personalisation Generativeness Transferability
of practice, taking ownership of new language?
 Exploitation of students' experience
 Personal transfer activities
 Suggestions for extra-curricula practice
 Collecting examples, keeping a scrapbook, preparing a
portfolio
1.4.2.5. Interaction Management
The type of interaction generated should take into account the Flexibility individualisation
learning experience of the learners, the teaching experience of the attractiveness sensitivity
teachers and the approach to language learning being applied, as
reliability consistency
well as the learning objectives.
An appropriate balance should be struck between the main
interaction modes: participation partnership
 teacher-directed class work; socialisation social skills
1.4.2.5.1.
intercultural
 interaction in the target language in small groups
1.4.2.5.2. awareness
 pair work
1.4.2.5.3.
 individual work
1.4.2.5.4.
 presentation by learners.
1.4.2.5.5.

1.4.2.5.5.1. What steps ensure a positive balance of activity? Attractiveness sensitivity


participation partnership
socialisation social skills
1.4.2.5.5.1.1.  Has an appropriate balance been struck between teacher- Attractiveness Variety
directed class work, interaction in the target language in small participation partnership
groups, pair work, individual work, and presentations by socialisation social skills
learners?
 Suggestions in teachers' notes
 Use of observation forms and checklists to log and analysis
instances/time taken in different kinds of interaction using
the programme / materials
 Explanation to students of the importance of group work
 Training of teachers in organising and monitoring group
work
57

 Does the emphasis in exercises, tasks and games encourage


co-operation rather than selfish competition?
 Well-conceived activities for pair work
1.4.2.5.5.1.2.
 Opportunities for group problem-solving (language
Socialisation Social skills
problem/puzzle or discussion task)
Participation Partnership
 Opportunities for a"hot line to teacher/strong colleague" to
help a weaker learner catch up
 Creative group work
 Is there a clear and differentiated policy on correction which is
effectively communicated so as to encourage learners to focus
on fluency on some occasions?
1.4.2.5.5.1.3.
 Clear guide-line: accuracy-focus / fluency-focus
Transparency Clarity of Aims
 Suggestions for teacher monitoring of fluency practice
Clarity of
 Use of video
achievement
 Teacher training on constructive feedback

1.4.2.5.5.2. What is undertaken to ensure that teachers with less Reliability Practicability
experience or ability will be able to cope with the demands on
classroom management posed by the type of interaction
demanded?

1.4.2.5.5.2.1.  Is there sufficient guidance provided to teachers? Reliability Consistency


 Teachers' notes Methodological
Integrity
 Opportunity to discuss techniques
 Opportunities for peer observation and feedback
Practicality
1.4.2.5.5.2.2.  Has the programme/material been tested on a representative Reliability
group of learners?
 Informal Trialling by a few teachers
 Formal piloting
 Formal piloting with addition of comments, tips and notes

1.4.2.6. Assessment
Assessment in the Design Phase is taken to refer to procedures and
materials that are designed to facilitate effective learning.
Assessment of the product of learning is considered in Outcomes.
Techniques to monitor learning during lessons and give
appropriate feedback are considered in Implementation.
The aspects of assessment considered here therefore concern:
 Placement testing, to ensure that learners receive a
1.4.2.6.1.
programme appropriate for them
 Diagnostic testing, to identify which particular skills and
1.4.2.6.2. language points need to be addressed
1.4.2.6.3.  Achievement testing, to motivate revision, consolidate
58

learning and monitor progress during the course.


In a learner-centred approach, assessment is carried out by the
teacher and the students alike as the ability to assess oneself is an
important step in the direction of autonomous learning.
1.4.2.6.3.1. What is done to ensure that assessment is conducted in a reliability Methodological
systematic way? integrity
consistency
attractiveness variety

1.4.2.6.3.1.1.  Is a placement test used to inform optimal matching of Reliability Consistency


resources to needs?
Efficiency Ergonomy
 Validated entry test
 Sample of communicative performance
 Self-assessment
 Do placement tests also fulfil a diagnostic function?
1.4.2.6.3.1.2. Relevance Learner
 Mistakes below current level logged for remedial practice Centredness
 Mistakes collated by grammatical themes for a group Individualisation
Flexibility
 Profile reported from interview (e.g. Range, Accuracy,
Fluency)
 Reading / listening text with simple "True / False" questions

 Is there an assessment for consolidation at the end of each


Clarity of
1.4.2.6.3.1.3. unit? Transparency
achievement
 Vocabulary quiz Reliability
Consistency
 Reading / listening text with simple "True / False" questions
 Reinforcement of new grammar
 Test constructed from homework mistakes
1.4.2.6.3.1.4.  Is there more holistic assessment at certain points during the
Clarity of
course?
Transparency Achievement
 At the end of a group of units forming a module
 At key points in the course / year (half term; end term)
 Is there an appropriate balance of assessment activities?
1.4.2.6.3.1.5. Attractiveness Variety
 Performance as well as knowledge
Reliability Internal coherence
 Spoken as well as written
Methodological
 Natural open-ended expression as well as narrowly integrity
controlled standardised items
 Student-student interaction (or media-led) as well as
teacher-student
 Self-assessment (with checklists, rating scale) as well as
Teacher assessment
59

1.4.2.6.3.2. What steps are undertaken to ensure that the assessment Reliability Methodological
procedures are valid? integrity

1.4.2.6.3.2.1.  Do assessment procedures mirror the learning objectives of Reliability Internal coherence
the programme/material? (content validity)?
 Content recently learnt rather than tricky questions
 Activities as well as content
 Weighting in relation to priorities in objectives
 Do assessment procedures reflect a theoretically grounded
1.4.2.6.3.2.2. Reliability Methodological
view of the nature of knowledge and language learning which
integrity
underpin the programme / material (construct validity)?
 Receptive, interactive and productive uses of language
 Meaning in context
 Underlying competences (linguistic usage; socio-linguistic
appropriateness; discourse organisation)
 Performance features (fluency, flexibility, interactional
ability, control)
 Are the results fair and consistent (reliability)
Accountability
1.4.2.6.3.2.3. Reliability
 Statistical analysis for tests which affect learners' futures
Ergonomy
Efficiency
 Standardisation training for teacher judgements

1.4.2.6.3.3. What steps are taken to involve learners into the assessment generativeness Cognitive
process? development
participation
sharing
responsibility

1.4.2.6.3.3.1.  Are learning objectives made transparent to the learners? Participation Involvement
 Summary of objectives and assessment criteria given to Partnership
learners
 Objectives explained at beginning of unit / lesson
 Learners involved in constructing test
 Are learners asked to record and contribute their reactions and
1.4.2.6.3.3.2. Participation Involvement
experiences?
Partnership
 Feed back feelings about materials / programme
 Maintain a diary
 Formulate a personal project
 Are there awareness-raising activities
1.4.2.6.3.3.3. Generativeness Cognitive
 Codes for classifying mistakes in written work
development
Participation
 Questionnaire-led discussion of strengths and weaknesses
Involvement
 Observation of neutral video recordings and discussion of
Partnership
performance
60

 Observation of videos of self / own group and discussion of


performance
 Are there opportunities for self-assessment?
1.4.2.6.3.3.4. Participation Involvement
 Global self-assessment onto a rating scale
Generativeness Partnership
 Self-assessment on a checklist of the communicative
Cognitive
objectives
development
 Comparison of self-assessment with regard to a particular
task with actual performance in the task concerned
1.5 Presentation
Presentation refers to the packaging of contents and materials and
the effect this has on the user. The appearance of the
programmes / materials has at least two dimensions: an aesthetic
and a functional one. The first appeals to the senses of the
learners, in particular to their visual sense. Illustrations, lay out
and colours all add to the attractiveness of programmes / materials
and thus foster the motivation to use it. The functional dimension
covers the accessibility of the material, i.e. the way the learner is
guided and helped through it.
1.5.1 What steps have been taken to ensure an appealing
appearance?

1.5.1.1  Does the presentation take account of current, relevant Attractiveness User Friendliness
aesthetic norms and expectations? Sensitivity
 Use of guidelines (e.g. on mixing colours, fonts etc.)
 Development by trained professionals
 Consultation with focus groups: Do people like it?

1.5.2 What has been done to ensure that the programme / materials
are accessible and practical?

1.5.2.1.  Is the package of materials a practical size? Efficiency Ergonomy


 Production cost in context
 Number of different elements (not too many, not too few)
 Portability of different elements (weight, dimensions)
 Time required for familiarisation, for a session
1.5.2.2.  Can people cope with the level of complexity of the Attractiveness User Friendliness
presentation?
 User-friendly overview of stages and parts
 Effective use of layout, colour etc. to aid comprehensibility

 Do the different media used have an acceptable technical


1.5.2.3. Attractiveness User Friendliness
fidelity?
Efficiency Ergonomy
 Studio quality audio and video recordings
61

 Professional quality script writing and acting


 Sharply focused photographs
 Legible, attractive type fonts
 Stability of computer materials (e.g. automatic back-ups)

1.5.3 Illustration
In language learning, pictures, drawings, photographs and symbols
facilitate comprehension and provide a strong motivational
element by stimulating the learners' imagination, raising their
curiosity or addressing emotions.
For illustrations to be efficient they have to meet the function
which was assigned to them. Photos or drawings, for example,
which are meant to explain the meaning of a new item have to be
unequivocal, while a photo that is meant to make the learners talk
to each other should leave room for speculation. This implies that
pictures and texts should be matched clearly to avoid
misinterpretation. (Neuner 1996: 86).
One has to find equilibrium between what is familiar to the
learners and what seems to be ”foreign”. The target group needs to
be able to relate to the illustrations, but at the same time these
should raise learners' interest and curiosity. Illustrations should
avoid confirming clichés or prejudices.
1.5.3.1. What has been done to ensure that illustrations are relevance appropriateness
appropriate and effective?
reliability internal coherence

1.5.3.1.1.  Are the illustrations functionally effective and relevant to the


learning activities?
Reliability Internal coherence
 Pictures, graphics, animations, video clips etc. tested for
1.5.3.1.2.
didactic effectiveness
 Symbols/icons intuitive; user-friendliness tested
 Help always accessible to explain symbols/icons
 Are the illustrations appropriate to the age, cultural values, and
social background of the target group? Relevance Appropriateness
 Consultation on appropriateness of pictures, graphics,
animations, video clips etc.
 Adaptability to the learning style and level of the learner

1.5.4. Layout
The layout must meet both aesthetic and functional demands. It
should be easy to follow and clear, using visual and graphic
elements to highlight the start and end of a unit and to attract
attention to particular points. Ideally, the layout reflects the
principles underlying the organisation of the course. However, in
order to be effective it also has to meet the 'taste' of the target
62

group.

1.5.4.1. What steps have been taken to ensure that the layout of the
material has a relevant didactic function?

1.5.4.1.1.  Is there a clear link between the different components Reliability Internal coherence
 Text linked to appropriate pictures
 Symbols clear
 Is the layout designed so as to contribute to a more effective Reliability Internal coherence
realisation of the learning objectives?
1.5.4.1.2. Consistency
 For print: focal points highlighted
 For media: screen layout facilitates links; feedback and help
availability supports when requested but does not distract

1.5.5. Colour and music.


Colours can be employed to attract attention to specific points,
e.g. to focus on certain language points, or to illustrate the
structure of the material and thus help the learners to find more
easily what they are looking for. For example, the grammar
section of a book could be in a different colour from the activities.
Such distinctions should be self explanatory and consistent.
Music can help to structure activity, vary pace and intensity or
provide opportunities for relaxation.

1.5.5.1. What has been done to ensure that colours and/or music reliability consistency
facilitate the learning process in an appropriate manner? attractiveness sensitivity

1.5.5.1.1.  Are colours used in a consistent way Reliability Internal coherence


 To help navigation
1.5.5.1.2  To focus on a certain element
 Is the integration of music (if applicable) appropriate? Attractiveness User friendliness
 Tested out on target group
 Possible to turn it down or off
 Sound fidelity such as to enhance rather than detract from
experience
 Used consistently to structure the experience, mark a
boundary
1.5.6. Navigation
63

Navigation describes the means by which the learners are helped


to find their way through programmes / materials. Navigational
aides inform the user how to use the resources available.
Navigation can be divided into
 starting procedures (e.g. software installation);
 means for an overall orientation such as a table of contents or
a menu;
 information that is provided to guide the learners through
programmes or materials (clear signposting);
 different paths learners can choose according to their
individual learning types and styles;
 the possibility for the learner/teacher to vary the pace of
learning according to individual or group needs and
expectations.

1.5.6.1. What has been done to ensure easy navigation through the
material?
attractiveness user friendliness
flexibility individualisation
1.5.6.1.1.  Is the software provided with a paper manual which explains
reliability
installation and navigation procedures? internal coherence
 Explanation easy to understand
 In-built graphics to facilitate comprehension
 Is the installation procedure easy and intuitive? attractiveness
user friendliness
1.5.6.1.2.  Autorun
 Uninstall available
 Help-on function to solve installation problems
 Wizard GUI (Graphic User Interface) available
 Internet access procedures explained in the welcome/home
page
 Internet access procedures reliable and easy to follow
Attractiveness
 Is the software provided with an on-line guide? User friendliness
1.5.6.1.3.
 "Help" function easily accessible
 Sound and graphics facilitating the use
 Possibility of printing parts of the guide
 Guided tour function available
Transparency
 Is the table of contents/menu clear and understandable to the Clarity of Aims
1.5.6.1.4.
user?
 Is there an index and a contents menu?
 Is on-screen help available?
Attractiveness
 Are all relevant parts of the material / application easily User-friendliness
1.5.6.1.5. accessible in all phases of the learning process?
 Possibility to take over the programme from any point of
the programme (e.g. from the point reached in previous
64

sessions)
 Exit function at each point of the programme
1.5.6.1.6.  Are the navigation possibilities (if applicable) adaptable to Flexibility Individualisation
different learner types?
 Shortcuts available to pass easily from one unit or part of
the programme to others (i.e. without being obliged to
stick to a part or exercise till the end).
 Alternative (individual) routes through the material
foreseen.

1.5.6.1.7.  Is navigation supported/enhanced by appropriate use of Attractiveness User friendliness


symbols, icons, and colours?
Transparency Clarity of
 Buttons and icons cover all relevant functions
Presentation
 Function of buttons, menus and icons self-evident
User friendliness
 Buttons, menus and icons easily accessible Attractiveness
1.5.6.1.8. Practicality
 For multimedia language applications: Is a ”history” facility Reliability
(or similar accessories) implemented in order to prevent the
Internal coherence
1.5.6.1.9. user from getting ”lost in hyperspace”? Reliability
 Is the navigation consistent and coherent throughout the whole User-friendliness
Attractiveness
application?
1.5.6.1.1.0. Adaptability
 For multimedia: Is there an audio option? Flexibility
 Audio quality good
 Professional speakers
 Audio can be controlled by user.
 Colours and size of text can be altered by user
 Voice can be recorded.
 User’s recordings can be compared with a master example
(mother tongue)
 Differences between user’s performance and standard
performance shown (e.g. via spectrogram)
65

2. Implementation

IMPLEMENTATION
Quality in the implementation stage implies providing a positive affective atmosphere and a supportive
learning environment in which the realisation of the design is monitored and adapted flexibly to the
evolving needs of the learners.
During

This stage involves: affective management, monitoring activities, and support measures.
66

PRINCIPLE SUB-
PRINCIPLE
A perfect programme design is no guarantee of the
provision of a successful service. Designs have to
be realised in a context and there are three issues
central to the determination of the quality of that
realisation in that particular context:
- affective management
- monitoring
- support
A prerequisite for a successful implementation is a
consideration of the "innovation gap" implied by
the introduction of the programme / materials. The
extent of this gap will affect the initial balance in
the investment of resources and personnel. Where
the degree of innovation is relatively low, the focus
might be on monitoring. When the degree of
innovation is high, more resources will need to be
devoted to support systems. Whichever is the case,
there is a need for affective management.
2.1 What steps are taken to estimate the extent of
the "innovation gap" represented by the
programme / materials?

Participation Partnership
2.1.1  Have teachers been involved in the development
of the programme / materials?
 Teachers included in core development team
 A range of teachers consulted at key points
in the development
 Channels for teachers to contribute ideas
during the development process
Participation Partnership
 Have teachers been consulted in planning the
implementation of the programme / materials?
2.1.2
 Information sessions
 Workshops
 Working groups to suggest adaptations /
teachers' notes etc.
Relevance Appropriateness
 Were visits made to observe the current Reliability Practicality
situation prior to the introduction of the
2.1.3
programme / materials?
 Observation of classes /study sessions
 Evaluation of resources
 Discussions with managers and teachers
 Has a realistic budget for implementation been Efficiency Cost-Effectiveness
2.1.4 worked out?
 Teacher / learner familiarisation
67

 Teacher / learner training in innovative


aspects
 Environmental implications
 Equipment needs

2.2 Affective Management


One crucial aspect of communicative language
Relevance Learner
teaching, as described in Section E1 on Design, is
Centredness
the presentation of relevant language based upon an
analysis of learner needs. Such an approach has
been described as ”learner-centred”. It is,
however, fundamental to provide in addition an
Attractiveness Sensitivity
environment which lowers affective blocks to
Interactivity
learning and presents a stimulating achievable
challenge if language learning is to be successful.
Socialisation Intercultural
Successfully learning a language cannot be fully
Awareness
separated from socio-cultural factors and a
willingness to develop one’s own personality. Such
Participation Involvement
development can be perceived defensively as a
Partnership
threat to the existing personality unless steps are
taken to reassure the learner. Awareness of the
complexity of the psychological processes involved
in language learning and the need for (conscious or
subconscious) management of affective factors has
given rise to the expression ”learning-centred” to
describe approaches and techniques designed to
create such a positive affective atmosphere.
2.2.1 What steps are taken to ensure that learning is a Attractiveness Interactivity
stimulating, enjoyable, rewarding experience? Sensitivity

Attractiveness Sensitivity
2.2.1.1  Is the learning environment designed in such a
Efficiency Ergonomy
way as to make learners feel at ease?
Attractiveness Sensitivity
 Classrooms spacious and luminous
 Equipment designed in such a way that
facilitates group working
 In case of self-directed learning, quiet
environment
2.2.1.2  Is the approach used consciously learning- Attractiveness Sensitivity
centred, i.e. focused on creating a relaxed Participation Involvement
atmosphere for effective learning? Partnership
 Action taken early in the course to establish Socialisation Intercultural
a virtuous circle of enjoyment - success - Awareness
enjoyment – success
 Approach chosen avoids strictly prescribing
steps and reactions on the part of the learner
 Approach considers the learner holistically
as an autonomous personality

 Are teachers trained to be aware of affective


68

issues and to be capable of dealing with them in Attractiveness Sensitivity


a differentiated manner? Participation Involvement
2.2.1.3
Socialisation Intercultural
 Provision made to identify and help
Awareness
learners who are unsettled or unhappy
 Account taken of the ”difficult” learner
who rejects the affective approach
(considering it a ”wishy-washy” stuff)
 Learners helped to regard their culture in a
relaxed, affectionate, confident way which
allows them to experiment with becoming
slightly ”other
 Teacher accepts his/her role as a
“counsellor” rather than as a traditional
teacher

There are numerous ways in which affective


management could be classified, but most
techniques fall into one of the following five
categories: Centring, Matching, Stimulating and
Encouraging.
2.2.1.3.1. Framing and Centring
Whether one is talking about a piece of software, a
broadcast, a lesson or an intensive course, some
form of warm up activity serves to frame the
learning experience. Such activities often involve
an element of ritual repetition and often exploit
music or physical movement. They help to set the
learning experience apart from what has come
before and to help the learner to free his/her mind
and tune in.
In a group situation bonding or re-bonding of the
group is helped by an opportunity to chat, by ice-
breaker activities. Concentration is helped if
appropriate information (what this lesson is about)
and decisions (what shall we do next?) are shared
with learners. Finally, concentration is helped by
pauses for recuperation, by lighter activities
interspersed in the programme, by appropriate
variety of pacing and intensity in the exploitation of
the subject matter.
2.2.1.3.1.1 How is the start of a new unit or of a new phase
in a unit signalled? What steps are taken to
” frame ” the experience?

Clarity
2.2.1.3.1.1.1.  Are parts presented as distinct and yet related to Transparency
Presentation
each other? Reliability
Internal coherence
 Start of new units signalled
 Conclusion of an activity signalled (e.g.
concluding exercises, small tests, summary
etc.)
69

 Is the calming and reassuring nature of a


familiar a warm-up ritual exploited? Attractiveness
2.2.1.3.1.1.2.
Sensitivity
 Warm up activities used to help groups bond
 Relaxing activities
 Musical introduction
 Ice-breaking stories or jokes
 Is a sensible balance struck between familiarity
and novelty? Generativeness
 Repetition used to provide security Attractiveness Integration
2.2.1.3.1.1.3. Variety
 Slight variations introduced to provide
novelty

2.2.1.3.1.2. Is variable pacing signalled: times to concentrate


hard and work, times to be open, receptive and
let things happen?

2.2.1.3.1.2.1.  Is the pace and intensity of learning activity Attractiveness Variety


varied in a consistent, systematic manner? Flexibility Adaptability
Techniques provided to help the teacher /
learner to change working rhythms
 Pauses (chatting, listening to music,
relaxing) foreseen as a part of the
teaching/learning process
Flexibility Adaptability
 Is a sensible balance struck between times to
2.2.1.3.1.2.2. Attractiveness Variety
concentrate and times to be open and receptive?
 Games, puzzles etc. used to break phases
 Extensive listening/reading activities used
for interrupting practice
 Video exploited in this way
Songs exploited in this way
 Is account taken of learners with a short
attention span?
2.2.1.3.1.2.3. Flexibility Individualisation
 Large number of different activities Participation Involvement
provided
 Activities designed in order to actively
involve the learners
 Is account taken of talented learners who surge
ahead? Flexibility Individualisation
Adaptability
 More challenging version of the task made
2.2.1.3.1.2.4. available if appropriate
 Follow up activities or references (e.g. to
books, CDs, WWW references)
2.2.1.3.1.3. What steps are taken to effect a neat closure?
70

2.2.1.3.1.3.1.  Is there conscious closure - as opposed to an Transparency Clarity of


unplanned end of unit / lesson? Presentation
 Reflection activity provided at the end of
each unit or group of units
 More conventional closure (quizzes,
progress tests) provided
Transparency Clarity of
2.2.1.3.1.3.2.  Are steps taken to ensure that all the learners are
Presentation
at a coherent stopping point at the end of the
unit / lesson?
 Particular symbols or actions used
consistently to signal new activity, changes
in phase or pacing, and closure
 Music used in signalling
 Opportunity for physical movement between
phases
2.2.1.3.2. Matching
Design decisions concerning specification and
presentation of content and activities are generally
made in relation to the target group rather than the
individual learner. The matching of topics,
activities etc. in a course to the needs and interests
of, say, 14 year old Irish secondary school children,
and the matching of particular activities to
particular linguistic themes, may happen at the
Design Stage of producing a new course book.
However, the exploitation by the teachers of the
book in planning and carrying out particular lessons
with it, and the exploitation by the learner of self
study material in the book involve sensitive
implementation decisions.
A good teacher, and also a learner, makes these
decisions intuitively. Good material allows itself to
be used flexibly in order to enable the ”self-
directed” teacher or learner to do so. Programmes
and materials should support and guide those
teachers and learners who are less capable of
appreciating such points intuitively.
2.2.1.3.2.1. What advice is given to the learner, what
support is given to the new teacher, to help them
to make a suitable choice of objective, activity,
theme or whatever?

2.2.1.3.2.1.1.  Are clear criteria defined to help select topics Transparency Clarity of Aims
and choose activities?
 Topics and activities clearly ordered and
classified
 Range of possible aims indicated
 Topics and activities embedded in a clear
71

plan for the unit


Participation
2.2.1.3.2.1.2.  Is there an opportunity for learners to decide on Partnership
topics and activities?
 Range of activities presented for each skill
 Scope of the activity
(individual/pair/group; written/oral; length
of activity etc.) clearly indicated
 Learner can propose activities
 Individually designed homework
 Is provision made to ensure that less
experienced learners / teachers are supported in
their decisions? Attractiveness
2.2.1.3.2.1.3. User Friendliness
 Function of each
activity indicated
 Examples included
showing how to exploit the activities
 Sets of ”recipes” and
”pathways” for well-trodden routes
 Are instruments in place to evaluate the extent
to which planned implementation matches
needs and interests and to identify wrong Efficiency Ergonomy
2.2.1.3.2.1.4. decisions? Attractiveness User-Friendliness
 New teachers paired with a more
experienced colleague
 Focused attention given to new learners
 Tips available on how to make decisions and
exploit resources
2.2.1.3.2.2. What steps are taken to ensure that groups
formed (as classes or as small groups within
classes) will be appropriate, successful learning
environments?

2.2.1.3.2.2.1.  Have learner characteristics been taken into Flexibility Individualisation


account in forming groups:
Efficiency Ergonomy
 Language level
 Needs
 Learning styles
 Age
 Social and cultural background?
 Is provision made to identify and correct or
compensate for unhappy group placement?
 ”Debriefing” meeting after a placement to
notify misplacements
 Standard procedure to correct such mis-
diagnoses with new placement
72

 Is there occasionally a chance to work with Efficiency Ergonomy


different people?
2.2.1.3.2.2.2.
 Exchange of experiences foreseen (tandem
partners, other classes etc.)
 Connections with the “outside world”
foreseen (internet, project work “on the
field” etc.) Partnership
Participation
2.2.1.3.2.2.3.
2.2.1.3.3. Stimulating
Successful learning involves a mixture of work and
fun. The earnestness often displayed by very small
children in play indicates that play can be very
purposeful social or work simulation. Conversely
neurological research suggests that intelligence can
be stultified by repetitive drilling devoid of
personal enjoyment.
Good teaching stimulates hard work by engaging
curiosity and creativity and harnessing them in
practice activities. Most play involves a strong
element of ritual repetition; adult board and card
games have repetitive structures with turns in which
statements are often made in standardised formulae.
An essential part of learning a language at the
lower levels of proficiency involves acquiring an
element of automaticity. This automaticity can be
effectively developed through repetition in
”grammar games” or personal quizzes in which
motivation and attention are harnessed by play
instincts and sustained by the excitement generated.
Simulations, whether simple role-plays of
shopping, simulated business meetings or grandiose
Internet-linked school projects running imaginary
countries, exploit both play instinct and ambition to
give a personally relevant platform for language
practice. To be effective, however, such games,
simulations and role-plays must be sensitively
matched to the profiles of the participants, as
discussed in the previous section.
2.2.1.3.3.1. What steps are taken to ensure that learning is
fun and that the repetition necessary for
automatisation is achieved in a stimulating
rather than stultifying manner?

2.2.1.3.3.1.1.  Is purposeful play attributed a systematic role in Attractiveness Interactivity


the course?
 Drama used to repeat and perfect
performances
 Opportunity to play with grammar
 Opportunity to play with vocabulary
73

 Opportunity to play with communicative


situations
 Is repetition exploited in a playful, enjoyable
manner?
Attractiveness Interactivity
 Games which are ” communicative drills ”
2.2.1.3.3.1.2. occur regularly
 Poetry used as a means of harnessing
creativity and repetition
2.2.1.3.3.2. What opportunities are given to be
experimental, to take risks on particular
occasions?
2.2.1.3.3.2.1.  Are there opportunities to dive in and speak,
argue, express yourself?
Participation Personal Interest
 Warm secure environment for such risk-
Attractiveness Sensitivity
taking
 Role-play (simulating an everyday situation;
representing a viewpoint in a discussion)
used to provide a safe risk-taking
environment
 Games used to exploit competitive
playfulness
2.2.1.3.3.3. Are steps taken to raise curiosity?

2.2.1.3.3.3.1.  Are topics presented that are new and Attractiveness Variety
unfamiliar to the target group in question?
 Culturally different
behaviour
 Innovative ways of
looking at linguistic aspects
 Puzzling situations

2.2.1.3.3.4. What has been done to harness learners’


creativity?
 Is use made of the arts and the media?
2.2.1.3.3.4.1.  Visual arts exploited Attractiveness Variety
Interactivity
 Film used as a springboard to discussion
 Experimental music used to encourage
expression
 Multimedia used in a interactive way
 Are there opportunities for personal writing? Personal Interest
Participation
 Opportunity to keep a diary
2.2.1.3.3.4.2.
 Penpals / correspondence with exchange
schools encouraged
74

 Chances to script scenes, situations


2.2.1.3.3.5. What has been done to harness learners’
ambition to improve their language competence?

2.2.1.3.3.5.1.  Are there opportunities to branch out and follow Participation Personal Interest
up a work-related or hobby interest?
 Learning centre resources available to
learners who wish to follow up a work-
related or hobby interest
 Personal projects
 References available to information sources
in the language environment
 Internet facilities

2.2.1.3.4. Encouraging
Good language teaching involves encouraging in
the sense of challenging, but also a considerable
amount of reassuring of less confident learners.
Materials and programmes can build in such
reassurance by showing learners what they can do.
It is not just children who respond positively to
praising and rewarding when they are successful or
who need emotionally sensitive feedback when they
are unsuccessful. Learners who are unsuccessful, or
who are held back by the material or the level of a
course become frustrated. Quality assurance
implies ways of dealing with frustration when it
arises.
A qualitative approach to affective management
sets the scene for successful learning. Furthermore,
by making the experience an enjoyable one it
increases motivation and creates a virtuous circle in
which enjoyment of success increases motivation
which in turn increases success - and enjoyment of
it. However, knowing that learners are enjoying
success and are not experiencing frustration implies
monitoring to find out what is actually happening.
2.2.1.3.4.1. What steps are taken to show learners what they
have achieved, what they can do in the language?
2.2.1.3.4.1.1.  Is there a recapitulation at the end of each unit?
 Role plays or activities used to sum up Transparency Clarity of
situations and skills already practised Presentation
Clarity about
 Simple achievement tests provided
Achievement
 Team quizzes or panel games
 Do learners get praise and emotional / symbolic
2.2.1.3.4.1.2. Attractiveness Sensitivity
rewards for success?
 Credits or points to add
 Applause or
75

congratulations (multimedia)
 Increased autonomy Generativeness Cognitive
(self-directed learning) Development
2.2.1.3.4.1.3.
Clarity about
 Are awareness raising activities included in
Transparency Achievement
order to help learners to identify their
achievements?
 Feed-back discussion on performance
 Repairing procedures (non-intrusive
provision of correct term)
 Introspective procedures (video or audio-
recorded)
 Diary keeping

2.2.1.3.4.1.4.  Are there opportunities to experience Generativeness Transferability


achievement in real life?
 Real life performance of tasks (e.g. field
work; telephone)
 Joint or individual product which
demonstrates achievement (e.g. scrap book;
video recording; magazine)
2.2.1.3.4.2. What steps are taken to identify learners having
difficulty and give them sensitive, appropriate
and encouraging feedback?

2.2.1.3.4.2.1.  Are steps taken to identify less successful Flexibility Individualisation


learners? Transparency Clarity about
Achievement
 Records of learners performance kept,
including participation in classroom
activities
 Regular discussion on teaching/learning in
order to detect learning difficulties
 Are steps taken to support learners who tried but
still have difficulty? Individualisation
Flexibility
2.2.1.3.4.2.2.
 Extra resources deployed to support learners Adaptability
who tried but who have difficulty
 Extra practice offered to such learners if
necessary
 Alternatives for such learners if necessary as
regards: course; different media; lesson
phasing etc.
 Are steps taken to underpin the motivation of
learners who have not invested the necessary Adaptability
Flexibility
2.2.1.3.4.2.3. time, effort or attention in order to achieve a Partnership
workable relationship? Participation

 Guidance and ongoing support in the form of


a realistic learning contract provided
76

 Alternatives for such learners if necessary as


regards: course; different media; lesson
phasing etc.
 Teachers encouraged to switch tactics in
order to raise motivation
 Are steps taken to deal with frustration caused
by poor achievement or difficulties with other Socialisation Social skills
course participants or teachers?
Attractiveness Sensitivity
2.2.1.3.4.2.4.
 Awareness raising counselling on
individual learning styles and difficulties
 Possibility and time necessary to express
frustration in oral or written form not in
front of others (“anonymously”)

2.3 Monitoring
Successful monitoring by the teacher leads to a
efficient fit between the service provided and both
the perceived and observed needs of the learner(s),
which should lead to a high degree of success and
satisfaction.
Successful self-monitoring by the learner with or
without prompting from the teacher or the
computer software in a media-led course should
lead to autonomous learning as well as a high
degree of success and satisfaction.
Monitoring in this sense encompasses four steps:
1. Observation
2. Assessment
3. Feedback
4. Adjustment

2.3.1. What measures are undertaken to enable


teachers and learners to carry out effective
monitoring?

2.3.1.1.  Are teachers and/or open learning centre Efficiency Ergonomy


counsellors in a position to carry out
monitoring?
 Training to sensitise and give techniques
 Technical support (e.g. with video)
 Time available to talk to learners

Generativeness Cog. Development


 Are learners encouraged to share responsibility
Transparency Clarity about
2.3.1.2. for monitoring their own work and the
77

programme as a whole? Achievement


 Treated as actors in the monitoring cycle and
partners in the planning process
 Informed about aims, plans and their actual
realisation
 Trained in strategies and techniques for self-
monitoring
2.3.2. Observation
Effective monitoring requires an ability to see what
is really going on rather than what you intended to
go on or what you assume to be going on. This
requires a conscious, detached act of looking (or
data collection).
- In relation to teacher training, this may imply
team teaching and peer observation
- In relation to a school, regular observation of
lessons possibly meetings with learners or the
collection of learner questionnaires may be
helpful.
- In relation to new materials, observation/logging
of use of the material possibly meetings with
teachers/learners or learner questionnaires may be
useful.
Learners following autonomous courses can be
helped by “reflection points” built into the
programme. At such points, the learner is
encouraged by structural questions to consider how
they are getting along with the materials.
2.3.2.1. What measures are undertaken to observe on an
ongoing basis what is actually happening in the
course?

2.3.2.1.1.  Is there a phase towards the end of each lesson, Transparency Clarity of
each teaching unit, when learner performance Achievement
is consciously observed to monitor teaching Clarity of
success? Presentation
 Planned monitoring phases at the end of
units
 Simple logs of lessons /activities
 Review lessons at the end of modules
 Does the teacher (for any actual course) take a
step back at certain points in the course to
observe the performance of the learners? Efficiency Ergonomy
2.3.2.1.2.  Systematic collection of teacher /learner
comments on activities/materials foreseen
 Video recordings of lessons used for
monitoring and feedback
 Does management program (for software)
78

provide global feedback at certain points in the


course on the performance of individual
learners?
2.3.2.1.3. Transparency Clarity of
 Meetings with learner representatives
Achievement
Participation
 Meetings with teachers Partnership
 Questionnaires from learners collected from
time to time
 Are there reflection points for the learner to take
a step back to see where they are and where
they want to be?
 Simple logs of lessons /activities
Participation Partnership
 Video recordings of lessons for monitoring
Transparency Clarity about
2.3.2.1.4. and feedback
Achievement

2.3.2.1.5.  Are measures taken to go beyond mere data Generativeness Cognitive


collection and to use the experience of Development
monitoring to achieve a shift in perspective?
 Peer observation by teachers
 Team teaching/learning
 Qualitative consideration of the types of
interactions being generated, the number and
length of interventions made by learners, the
time the learner produces language and/or
works focused on task
 Quantitative analysis of the types of
interactions being generated, the number and
length of interventions made by learners, the
time the learner produces language and/or
works focused on task
2.3.3. Assessment
Assessment in this sense is achievement and
diagnostic assessment.
- How is the person making progress in general?
- How much of what was being aimed at has been
achieved?
- What are the continuing needs?
Techniques used in both course materials and
classrooms might include: progress tests; ”Now you
can do it” activities; self-assessment questionnaires.
Teachers will in addition monitor - and perhaps
keep notes on - performance in class, homework
and project work. Institutions may ensure fixed
”assessment points” at particular points in the
course with more formal assessment activities,
including proficiency tests and reporting of results.
79

2.3.3.1. How is in-course diagnostic and achievement


assessment organised?
2.3.3.1.1.  Is there an ongoing system to track individual Transparency Clarity about
progress? Achievement
 Progress assessed at the end of each unit
 Progress assessed at mid-course
 Progress assessed at the end of the course
 Is there a ongoing system to track individual
satisfaction?
 Regular tutorials for individual students
Sensitivity
2.3.3.1.2. Attractiveness
 Feedback sessions for groups in regular
Involvement
intervals Participation
 Is there an ongoing system to track the
development of teachers’ competences?
 Observation procedures by peers and/or
inspectors
2.3.3.1.3. Efficiency
 Forms of continuous training Ergonomy
Socialisation
 Teachers meet on an informal basis to Social Skills
exchange experience Generativeness
Cognitive
 Is diagnostic and achievement assessment Development
carried out regularly?
 Written homework regularly collected,
corrected and analysed diagnostically
Transparency
2.3.3.1.4. Clarity about
 Interim tests
Achievement
 Individual feedback on spoken performance
during classes
 Does diagnostic information feed back into the
programme?
 Low key quizzes and tests carried out
regularly
Efficiency Ergonomy
2.3.3.1.5.  Regular performance experiences ("Now you Reliability Consistency
can do it")
 Homework
2.3.3.2. What criteria are used to assess progress?

2.3.3.2.1.  Are ”key stages”, milestones, levels of Transparency Clarity of Aims


attainment or modules defined in terms of
outcomes?
 Definition of levels of attainment
 Grids specifying outcomes for different
skills and abilities
 Checklists of competences covered by
different modules
 Is account taken in the definition of the need to
80

2.3.3.2.2. encourage creative, personal use of language? Generativeness Transferability


 Definition of relevant ”real life tasks” that Attractiveness Interactivity
the learner should be able to do on Transparency Clarity of Aims
completion of a stage Relevance Learner
Centredness
 Definition for each stage of qualitative
criteria: the standard of performance
expected in terms of different aspects of
communicative language ability
 Definition of key target language (functions,
structures, areas of vocabulary) for each
stage
 Definition of key communication strategies
for each stage (e.g.: getting into the
conversation, coping with difficulties,
compensating for shortcomings, getting and
giving help and clarification)
2.3.3.2.3.  Is there an opportunity for self-assessment with
the criteria used for assessment?
Socialisation Intercultural
 Self-assessment leading to discussion Awareness
 Self-assessment compared to performance
in related assessment tasks
Clarity about
Transparency
 Are criteria set for inter-cultural objectives? Achievement
2.3.3.2.4.
 Providing authentic
representations of foreign culture.
 Integrating multiple
perspectives on relevant topics (intercultural
comparison)
 Overcoming cultural
barriers
 Recognising foreign
values and norms
2.3.4. Feedback
Feedback is made possible by conscious
achievement/diagnostic assessment of the learners.
Feedback is here used in the sense of feedback to
the learner on language performance. Such
feedback should include suggestions for extension
or remediation activities and / or for the adoption of
other techniques and strategies.
Some features which distinguish quality in such
feedback are:
- appropriateness of the timing;
- accuracy, comprehensibility and personal
relevance of the information;
- selection of the right level of detail - setting
sensible priorities;
- sensitivity.
2.3.4.1. What policy is adopted as regards the question of
when and how much to correct mistakes?
81

2.3.4.1.1.  Is it made clear to learners when they will be Transparency Clarity of Aims
given feedback (positive or negative) and when
not?
 Regular individual feedback given
 Video camera available to monitor
performance in communicative activities
2.3.4.1.2.  Do learners sometimes have an opportunity for
unfettered expression without fear of negative Attractiveness
Sensitivity
feedback?
 Feedback given related to the wider criteria
used in assessment as well as just to
mistakes
 "Free sessions” foreseen in which learners
can freely express themselves without being
corrected, except on request
2.3.4.1.3.  Are steps taken to prevent the learner being Consistency
swamped by feedback (techniques used in Reliability
fluency activities in order to give delayed
feedback without disrupting the activity)?
 Error cards
 Teacher checklists
 Video recording, selection of interesting
clips

2.3.4.2. What form of language awareness training is


help students profit from feedback?

2.3.4.2.1.  Are techniques used to help learners recognise, Attractiveness Sensitivity


lose their fear of and learn from their mistakes?
 Mistakes code used when correcting written
work
 ”Spot the mistake” and correction exercises
and games
 Criteria used in assessment made known to
the learners
 ”Neutral” scripts and/or videos showing
other, comparable learners’ performances
used to make learners more aware of
qualitative criteria
Cognitive
 Are techniques used to train learners to repair Generativeness Development
2.3.4.2.2. communication breakdowns?
 Changing topics
 Asking for help
 Negotiating meaning
82

 etc.
2.3.4.3. What steps are taken to provide personalised
feedback which leads to effective improvement?

2.3.4.3.1.  Are learners helped through specific steps to Attractiveness Sensitivity


become aware of their strengths?
Participation Partnership
 Selective, personalised praise for their
principal language strengths
 Learners helped to form learning priorities
on the basis of feedback
 Learners helped to develop an appropriate,
realistic hit list of problems to tackle
2.3.5. Adjusting service provision
After observing what is really happening (1);
determining what is working and what is not
working (2 & 3); giving feedback on what
happened and on proposed changes (4) the final
step is to realign the Design to better meet the
needs of the group. Adjustments to the specification
may be necessary, or it may be aspects of the
presentation which need changing.
2.3.5.1. What steps are taken to ensure that the
programme is adjusted sensitively to the ongoing
needs and interests of the learner(s)?

2.3.5.1.1.  Is training provided to ensure that the actors in Transparency Clarity of Aims
the process are aware which aspects of the Clarity of
programme are fixed and integral, and thus Rationale
cannot be changed?
 Reference to teacher handbook
 Reflecting on the methodological premises
of the programme
 Is training provided to ensure that the actors in
2.3.5.1.2. Transparency Clarity of Aims
the process are aware which aspects of the
Flexibility Adaptability
programme are flexible and intended to be
adjusted sensitively?
 Provision made for speeding up or slowing
down, to increase or reduce opportunities for
consolidation and reflection
 Provision made for altering the proportion of
different activities or to abandon an activity
type which is unsuccessful
Efficiency
2.3.5.1.3. Ergonomy
 Is training provided to ensure that the actors in
the process are confident and competent enough
to adjust planning to take account of
circumstances?
 Standard processes to amend a course plan
and inform those affected
83

 Support systems to help an inexperienced


teacher / user faced with the need for a
radical change of plan
2.3.6. Support
Over and above the support offered by the teacher,
there are three main categories of support system:
1. People who can help
2. Information tools
3. Awareness-raising and Self-training tools
2.3.6.1. What steps have been taken to ensure co-
ordination and cross-referencing between
different types of support?

2.3.6.1.1.  Are steps taken to inform users of the existence Transparency Clarity of
and function of different forms of support? Presentation
 Face-to-face
information
 Leaflets or brochures
 On-line information
Transparency Clarity of
2.3.6.1.2.
 Are users informed of the limitations to Presentation
support?
 Range of activities
supported
 Time limits
 Learner
responsibilities
2.3.7. People who can help
Whether one is talking about distance learning, a
multimedia package, a language learning stay
abroad, a secondary school language course, there
are certain questions which are indicators of the
quality of service provided:
- Someone to phone up or go and see when
things go wrong (a ”tutor”)
- Someone who can help with routine
administrative issues which may baffle the
learner (”service personnel”)
2.3.7.1. What support personnel are available if the user
cannot cope?

2.3.7.1.1.  Is there someone available to help when a Attractiveness User Friendliness


learner is unable to navigate the system, find Efficiency Ergonomy
their way around or cope with organisational
demands on them?
 Someone available to help when learners
have a question they cannot answer by
84

themselves
 Specification of how quickly help will be
provided in such a case
 Is there someone available to help with extra
2.3.7.1.2. User Friendliness
attention when a learner is not making progress?
Attractiveness
Individualisation
 Routing system to refer such learners to a
Flexibility Ergonomy
regular support system (e.g. a one-to-one
Efficiency
tutorial to help organise them)
 Reference to a supervisor
2.3.8. Information tools
The simplest information tools are a contents page
and an index. The presence or absence of an index
to a book is one indicator of the seriousness of a
publisher producing professional or academic
literature. The Navigation referred to in Section E1
on Design is the more effective the more tried and
tested such ”Help” facilities are. The range of such
information facilities for a system (multiple points
of entry) is one aspect of quality, but reliability of
information, ease of use and speed of retrieval are
probably more crucial.
2.3.8.1. What orientation tools are learners given?

2.3.8.1.1.  Is an overview of course objectives provided? Transparency Clarity of Aims


 "Milestones”: stages in the course when
there is a summary of what has been learnt
so far and what is coming in the next module
 Overview of unit/daily/weekly objectives
and content
 Resume of unit content at the end of a unit?
 Cross-referencing to what came before and
what may come afterwards
 Reference section or facility (with a
grammar reference list and vocabulary
summaries).
2.3.8.2. What training is learners given in using these
orientation tools?

2.3.8.2.1.  Are steps taken to ensure that learners can use Attractiveness User Friendliness
self access tools such as indices, manuals and
facilities effectively?
 Steps taken to ensure that learners are able
to look up, explore and practise grammar
points on their own
 Steps taken to ensure a "phrase book"
function: that learners can look up and
practise language for a particular situation
85

they are going to find themselves in


 Cross-referencing between comprehension
texts and language practice exercises
2.3.9. Awareness-raising and Self-training tools
Teachers’ handbooks explaining how to exploit
course books are the first practical guide to their
profession for many people teaching languages.
Self-directed use of materials can also be enhanced
by the provision of tips and advice to the learners to
encourage them to adopt strategies suitable for
them for the task at hand. Selection of materials in
open learning centres is more effective if people are
on hand to give those who need it appropriate
advice in how to look for and select material.
In language learning, use of support tools often
involves understanding the metalanguage used to
organise them. Provided that this metalanguage is
well thought out, it can also increase Language
Awareness. Thus, in a seeming paradox, provision
of support systems can encourage more self-
direction towards autonomous learning, whilst an
absence of support and advice can prevent the
majority of learners from developing the meta-
skills they would need to become more
autonomous.
2.3.9.1. What awareness-raising and self-training tools
are available?
2.3.9.1.1.  Are there teachers' and/or learners' guides to
accompany the course?
Attractiveness User Friendliness
 Typical scenarios which the learner can use
Generativeness Cognitive
as a basis to develop their own pathway
Development
 Sequenced tutorial offered showing all the
elements of the system
 Fast track facility to help more independent
learners answer a specific question
86

3. Outcomes

OUTCOMES
The quality of the outcomes can be evaluated in terms of the gains in competences and awareness in
relation to the resources and conditions concerned and also in terms of the satisfaction of the stakeholders
involved.
Afterwards

When evaluating results obtained, one can differentiate between successful learning of the language
concerned (Success); reactions to the learning process (Satisfaction); and the achievement of
broader educational aims (Educational Bonus).
87
88

The third stage of quality management in the language


learning/teaching process, i.e. OUTCOMES is concerned with the
assessment of the effectiveness of the programme in terms of
assessing the results obtained.
When evaluating results obtained, one can differentiate between
successful learning of the language or languages concerned
(Success); the achievement of broader educational aims
(Educational Bonus), and reactions of stakeholders to the
programme (Satisfaction).
In planning an evaluation of outcomes a variety of techniques
should be applied to capture significant differences of perspective:
qualitative and quantitative, objective and subjective, structured
and open-ended, etc.
3.1 What steps have been taken to carry out a comprehensive and
objective evaluation of the programme/materials?

3.1.1.  Has feedback been collected from learners? Efficiency Ergonomy


 Large samples of learners asked to complete questionnaires Relevance Learner Centredness
 Read aloud simultaneous protocols in which learners record
their reactions whilst following the course or materials
 Retrospective protocols, in which learners later fill out a
report on their reactions to the programme and materials
 Structured interviews with a sample of learners

3.1.2.  Has feedback been collected from teachers and resource centre Efficiency Ergonomy
managers?
Relevance Learner Centredness
 Records of how the materials were used
 Report sheets on specific (e.g. experimental) aspects of the
materials
 Questionnaires
 Structured interviews with a sample of teachers

3.1.3.  Have outside independent experts been asked to review and Efficiency Ergonomy
audit the programme or materials?
 Report sheets on specific (e.g. experimental) aspects of the
materials
 Questionnaires
 Structured interviews
89

3.1.4.  Has there been structured observation, recording and analysis of Efficiency Ergonomy
the programme or materials in operation?
Relevance Learner Centredness
 Observation to analyse the patterns of interaction generated
Relevance
 Discourse analyses of the type of activity generated
 Analysis of speech transcripts or writing samples to study
Appropriateness
the extent of the acquisition of language forms?
Reliability
Accountability
 Analysis of speech transcripts or writing samples to study
Transparency
variation in the complexity, accuracy and fluency of
language generated by tasks of different types
Consistency
 Analysis of speech transcripts or writing samples to study
Clarity of Rationale
use of communication strategies

3.2 Assessment of Language Learning Success


Language learning success on the part of the learners can be
assessed with regard to:
1. Internal criteria: achievement of the objectives stated.
2. External criteria: measurable improvement in language
proficiency.
3. Real life competence gains - observable improvement in
actual real life language use.
4. Depth of acquisition:
(a) long term acquisition (sustainability);
(b) ability not only to reproduce models but also to cope with
tasks not specifically encountered (transferability).
3.2.1. What perspective on the assessment of language learning
attainment is taken in the programme / materials?
3.2.1.1.  Is the achievement of the stated objectives assessed (internal Transparenc Clarity about
perspective)? y Achievement
 Progress tests
 Teacher impression
 Self-assessments
 Is the gain in language proficiency assessed? Clarity about
3.2.1.2.
Achievement
 Comparing teacher assessments before the course with Transparenc
performance on an examination after the course y
 Comparing results on pre-course tests to those on post
course tests
 Comparing self-assessments before and after the course
 Is observable improvement in actual real life language use
Transferability
recorded?
Generativene
 Testimonials
3.2.1.3. ss
 Self-assessments
 Documentation
Integration
 Is depth of acquisition taken into account?
90

 Proficiency tests taken 6 months later (sustainability)


 Tasks not specifically encountered (transferability) Generativene
ss
3.2.1.4.

3.2.2 Success in Relation to Internal Criteria


For much of the learning process assessment of success is
concerned purely with the degree of achievement or mastery of
the selected course objectives. Such assessments are ”low stakes”:
no career decisions rest on the results so technical questions of
validity and reliability are not central. What is important in such
achievement testing is that there is a clear, coherent relationship
between what was presented and what is assessed, that
opportunities for meaningful language use are offered as well as
just tests of language knowledge.
3.2.2.1. What steps are taken to ensure clear coherence between
learning objectives and assessment tasks?

3.2.2.1.1.  Are the course content specifications (linguistic content, text Transparenc Clarity of Aims
types) for the module or unit used to provide a specification for y
test development?
 Tests designed according to the
linguistic progression used
 Tests reflect the text typology used
 Tests correspond to the declared aims
of the units
Consistency
 Are the course activity specifications (skills, pedagogic and Reliability Internal coherence
3.2.2.1.2.
real life tasks) used to provide a specification for test
Clarity of Aims
development?
Transparenc
 Regular informal assessment of spoken interaction y
 Regular informal assessment of writing
 Listening tests reflecting learning aims
Consistency
 Reading tests reflecting learning aims Interactivity
Reliability
 Is formative assessment integrated into the course?
Attractivene
3.2.2.1.3.  Regular progress tests ss
 Continuous assessment in relation to
course objectives
 Opportunities for self-assessment in
relation to course objectives

3.2.3. Success in Relation to External Criteria


Outside stakeholders, however, are primarily interested in
assessment of success in relation to external criteria. Such
assessment presupposes that a clear, coherent and consistent
formal procedure has been devised and implemented for the pre-
91

course and final assessments in order to properly estimate the gain


in communicative language proficiency.
Such formal proficiency testing requires:
- a balanced, theoretically motivated assessment model operating
within a common framework of tasks, standards, criteria and
marking procedures;
- effective communication of the kinds of tasks, the standards and
the procedures to those people involved;
- consistent data collection which enables direct comparison of
data (e.g. pre- and post course);
- accurate record keeping.

Since the results of proficiency tests may be used to make


decisions affecting learners’ futures, such assessment can be
considered relatively ”high stakes”. It should therefore
demonstrate generalisability beyond the internal concerns of the
learning context through a degree of:
- reliability: that results are consistent and replicable;
- validity: that the results reflect the abilities they are intended to
reflect;
- comparability: that some form of equating has been undertaken
to relate results to external benchmarks.

3.2.3.1. How is proficiency gain reported?


3.2.3.1.1.  Is ability assessed in relation to some yardstick or scale
covering the relevant spectrum of proficiency?
Relevance Accountability
 Definitions of different qualitative aspects of performance
for the different levels Clarity of Aims
Transparenc
 Comparison between pre-course and post-course y
proficiency on the same scale
 Samples of performance illustrating the different levels of
the scale
 Are results reported on a scale of proficiency descriptors
explaining the meaning of the levels on the scale to users? Clarity of
3.2.3.1.2. Transparenc Presentation
 Descriptors describe concrete features of performances
y
 Descriptors on the scale constitute independent criteria
statements about what the learner can do, rather than using
normative terminology
 Transparent, brief descriptors
3.2.3.2. What steps are taken to ensure that the assessment model is
capable of generating generalisable results?
92

3.2.3.2.1.  Is the assessment model related to theoretical models of Reliability Methodological


language competence and language use? Integrity
 A range of discourse/text types systematically sampled to
minimise task effects
 A range of response formats systematically used to
minimise test method effects
 Are assessment procedures standardised in order to produce
consistent results?
3.2.3.2.2.
 Standardised marking / rating criteria
 Standardised marking / rating procedures
 Standardised data collection and record keeping processes

3.2.3.3. What steps are taken to ensure the reliability of results?


93

3.2.3.3.1.  Are test items pre-tested before being used for assessment? Reliability Consistency
 Trialling with a class at the appropriate level
Reliability Methodological
 Piloting with a range of classes Integrity
 Consultation with teachers on correct answers, adequacy of
instructions etc.
 Is test data analysed?
3.2.3.3.2.
Reliability Methodological
 Facility values used to refine tests by removing very easy
Integrity
or very difficult items
Relevance Accountability
 Reliability coefficients
Transparenc Clarity about
 For assessing productive skills, are assessors trained in relation
y Achievement
3.2.3.3.3. to criteria defining standards of performance for different
aspects of proficiency?
 Criterion descriptors defining standards of performance
developed by analysing samples of performance with
expert informants
 Criterion descriptors: clear, concrete, stand-alone of
observable features of performance?
 Allocation of descriptors to different bands done through
analysing data from workshops with informants
 Allocation of descriptors to different bands through
statistical analysis of the way the descriptors are actually Consistency
interpreted (e.g.; Rasch rating scale model). Reliability
Methodological
 Are moderating procedures used to minimise subjectivity in Integrity
3.2.3.3.4. the assessment of productive skills?
 Rater training / tuning in sessions employed before
assessment
 Raw scores from different groups checked and compared
by colleagues or chief examiners
 Double marking
 Subjectivity systematically eliminated through statistical
techniques
3.2.3.4. What steps are taken to ensure validity?
94

3.2.3.4.1.  Is there a qualitative analysis of the match between the content Reliability Linguistic Integrity
of the test and the content of the target domain of language
Consistency
use?
Practicality
 Identification of test tasks with particular skills and content
areas confirmed through consultation of experts
 Identification of test tasks with particular skills and content
areas confirmed through qualitative analysis
 Identification of test tasks with particular skills and content
areas confirmed through appropriate statistical analysis
(e.g. internal correlations of scores on subtests; multi-trait
multi-method analysis)
 Is there an analysis to discover whether tasks and test questions Consistency
3.2.3.4.2. assess what the authors intended by investigating the problem- Reliability
solving processes of learners as they take the test? Methodological
Integrity
 Learners interviewed about how they did the test
Relevance Appropriateness
 Retrospective protocol
 Think-aloud protocols used to record what learners think
they are doing as they complete the task
 Other (e.g. electronic) techniques used to log what learners
do while completing the tasks
 Is there a statistical analysis of the internal structure of
3.2.3.4.3. assessments: that each item or phase of the task contributes to Reliability
the overall construct being assessed? Consistency

 Conventional statistical techniques (e.g. point biserials) Internal Coherence


used to identify ”outliners” not contributing to the main
construct
 More advanced technical techniques (e.g. factor analysis,
multiple regression, generalisability theory,
multidimensional scaling) used to investigate the way in
which different aspects tested combine and inter-relate
 Item response theory used to model the fit and consistency
of individual items in relation to the construct being
assessed in the test
 Is there a statistical analysis of the strength of the correlation
3.2.3.4.4 between results on these assessments and on external, Reliability Methodological
validated assessments testing the same construct? Integrity
 Simple experiments with small samples to calculate Consistency
correlation coefficients between the test and other measures
 Large scale comparability study using more sophisticated
statistical techniques to investigate relationships with an
assessment taken as an external benchmark
95

3.2.3.4.5.  Is information collected about the usefulness and accuracy of Relevance Accountability
the assessment results?
Appropriateness
 Receiving institutions (e.g. university departments,
employers) asked to report their experience with the
assessment results
 Former test-takers asked to report how their later
experience related to the assessment results
 Predictions of future success in use of language for study or
work investigated statistically
3.2.3.4.6. Relevance Accountability
 Is information collected about the consequences of the
Learner
assessments?
Centredness
 Steps taken to discover how end-users (receiving Transparenc Clarity of Aims
institutions) interpret assessment results y
 Steps taken to identify and take into account unintended
external uses of test results
 Teachers asked to give their views on the effects of the
assessments on teaching
 Systematic investigation of such ”backwash” effects

3.2.3.5. What steps are taken to provide comparability between


assessments?
96

3.2.3.5.1.  Are assessments made of performance on tasks selected in Methodological


Reliability
relation to a consensual common framework on the basis of Integrity
common standards?
Efficiency Ergonomy
 Equivalences established on the basis of consulting experts
 Equivalences established on the basis of formal comparison
of specifications and performance samples by a standards
team
 Equivalences established on the basis of comparing
performances of groups of learners
 Equivalences established on the basis of formal statistical
equating
 Is the difficulty of assessment tasks and test items established
to calibrate them to the same scale?
3.2.3.5.2. Methodological
 Significance of test results determined by their Reliability Integrity
development from content specifications Transpare Clarity of
ncy Presentation
 Significance of test results determined by consulting
experts Clarity of Aims
 Difficulty of tests and tasks investigated by considering
samples of performance on the tests / tasks
 Difficulty on test items and test asks established with a
measurement model to create an item bank
 Are results on different assessments moderated to reduce the Methodological
effects of differences in administration conditions and in the Reliability Integrity
interpretation of standards by different centres or individual
3.2.3.5.3. assessors? Efficiency Ergonomy
 Test / exam results used to provide benchmarks to guide
looser forms of assessment
 Investigation and smoothing of conventional statistics
(means and standard deviations) used to identify and adjust
for such systematic error
 Many-faceted Rasch model analysis or an equivalent
statistical technique used to place all results on the same
measurement scale

3.2.4. Success in terms of Real Life Competence


Gains
Subjective ”soft data” can be very useful to supplement objective
”hard data” in assessment. Learners have their subjective
impressions of their success when they try to use what they have
learnt in the real world. End-users (employers, receiving
educational institutions) have their impressions of the performance
of the learners concerned.
Such self-assessments and outsider confirmation of successful
language use can be collected during a course; in relation to
exchange visits or immersion in a (simulated or real) foreign
language environment, and / or after a course. Statements made by
learners, teachers, and employers or native speakers with whom
learners have come into contact can be supported by documentary
97

evidence in the form of:


- samples of work including scrapbooks, diaries etc. and products
created in personal projects;
- attestations of language learning experiences;
- testimonials on the quality of performance in those experiences.
3.2.4.1. What forms of ”soft evidence” of success are learners
encouraged to collect?
3.2.4.1.1  Are there opportunities for self-assessment in relation to real Relevance Learner
life tasks and experiences? Centredness
Generativen
 Field work tasks which can be the subject of such self- ess Transferability
assessments during exchange visits or immersion
experiences
 Simulated authentic tasks in the learning environment
which can be the subject of specific self-assessments in
relation to those tasks
 Is a structure provided for learners to record samples of work?
3.2.4.1.2. Participatio
 Personal writing given value by being included in a Personal Interest
n
portfolio?
Clarity about
 Personal projects and course work given value by being Transparenc Achievement
included in a portfolio y
 Is the status of such work samples made clear?
 Distinction made between best and typical work
3.2.4.1.3.
 Distinction made between individual and joint products Clarity of Aims
 Distinction made between a first draft handed in and a Clarity of
product finely honed after feedback from the teacher Transparenc Presentation
y

3.2.4.2. What forms of ”soft evidence” of success are collected from


outsiders or compiled by an electronic monitoring program?
3.2.4.2.1.  Does the programme or materials provide field work tasks Relevance Accountability
which can be the subject of such testimonials during exchange
visits or immersion experiences?
 Scrapbooks, logs, and diaries
 Interviews
 Social events
 Is a structure provided to record attestations and testimonials?
Efficiency Ergonomy
3.2.4.2.2. Accountability
 Templates provided to encourage outsiders to give useable
information
Relevance Learner
 Personal record of achievement; dossier in which such Centredness
attestations and testimonials can be stored

3.2.5 Assessment of Depth of Acquisition


Assessment of success involves collection of data at a particular
point in time, usually at the end of a course. Yet many learners
98

tend to lose some of their ability after the course in a phenomenon


known as ”Language Loss”.
Language loss is greater when programmes encourage an
unbalanced learning style (focus on accuracy and little fluency;
concentration on fluency with little focus on form).
Language loss can also be expected to be greater in relation to a
programme relying on the reproduction of models without the
systematic development of a capacity to transfer learnt knowledge
to personal areas of interest and to new situations.
3.2.5.1. What steps are taken to estimate the depth of acquisition?

3.2.5.1.1.  Do assessment tasks require learners to transfer and recombine


Generativene
what they have learnt to apply it to new situations? Transferability
ss
 Reformulation tasks
 Integrated tasks combining different skills trained
 New scenarios in which to perform skills and tasks
encountered
 Does the assessment include tasks which require learners to Generativene
Transferability
3.2.5.1.2. react spontaneously and interactively in unpredictable, open- ss
ended situations?
Attractivene
Interactivity
 Virtual scenarios with unfamiliar interactions ss
 Spontaneous internet interaction
 Does the assessment include tests of underlying language
3.2.5.1.3.
competence in addition to assessing performance?
 Phonological tests Linguistic Integrity
Reliability
Textual Integrity
 Grammar tests
 Vocabulary tests
 Discourse-based tests of cohesion/coherence
 Does assessment include tasks in which learners have to solve
problems in real time? Generativenes Cognitive
3.2.5.1.4.
s Development
 Discussion tasks broad enough in scope to require learners
Transferability
to mobilise all their resources rather perform learnt
language
 Steps taken to ensure that tasks are challenging
 Discussion tasks in which learners have to discuss how to
go about a task
 Integrated skills tasks which require learners to take points
from different sources and process them in the process of
producing a response
3.2.5.2. What steps are taken to estimate the effects of language loss?
99

3.2.5.2.1.  Are follow up studies undertaken to assess language Generativenes Integration


proficiency more than 6 months after the end of the course? s
 Follow up assessment of fluency as well as language
knowledge
 Follow up assessment includes tasks requiring real time
processing in order to assess effects on the complexity and
accuracy of language produced

3.3. Assessment of Educational Bonus


Evaluations should be flexible enough to capture positive
educational outcomes other than language proficiency on which
data can be collected through attitudinal questionnaires, diaries,
interviews and teacher observations. Examples of such outcomes
may be
 increased inter-cultural awareness;
3.3.1.
 learning to learn skills;
3.3.2.
 more effective communication strategies;
3.3.3.
 improved self confidence and self esteem;
3.3.4.
 higher motivation.
3.3.5.
3.3.5.1. What steps are taken to include evaluation of educational
bonuses in assessment?
3.3.5.1.1  Are learners asked to give feedback on personal gains? Transparenc Clarity about
y Achievement
 Questionnaires
 Diaries or log books
 Follow up interviews
Clarity about
3.3.5.1.2.  Is a sample of teachers consulted?
Achievement
Transparenc
 Questionnaires
y
 Diaries
 Follow up interviews
3.3.5.1.3.  Is a sample of parents / sponsors / employers consulted? Clarity about
Achievement
 Questionnaires
Transparenc
 Invited comments y

3.3.5.2. Inter-cultural Awareness


The increasing ease of world travel, the trends towards integration
in Europe and the contributions of immigrants to economies and
cultures means that inter-cultural awareness is an aspect of
education of ever increasing important.
One of the reasons for encouraging foreign language learning is to
promote a wider perspective and help overcome social and
psychological barriers to understanding and communicating with
people from other cultures.
100

3.3.5.2.1. What steps are taken to assess inter-cultural educational


bonuses?
3.3.5.2.1.1  Are gains in inter-cultural awareness included in assessment? Socialisation Intercultural
. Awareness
 Inter-cultural aspects included in assessment instruments
 Attitudinal questionnaires completed by learners
 Consultation with hosts after an exchange trip
3.3.5.3. Learning to learn

Language learning has long been included in the curriculum to


fulfil the broader educational aim of broadening learners minds
and encouraging logical thinking. Hence the inclusion of Latin,
and Ancient Greek, in school curriculum long after they ceased to
be used as languages of communication.
Learning a modern foreign language also:
- Broadens the mind;
- Encourages structured thinking;
- Trains the memory, associative and lateral thinking;
- Trains an ability for analysis;
- Develops learning strategies and techniques which are of
general application.
3.3.5.3.1.  What steps are taken to assess learning to learn?

3.3.5.3.1.1  Are gains in language learning skills included in assessment? Generativen Cognitive
. ess Development
 Language awareness, correction tasks included in tests
 Retrospective questionnaires used to assess learning
strategies
 Analysis of discussions about how to learn a language
 Learner diaries recording strategies used
 Teacher questionnaires recording general observations
 Follow up interviews: learners / teachers?
 Are gains in general learning skills included in assessment? Generativene
Cognitive
3.3.5.3.1.2  Aptitude aspects included in assessment (memory tests, ss
Development
. analysis tasks)
 Learner questionnaires
 Consultation with teachers of other subjects
 Consultation with vocational trainers / employers

3.3.5.4 Communication strategies

Communication strategies involve an ability to achieve and


maintain communication by maximising the exploitation of
available resources and compensating or finding ways to get
around a lack of knowledge or skill.
101

The development of such an ability to juggle resources is essential


to effective foreign language use. In addition, the learner transfers
improved communication skills back to situations involving
mother tongue language use, which leads to a general increase in
self-confidence.
3.3.5.4.1.  What steps are taken to assess communication strategies?

CORE INDICATORS
3.3.5.4.1.1  Are gains in effective communication strategies included in Cognitive
Generativen
. assessment? Development
ess
 Communication strategies valued positively in performance
assessment criteria
 Transcripts of learner interaction analysed to investigate
strategy use
 Teacher questionnaires recording general observations
3.3.5.5. Self confidence & motivation

The feeling of accomplishment generated by effective foreign


language learning, the acquisition of an ability to communicate in
another tongue, has a direct feedback on self confidence and self
esteem. Language learning experiences should engender such a
feeling of success in order to motivate the learner to continue
studying the language. Such motivation may also lead learners to
wish to widen their experience of the language (e.g. by organising
a journey) or continue their studies autonomously after the course.
3.3.5.5.1. What steps are taken to assess gains in self confidence?

3.3.5.5.1.1  Are gains in self confidence and self esteem included in Attractivene Sensitivity
. assessment? ss
 Questionnaires before and after the course
 Recording feelings in diaries
 Follow up interviews with a sample of learners
3.3.5.5.2. What steps are taken to assess motivation?

CORE INDICATORS
3.3.5.5.2.1  Are gains in motivation included in assessment? Attractiveness Sensitivity
.
 Attitudinal questionnaires
 Elicited comment from teachers
 Analysis of attendance and re-enrolment patterns
 Investigation of take up of extra-curricula activities

3.4 Assessment of Satisfaction


Satisfaction can be considered from the point of view of both
those receiving the service (learners, etc.) and those involved in
using the material or programme to provide the service (teachers
etc.). Data can be collected with questionnaires, diaries, structured
interviews and, in some learning contexts, re-enrolment numbers
102

or, to some extent, through sales figures and enquiries.


3.4.1. What steps are taken to ensure that users are satisfied?
3.4.1.1.  Have the relevant stakeholders been identified and addressed Relevance Learner
appropriately? Centredness
Appropriateness
 Learners
Accountability
 Teachers
 Employers/local community
 Educational authorities
 Are resources made available to elicit and analysis comment? Efficiency
3.4.1.2. Ergonomy
 Questionnaires
 Interviews
 Focus groups
Efficiency Ergonomy
 Are dissatisfied users offered adequate compensation? Flexibility
3.4.1.3. Individualisation
 Change of course /teacher
 Partial refund
 Free course
 Full refund

3.4.1.4. E3.3.1 Satisfaction with Progress


For the learner, the central question is whether he/she is satisfied
with the progress made.
3.4.1.4.1. What steps are taken to ensure that expectations of possible
progress raised by promotional material are met?
3.4.1.4.1.1  Are statements about progress made in promotional literature Reliability Consistency
. realistic? Efficiency
Cost-Effectiveness
 Based upon analysis of progress made by previous
learners
 Based upon a market analysis of costs/benefits
 Compared with certificate scales
 Checked against official curricula and syllabi
 Has the learner’s satisfaction been checked? Attractivene
 Learners asked in a questionnaire whether they are happy ss Sensitivity
3.4.1.4.1.2
with the progress they have made
.
 Learners invited to compare their exit level with their
starting level in making this statement?

3.4.1.5 Satisfaction with Learning Activities


To what extent were expectations about teaching and course
activities met?
103

3.4.1.5.1. What steps are taken to ensure that expectations about


teaching and course activities are met?

3.4.1.5.1.1  Are learners asked in a questionnaire whether statements about Cost-effectiveness


Efficiency
. teaching, qualifications, class sizes etc. made in promotional Ergonomy
literature are adhered to?
 The teaching Attractivenes Variety
s Sensitivity
 The amount of support they received
 The monitoring of their performance Clarity about
and progress Transparenc Achievement
y
 The balance of activities in the
Practicality
programme
Reliability
 The exercises and pedagogic tasks Transferability
Generativen
 The opportunities for real life tasks
ess
and projects

3.4.1.6 Satisfaction with Facilities


3.4.1.6.1. To what extent were the pre-course commitments regarding
the characteristics of the learning context and facilities
fulfilled?
3.4.1.6.1.1  Did the organisational and logistical facilities of the course
comply with the features stated:
Ergonomy
Efficiency
 Administrative formalities (enrolment, payment etc.)
Cost-Effectiveness
 premises (soundproofing, signposting etc.)
 furniture/equipment at disposal
 maintenance of equipment

3.4.1.7. Satisfaction with Modalities


104

3.4.1.7.1.  Did the language learning modalities used in the course Transparency Clarity of Aims
comply with the features stated?
Consistency
Reliability
 Duration of lessons/units/course
Appropriateness
 Sequencing of lessons/units Relevance
 Types of assessment
 Exploitation of assessment results
 Did the language learning materials and equipment actually
used comply with the stated features?
 Types of teaching/learning materials/aids
3.4.1.7.2 Efficiency
 Complementary resources (multimedia centre, library, etc.) Ergonomy
 Extra-course activities (tutoring, homework, etc.)

3.4.1.7.3.  Did the qualifications of language teaching staff comply with Efficien Cost-Effectiveness
what was stated?: cy Ergonomy
 Teachers’ experience with the target public
 Teachers’ general language or LSP expertise and
experience
 Teachers’ experience in using the required
materials/equipment
105

F References

AFNOR (1998), Projet de norme internationale ISO 10015 – Management de la qualité – Lignes
directrices pour la formation. Paris (X 50 V – doc. N. 31).
Alderson, J./Beretta, A., (eds.) (1992), Evaluation Second Language Education, Cambridge, CUP.
Anderson, J. R. (1983), The Architecture of Cognition, Cambridge Mass..
Bredella, L./Christ, H. (eds.) (1995), Didaktik des Fremdverstehens, Tübingen.
Byram, M./Zarate, G. (eds.) (1997), The sociocultural and intercultural dimension of language
learning and teaching, Strasbourg.
Dickinson, L. (1987), Self-instruction in Language Learning, CUP.
EAQUALS (19953), A Pan-Europeans Inspection Scheme for Quality Language Providers. Trieste.
Ehrman, M.E. (1996), Understanding Second Language Difficulties, Thousand Oak CA.
Engel, U. et alii (1977/1979), Mannheimer Gutachten zu Lehrwerken Deutsch als Fremdsprache. 2
Vols. Heidelberg,
Feigenbaum, A. (1983), Total Quality Control, McGraw Hill, New York.
FRAMEWORK (1996), Modern Languages: Learning, Teaching, Assessment. A Common European
Framework of reference, Strasbourg.
Fruhauf, G./Coyle, D./Christ, I. (eds.) (1996), Teaching Content in a Foreign Language. Practice and
Perspectives in European Bilingual Education, Alkmaar.
Gardner, R (1985), Social Psychology and Second Language Learning: The Role of Attitude and
Motivation, London.
Gardner, R. /Lambert, W. (1972), Attitudes and Motivation in Second Language Learning,
Rowley/Mass.
Goethe-Institut, (1990) Handbuch für Spracharbeit. Teil 3: Lehrwerkanalyse, Goethe-Institut,
München.
Goleman, D. (1996), Emotional Intelligence, London.
Hawkins, E. (1984), Awareness of Language, Cambridge.
Holec, H. (ed.) (1988), Autonomie et apprentissage autodirigé, Strabourg, Conseil de l’Europe.
Hopkins, (1996), Guide for textbook and Materials Writers, Council of Europe, Strasbourg.
Horwitz, E./Young, D (1991), Language Learning Anxiety: from Theory Research to Classroom
Implications, Englewood Cliffs NJ.
Kast/Neuner (Hrsg.) (1994), Zur Analyse, Begutachtung und Entwickung von Lehrwerken.
Berlin/München.
Lynch, B. K. (1996), Language Program Evaluation, CUP.
Morgatroyd. S. and C. Morgan (1992), Total Quality Management and the School.
Buckingham/Philadelphia.
Negro, G. (1996), Organizzare la qualità nei servizi, Il Sole 24Ore, Milano.
Neuner, G./Hunfeld, H. (1993), Methoden des fremdsprachlichen Deutschunterrichts, Berlin.
OECD, (1992) High Quality in Education and Training, Paris, OECD 1992.
O’Malley, J./Chamot, A. (1990), Learning Strategies in Second Language Acquisition, Cambridge.
106

Reeves, N./Wright, C. (1996), Linguistic Auditing . A guide to identifying Foreign Language


Communication Needs in Corporation, Clavedon.
Richards, J./Lockhart, C. (1994), Reflective Teaching in Second Language Classroom, CUP.
Richards, J/Rodgers, T (1986), Approaches and Methods in Language Teaching, Cambridge.
Richterich, R. (1973), ’A model for the definition of language need of adults learning a modern
language”, Systems Development in Adult Language Learning, Strasbourg, Council of Europe.
Richterlich, R./Chanterel, J. (1977), Identifying the Needs of Adults Learning a Foreign Language,
Strasbourg, Council of Europe.
Sercu, L./Jensen, A./Jæger, K./Lorentsen, A. (eds.) (1995), Intercultural Competence, 2 Vols.,
Aalborg.
Serra Borneto, C. (ed.) (1998), C’era una volta il metodo, Roma.
Skehan, P. (1989), Individual Differences in Second-Language Learning, London.
Skehan, P. (1998), A Cognitive Approach to Language Learning, OUP.
Stern, H.H. (1983), Fundamental Concepts of Language Teaching, Oxford.
Tarone, E./Yule, G (1989), Focus on the Language Learner, Oxford.
Tomlinson, B. (ed.) (1998), Materials Development in Language Teaching. Cambridge.

You might also like