Quality Guide en
Quality Guide en
Quality Guide en
Guide21
Produced by
Jean-Claude Lasnier, A.C.F.C.I./A.G.E.R.C.E.L., France. agercel@porte-oceane.com
Petra Morfeld, International Certificate Conference, Germany. ICC_europe@compuserve.com
Brian North, Eurocentres Foundation, Great Britain. bnorth@eurocentres.com
Carlo Serra Borneto, University of Rome, Italy. serra@uniroma3.it
Preben Späth, PS Sprog & Edb, Denmark. p.spaeth@mail1.stofanet.dk
10/09/99
________________________________________________________________________________
Contents
C The Principles
D The Stages
F References
3
Preface
The idea to develop a guide that helps different target groups to identify aspects of quality with
regard to modern language teaching and learning is directly connected with one of the objectives
of the ”White Paper” Teaching and learning. Towards the learning society
(Brussels/Luxembourg 1996). The fourth general objective in the White Paper is ”Proficiency in
three Community languages”, in relation to which the European Commission proposes the
”introduction of assessment systems (including elaboration of quality indicators) and quality
guarantee systems, covering the methods and materials used to teach Community
languages”(page 68).
This guide is divided into five sections. The aims of the guide are introduced in Section A, while
Section B presents the background of our approach to quality and the structure underlying the
guide. Section C then introduces a set of general principles of quality. A presentation of the
three stages of quality management in the learning/teaching process is undertaken in Section
D. In the main body of the Guide, Section E, the general principles of quality are applied to an
analysis of these three stages. Section E1 deals with Design, Section E2 with Implementation
and Section E3 with Outcomes. Each of these stages is analysed into components (called
"sections") and sub-components (called "elements"). The general principles of quality are used to
formulate and cross-reference indicators relevant to the components of each stage of the
teaching/learning process.
A version of this guide is published as a CD-ROM. This CD-ROM is an interactive tool which
tracks the answers given to the questions in Section E and so enables the user to profile his/her
language learning provision at the three stages of the learning process: before (Design), during
(Implementation), and after (Outcomes). These profiles graphically portray strengths and
weaknesses and offer feed back in quality planning and quality control, which is available as a
print out. An abstract profile in relation to the general principles of quality introduced in Section
C is also offered.
For instance, in Section E1 (Design) the main aspects which have to be taken into account when
starting to write a course book or to set up a new language programme are described on the basis
of quality principles. Sets of questions are formulated which can be used as a checklist to make
sure that none of the relevant aspects have been overlooked or to find out to which extent the
different principles of quality have been taken into account.
4
3.A practical aim to provide an evaluation tool for teachers/educationalists and project managers to use
in assessing programmes and materials and increase motivation in giving a focus to efforts to increase
quality in the learning/teaching process.
The guide can also be employed as an instrument for the evaluation of existing programmes or materials,
e.g. by teachers who have to decide if a course book or a CD-Rom meets certain standards or by
programme designers or educationalists who want to know if a course has been successful in terms of
results and satisfaction on the part of their clients or stakeholders.
Introduction
In recent years the concept of and the concern for quality has shifted from the production branch more and
more into service and education, requiring differentiation with respect to the sectors involved. For instance,
quality in service is characterised by limited measurability, context sensitivity, heterogeneity, relevance of
the human factor, the affective dimension, strong relevance of the client’s needs etc. (cf. Negro 1996). The
concept of quality in modern language teaching and learning [MLTL] cannot rely on a consistent
tradition except for the more specific issue of course evaluation (cf. Alderson/Beretta 1992, Lynch 1996).
Only in recent years has it been applied in practical learning contexts, for example in EAQUALS (1995)
and O.E.C.D. projects (OECD 1992). EAQUALS [European Association for Quality Language Services] is
an attempt to develop a widely recognised set of standards with regard to the services necessary to organise
and run language courses. Comparable schemes are EXCELLANGUES or initiatives from other providers
of language training such as Eurocentres, International House, Bell School etc. Quality is also advocated in
a number of works produced for textbook writers in order to help them to design better materials (e.g.
Goethe-Institut 1990; Kast/Neuner 1994; Hopkins 1996).
Generally speaking, quality in modern language teaching and learning has to face a very specific issue: the
”cognitive factor”. Language learning is a process which affects and is affected by the cultural-cognitive
structure of the subjects involved. This has several implications on the concept of quality: psychological
and affective components should be considered a major issue. Such variables - for example needs,
motivation, social conditions, cultural barriers, social learning environment, cognitive styles and
preferences, subjective reactions etc. - are by their nature context bound. The traditional educational view
is that these variables cannot be assessed - or can be assessed only in a very limited sense - using
quantitative measurements. In this field, the ratio of ”soft”, qualitative (as opposed to ”hard” quantitative)
factors and data is higher than in any other kind of quality assessment context; higher even than in relation
to general education, which affects the cognitive structure in a differentiated, often less dramatic way.
identified and taken into account before quality can be achieved. The extent to which these needs are met
serves as a measure of the satisfaction of the client. Accordingly, the product/service will be constantly
adjusted and improved in order to reach an optimal matching of the users’ needs and the attributes of the
product/service itself.
The issue of needs orientation and constant improvement was taken up and further developed into the
"Total Quality Management" (TQM) approach. The TQM philosophy implies that the satisfaction of the
clients’ needs is the main concern of all parties involved in the production or delivery of a product/service.
Moreover, a fundamental feature of TQM consists in carefully monitoring every step in the process of
generating goods (production cycle) with relation to defined quality indicators. Such indicators
operationalise general principles of quality such as efficiency or transparency in the context of the
subprocesses (stages) concerned.
The challenge in transferring such a notion to MLTL or any other educational field is to apply principles
which have been successful in the original contexts in a differentiated rather than mechanistic manner in
order to put humanistic, interpersonal, cognitive, and affective factors at the centre of concern. On the
other hand, educational and - more specifically - MLTL theories have developed a range of general
propositions and methodological suggestions which themselves can be interpreted as principles of quality.
Therefore, in order to develop an independent concept of quality for MLTL, the findings of both
approaches have been taken into consideration and adjusted to form the theoretical background of the
guide.
The principles
The theoretical concept of the guide is based upon nine general principles of quality and their relationship
to the language learning and teaching process. The principles are broad superordinate notions which have
been developed by relating widely used concepts of quality to current MLTL theories and adjusting them
for our purposes. Each principle is divided into several sub-principles which represent its different
dimensions. The principle of Transparency, for example, consists of the four sub-principles Clarity of
Aims, Clarity about Achievement, Clarity of Presentation, and Clarity of Rationale. The demand for
comprehensive and comprehensible information about the objectives of a programme/material, about the
progress one has made and about the methods which have been applied to achieve these objectives has not
only become an objective in quality programmes of language schools etc. but is also implicit in educational
concepts like learner autonomy or reflective learning. Definitions for the nine principles are given in the
next chapter.
The stages
As is the case in comparable quality assurance schemes (e.g. ISO norms, TQM etc.) this guide
refers to three distinct stages of the process of provision: Design (before), Implementation (during),
Outcomes (afterwards). The characteristics and functions of these three stages can be summarised
as follows:
Design
The process of analysing and planning to provide for needs in context and of presenting the
resultant content and activities in an attractive and effective manner;
Implementation
The process of providing a positive learning environment and of adapting planned provision
flexibly and appropriately to circumstances (translating design into action);
Outcomes
The process of assessing gains in context in relation to resources and of ensuring satisfaction on
the part of the actors involved.
Each stage can be divided into smaller sections covering the different aspects of the teaching and learning
process. These sections are then themselves sub-divided further into "elements". This conceptual division
of the process is reflected in Section E which is divided into three parts reflecting the three stages.
6
Needs analysis
Profiles
Every indicator is connected to one or more related sub-principles. By ticking off the indicators the user
scores points with regard to the principle(s) in question. The final score for each principle is then calculated
7
with regard to the number of questions referring to it and the responses given by the users. The resulting
percentage is shown for each principle and gives an overview of the extent to which the principles are
represented in the particular programme/materials. The profile in relation to principles of quality offered by
the CD ROM is illustrated in Figure 2.
Print outs
The user is able to print the colour profiles and is also offered the option of two kinds of print
out. One prints to screen or printer a record of the low scoring indicators associated with
principles and sub-principles which were rated relatively low (less than 50%) in the evaluation.
The other print out reports low scoring indicators by segments (sections and elements) of the
stage profiles.
These print outs offer the user a diagnostic checklist of points which may be worth investigating
in order to improve the overall quality and effectiveness of the programme or materials.
Summary
Figure 4 below summarises the structure of the guide as discussed so far.
9
C1. Rationale
”Definitions of quality vary and, to some extent, reflect different perspectives of individual and society”
(Green: 17).
Given this caveat, we have tried to develop a concept of quality based on a set of general principles,
which should subsume ”different perspectives” and at the same time reflect common tendencies of our time
both in economic and in educational (especially Modern Language Teaching and Learning = MLTL)
theory.
In our approach, general principles of quality are very broad superordinate notions which represent a
distillation of the experience of the authors as well as of concepts commonly used in economics and
MLTL. They can be applied to any step of the teaching/learning process and provide an opportunity for
reflection and re-thinking the process itself in order to provide an improved design, as described above.
They have been revised and elaborated in the light of feedback during the consultation stage of the project.
A list of nine general principles of quality is presented below. They will be defined together with their
corresponding sub-principles which more specifically illustrate the different dimensions implied.
General principles belong to the ’upper’ theoretical layer of the model illustrated in Figure 4. Accordingly,
on the one hand they reflect theoretical premises (see below). On the other hand, they are conceptually
linked to the next layers in so far as the corresponding sub-principles are related to each aspect of the
learning and teaching process and to the quality indicators. Therefore, the sub-principles are very important
to ensure the consistency and coherence of the whole model.
The complete list of principles identified is the following:
1. Relevance
2. Transparency
3. Reliability
4. Attractiveness
5. Flexibility
6. Generativeness
7. Participation
8. Efficiency
9. Socialisation
C2.1 Relevance
Relevance refers to the appropriateness of an issue in terms of its accordance with the needs,
objectives and purposes of the users (learners, teachers, stakeholders etc.). Relevance can apply
to content as well as methodology.
The principle of relevance comprises the following sub-principles:
Learner Centredness
Programmes and materials take account of the personal characteristics of the learners and/or
their professional objectives (referring to: content, situations, language, skills). This requires a
good knowledge of the learners and of their needs and differentiation in the selection and
organisation of programmes and materials to meet them.
accountability
Programmes and materials meet social and political needs (referring to general educational aims).
Accountability is also highly relevant in terms of materials and programmes for more specific target groups
such as commercial clients and international exchange programmes.
appropriateness
Programmes and materials suit the learning/teaching context. Approaches, materials, examples and
activities take into account the social, cultural and educational characteristics of the learner (i.e. age, level,
customs).
Theoretical background: Relevance is a function of needs. The adjustment of a product or service to the
needs of the customers/clients is an essential concept of the TQM philosophy, especially in connection with
quality in service (Negro 1996). Needs can be implicit, i.e. so obvious that they may not even be
mentioned; they can be explicit (for example as a result of an audit); and finally they can be latent, i.e. the
client might not envisage particular aspects overtly, but would appreciate them if offered. The orientation
towards the needs of the learners in MLTL theory has a long tradition in E.U policy (see Richterich 1973;
Richterich/Chanterel 1977; cf. also Tarone /Yule 1989) and refers to individual as well as to cultural and
socio-political issues (cf. Reeves,/Wright 1996).
C2.2 Transparency
Transparency, as applied to programmes and materials, describes those features which
(cognitively) contribute to facilitating access and use.
The principle of transparency comprises the following sub-principles:
clarity of aims
Programmes and materials clearly indicate the learning objectives. In other words the user is
given the opportunity to determine whether the material or programme or learning
environment actually suits his or her needs. Furthermore, the skills taught and the target level
should be clearly specified.
clarity about achievement
Programmes and materials help learners become conscious of areas of success and of
weakness and inform them about progress made in relation to the learning objectives.
clarity of presentation
Programmes and materials present a clear and logical structure with appropriate,
comprehensible instructions and layout. Some relevant issues are whether progression is
implicitly or explicitly stated, and whether tasks, exercises and simulations clearly indicate
what the learners should do.
clarity of rationale
Programmes and materials provide the teacher with a methodological rationale, explaining
12
the language learning approach(es) used. For teachers it is important that the assumptions and
aims of the methodology are clearly explained.
Theoretical background: Transparency is a central issue in the client-led approach of TQM: the client
should be best informed about the characteristics of a product or service, in order to be able to make
choices, compare performances, and complain in case of unfulfilled promises. Similarly, transparency is a
key issue with regard to the organisation of language learning courses at private or non-private language
schools (cf. in particular EAQUALS). Transparency is also topical in approaches which aim at autonomous
learning (cf. Dickinson 1987; Holec 1988;) and in all forms of reflective teaching and learning (cf.
Richards/Lockhart 1994).
C2.3 Reliability
Reliability refers to the internal consistency of programmes and materials and the dependability of
contents and methods, thereby facilitating the implementation of the teaching/learning process.
The principle of reliability comprises the following sub-principles:
consistency
Programmes and materials show consistency between stated objectives and actual realisation.
A course purporting to provide the development of spoken language should, for example,
provide extensive opportunities for free fluency practice, and material for practising listening
to native-speakers interacting.
internal coherence
Programmes and materials are well constructed with a consistent inter-relationship between
their parts. Presentation, explanation, practice, exercise, feedback etc. should be coherently
inter-related.
methodological integrity
Programmes and materials reflect validated methodological premises in the selection of
activities, exercises etc. If a functional approach is chosen, then the situations and exercises
offered should be in line with that methodology and not reflective of a grammar-translation
approach with new labelling.
linguistic integrity
Programmes and materials present models of good, current language usage with correct and
socio-linguistically appropriate examples. The concept of authenticity is important here, as is
the need for idiomatically and phonetically correct language based on native speaker usage.
textual integrity
Programmes and materials respect the integrity and authenticity of the relevant genre in the
creation, adoption, and adaptation of texts. This does not mean that only authentic and
unedited texts can be presented, rather it implies that texts should be written, edited or
simplified in a way which retains key linguistic features.
factual integrity
Programmes and materials present accurate information, examples and statements and
authentic social behaviour.
practicality
Exercises included in programmes and materials can be successfully completed by the
learners concerned. One has to be sure that the exercises actually work, and that their degree
of difficulty corresponds to the language level of the learners concerned.
Theoretical background: The issue of reliability is reminiscent of the same concept in quality control
procedures, which have been developed as a part of quality management. Quality management also
includes quality assessment procedures. The ISO-norms play a leading role in this respect (cf. AFNOR
1998). In our context, the notion of reliability is mainly related to the planning phase of the
teaching/learning interaction, i.e. it refers to the design of programmes and especially materials. Therefore,
this principle has been elicited from the analysis of works concerned with the development of materials (cf.
Tomlinson 1998) the analysis of textbooks (i.e. Engel 1977/79, Kast/Neuner 1994), as well as from
13
discussions of teaching and learning methods and approaches (for instance Stern 1983, Richards/Rodgers
1986, Neuner/Hunfeld 1993, Serra Borneto 1998).
C2.4 Attractiveness
The principle of attractiveness summarises all the features of programmes and materials which
are appealing to the user and therefore contribute to enhance his/her motivation.
The principle of attractiveness comprises the following sub-principles:
user friendliness
Programmes and materials are easy to use (easily accessible, with a user-friendly interface
and appropriate illustrations). This issue is particularly relevant to multimedia programmes
and for autonomous learning.
interactivity
Programmes and materials create a dialogue with the learner, in that they offer facilities and
feedback, stimulate curiosity and generate an atmosphere of play.
variety
Programmes and materials are characterised by a range of activities, types of interaction and
working rhythms. Motivation in learning is enhanced by switches of emphasis.
sensitivity
Programmes and materials take the affective dimension into consideration, creating an
attractive environment and motivating activities to help create a relaxing atmosphere.
Furthermore, it is important for any learning environment that the learners’ progress is
encouraged, that correction does not stultify the desire to learn and that there is attention and
patience for weaker learners.
Theoretical background: Attractiveness, as intended here, is a notion connected with the client-
led TQM approach and with marketing principles in general. TQM, especially if applied to
services (as opposed to products) has a strong user-oriented bias (Negro 1996). The MLTL
interpretation of this issue is influenced by motivational theories of learning (e.g.
Gardner/Lambert 1972, Gardner 1985), the autonomous learning movement as well as by the
methodological discussion of affective approaches, which has been partly taken over by
communicative or action-oriented approaches (cf. FRAMEWORK 1996 among others).
C2.5 Flexibility
Flexibility accounts for the individual modes (both cognitive and affective) in approaching the
teaching/learning process. It refers to features of programmes and materials which are sensitive
towards the individual specificity of the user, including group differentiation in classroom
environments.
The principle of flexibility comprises the following sub-principles:
individualisation
Programmes and materials take account of learner characteristics like former learning
experience, learning styles, disposition for autonomous learning, etc. The material or
programme should allow different types of learners to use different activities or techniques to
approach a topic or problem.
adaptability
Programmes and materials allow for expansion, reduction, easier and more challenging
adaptation and exploitation for different learning purposes. The material or programme
should be open and flexible enough to allow the individual learners to work more in depth
with particular items.
14
Theoretical background: Educational theories in general and studies on the psychology of FL learning
(especially individual characteristics of learning) account for the notion of individualisation (cf. Skehan
1989, Horwitz/Young 1991, Ehrman 1996). While individualisation refers to the type of activities and
exercises, the sub-principle of adaptability is related to the number and the scope of activities, allowing the
learners to determine their own pace (cf. Tomlinson 1998).
C2.6 Generativeness
Generativeness accounts for the open-ended character of programmes and materials which
facilitate cognitive development and the transfer of what has been learnt in one context to more
general tasks and/or other contexts.
The principle of generativeness comprises the following sub-principles:
transferability
Programmes and materials encourage the transferability from controlled through guided to
free activities and the transferability of strategies, skills and contents to different contexts in
and outside the learning environment.
integration
From a linguistic point of view, programmes and materials build onto previous knowledge
(progression) and help learners to relate concepts.
language awareness
Programmes and materials help learners to develop awareness of the way language operates
and to pay attention to the way they use language and the way language is used around them.
cognitive development
Programmes and materials provide opportunity for learning to learn (problem solving,
strategy training, etc.) and promote the awareness of these aspects in the learners. By
explaining clearly what is actually going on in the learning process, and by providing the
learners with opportunities for reflection materials and programmes contribute to the
awareness of what learning is about.
Theoretical background: Since Generativeness is mainly concerned with the elaboration and processing of
(linguistic) knowledge, the main theoretical source for this principle are cognitive theories of learning (see
for instance Anderson 1983; Skehan 1998) as applied to language teaching and learning, for example the
notions of awareness raising (Hawkins 1984) and strategy training (O’Malley/Chamot 1990)
C2.7 Participation
Participation is concerned with opportunities to make choices and to share responsibility in the
process of teaching/learning.
The principle of participation comprises the following sub-principles:
involvement
Programmes and materials allow learners to be fully and actively involved in the learning
process. This requires that the activities - both in the cognitive and affective dimension - are
designed in such a way that the learner at any time has the knowledge and skills required for
taking part in the learning process.
personal interest
Programmes and materials allow learners to bring their interests, opinions and experiences to
the learning process, thereby making it personally meaningful. The materials and
programmes may achieve this by challenging, provoking, reassuring and confirming attitudes
the learners have.
partnership
Programmes and materials encourage learners to make choices, contribute to decisions, and
share responsibility for their learning. In some learning environments this may imply that
15
learners are consulted in the process of planning the next module. In others it may lead to the
definition of a learning contract - to do particular tasks, reach a particular level.
Theoretical background: Participation is one of the principles that seems self-evident. Everything that is
directly related to the learner and his/her interests, experiences or emotions is of personal relevance and
thus not only more motivating but also easier to remember. This is due to the fact that the affective
dimension seems to support the storage and retrieval of information (Goleman 1996) but also because
learning material that is related to the knowledge of the learners can be more easily processed than
something that is beyond their experience or imagination (O’Malley/Chamot 1990).
Partnership is the first step in direction of autonomy. The emancipation from the teacher is a central
demand among educationalists and foreign language teachers aiming to help individuals take on
responsibility for their own (learning) actions.
C2.8 Efficiency
Efficiency is an “economic variable“ relating resources to results.
The principle of efficiency comprises the following sub-principles:
cost-effectiveness
Time and resources are carefully exploited in the realisation of programmes and materials.
This aspect implies that there is a good match between the investment (in terms of money,
energy and time) and the results.
ergonomy
Programmes and materials demonstrate an optimal matching of requirements to needs in a
given context in order to generate an outcome.
Theoretical background: The concept of Efficiency has been adopted from the economic theory of
Management, Quality Control theory and TQM (Feigenbaum 1983). The way it is used here is strongly
reminiscent of its original use. It brings the economic factor into focus, the fact that language learning and
teaching has to be paid for and that prices have to cover the expenses of the provider of
programmes/materials on the one hand and yet be attractive for the potential client (e.g. learners) on the
one hand.
C2.9 Socialisation
Socialisation concerns “added value“ to a language programme which pursues the development
of additional skills beyond purely language competencies.
The principle of socialisation is fulfilled if programmes and materials pursue linguistic achievements
together with:
social skills
Programmes and materials aim at promoting the social skills of the learners such as the
ability to co-operate with others or to develop empathy. In addition to that, the learners are
encouraged to develop new concepts and to look at things from a different perspective.
intercultural awareness
Programmes and materials ask learners to reflect upon the knowledge and understanding of
their own culture as well as the cultures of other countries and their personal attitude towards
them. It is important to develop sensitivity towards the different and the “other“, and to
provide bilingual education where appropriate. Activities can include considering
“contrastive“ cultural patterns and discussing cross-cultural behaviour.
16
C3 Grouping of principles
The general principles listed above have been developed as a conceptual framework for quality in
language learning. It should be remarked that virtually no principle alone can assure quality
standards but that the notion of quality arises as the result of the convergence of as many principles
as possible in particular programmes and materials. Any conceptual framework in the context of
modern language teaching and learning has to include three main ”conceptual” dimensions of the
teaching/learning process which determine the conditions of its realisation:
a) the teacher/learner,
b) the context,
c) the means.
The teacher and the learner constitute the central dimension in the teaching/learning process, since
they represent the main actors involved. As far as the concept of quality is concerned, their
perspective can be summarised in following terms.
from the point of view of the learner quality implies:
1) opportunity for choice with optimal matching of provision to needs;
2) satisfaction and achievement;
from the point of view of the teacher (or: provider):
1) consideration for the needs and expectations of the users;
2) provision of a well-organised, efficient, user-friendly product or service.
The context represents the conditions under which the teaching/learning process takes place
determining the frame of reference and the departure point of any reflection on quality. The
variables influencing quality in this respect are (cf. FRAMEWORK 1996, 20-24):
the domain in which the teaching/learning process takes place (personal/public/occupational/
educational);
the situations which arise in a particular domain (institution, location, persons involved,
environment, events and operations);
physical, social, time conditions and constraints;
the user/learner’s mental context (cognitive factors, intentions, reflections, interests, needs and
motivations).
The means can be equated with the ”methods and materials ”as they are mentioned in the ”White
Paper”. They also constitute a major factor of quality in the realisation of the learning/teaching of
modern languages.
Starting from these basic reflections, the principles can be grouped according to the "conceptual
dimensions". From this point of view, the principle of Relevance becomes central, since it
applies to all three dimensions. The other principles are connected with one or two dimensions
and contribute in establishing an interface between them. Figure 5 illustrates this point, showing
the groupings of the principles centred on the principle of Relevance.
17
The box on the left contains those principles that concern the interface between the learner and the
material or programme in question. Clarity, User Friendliness and Motivation are some of the
aspects in this domain. The top box includes the principles of quality that concern the material or
programme ”per se” and the variables that are related to them, i.e. the learning environment and the
group of learners in question. The right hand box groups together those principles of quality that
deal with the personal and social aspects of the language learning/teaching process in general
(including cognitive and affective variables) and the way in which this is being taken account of
within the means used.
18
DESIGN IMPLEMENTATION
Quality in the design stage Quality in the implementation
relates to the specification of stage implies providing a
the learning content and positive affective atmosphere
activities on the basis of the and a supportive learning
analysed needs of the learners,
plus the presentation of these environment in which the
realisation of the design is
content and activities in the monitored and adapted flexibly
programme or materials. to the evolving needs of the
learners.
Before During
OUTCOMES
The quality of the outcomes
can be evaluated in terms of the
gains in competence and
awareness in relation to the
resources and conditions
concerned and also in terms of
the satisfaction of the
stakeholders involved.
Afterwards
Each stage of quality is divided into sections which represent the main aspects of the stage in
question; each section is further subdivided in elements which illustrate the specific contents of the
sections. Various types of questions, indicators, and examples are attached to each element (see
Section E). They are intended to serve both as an awareness raising tool and as a means to check
quality.
D2.1. DESIGN
Quality in the design stage relates to the specification of the learning content and activities on the
basis of the analysed needs of the learners including the presentation of these content and activities
in the programme or materials.
Design is divided into four sections (Content, Activities, Organisation, Presentation) and the
corresponding elements:
Content refers to the selection of the linguistic content, i.e. functions and notions, grammar, and
vocabulary, the kinds of texts and the themes to be covered.
Activities is used here to cover all types of actions carried out by the learners in order to learn a
new language. This comprises the skills, listening, reading, speaking, writing, interacting,
mediating (cf. Framework), as well as exercises and pedagogic tasks, "real life" tasks and
projects, and learning strategies.
Organisation at this stage refers to the inner structure of programmes/materials. It concerns the
design of the individual units as well as the curriculum design as a whole; the assessment
procedures used to evaluate the achievement of the objectives, and the way interaction
among the learners is organised.
Presentation refers to the packaging of contents and materials and the effect it has on the user. It
concerns both the aesthetic and the functional dimension of the material/programme and
includes primarily the following aspects: navigation, illustrations, layout, and colour &
music.
D2.2. IMPLEMENTATION
Quality in the implementation stage implies providing a positive affective atmosphere and a
supportive learning environment in which the realisation of the design is monitored and adapted
flexibly to the evolving needs of the learners.
Affective management includes steps to ensure that the learning process is an enjoyable,
stimulating and rewarding experience. Relevant concepts are framing & centring (e.g.
warming up, variable pacing, recuperating), matching (topics and activities to the specific
learners), stimulating (the curiosity and creativity of the learners), and encouraging (i.e.
challenging as well as praising and rewarding).
Monitoring activities include observation (e.g. of the performance of the individual learner or of
the progress of the teaching process), assessment (i.e. achievement in relation to the starting
point and/or the time and resources available etc.), feedback (to the learner on language
performance), and adjusting provision (e.g. of the presentation or the activities).
Support measures include people who can help, if things go wrong (apart from the teacher, i.e.
"tutors", service personnel), information tools (e.g. content overview, reference material),
20
awareness-raising & self-training tools (teachers’/learners’ guide, fast track facility to help
more independent learners etc.).
D2.3. OUTCOMES
Outcomes is concerned with the assessment of the effectiveness of the programme / material with
regard to the results obtained. The quality of the outcomes can be evaluated in terms of the
objective gains in competencies and awareness in relation to the resources and conditions given on
the one hand and in terms of the subjective satisfaction of the learners or the stakeholders involved
on the other.
One can differentiate between successful learning of the language concerned (Success), the
reactions of learners and stakeholders to the programme/material (Satisfaction), and the
achievement of broader educational aims (Educational Bonus).
Success from the learners’ point-of-view is evaluated by means of internal criteria (achievement of the
objectives stated, assessed in e.g. progress tests, teacher & self-assessments), external criteria
(measurable improvement in language proficiency established by comparing results on pre-course
proficiency tests to those on post-course tests or examinations), real life competence gains
(observable improvement in actual real life language use recorded in e.g. testimonials, self-
assessments and documentation), and depth of acquisition (long term acquisition and ability to cope
with new tasks).
Satisfaction can be considered from the point of view of both those receiving the service (learners,
etc.) and those involved in using the material or programme to provide the service (teachers
etc.). Satisfaction is measured by evaluating e.g. modalities (duration, sequencing, logistics),
facilities (materials, resources, support), activities (tasks, interaction, learning strategies),
and progress.
DESIGN
Quality in the design stage relates to the specification of the learning content and activities on the basis of the
analysed needs of the learners including the presentation of these content and activities in the programme or
materials.
Before
The following diagram gives an overview of the different aspects which have to be considered when planning
programmes / materials. The four main areas are
content specification: a description of the linguistic and non-linguistic contents;
activity specification: a description of different types of activities for practising and using the language;
organization: how the different parts are put together to form a coherent and consistent product;
presentation: the means used to meet both functional and aesthetic objectives.
Each of these areas is again subdivided into sub-categories, the content of which will be illustrated in detail in the
following section.
22
Quality in modern language teaching and learning implies in our view a learner-centred approach. Learner-
Centredness means here that the needs and expectations of the learners form the point of departure for the selection of
contents and activities and for the way these are organised and presented within the given context.
Content and methodology have to find a balance between the needs of the individual and the needs of society as a
whole. While the former refers to different learning styles and experiences, expectations, attitudes and reasons for
language learning, the latter concerns the needs of the economy and/or the stakeholders on the one hand and general
educational needs on the other. Educational needs cover competences such as social and intercultural competence.
This competence involves the willingness to interact with others and the ”ability to handle social situations” (van Ek
1976: 2) within one's own culture as well as in interaction with members of other cultures.
1.1.1 What has been done to ensure that the characteristics of the
target group have been adequately taken into account?
1.1.2 What has been done to ensure that the learning environment
has been adequately taken into account?
1.1.2.1 Has the relevant learning environment been identified: relevance appropriateness
relevance appropriateness
Distance course (”correspondence”);
relevance appropriateness
Home study?
relevance appropriateness
1.2 Content Specification
Content as it is used here refers to the linguistic learning
objectives (functions, notions, grammar and vocabulary), the
themes and the texts addressed during the course.
Being able to understand and express oneself in a foreign language
is the ultimate goal of communicative language teaching and
learning. Programmes / materials should provide learners with the
linguistic and strategic means to use the foreign language linguistic
appropriately and successfully in the situational context that is integrity
reliability
relevant to the group of learners in question.
relevance learner-
Appropriately means that learners need to be able to say the right Centredness
thing at the right time. To do that, they have to be able to take the
cultural and the social background against which communication
intercultural
takes place into account. Learners therefore have to be equipped socialisation
awareness
with sufficient awareness of or knowledge about social
conventions, social rituals and politeness conventions to be able to
avoid or repair misunderstandings.
Successfully means that the learning environment and the time and
resources available need to be taken into account when efficiency economy
determining the learning load: the amount of content. In addition,
since people learn in different ways, success requires enough flexibility individualisatio
variety to allow learners and teachers to cater for different n adaptability
interests, learning styles and paces of learning. participation personal
interest
Relevant means that the content specification should be based relevance learner-
upon the present and future communicative needs of the group of Centredness
learners concerned.
1.2.1 What has been done to ensure that the choice of linguistic and
thematic content is relevant for the learners?
1.2.1.1 Has a formal analysis of needs been conducted for this learning relevance learner-
population as a whole? Centredness
Questionnaire
Interviews with a sample of respondents
Have the needs and interests of this particular group been learner
1.2.1.2 relevance
investigated? Centredness
Questionnaire participation personal
interest
Interviews with a sample of respondents
Discussion with the group
Consultation of relevant resources such as Waystage Level,
Threshold Level etc.
24
1.2.2 What steps ensure that the choice of the linguistic and
thematic content is relevant to present or future vocational
requirements?
1.2.3 What has been done to ensure that the language used in
programmes/materials represents correct, appropriate,
current usage?
1.2.4.1 Has any factual content been checked? reliability factual integrity
Sources given and checked for reliability
Facts and dates regularly updated
1.2.5 Themes
Language is normally used as a means for conducting interactions
25
1.2.5.1. What has been done to ensure that the choice of themes is
interesting for the learners?
1.2.5.2 What steps ensure that the choice of themes take educational
and / or intercultural aspects into account?
1.2.5.2.1 Raise awareness of cultural and gender stereotyping? socialisation social skill,
intercultural
Comparing and contrasting conditions and patterns of
awareness
behaviour
Pointing out instances of inter-cultural differences in
materials and scenarios used for teaching
1.2.5.2.2
Offer opportunities for cross-curricula links appropriate to the transferability
generativeness
level?
cognitive
Integrating information from other subjects (history, development
geography) into language lessons
Teaching an aspect of another subject as a limited
immersion experience
Learning another subject through the medium of the foreign
language
Exploiting the internet
1.2.6 Texts
The term 'text' refers to spoken as well as to written language. generativeness
Texts play three main roles in foreign language learning and participation
teaching:
integration
- a source for comprehension practice and a means of getting
personal
26
A wide range of text types and genres appropriate to the needs and attractiveness variety
expectations of the target group in question is necessary.
relevance learner-
Furthermore, text types and genres should also be in line with the
Centredness,
methodological approach adopted and be chosen in such a way
appropriateness
that the users are given the opportunity to practise different
reading or listening skills.
reliability
If texts are adapted for didactic reasons, the original features of method;
the genre need to be preserved in order to help learners recognise integrity
and, if appropriate, acquire such features. Language and current
reliability textual integrity
factual information in texts also needs to be correct and up-to-
date. linguistic
integrity
reliability
factual integrity
reliability
1.2.6.1 What steps have been taken to choose texts which will increase
motivation?
1.2.6.1.1 Do the texts meet the learners personal interests and participation personal
emotions? interest
Fields of professional interest
Music, culture
Hobbies / personal interests
Love / romance interest
Current issues
Is a variety of different kinds of texts and documents variety
1.2.6.1.2 attractiveness
provided?
personal
participation
Short anecdotes interest
Stories
Information carrying text
Texts from encyclopædias, travel guides etc.
Newspaper articles
Editorials / short essays
Cartoons
Comics
Poems
Songs
1.2.6.2 How are the texts used in the programme / materials matched
to didactic purpose?
27
1.2.6.2.1 Is the selection of texts consistent with the way texts are to be reliability methodological
exploited in the didactic methodology adopted? integrity
Dialogues for a situational / audio visual approach
Model presentation texts for a classic- inductive approach
(presentation – comprehension - pattern practice - transfer)
Authentic source materials for a needs-based approach
Amusing drawings, cartoons, short stories, drama scripts for
a "learning-centred" story-based approach
Literary texts for a grammar-translation method
methodological
Are appropriate texts used to present language forms in a reliability
integrity
meaningful context?
1.2.6.2.2
Presentation texts which illustrate use of the target forms
Texts for comprehension with occurrences of the target
forms
Dialogues, prompt sets for fluency practice methodological
reliability integrity
Texts for language awareness activities
1.2.6.2.3
Are appropriate texts selected to practise specific reading
skills?
Searching for specific information and/or matching
information to needs
(authentic notices, adverts, leaflets, brochures etc.)
Reading for main ideas
(short articles and prose texts; longer prose texts)
Reading for detailed comprehension
(short articles and prose texts)
Reading to locate information
(longer factual and opinion-bearing texts; reference works)
Reading for fluency (extensive reading)
(longer texts, short stories etc.)
28
1.2.6.2.4 Are appropriate texts selected to practise specific listening reliability methodological
skills? integrity
Listening for specific information and/or matching
information to needs
(announcements, recorded messages, news broadcasts,
adverts, etc.)
Listening for main ideas
(conversations, talks, presentations, broadcasts etc.)
Listening for detailed comprehension
(conversations, extracts from short talks / presentations,
documentaries etc. )
Listening for fluency (extensive listening)
(short stories, songs etc.)
involvement
1.2.6.2.5 Has the appropriateness of the difficulty and complexity of participation
the texts for the target group been checked? textual integrity
reliability
Consultation with appropriate teachers practicality
reliability
Comparison of vocabulary and structures with content
inventories (e.g. Threshold Level; examination specifications)
or linguistic inventories (corpora; frequency lists etc.)
Analysis of cognitive complexity – information density,
rhetoric pattern, presence or absence of distracting
information- etc.)
Trialling with representative classes.
1.2.6.3 What steps have been taken in order to ensure the integrity of
the different types of texts chosen?
1.2.6.3.1 Has each adapted text been checked to guarantee that the
specific textual conventions of the genre concerned (cohesion,
reliability textual integrity
coherence, structuring, signalling etc.) have been preserved?
Conscious comparison by the materials developer
Consultation with experts in field
A discourse analysis / text linguistics analysis
1.2.6.3.2
Has each custom written text been produced so as to comply
with the textual conventions of the genre concerned (cohesion,
coherence, structuring, signalling etc.)?
reliability textual integrity
Conscious comparison by the materials developer
Consultation with experts in field
A discourse analysis / text linguistics analysis
1.2.71.1.1 Was a curriculum specification used to list language functions relevance learner-
relevant to the learners concerned and to link functions to centredness,
grammatical structures appropriate to the level concerned? appropriateness
reliability
Collating relevant existing sources (course books, linguistic integrity
syllabuses from other projects etc.)
Consultation of standard reference works (e.g. Threshold
Level)
Derivation from own needs analysis
Is language presented in a functional context?
1.2.71.1.2
Real life situations appropriate to these learners
reliability
linguistic integrity
Realistic purposes
methodological
Identification of new language with relevant intentions and integrity,
appropriate moves in the situation. consistency
Are learners presented with and encouraged to collect
1.2.71.1.3 alternative ways of expressing things?
transferability,
Substitution table of alternative forms (e.g. Can you, Could cognitive
30
1.2.71.1.4 Are notions (e.g. possibility, probability, time, shape etc. ) reliability methodological
developed systematically? integrity,
internal coherence
Situations used as a vehicle for presentation
Associated with related functions
Recycled and extended at intervals
methodological
Is the acquisition of language to express different notions
1.2.71.1.5 reliability integrity,
linked to the organisation of relevant linguistic forms (e.g.
internal coherence
time/tense; space/prepositions, shape/adjectives;
possibility/modals)?
Notions presented with related structures
Cross referencing provided in a chart or index
Notions recycled and linked to new language as new
language is introduced
1.2.71.1.6
Are learners encouraged to “unpack“ and analyse functional
language
phrases learnt as prefabricated chunks? generativeness
awareness
Grammatical patterns occurring in such chunks pointed out
as they are encountered in the syllabus
Language awareness-raising activities to identify underlying
grammatical rules
1.2.7.1.2 What has been done to ensure that pragmatic features are
adequately represented?
Routines (verbal/non-verbal/paraverbal)
Rituals (verbal/non-verbal/paraverbal)
Gambits
Etc.
1.2.7.1.2.4 Discourse signalling reliability textual+linguistic
integrity
Cohesion markers
Evidence of coherence
Openings and closures
Etc.
1.2.7.2 Vocabulary
Vocabulary refers to all lexical items including multi-word chunks relevance learner-
and idiomatic expressions. Vocabulary should be selected centredness,
carefully to focus on items that have a high priority for the target accountability
group in question. flexibility individualisation
Learners should be provided with sufficient comprehension generativeness cognitive
materials to acquire a wide passive vocabulary in addition to their development
learnt repertoire, and encouraged to note interesting new
language
expressions to build up a personalised wordlist. generativeness
awareness
In addition to this, learners need practice in relating words to each
other e.g. recognising and building derivatives from word roots;
associating similes and opposites. flexibility
individualisation
Finally, varied storage and retrieval techniques should be provided attractiveness variety
to cater for different learning styles and keep up motivation levels. generativeness
cognitive
development
1.2.7.2.2 What has been done to ensure that the approach to vocabulary
work is clear and effective?
methodological
1.2.7.2.2.1 Is vocabulary practised or activated in a systematic and varied reliability
integrity,
way?
internal coherence
Vocabulary exercises consistently connected with the other practicality
attractiveness
activities variety
A systematic progression in the treatment of vocabulary
Regular recycling of vocabulary
variety
Are learners introduced to a variety of techniques for individualisation
1.2.7.2.2.2 vocabulary association and memorisation in order to cater for
different learning styles? attractiveness
cognitive
flexibility
Mind-maps development
Lexical sets generativeness
Flow charts
Tree structures
Contrasting lists
clarity of
Etc. presentation
Are vocabulary items listed?
1.2.7.2.2.3
General list at the end of material transparency
1.2.7.3 Grammar
Grammar describes the system of linguistic rules that underlies the
target language. Criteria are needed to decide how these rules are
selected, described and presented to the group of learners in
question. Equally important are the activities that are carried out
to ensure the consolidation and retention of new input.
1.2.7.3.1.1 Is the form of language description chosen in line with the reliability consistency
overall approach taken in the programme/material? methodological
integrity
The use of language rather than description of language is
the basis of categorisation
Grammatical structures related to functions
Appropriate reference works
Do the forms chosen illustrate a wide functional range for a
1.2.7.3.1.2 variety of contexts? generativeness transferability
Selection based on an analysis of functions necessary in the
contexts (ContextFunctions necessaryGrammatical
exponents
Cross-referencing during planning of grammatical forms to
the different functions and contexts
Provision in materials of tables matching forms to functions
1.2.7.4.1 Is pronunciation and intonation a specific topic in the design of reliability consistency
the programme/material?
Pronunciation drills (on cassette) for
word stress
sentence stress
thought groups
pitch
35
voicing
stops and continuants
aspiration
intonation
Intonation markers (colour, arrows, contours, etc.) added to
written texts
1.2.7.4.2
Is authentic material providing an appropriate variety used for
reliability linguistic integrity
pronunciation/intonation reference?
Radio and TV example
Different genres (News, drama, interviews, etc)
1.2.7.4.3
Has the social, emotional and intercultural relevance of
pronunciation been dealt with?
Examples of misunderstandings socialisation intercultural
awareness
Examples of lack of social acceptance due to pronunciation generativeness
problems language
awareness
Is the importance of rhythm and speed in fluency stressed?
1.2.7.4.4
Practice in placing emphasis on key message by change of
stress and rhythm reliability
methodological
integrity
Catenation practice (linking non-stressed sounds and words)
1.3.1.1 Have the different types of activities been cross-checked with relevance Learner
the results of the needs analysis? Centredness;
accountability
Has the social and cultural background of the learners been
1.3.1.2 relevance
taken into consideration in selecting or designing the appropriateness
activities?
1.3.2 What steps have been taken to ensure that the main
characteristics of activities are taken into consideration ?
1.3.2.1 Have the purpose and aims of the task been specified:
Purpose: generating motivation, consolidating the weeks transparency clarity of aims
work; assessing learning reliability consistency
Learning objective: skill, language point, strategies
1.3.3 What steps are taken to ensure that activities are success-
oriented?
1.3.3.1 Has account been taken of the circumstances in which the
activity will be used?
relevance appropriateness
Materials, data, equipment, realia available reliability practicality
efficiency ergonomy
Duration
flexibility cost-effectiveness
Amount of freedom/constraint generativeness adaptability
flexibility internal coherence
Availability of tools, supports and help facilities
attractiveness integration
participation user-friendliness
involvement,
personal interest
method. integrity
Is the task adequately prepared and followed up so that
1.3.3.2
learners understand what the purpose of the task is? user friendliness
attractiveness
Adequate pre-task preparation cog. development
generativeness
clarity of
Guidance / notes for the learner transparency
presentation
Post-task de-briefing and feedback reliability method. integrity
generativeness integration,
flexibility transferability,
participation cognitive
development
individualisation
involvement
1.3.3.3 Have exercises and tasks been tested for practicality? reliability practicality
Clear, comprehensive instructions generativeness cog. development
participation involvement
Check for "bugs" (illogicality, incomplete editing, incorrect
transparency clarity of
language)
presentation and
Check that relevant learners could complete the task of aims
Check that learners could work independently when
necessary
37
1.3.4 Skills
Skills have traditionally been considered in terms of the 1960s
“four skills” model. Skills refer to the use of language – language
proficiency traditionally being considered a combination of
knowledge and skill.
Skills are rarely encountered in isolation, but the distinction
between the four skills continues to have a psychological reality:
being a fluent speaker may be very different from being a fluent
writer. For this reason, this section adopts a classification
approach which is a compromise between the four skills and the
approach in the Common European Framework.
1.3.4.1. What has been done to take different learner characteristics
into account?
1.3.4.1.1 Have exercises and activities been designed in order to be relevance appropriateness
appropriate for the personal characteristics of the learners:
participation personal interest,
Age involvement
Level
flexibility
individualisation
Interests
Learning experience
Learning style?
Is the cultural background of the target group considered in the
organisation of skills practice?
1.3.4.1.2. relevance
appropriateness
CORE INDICATORS
1.3.4.2.1. Are learners introduced to relevant practice and performance
techniques for each skill? generativeness cognitive
development
For spoken texts
For written texts reliability
methodological.
For watching a programme integrity
For initiating a conversation
For keeping up a conversation
For overcoming a spoken communication problem
For writing a letter
Etc.
1.3.4.3 What steps ensure that the skills training is efficient in relation
to time and means available?
1.3.4.3.1 Is the skills training integrated into the curriculum with links efficiency ergonomy
to other activities in the unit?
reliability internal coherence
Listening and reading texts integrated round a theme
Use of information from receptive practice in productive
practice
Integration of information from different sources
1.3.4.3.2 relevance appropriateness
1.3.4.3.3.1.1 Are activities provided which help the learner to achieve a reliability consistency,
fluent command of new language being learnt?
methodological.
Warm-up activities such as description of situation, integrity
introduction of reference vocabulary, brain storming in
mother tongue etc
Pair work with 4 line dialogues
Prompt cards or flow charts outlining "moves" for
improvised dialogues
Role plays
Games
Quizzes
Etc.
Are activities provided which help the learner to integrate new
1.3.4.3.3.1.2 generativeness
language into their overall repertoire?
transferability,
Information gap exercises, integration
reliability
Problem solving tasks consistency,
methodological.
Consensus and discussion tasks
integrity
Simulations and role-plays offering a variety of contexts and
(social) roles
Are activities offered which develop learners’ sensitivity to the
appropriateness of certain items/patterns/registers in a specific generativeness
context?
Activities which help learners to identify speech functions lang. awareness
1.3.4.3.3.1.3 and intention
Activities exploiting photos, illustrations, videos etc. to
sensitise learners to non-linguistic contextual cues
Comparison of politeness convention, discourse markers
etc. in the first language and in the target culture
Language awareness activities on dependence of language
on context, role and status of speakers
Are speaking activities connected with or embedded in other
activity types? generativeness
Follow up or precursor to language practice reliability
1.3.4.3.3.1.4
Use of listening / video / reading text as a stimulus for transferability,
speaking practice cognitive
development
Integrated skills activities, e.g. "in tray" activities, case internal coherence
studies etc.
Project work
generativeness
Do programmes / materials provide tasks which by providing a cognitive
1.3.4.3.3.2.2 communicative urgency encourage learners to apply these development
strategies? transferability
Rephrasing, overgeneralising, substituting a similar word participation
Using non-verbal signals such as gestures, mime, sounds
involvement
Trying to invent foreign sounding versions of first language
words;
Trying to create new words
1.3.4.3.4. Writing
Writing practice has two quite different uses in a language
learning programme:
transparency clarity of
(a) Consolidation for language development.
rationale,
(b) Development of writing skills.
clarity of aims
Writing skills can also be seen from two quite different
perspectives, in relation to:
relevance
(a) the acquisition of the rhetoric and socio-linguistic conventions Learner
associated with important, relevant genre (e.g. business Centredness,
letters) accountability
(b) the development of an ability to interact with others, to reflect
and express oneself in a creative and personal way
textual integrity
Writers have to organise their ideas in a systematic way and have reliability
ling. integrity
to be precise to make themselves understood because they cannot
take so many things as "givens" in the context. In addition to this,
spelling and grammar mistakes are more obvious in a written than reliability
Ling. integrity
in a spoken text. Writing in a foreign language thus requires textual integrity
planning at the beginning and constant reviewing of expressions,
content and organisation before the text is given its final form. relevance
appropriateness
The degree of assistance learners receive for content, organisation flexibility
and language depends on the needs and linguistic level of the individualisation
target group. Learners are often guided from controlled practice to
free production, but with some groups of learners this order of
activities might be reversed.
1.3.4.3.5 Listening
Understanding spoken language requires interaction between the
listener's knowledge of the world and expectations (schemata) and
generativeness integration
the information being transmitted by the speaker. The listener has
first to recognise sounds and identify the meaning of words and
sentences. On the basis of their existing knowledge and
experiences the listener then forms hypotheses of what is going to
be said next and adjusts these hypotheses constantly as new reliability methodological.
information comes in. Non-linguistic cues and prosodic elements, integrity
e.g. intonation, stress, rhythm, background noises and the
situational context help the listener in cases of ambiguity and
uncertainty.
1.3.4.3.5.1. How does listening practice help the learner to decode and
chunk speech under different conditions?
Do programmes / materials provide:
Exercises which help the learners to automatise decoding
processes?
1.3.4.3.5.1.1. reliability methodological.
Training in recognising sounds, minimal pairs etc. integrity
Training to help learners "chunk" incoming text
Training to compensate for interference, background noise,
poor fidelity
generativeness
integration
Pre-listening activities to activate existing knowledge? reliability cog. Development
1.3.4.3.5.1.2. methodological.
Activities to demonstrate the importance of inferring and
Integrity
anticipation
practicality
Activities to identify the purpose of the listening and select
an appropriate "while listening" task
Tasks to help learners to activate a "frame" for the situation
reliability
in order to form hypotheses
generativeness methodological.
Integrity
practicality
integration
cog. Development
1.3.4.3.5.2 How does listening practice help the learner to understand
messages from a variety of sources?
43
1.3.4.3.6 Reading
There are certain similarities in the processes involved in listening generativeness integration
and reading, both being types of reception. However, the fact that
generativeness cog. development
the whole text is available makes reading a more conscious
activity, with a wider variety of strategies with which to approach generativeness cog. development,
a text for different purposes. Given the attainment of a certain transferability
linguistic threshold, a good first language reader can be
encouraged to transfer such strategies.
1.3.4.3.6.1 How does reading practice help the learner connect what they
are reading to previous knowledge and activate appropriate
strategies to understand the text?
Do programmes / materials provide:
1.3.4.3.6.1.1 Pre-reading activities to activate previous knowledge? generativeness Integration
cog. development
Tasks to encourage inferring and anticipation
reliability methodological.
Tasks encouraging the learner to exploit cues in layout, integrity
titles, paragraphing, , source of text, etc. to form inferences
44
Tasks have to be practical and feasible within the given context relevance appropriateness
and have to be designed in such a way that instructions are clear reliability practicality
and precise. Institutional or other constraints such as length of
efficiency ergonomy
lessons, equipment, etc. also need to be taken into account.
46
1.3.6.1 What steps are taken to ensure that "real life" tasks and
projects help to bridge the gap between classroom practice and
real language use?
1.3.6.1.1 Do the tasks simulate language use in the real world? generativeness transferability
socialisation social skills
Discussing how to approach a problem
Gathering Information from written or recorded documents
Gathering information from (native speaker) informants
Presenting ideas and arguments in a meeting
Presenting proposals and findings to an audience
Reacting to ideas / proposals / findings etc.
Etc.
1.3.7.3.1 What has been done in order to stimulate the use and training
of strategies?
Do programmes / materials include
1.3.7.3.1.1. Procedures to find out about existing strategies?
Teacher observation checklists generativeness cog. development
Questions to help learners reflect on current strategies transparency clarity of
presentation
Questions to help learners make links to other skills and
forms of learning
Learner diaries,
Class discussions
Etc
Suggestions for using and/or training new strategies?
1.3.7.3.1.2.
cog. development
generativeness
Reading differently for different purposes adaptability
flexibility
Ways of using radio, TV, songs to help listening
Ways of being active in the learning process
Ways of memorising in a differentiated way to help
retrieval (key word techniques, imaging, associograms,
making diagrams, making tables, hierarchical diagrams)
1.3.7.3.2 What has been done in order to help learners find out how,
when, where, how often they learn most efficiently?
CORE INDICATORS
Do programmes / materials: generativeness cog. development
participation partnership
1.3.7.3.2.1 Encourage learners to set themselves objectives?
Presentation of ways of personalising content being learnt
Provision of schemes, charts, indexes to help planning
Suggestions on setting priorities
1.3.7.3.2.2 Introduce learners to self-assessment techniques?
generativeness cog. development
Observation activities participation partnership
transparency clarity about
Scales and checklists to rate performances
achievement
Comparison of self-assessment with performance on an
actual activity
Encourage learners to try out different learning techniques and
1.3.7.3.2.3
modes to find those that suit them best?
Wide range of techniques presented industrialisation
flexibility
cog. development
Linking of techniques to learning styles generativeness
partnership
participation
Feed-back suggesting an appropriate pathway through the
programme / material
Opportunities for discussion and reflection
1.4 Organisation
Organisation refers to the structure of programmes / materials. It
describes the principles according to which the content is
organised throughout the programme / material (Curriculum
design), and the organisation of contents within a unit (Unit
design).
Organisation concerns the co-ordination of Content and Activities
and is signalled through Presentation. Essentially, Organisation
involves:
- Chunking: grouping small bits into meaningful elements, and/
or breaking down large areas into elements
- Integrating: matching activities and content; mixing and
synthesising different strands of content and different types of
experience
- Sequencing: presenting things in a logical, flexible order
- Referencing: showing how parts relate to each other and to
the whole.
50
1.4.1 What steps have been taken to ensure that chunking and
integrating is successful?
1.4.1.1 Has the adequacy of connections and sequences been checked? Reliability Internal coherence
Organisation of content and activity specifications into
hierarchical lists
Classification of exercises and activities with categories
from the content and activity specifications
Use of a database to classify elements in order to identify
potential links
Use of a flowchart during authoring to plot connections
graphically
Has the plausibility and acceptability of matches between
1.4.1.2 Reliability Methodological
content and activity or of media and skill been checked?
integrity
Relevance
Workshops brain-storming matches and connections
Appropriateness
Consultation with teachers/ learners
1.4.1.3
Has the feasibility of the size of chunks and sophistication of
Practicality
the matching been checked in relation to cognitive load? Reliability
Appropriateness
Consultation with teachers/ learners Relevance
Observation of pilot use
1.4.2 What steps have been taken to ensure that important elements
have not been omitted?
1.4.2.1 Have exercises and activities been cross-referenced to Reliability Consistency
categories from the content specifications?
Manual collation
Use of database sorting, to identify inadequate coverage.
Consultation with teachers/ learners
Have vocabulary glossaries, grammar summaries, lists of
1.4.2.2. Reliability Consistency
functions, been checked to ensure that all exponents occurring
are included?
Manual collation
Use of a database sorting to identify inadequate coverage.
Consultation with teachers / learners
1.4.2.3. Curriculum design
The term curriculum design is generally used to refer to the
definition of appropriate objectives at successive levels, the
organisation of those objectives into strands, the integration of the
strands into didactic units, the provision of suitable activities and relevance appropriateness
materials to practise and learn the knowledge and skills
flexibility individualisation
concerned, and the assessment of the extent to which the
adaptability
objectives are mastered.
appropriateness
The organisation of the curriculum is dependent on the conditions relevance
cost-effectiveness
and constraints of the learning context. Issues to be considered efficiency
51
include: cost-effectiveness
- the time and resources available for the course as a whole, for
each strand and for each lesson;
relevance
- the rhythm and intensity of the course (2 hours a week / 20
hours a week) transparency
Clarity of Aims,
- the roles of different learning modes (class work, homework, clarity of rationale
media-led self study, project work, etc).
reliability
To help learners to orient their learning, set priorities and made consistency
informed choices, the principles and objectives underlying the
curriculum should be made transparent and applied in a consistent
way.
1.4.2.3.1. What has been done to ensure that the curriculum design is
transparent and coherent?
1.4.2.3.1.1. Does the curriculum present and describe an adequate set of Transparency Clarity of Aims
Levels/Stages?
Reliability Internal coherence
Objectives for the different Levels/Stages developed from
Relevance Learner-
the actual achievement of the learners concerned
centredness
Workshops undertaken to ensure that teachers/learners can
Appropriateness
see the difference between Levels or Stages
Accountability
Progress traced to check the Stages/Levels represent a
development path
Does the curriculum present and describe an adequate set of Clarity of Aims
1.4.2.3.1.2. Transparency
Content Strands?
Appropriateness
Relevance
Theoretical basis to the selection and grouping
Method. Integrity
Reliability
Consultation of models (e.g. the Common European
Framework)
Feedback to check that teachers/learners can understand and
work with the categories
1.4.2.3.3.1. Have steps been taken to check that the learning load and Reliability Practicality
number of exponents offered is realistic and challenging?
Consultation with teachers
Piloting on groups representative of the range of ability
Investigation of workload in lesson phases, at home
Teacher feedback
Learner feedback
1.4.2.3.3.2.
Have criteria such as frequency of use, a wide range of
Relevance Learner-
possibilities for application, etc. been taken into consideration
centredness
to restrict the learning load and set priorities? Transparency
Clarity of Aims
Low priority exponents been cut out or relegated to a
reference section
Core practice and extra practice separately identified
Consolidation exercises on core content for work at home
provided
Have exponents been explicitly marked as to their importance
Learner-
1.4.2.3.3.3. for reception, production or their usefulness for examinations? Relevance
centredness
Systematic exposure to exponents learners should recognise Accountability
but not necessarily master actively
Identification of aspects relevant for examination
Is the language used to describe or explain rules and structures
1.4.2.3.3.4. Relevance Appropriateness
adjusted to the knowledge of the target group?
Reliability Practicality
Consultation with teachers
53
1.4.2.3.5. What steps are taken to ensure that the relevance of an activity transparency Clarity of Aims
to the objectives is clear to the user?
clarity of
presentation
Transparency Clarity of Aims
1.4.2.3.5.1. Are learners given an overview of the curriculum as well as the clarity of
information they need for the purpose at hand? presentation
Explanation of each content strand
Overview offered of each Level/Stage
Overview of the Levels/Stages as a whole
Is there an explanation of how the organisation of the course Clarity of Aims
1.4.2.3.5.2. Transparency
serves the achievement of the learning objectives in question?
clarity of
Table of contents showing objectives presentation
Introduction explaining content strands and objectives
Summaries at key points to aid self-assessment
1.4.2.3.6. What steps are taken to ensure that the organising principles transparency Clarity of Aims
are clear to the teacher? clarity of
presentation
1.4.2.3.6.1. Are teachers given an adequate overview of the curriculum as Transparency Clarity of Aims
well as the information they need for the purpose at hand?
clarity of
Explanation of the content strand presentation
Overview of the Levels/Stages as a whole
Is there adequate documentation at differing levels of detail for
54
1.4.2.4.1. Which steps ensure that the organisation of the units meets the relevance Appropriateness
conditions and constraints of the learning context?
1.4.2.4.2. What has been done to make the structure and the objectives transparency Clarity of Aims
of the units transparent to the learners? clarity of
presentation
Transparency Clarity of Aims
clarity of
1.4.2.4.2.1. Are learning objectives stated explicitly for each unit?
presentation
Appropriate title, headings and sub-headings
Objectives listed in the table of contents
Index / Grammar summary / Word list at the end
1.4.2.4.2.2. Are units easy to distinguish from each other? Transparency
Clarity of Aims
Beginning and end clearly marked clarity of
presentation
Standardised presentation
1.4.2.4.3. What has been done to ensure that learners' differing level and flexibility individualisation
learning styles are catered for? adaptability
1.4.2.4.3.1. Is the structure of the unit open enough to expand or reduce Flexibility Adaptability
content according to the needs of a particular target group?
Optional exercises
Core exercises identified
Easier exercises first
Suggestions for supplementary activities
Are different learning modes catered for?
1.4.2.4 3.2. Flexibility individualisation
Range of inputs (audio, print, illustration, video)
Range of text types (story-based; expository; interactional)
Range of activity types (accuracy-focus; fluency focus)
Balance of conscious practice / games
56
1.4.2.4.4.1. Are there structured steps between presentation - controlled Generativeness Integration
practice - free practice (and/ or the reverse) to foster
acquisition of new exponents?
Controlled-Guided-Free practice
Careful slow practice - Fast spontaneous use
Focus on form through pre-activities and notification of
post-activities
1.4.2.4.4.2. Are there suggestions for transfer activities, for personalisation Generativeness Transferability
of practice, taking ownership of new language?
Exploitation of students' experience
Personal transfer activities
Suggestions for extra-curricula practice
Collecting examples, keeping a scrapbook, preparing a
portfolio
1.4.2.5. Interaction Management
The type of interaction generated should take into account the Flexibility individualisation
learning experience of the learners, the teaching experience of the attractiveness sensitivity
teachers and the approach to language learning being applied, as
reliability consistency
well as the learning objectives.
An appropriate balance should be struck between the main
interaction modes: participation partnership
teacher-directed class work; socialisation social skills
1.4.2.5.1.
intercultural
interaction in the target language in small groups
1.4.2.5.2. awareness
pair work
1.4.2.5.3.
individual work
1.4.2.5.4.
presentation by learners.
1.4.2.5.5.
1.4.2.5.5.2. What is undertaken to ensure that teachers with less Reliability Practicability
experience or ability will be able to cope with the demands on
classroom management posed by the type of interaction
demanded?
1.4.2.6. Assessment
Assessment in the Design Phase is taken to refer to procedures and
materials that are designed to facilitate effective learning.
Assessment of the product of learning is considered in Outcomes.
Techniques to monitor learning during lessons and give
appropriate feedback are considered in Implementation.
The aspects of assessment considered here therefore concern:
Placement testing, to ensure that learners receive a
1.4.2.6.1.
programme appropriate for them
Diagnostic testing, to identify which particular skills and
1.4.2.6.2. language points need to be addressed
1.4.2.6.3. Achievement testing, to motivate revision, consolidate
58
1.4.2.6.3.2. What steps are undertaken to ensure that the assessment Reliability Methodological
procedures are valid? integrity
1.4.2.6.3.2.1. Do assessment procedures mirror the learning objectives of Reliability Internal coherence
the programme/material? (content validity)?
Content recently learnt rather than tricky questions
Activities as well as content
Weighting in relation to priorities in objectives
Do assessment procedures reflect a theoretically grounded
1.4.2.6.3.2.2. Reliability Methodological
view of the nature of knowledge and language learning which
integrity
underpin the programme / material (construct validity)?
Receptive, interactive and productive uses of language
Meaning in context
Underlying competences (linguistic usage; socio-linguistic
appropriateness; discourse organisation)
Performance features (fluency, flexibility, interactional
ability, control)
Are the results fair and consistent (reliability)
Accountability
1.4.2.6.3.2.3. Reliability
Statistical analysis for tests which affect learners' futures
Ergonomy
Efficiency
Standardisation training for teacher judgements
1.4.2.6.3.3. What steps are taken to involve learners into the assessment generativeness Cognitive
process? development
participation
sharing
responsibility
1.4.2.6.3.3.1. Are learning objectives made transparent to the learners? Participation Involvement
Summary of objectives and assessment criteria given to Partnership
learners
Objectives explained at beginning of unit / lesson
Learners involved in constructing test
Are learners asked to record and contribute their reactions and
1.4.2.6.3.3.2. Participation Involvement
experiences?
Partnership
Feed back feelings about materials / programme
Maintain a diary
Formulate a personal project
Are there awareness-raising activities
1.4.2.6.3.3.3. Generativeness Cognitive
Codes for classifying mistakes in written work
development
Participation
Questionnaire-led discussion of strengths and weaknesses
Involvement
Observation of neutral video recordings and discussion of
Partnership
performance
60
1.5.1.1 Does the presentation take account of current, relevant Attractiveness User Friendliness
aesthetic norms and expectations? Sensitivity
Use of guidelines (e.g. on mixing colours, fonts etc.)
Development by trained professionals
Consultation with focus groups: Do people like it?
1.5.2 What has been done to ensure that the programme / materials
are accessible and practical?
1.5.3 Illustration
In language learning, pictures, drawings, photographs and symbols
facilitate comprehension and provide a strong motivational
element by stimulating the learners' imagination, raising their
curiosity or addressing emotions.
For illustrations to be efficient they have to meet the function
which was assigned to them. Photos or drawings, for example,
which are meant to explain the meaning of a new item have to be
unequivocal, while a photo that is meant to make the learners talk
to each other should leave room for speculation. This implies that
pictures and texts should be matched clearly to avoid
misinterpretation. (Neuner 1996: 86).
One has to find equilibrium between what is familiar to the
learners and what seems to be ”foreign”. The target group needs to
be able to relate to the illustrations, but at the same time these
should raise learners' interest and curiosity. Illustrations should
avoid confirming clichés or prejudices.
1.5.3.1. What has been done to ensure that illustrations are relevance appropriateness
appropriate and effective?
reliability internal coherence
1.5.4. Layout
The layout must meet both aesthetic and functional demands. It
should be easy to follow and clear, using visual and graphic
elements to highlight the start and end of a unit and to attract
attention to particular points. Ideally, the layout reflects the
principles underlying the organisation of the course. However, in
order to be effective it also has to meet the 'taste' of the target
62
group.
1.5.4.1. What steps have been taken to ensure that the layout of the
material has a relevant didactic function?
1.5.4.1.1. Is there a clear link between the different components Reliability Internal coherence
Text linked to appropriate pictures
Symbols clear
Is the layout designed so as to contribute to a more effective Reliability Internal coherence
realisation of the learning objectives?
1.5.4.1.2. Consistency
For print: focal points highlighted
For media: screen layout facilitates links; feedback and help
availability supports when requested but does not distract
1.5.5.1. What has been done to ensure that colours and/or music reliability consistency
facilitate the learning process in an appropriate manner? attractiveness sensitivity
1.5.6.1. What has been done to ensure easy navigation through the
material?
attractiveness user friendliness
flexibility individualisation
1.5.6.1.1. Is the software provided with a paper manual which explains
reliability
installation and navigation procedures? internal coherence
Explanation easy to understand
In-built graphics to facilitate comprehension
Is the installation procedure easy and intuitive? attractiveness
user friendliness
1.5.6.1.2. Autorun
Uninstall available
Help-on function to solve installation problems
Wizard GUI (Graphic User Interface) available
Internet access procedures explained in the welcome/home
page
Internet access procedures reliable and easy to follow
Attractiveness
Is the software provided with an on-line guide? User friendliness
1.5.6.1.3.
"Help" function easily accessible
Sound and graphics facilitating the use
Possibility of printing parts of the guide
Guided tour function available
Transparency
Is the table of contents/menu clear and understandable to the Clarity of Aims
1.5.6.1.4.
user?
Is there an index and a contents menu?
Is on-screen help available?
Attractiveness
Are all relevant parts of the material / application easily User-friendliness
1.5.6.1.5. accessible in all phases of the learning process?
Possibility to take over the programme from any point of
the programme (e.g. from the point reached in previous
64
sessions)
Exit function at each point of the programme
1.5.6.1.6. Are the navigation possibilities (if applicable) adaptable to Flexibility Individualisation
different learner types?
Shortcuts available to pass easily from one unit or part of
the programme to others (i.e. without being obliged to
stick to a part or exercise till the end).
Alternative (individual) routes through the material
foreseen.
2. Implementation
IMPLEMENTATION
Quality in the implementation stage implies providing a positive affective atmosphere and a supportive
learning environment in which the realisation of the design is monitored and adapted flexibly to the
evolving needs of the learners.
During
This stage involves: affective management, monitoring activities, and support measures.
66
PRINCIPLE SUB-
PRINCIPLE
A perfect programme design is no guarantee of the
provision of a successful service. Designs have to
be realised in a context and there are three issues
central to the determination of the quality of that
realisation in that particular context:
- affective management
- monitoring
- support
A prerequisite for a successful implementation is a
consideration of the "innovation gap" implied by
the introduction of the programme / materials. The
extent of this gap will affect the initial balance in
the investment of resources and personnel. Where
the degree of innovation is relatively low, the focus
might be on monitoring. When the degree of
innovation is high, more resources will need to be
devoted to support systems. Whichever is the case,
there is a need for affective management.
2.1 What steps are taken to estimate the extent of
the "innovation gap" represented by the
programme / materials?
Participation Partnership
2.1.1 Have teachers been involved in the development
of the programme / materials?
Teachers included in core development team
A range of teachers consulted at key points
in the development
Channels for teachers to contribute ideas
during the development process
Participation Partnership
Have teachers been consulted in planning the
implementation of the programme / materials?
2.1.2
Information sessions
Workshops
Working groups to suggest adaptations /
teachers' notes etc.
Relevance Appropriateness
Were visits made to observe the current Reliability Practicality
situation prior to the introduction of the
2.1.3
programme / materials?
Observation of classes /study sessions
Evaluation of resources
Discussions with managers and teachers
Has a realistic budget for implementation been Efficiency Cost-Effectiveness
2.1.4 worked out?
Teacher / learner familiarisation
67
Attractiveness Sensitivity
2.2.1.1 Is the learning environment designed in such a
Efficiency Ergonomy
way as to make learners feel at ease?
Attractiveness Sensitivity
Classrooms spacious and luminous
Equipment designed in such a way that
facilitates group working
In case of self-directed learning, quiet
environment
2.2.1.2 Is the approach used consciously learning- Attractiveness Sensitivity
centred, i.e. focused on creating a relaxed Participation Involvement
atmosphere for effective learning? Partnership
Action taken early in the course to establish Socialisation Intercultural
a virtuous circle of enjoyment - success - Awareness
enjoyment – success
Approach chosen avoids strictly prescribing
steps and reactions on the part of the learner
Approach considers the learner holistically
as an autonomous personality
Clarity
2.2.1.3.1.1.1. Are parts presented as distinct and yet related to Transparency
Presentation
each other? Reliability
Internal coherence
Start of new units signalled
Conclusion of an activity signalled (e.g.
concluding exercises, small tests, summary
etc.)
69
2.2.1.3.2.1.1. Are clear criteria defined to help select topics Transparency Clarity of Aims
and choose activities?
Topics and activities clearly ordered and
classified
Range of possible aims indicated
Topics and activities embedded in a clear
71
2.2.1.3.3.3.1. Are topics presented that are new and Attractiveness Variety
unfamiliar to the target group in question?
Culturally different
behaviour
Innovative ways of
looking at linguistic aspects
Puzzling situations
2.2.1.3.3.5.1. Are there opportunities to branch out and follow Participation Personal Interest
up a work-related or hobby interest?
Learning centre resources available to
learners who wish to follow up a work-
related or hobby interest
Personal projects
References available to information sources
in the language environment
Internet facilities
2.2.1.3.4. Encouraging
Good language teaching involves encouraging in
the sense of challenging, but also a considerable
amount of reassuring of less confident learners.
Materials and programmes can build in such
reassurance by showing learners what they can do.
It is not just children who respond positively to
praising and rewarding when they are successful or
who need emotionally sensitive feedback when they
are unsuccessful. Learners who are unsuccessful, or
who are held back by the material or the level of a
course become frustrated. Quality assurance
implies ways of dealing with frustration when it
arises.
A qualitative approach to affective management
sets the scene for successful learning. Furthermore,
by making the experience an enjoyable one it
increases motivation and creates a virtuous circle in
which enjoyment of success increases motivation
which in turn increases success - and enjoyment of
it. However, knowing that learners are enjoying
success and are not experiencing frustration implies
monitoring to find out what is actually happening.
2.2.1.3.4.1. What steps are taken to show learners what they
have achieved, what they can do in the language?
2.2.1.3.4.1.1. Is there a recapitulation at the end of each unit?
Role plays or activities used to sum up Transparency Clarity of
situations and skills already practised Presentation
Clarity about
Simple achievement tests provided
Achievement
Team quizzes or panel games
Do learners get praise and emotional / symbolic
2.2.1.3.4.1.2. Attractiveness Sensitivity
rewards for success?
Credits or points to add
Applause or
75
congratulations (multimedia)
Increased autonomy Generativeness Cognitive
(self-directed learning) Development
2.2.1.3.4.1.3.
Clarity about
Are awareness raising activities included in
Transparency Achievement
order to help learners to identify their
achievements?
Feed-back discussion on performance
Repairing procedures (non-intrusive
provision of correct term)
Introspective procedures (video or audio-
recorded)
Diary keeping
2.3 Monitoring
Successful monitoring by the teacher leads to a
efficient fit between the service provided and both
the perceived and observed needs of the learner(s),
which should lead to a high degree of success and
satisfaction.
Successful self-monitoring by the learner with or
without prompting from the teacher or the
computer software in a media-led course should
lead to autonomous learning as well as a high
degree of success and satisfaction.
Monitoring in this sense encompasses four steps:
1. Observation
2. Assessment
3. Feedback
4. Adjustment
2.3.2.1.1. Is there a phase towards the end of each lesson, Transparency Clarity of
each teaching unit, when learner performance Achievement
is consciously observed to monitor teaching Clarity of
success? Presentation
Planned monitoring phases at the end of
units
Simple logs of lessons /activities
Review lessons at the end of modules
Does the teacher (for any actual course) take a
step back at certain points in the course to
observe the performance of the learners? Efficiency Ergonomy
2.3.2.1.2. Systematic collection of teacher /learner
comments on activities/materials foreseen
Video recordings of lessons used for
monitoring and feedback
Does management program (for software)
78
2.3.4.1.1. Is it made clear to learners when they will be Transparency Clarity of Aims
given feedback (positive or negative) and when
not?
Regular individual feedback given
Video camera available to monitor
performance in communicative activities
2.3.4.1.2. Do learners sometimes have an opportunity for
unfettered expression without fear of negative Attractiveness
Sensitivity
feedback?
Feedback given related to the wider criteria
used in assessment as well as just to
mistakes
"Free sessions” foreseen in which learners
can freely express themselves without being
corrected, except on request
2.3.4.1.3. Are steps taken to prevent the learner being Consistency
swamped by feedback (techniques used in Reliability
fluency activities in order to give delayed
feedback without disrupting the activity)?
Error cards
Teacher checklists
Video recording, selection of interesting
clips
etc.
2.3.4.3. What steps are taken to provide personalised
feedback which leads to effective improvement?
2.3.5.1.1. Is training provided to ensure that the actors in Transparency Clarity of Aims
the process are aware which aspects of the Clarity of
programme are fixed and integral, and thus Rationale
cannot be changed?
Reference to teacher handbook
Reflecting on the methodological premises
of the programme
Is training provided to ensure that the actors in
2.3.5.1.2. Transparency Clarity of Aims
the process are aware which aspects of the
Flexibility Adaptability
programme are flexible and intended to be
adjusted sensitively?
Provision made for speeding up or slowing
down, to increase or reduce opportunities for
consolidation and reflection
Provision made for altering the proportion of
different activities or to abandon an activity
type which is unsuccessful
Efficiency
2.3.5.1.3. Ergonomy
Is training provided to ensure that the actors in
the process are confident and competent enough
to adjust planning to take account of
circumstances?
Standard processes to amend a course plan
and inform those affected
83
2.3.6.1.1. Are steps taken to inform users of the existence Transparency Clarity of
and function of different forms of support? Presentation
Face-to-face
information
Leaflets or brochures
On-line information
Transparency Clarity of
2.3.6.1.2.
Are users informed of the limitations to Presentation
support?
Range of activities
supported
Time limits
Learner
responsibilities
2.3.7. People who can help
Whether one is talking about distance learning, a
multimedia package, a language learning stay
abroad, a secondary school language course, there
are certain questions which are indicators of the
quality of service provided:
- Someone to phone up or go and see when
things go wrong (a ”tutor”)
- Someone who can help with routine
administrative issues which may baffle the
learner (”service personnel”)
2.3.7.1. What support personnel are available if the user
cannot cope?
themselves
Specification of how quickly help will be
provided in such a case
Is there someone available to help with extra
2.3.7.1.2. User Friendliness
attention when a learner is not making progress?
Attractiveness
Individualisation
Routing system to refer such learners to a
Flexibility Ergonomy
regular support system (e.g. a one-to-one
Efficiency
tutorial to help organise them)
Reference to a supervisor
2.3.8. Information tools
The simplest information tools are a contents page
and an index. The presence or absence of an index
to a book is one indicator of the seriousness of a
publisher producing professional or academic
literature. The Navigation referred to in Section E1
on Design is the more effective the more tried and
tested such ”Help” facilities are. The range of such
information facilities for a system (multiple points
of entry) is one aspect of quality, but reliability of
information, ease of use and speed of retrieval are
probably more crucial.
2.3.8.1. What orientation tools are learners given?
2.3.8.2.1. Are steps taken to ensure that learners can use Attractiveness User Friendliness
self access tools such as indices, manuals and
facilities effectively?
Steps taken to ensure that learners are able
to look up, explore and practise grammar
points on their own
Steps taken to ensure a "phrase book"
function: that learners can look up and
practise language for a particular situation
85
3. Outcomes
OUTCOMES
The quality of the outcomes can be evaluated in terms of the gains in competences and awareness in
relation to the resources and conditions concerned and also in terms of the satisfaction of the stakeholders
involved.
Afterwards
When evaluating results obtained, one can differentiate between successful learning of the language
concerned (Success); reactions to the learning process (Satisfaction); and the achievement of
broader educational aims (Educational Bonus).
87
88
3.1.2. Has feedback been collected from teachers and resource centre Efficiency Ergonomy
managers?
Relevance Learner Centredness
Records of how the materials were used
Report sheets on specific (e.g. experimental) aspects of the
materials
Questionnaires
Structured interviews with a sample of teachers
3.1.3. Have outside independent experts been asked to review and Efficiency Ergonomy
audit the programme or materials?
Report sheets on specific (e.g. experimental) aspects of the
materials
Questionnaires
Structured interviews
89
3.1.4. Has there been structured observation, recording and analysis of Efficiency Ergonomy
the programme or materials in operation?
Relevance Learner Centredness
Observation to analyse the patterns of interaction generated
Relevance
Discourse analyses of the type of activity generated
Analysis of speech transcripts or writing samples to study
Appropriateness
the extent of the acquisition of language forms?
Reliability
Accountability
Analysis of speech transcripts or writing samples to study
Transparency
variation in the complexity, accuracy and fluency of
language generated by tasks of different types
Consistency
Analysis of speech transcripts or writing samples to study
Clarity of Rationale
use of communication strategies
3.2.2.1.1. Are the course content specifications (linguistic content, text Transparenc Clarity of Aims
types) for the module or unit used to provide a specification for y
test development?
Tests designed according to the
linguistic progression used
Tests reflect the text typology used
Tests correspond to the declared aims
of the units
Consistency
Are the course activity specifications (skills, pedagogic and Reliability Internal coherence
3.2.2.1.2.
real life tasks) used to provide a specification for test
Clarity of Aims
development?
Transparenc
Regular informal assessment of spoken interaction y
Regular informal assessment of writing
Listening tests reflecting learning aims
Consistency
Reading tests reflecting learning aims Interactivity
Reliability
Is formative assessment integrated into the course?
Attractivene
3.2.2.1.3. Regular progress tests ss
Continuous assessment in relation to
course objectives
Opportunities for self-assessment in
relation to course objectives
3.2.3.3.1. Are test items pre-tested before being used for assessment? Reliability Consistency
Trialling with a class at the appropriate level
Reliability Methodological
Piloting with a range of classes Integrity
Consultation with teachers on correct answers, adequacy of
instructions etc.
Is test data analysed?
3.2.3.3.2.
Reliability Methodological
Facility values used to refine tests by removing very easy
Integrity
or very difficult items
Relevance Accountability
Reliability coefficients
Transparenc Clarity about
For assessing productive skills, are assessors trained in relation
y Achievement
3.2.3.3.3. to criteria defining standards of performance for different
aspects of proficiency?
Criterion descriptors defining standards of performance
developed by analysing samples of performance with
expert informants
Criterion descriptors: clear, concrete, stand-alone of
observable features of performance?
Allocation of descriptors to different bands done through
analysing data from workshops with informants
Allocation of descriptors to different bands through
statistical analysis of the way the descriptors are actually Consistency
interpreted (e.g.; Rasch rating scale model). Reliability
Methodological
Are moderating procedures used to minimise subjectivity in Integrity
3.2.3.3.4. the assessment of productive skills?
Rater training / tuning in sessions employed before
assessment
Raw scores from different groups checked and compared
by colleagues or chief examiners
Double marking
Subjectivity systematically eliminated through statistical
techniques
3.2.3.4. What steps are taken to ensure validity?
94
3.2.3.4.1. Is there a qualitative analysis of the match between the content Reliability Linguistic Integrity
of the test and the content of the target domain of language
Consistency
use?
Practicality
Identification of test tasks with particular skills and content
areas confirmed through consultation of experts
Identification of test tasks with particular skills and content
areas confirmed through qualitative analysis
Identification of test tasks with particular skills and content
areas confirmed through appropriate statistical analysis
(e.g. internal correlations of scores on subtests; multi-trait
multi-method analysis)
Is there an analysis to discover whether tasks and test questions Consistency
3.2.3.4.2. assess what the authors intended by investigating the problem- Reliability
solving processes of learners as they take the test? Methodological
Integrity
Learners interviewed about how they did the test
Relevance Appropriateness
Retrospective protocol
Think-aloud protocols used to record what learners think
they are doing as they complete the task
Other (e.g. electronic) techniques used to log what learners
do while completing the tasks
Is there a statistical analysis of the internal structure of
3.2.3.4.3. assessments: that each item or phase of the task contributes to Reliability
the overall construct being assessed? Consistency
3.2.3.4.5. Is information collected about the usefulness and accuracy of Relevance Accountability
the assessment results?
Appropriateness
Receiving institutions (e.g. university departments,
employers) asked to report their experience with the
assessment results
Former test-takers asked to report how their later
experience related to the assessment results
Predictions of future success in use of language for study or
work investigated statistically
3.2.3.4.6. Relevance Accountability
Is information collected about the consequences of the
Learner
assessments?
Centredness
Steps taken to discover how end-users (receiving Transparenc Clarity of Aims
institutions) interpret assessment results y
Steps taken to identify and take into account unintended
external uses of test results
Teachers asked to give their views on the effects of the
assessments on teaching
Systematic investigation of such ”backwash” effects
3.3.5.3.1.1 Are gains in language learning skills included in assessment? Generativen Cognitive
. ess Development
Language awareness, correction tasks included in tests
Retrospective questionnaires used to assess learning
strategies
Analysis of discussions about how to learn a language
Learner diaries recording strategies used
Teacher questionnaires recording general observations
Follow up interviews: learners / teachers?
Are gains in general learning skills included in assessment? Generativene
Cognitive
3.3.5.3.1.2 Aptitude aspects included in assessment (memory tests, ss
Development
. analysis tasks)
Learner questionnaires
Consultation with teachers of other subjects
Consultation with vocational trainers / employers
CORE INDICATORS
3.3.5.4.1.1 Are gains in effective communication strategies included in Cognitive
Generativen
. assessment? Development
ess
Communication strategies valued positively in performance
assessment criteria
Transcripts of learner interaction analysed to investigate
strategy use
Teacher questionnaires recording general observations
3.3.5.5. Self confidence & motivation
3.3.5.5.1.1 Are gains in self confidence and self esteem included in Attractivene Sensitivity
. assessment? ss
Questionnaires before and after the course
Recording feelings in diaries
Follow up interviews with a sample of learners
3.3.5.5.2. What steps are taken to assess motivation?
CORE INDICATORS
3.3.5.5.2.1 Are gains in motivation included in assessment? Attractiveness Sensitivity
.
Attitudinal questionnaires
Elicited comment from teachers
Analysis of attendance and re-enrolment patterns
Investigation of take up of extra-curricula activities
3.4.1.7.1. Did the language learning modalities used in the course Transparency Clarity of Aims
comply with the features stated?
Consistency
Reliability
Duration of lessons/units/course
Appropriateness
Sequencing of lessons/units Relevance
Types of assessment
Exploitation of assessment results
Did the language learning materials and equipment actually
used comply with the stated features?
Types of teaching/learning materials/aids
3.4.1.7.2 Efficiency
Complementary resources (multimedia centre, library, etc.) Ergonomy
Extra-course activities (tutoring, homework, etc.)
3.4.1.7.3. Did the qualifications of language teaching staff comply with Efficien Cost-Effectiveness
what was stated?: cy Ergonomy
Teachers’ experience with the target public
Teachers’ general language or LSP expertise and
experience
Teachers’ experience in using the required
materials/equipment
105
F References
AFNOR (1998), Projet de norme internationale ISO 10015 – Management de la qualité – Lignes
directrices pour la formation. Paris (X 50 V – doc. N. 31).
Alderson, J./Beretta, A., (eds.) (1992), Evaluation Second Language Education, Cambridge, CUP.
Anderson, J. R. (1983), The Architecture of Cognition, Cambridge Mass..
Bredella, L./Christ, H. (eds.) (1995), Didaktik des Fremdverstehens, Tübingen.
Byram, M./Zarate, G. (eds.) (1997), The sociocultural and intercultural dimension of language
learning and teaching, Strasbourg.
Dickinson, L. (1987), Self-instruction in Language Learning, CUP.
EAQUALS (19953), A Pan-Europeans Inspection Scheme for Quality Language Providers. Trieste.
Ehrman, M.E. (1996), Understanding Second Language Difficulties, Thousand Oak CA.
Engel, U. et alii (1977/1979), Mannheimer Gutachten zu Lehrwerken Deutsch als Fremdsprache. 2
Vols. Heidelberg,
Feigenbaum, A. (1983), Total Quality Control, McGraw Hill, New York.
FRAMEWORK (1996), Modern Languages: Learning, Teaching, Assessment. A Common European
Framework of reference, Strasbourg.
Fruhauf, G./Coyle, D./Christ, I. (eds.) (1996), Teaching Content in a Foreign Language. Practice and
Perspectives in European Bilingual Education, Alkmaar.
Gardner, R (1985), Social Psychology and Second Language Learning: The Role of Attitude and
Motivation, London.
Gardner, R. /Lambert, W. (1972), Attitudes and Motivation in Second Language Learning,
Rowley/Mass.
Goethe-Institut, (1990) Handbuch für Spracharbeit. Teil 3: Lehrwerkanalyse, Goethe-Institut,
München.
Goleman, D. (1996), Emotional Intelligence, London.
Hawkins, E. (1984), Awareness of Language, Cambridge.
Holec, H. (ed.) (1988), Autonomie et apprentissage autodirigé, Strabourg, Conseil de l’Europe.
Hopkins, (1996), Guide for textbook and Materials Writers, Council of Europe, Strasbourg.
Horwitz, E./Young, D (1991), Language Learning Anxiety: from Theory Research to Classroom
Implications, Englewood Cliffs NJ.
Kast/Neuner (Hrsg.) (1994), Zur Analyse, Begutachtung und Entwickung von Lehrwerken.
Berlin/München.
Lynch, B. K. (1996), Language Program Evaluation, CUP.
Morgatroyd. S. and C. Morgan (1992), Total Quality Management and the School.
Buckingham/Philadelphia.
Negro, G. (1996), Organizzare la qualità nei servizi, Il Sole 24Ore, Milano.
Neuner, G./Hunfeld, H. (1993), Methoden des fremdsprachlichen Deutschunterrichts, Berlin.
OECD, (1992) High Quality in Education and Training, Paris, OECD 1992.
O’Malley, J./Chamot, A. (1990), Learning Strategies in Second Language Acquisition, Cambridge.
106