``I would have everie man write what he knowes and no more.

''ÐMontaigne

BRITISH J OURN AL O F ANAESTHESI A

Editorial I
Development and use of scoring systems for assessment of clinical competence

The assessment of clinical competence is one of the greatest current practice and seek out those components necessary
challenges facing medicine today. A useful construct for the for good performance. In this issue, Forrest and colleagues2
assessment of competence is provided in the Miller pyramid describe the use of a modi®ed Delphi technique to develop a
(Fig. 1).1 scoring system for assessment. First used in the 1950s,3 this
The ®rst two stages, `knows' and `knows how', can be technique seeks to gain a consensus from a group of experts
assessed using the traditional assessment tools of written in response to an open-ended question. It has become
and oral tests. However, `knowing' and `knowing how' popular in a wide range of ®elds, including economics and
clearly do not necessarily extrapolate to the application of social policy as well as health-care. The Delphi technique
knowledge in the workplace. To demonstrate clinical involves sending an initial questionnaire to a group of
competence, assessment at levels 3 and 4 becomes more identi®ed experts. This will generate a variety of responses
important, but also more challenging. Level 3, `shows how', and ideas, and these are collated and form the basis of the
is currently assessed by practical examinations, observed second questionnaire, which is sent to the same group of
long or short cases, or OSCE style examinations. However, experts. Subsequent responses are handled statistically to
the only way to assess level 4, `does', is to observe the produce relative frequencies, and continued review by the
practitioner at work in the real world. expert group gives them the opportunity to change their
Assessment tools developed for this purpose should allow mind or include any other items. This process is repeated
us to compare performance with some pre-existing standard. until consensus is reached, which may require as many as 10
They should also allow us to identify de®ciencies in the rounds, although this is often reduced when the modi®ed
performance of the person being assessed, so that subse- technique is used. Keeney and colleagues have recently
quent training can be targeted to the areas of greatest need. published a critical review of the use of the technique in
This requires that we break performance down into more nursing research.4 While the Delphi technique works well to
manageable and identi®able components, and this is identify the key technical skills involved in a process such
normally achieved using scoring systems. as rapid sequence induction, one of the drawbacks of asking
The ®rst challenge is therefore to identify the important experts to identify all attributes of clinical competence is
constituent parts of good medical practice that should make that they may not be conscious of the full range of these
components. Many aspects of expert performance have
up the elements of the scoring system. We can examine our
become intuitive5 and may not be readily accessible by
individuals. Much of anaesthetic practice involves a cogni-
tive component, which is also not readily accessible using
techniques such as a Delphi process. This poses the question
of how complete the Delphi technique can be in providing
the material from which a scoring system to assess clinical
competence will be compiled. Any scoring system that
attempts to address the assessment of clinical competence
clearly has to address both technical and non-technical
skills.
The technique of cognitive task analysis is designed to
access the key non-technical skills which underpin per-
formance, such as decision making, problem solving,
attention allocation, planning and workload management.
It uses a combination of methods, such as observation and
interview, to `unpack and make explicit the expert know-
Fig 1 The Miller pyramid. ledge that is often implicit and dif®cult for analysis to

Ó The Board of Management and Trustees of the British Journal of Anaesthesia 2002
 Editorial II

observe or for experts to verbalise'.6 These methods are systems of the future will not only be used to ensure the
clearly more time-consuming and specialist, but have been competence of our trainees, but they will also stand to
used extensively in other high-reliability industries and have become the tools of revalidation. Every one of us will then
been used more recently in anaesthesia to identify key be subjected to them. It is time to take assessment seriously.
components.7
The identi®ed components must be assembled into a R. J. Glavin
scoring system that satis®es the characteristics of a good N. J. Maran
assessment tool, namely, validity, reliability and feasibil- Scottish Clinical Simulation Centre
ity.8 The scoring system must then undergo pilot testing to Stirling Royal In®rmary
demonstrate that it has these characteristics. Forrest and Livilands Gate
colleagues have addressed all of these issues.2 The content Stirling FK8 2AU
validity (the extent to which the assessment representatively UK
samples what it is supposed to be measuring) of their
scoring system is high, given the methods used to develop
their tool. Construct validity (the extent to which a new
measure is related to speci®c variables in accordance with a References
hypothetical construct) is also high, given that they were 1 Miller GE. The assessment of clinical skills/competence/
able to demonstrate improved scores with increasing performance. Acad Med 1990; 65 (Suppl): S63±7
2 Forrest FC, Taylor MA, Postlethwaite K, Aspuell R. Testing
clinical experience. They also were able to demonstrate
validity of a high ®delity stimulator for assessment of
good inter-rater reliability, using multiple assessors to score performance: development of a technical performance scoring
the same performance. They have demonstrated the poten- system and its application in the assessment of novice
tial role of the simulator in the testing of an assessment tool, anaesthetists. Br J Anaesth 2002; 88: 338±44
and they established the feasibility of using their scoring 3 Grbich C. Qualitative Research in Health: an Introduction. London,
system to rate performance from videos of performance in UK: Sage Publications, 1999
the simulator. However, such a detailed scoring system may 4 Keeney S, Hasson F, McKenna HP. A critical review of the Delphi
be more challenging to use during real-time observation in technique as a research methodology for nursing. Int J Nurs Stud
2001; 38: 195±200
clinical practice.
5 Atkinson L. Trusting your own judgement (or allowing yourself
The ®nal challenge of developing a new assessment tool to eat the pudding). In: Atkinson L, Claxton G, eds. The Intuitive
is to ensure that the subjects have con®dence not only in the Practitioner. Buckingham, UK: Open University Press, 2000;
tool itself but also in those carrying out the assessment. This 53±65
becomes increasingly important as we venture into assess- 6 Seamster TL, Redding RE, Kaempf GL. Applied Cognitive Task
ment outside the traditional framework of knowledge and Analysis in Aviation. Aldershot, UK: Avebury Aviation, 1997
skills, and as the stakes involved increase. This requires that 7 Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Final
those carrying out the assessment have had suf®cient Report: Development of a Behavioural Marker System for
Anaesthetists' Non-Technical Skills (ANTS). [Grant Report for
training in its application to ensure not only that the
SCPMDE, project reference RDNES/991/C]. Aberdeen, UK:
assessment is fair but also that it is seen to be fair. The
University of Aberdeen, 2001
higher the stakes, the more important it is that all of these 8 Joint Centre for Education in Medicine. The Good Assessment
challenges have been met properly. Judgement of clinical Guide: A Practical Guide to Assessment and Appraisal for Higher
incompetence and its impact on the career of the subject Specialist Training. London, UK: Joint Centre for Education in
must be defensible from legal challenge. The assessment Medicine, 1997; 34±39

Ó The Board of Management and Trustees of the British Journal of Anaesthesia 2002