Paperless Examinations Project Phase 2 Report

PAPERLESS EXAMINATION PROJECT PHASE 2 REPORT EXECUTIVE SUMMARY

Background
The Paperless Examinations Project was initiated in late 2000 following discussions between CCEA and Edexcel. The two organisations recognised that the increasing developments in the use of technology for learning and formative assessment would lead, in due course, to evaluation of the ways in which technology could support high stakes public examinations. Both organisations also recognised that, while there were a number of tests and examinations, such as the DSA Driver Theory Test, being delivered using technology, there were no high stakes public examinations. In designing the first phase of the project, which ran from October 2000 to September 2001, CCEA and Edexcel were primarily concerned with researching the technological issues associated with delivering an electronic examination paper and in making assessment comparisons between undertaking an examination on paper and undertaking the same examination on screen. A report of the first phase of the project was published in late 2001.1 CCEA and Edexcel had always envisaged that the project would be planned in a number of phases and in October 2001 it was agreed that Phase Two of the project should be initiated and planned to run from January 2002 to April 2003. It was also envisaged that, as wider developments in the use of technology to deliver examinations took place, the outcomes from the project would be analysed against these developments and the overall project plan adapted as appropriate.

Project Plan for Phase Two

Design computer-based mock examinations in the three Sciences, Biology, Chemistry and Physics and Geography using a specification drawn from material in both the CCEA and Edexcel GCSE specifications and designed, as far as possible, to make the best use of the technology available rather than the transfer of paper based assessment to screen. Deliver certificated Basic/Key Skills multiple choice tests at levels 1 and 2 using tests provided by the Qualifications and Curriculum Authority (QCA).. Set up the necessary systems to author, design, deliver, collect and mark the papers on line, including the use of computer marking for both GCSE and Basic/Key Skills. Set up the necessary systems to provide the CCEA and Edexcel legacy systems with data to enable them to check, audit and certificate the results of the Basic/Key Skills Tests. Undertake online specialist marking for those GCSE questions that could not be computer marked. Trial the system in up to twenty schools in Northern Ireland and England for the GCSE papers and up to twenty FE and Training Provider centres in Northern Ireland and England for the Basic/Key Skills Tests. Work with a suitable technology partner to support the development of the GCSE examinations and to deliver the system.

CCEA/Edexcel (2001). Paperless Examinations Project. Phase One Report.

Paperless Examinations Project Phase 2 Report

Outcomes of Phase Two

The Technology Partner, Bradford Technology Ltd with support from FD Learning, was selected and contracted to provide the delivery system and to support the design of the GCSE questions. Twenty schools were recruited for the GCSE trials and sixteen centres for the Basic/Key Skills trials, including a number of University for Industry (UfI) and Army centres. Eight examinations were developed for the three Sciences and Geography; one at Foundation and Higher Tier for each subject. Six sets of Basic/Key Skills Tests were converted for use on screen from paper based tests provided by QCA. Two sets of trials took place in September 2002 for the Basic/Key Skills Tests and in February/March 2003 for the mock GCSE examinations. 1416 computer-based mock GCSE tests were taken, together with 346 Basic/Key Skills Tests. All the Basic/Key Skills Tests were marked electronically and approximately fifty percent of the GCSE questions were marked electronically. Ten markers worked from a marking centre for the GCSE examinations.. The software was adopted by CCEA and Edexcel to provide a further continuous trial of Basic/Key Skills Tests from March 2003.

Key Issues and Conclusions

Technological
The technology was able to deliver the online examinations very successfully, although it was less successful when used as a real time Internet delivery system. Whilst a number of centres needed support to get them up and running, this was less so than in Phase One and was less so for the GCSE centres that ran later in the project.2 Other than issues using the real time internet version, there appeared to be no discernable pattern to the issues that arose other than networks and servers that had been set up were running in a non standard way. Knowledge of the centres system by its support staff and the support these staff gave to the project proved to be more important than any individual software or hardware issue.

Training and Support

The value of briefing and training the centre staff to install the software and administer the tests was shown by the lack of problems experienced in those centres where this was undertaken effectively.

In the rollout since March 2003, support needs have become minimal.

Paperless Examinations Project Phase 2 Report

Pedagogical
The authoring process in bringing together the subject experts and the technical authors proved to be one of the most valuable learning elements of this phase. The authoring process raised upskilling and training issues if Awarding Bodies were to develop large numbers of onscreen examinations. Once confident of the facilities available, the authors were able to make good use of the technology to assess parts of the specifications in ways that were not possible with paper based examinations. The need to convert paper based Basic/Key Skills Tests for on screen use confirmed the findings of the Phase One report that converting from paper to screen is not the most effective method of creating or using the features available from on screen testing. Learners found all the tests accessible and there were no apparent IT barriers to completion; in the Basic/Key Skills Tests no learners indicated that they would have preferred to undertake the paper tests. Most centres conducted the tests in rooms already fitted out with computers. Whilst this caused some environment issues, these were not insurmountable. Markers generally liked the software now being used for marking, although there are still some issues stemming from the different marking process that need to be addressed.

Summary of Recommendations
Item and Test development
Authors for onscreen item development should be ICT competent and should have had used a range of multimedia materials both online and on CD-ROM in the classroom. A more freeform approach to authoring, while generating a lot of creativity, caused problems with the programming and screen creation. The awarding bodies should draw up guidelines concerning item development, software capability and procedures for authoring. The development of objective question styles for computer-based summative assessment is a professional skill. This should be offered as part of teacher training and/or within INSET provision in order that item design expertise can be developed. In fact, the Scottish Qualifications Authority (SQA) is currently developing an Advanced Certificate in E-assessment. A national qualification such as this would be of great value in ensuring homogeneity of standards. Authors will not have the same technical skills and facilities available as the technology partner. With regard to this, video conferencing could bridge the geographical distance of authors and item producers and allow more frequent and efficient reviewing of items and mark schemes. This would reduce costs and would ease the revision activities and the scrutiny of the near final product. Provision for the development of the widest possible range of item styles is required so that assessment design is not compromised by limitations within the system. A revision of the method in which templates, offered in a Microsoft Word format, were employed for item writing is recommended. A custom HTML based system with upload functionality and version control would be preferred. Scrutineers should experience a fully functional model of the exams with all technical requirements in place exactly as a candidate would experience it.

Paperless Examinations Project Phase 2 Report

Research into the potential for different item types should be utilised. Innovative items, which make increasing use of the technology, could be developed. As computer-based exams become more common, specimen discs of items could be developed. To provide Basic/Key Skills Tests more suitable for computer-based testing the issue of item design needs to be addressed and computer-based items developed from the beginning. In particular, audio for explaining instructions should be used within onscreen Basic/Key Skills Tests. Expertise in the areas of accessibility with regard to learners with special needs, for example visual impairment, needs to be introduced from the beginning. This is not only with regard to the assessment but also in the development of teaching resources and exemplar materials. The use of audio is particularly recommended. A greater use of reference material should be available onscreen where appropriate. The periodic table in the chemistry paper would be a good example. Calculators and other tools should be incorporated where this is allowed/appropriate.

Technology
Content
The range and complexity of custom items needs to have greater direction from the technical end, without compromising author creativity. Formal item authoring software should be provided to authors to allow them to create screens in a consistent fashion with upload functionality. This should replace the often awkward and inconsistent use of word documents.

Security
Further integration of ExamBase server with the web server to remove any potential security weaknesses should be implemented.3 Automated ExamBase server housekeeping functionality is needed (removing old data). Full and encrypted system logging is required to allow auditing of exam centre procedures.

Development
More thorough testing using structured procedures and in multiple environments is required at every stage of the development process.4 Although the administrative system worked effectively for the volume of candidates in the trial, MicroBoard needs an end-to-end redesign and development in the light of comments from users and scalability concerns.5

Installation
Specifications for centres need to be more stringent, notably no use of NT4 server operating system, and standards for proxy servers need to be developed.6

3 4

This has now been fully implemented in the latest version of the software (V2.0). Full development procedures have now been implemented. These include a register to ensure all centres are using the same version of the software and that any upgrades have been implemented. A specification for this upgrade is under development. NT4 is no longer being supported.

5 6

Paperless Examinations Project Phase 2 Report

Smoother and more integrated installation methods are required to allow easier self-installation in centres.7 A date of installation should be agreed with a centre and a technical representative of the technology partner should visit all centres. This is costly in terms of resources and travel but saves time in the end. Ideally, combined installation and training to the technicians in the use of the software on site should be introduced.8 A self-install option of the software for LAN-centres together with rigorous monitoring and testing procedures should be offered to centres for large-scale rollout.9 .pdf versions of documentation or training materials should be offered on a dedicated secure website, including forms. Posting hard copies of documentation should be avoided.9 With a large-scale rollout a detailed plan for installation needs to be in place. This would require documentation made available to all centres from the Awarding Body. In Northern Ireland, a rollout through a managed service such as Classroom 2000 (C2k)29 would help to avoid the occurrence of the installation difficulties encountered during the project.

Display
Centres should set ExamBase client machines to exactly 800 600 resolution to avoid the black border effect found with higher resolutions.

Test Administration and Delivery

The Instructions for the Conduct of Electronic Exams, based on the e-ICE instructions used in the Basic/Key Skills Tests pilot could be used to inform the standardisation of test administration and invigilation procedures between Awarding Bodies. An effective strategy for managing large-scale installation will be a crucial factor in determining the success of an effective rollout. In particular, there needs to be a strategy for on site testing to prove system integrity in advance of live tests being delivered.10 The centres operating with an Internet-based solution need to address the question where the server and its associated support functions should be hosted. It is important to be able to offer this alternative technological solution as it has the scalability to enable small, remote (such as outreach) centres or those with low levels of technical support to offer assessment.11 Registration procedures and the software need to be improved to accommodate last minute or on demand candidates. Much greater flexibility is required to provide testing on demand as the rigidity of holding sessions at specific times on specific dates within a window caused difficulties and deterred some learners from participating. Edexcel and CCEA need to define the requirements for on demand testing so that these issues can be addressed.

Since March 2003 over 135 centres (as at end of November 2003) have successfully self-installed the software using revised training and installation procedures. As a result of the successful self-installation processes, it is not considered necessary for a technical representative to visit all centres. All these recommendations have now been put into place with the rollout of the Basic/Key Skills Tests. An on site installation process is now being used successfully with the Basic/Key Skills rollout. The Internet-based solution is no longer being offered. See section 4.5.2.

9 10 11

Paperless Examinations Project Phase 2 Report

For future use of the system, Awarding Bodies and the Technology Partner need to develop a clear and stringent approval process for centres that includes the way in which the technology is managed and supported. Centres need to be clear about the responsibility they have to ensure the system is properly managed and supported (including adhering to deadlines for centres, for example for registration and installation procedures) in the same way that they are held responsible for the proper operation of the paper-based system. Patterns of centre support requirements (technological and learner support) need to be analysed on an ongoing basis so that future needs can be assessed.12 The software development undertaken by BTL has produced a robust solution that provides the facilities for centres to access tests and to run on-screen examinations. However, the project has shown the need for IT expertise to be available in centres. Centre staff, both technical and administrative, need to be fully trained to operate the system. In addition, it is vital that centres complete a rigorous testing programme prior to participating in live examinations. There would be great benefits if all centres were equipped with common IT platforms. This will be the case in Northern Ireland with the rollout of Classroom 2000 (C2k)29. Coupled with clear guidance from Awarding Bodies, this would ensure that centres able to deliver on-screen examinations in an effective way. Adult Basic/Key Skills Tests will need a much more streamlined and automated process to move results from MicroBoard to the Awarding Bodies legacy systems.13 A mobile test centre, for example a bus equipped with a wireless LAN, perhaps Internet Access and exam delivery software, would assist test centres that would wish to offer tests to small numbers of candidates on a non-regular basis but do not operate a LAN or have little ICT expertise available, from the installation/invigilation procedures and the costs included in the updating of their ICT system.14

Marking
The Awarding Bodies need to develop procedures for computer-based marking processes and mark centres. The MarkerBase system would be even more popular with examiners if they could mark onscreen at home, which would provide greater flexibility. MarkerBase can be configured for secure Internet access and has to undergo a testing procedure for remote marking. MarkerBase needs to be made even more user-friendly by implementing the markers' suggestions received during the GCSE marking pilot. MarkerBase needs to offer facilities for markers and supervisors (chief examiners) to communicate electronically and immediately. Furthermore, access for the supervisors to the facility of monitoring markers' work needs to be tested. A mobile mark centre, for example a bus equipped with a wireless LAN, perhaps Internet Access, and marking software, would offer savings in travel time and costs to examiners who at any stage during the marking process temporarily do not have secure Internet access or the technology at home. That way marking could continue under secure conditions and delays in marking resulting in late releases of results would be kept to a minimum. This mobile mark centre could be offered as a regular test centre, too, as well as a contingency for marking. Consideration of using wireless technology (laptops) should be included in forward planning/implementing of computer-based marking.

12 13 14

See section Key Issues and Conclusions, page 2. This has now been implemented (September 2003). Trials are now being undertaken both with a mobile bus equipped with an on board network and a portable wireless LAN based system.

Paperless Examinations Project Phase 2 Report

PROJECT OVERVIEW

1.1 Background
The Paperless Examinations Project was initiated in 2000, following discussions between CCEA and Edexcel. The two Awarding Bodies recognised the growing importance of technology within assessment and the likelihood that, in the medium term, there would be initiatives to introduce the use of technology within public examinations. Furthermore, in Northern Ireland the Department of Education document A Strategy for Educational Technology proposed the piloting of ICT enriched syllabuses and examining modes in one or two subjects.15 CCEA and Edexcel were aware that, while there were a very limited number of public examinations delivered using computers in the United Kingdom, there was no assessment technology being utilised in schools. None of the tests currently delivered using computers could be described as involving high stakes qualifications such as GCSE. The only large-scale use of IT for testing purposes was the DSAs driver theory test. This test, together with the TTAs skills tests for newly qualified teachers represented most of the IT testing being undertaken within public examinations in the United Kingdom. Government has shown considerable interest in the possible development of computer-based tests and onscreen examinations. There is little doubt that, within the next few years, the awarding bodies will be expected to introduce computer-based examinations. The CCEA/Edexcel project aimed to evaluate the technical feasibility and educational benefits of developing a system that allowed the introduction of computer-based assessment within the current practices used for high stakes examinations such as GCSE. The Paperless Examinations Project was designed to address the issues, both pedagogical and technological, that were likely to be faced by Awarding Bodies in implementing computer-based examinations. The intention was to use ICT to improve the processes associated with: designing, delivering, marking, and reporting of results

of examinations without changing the current rigour of the assessment. It was recognised that the introduction of ICT within teaching and learning will change the nature of the assessment process. Initially, a timescale of three to four years for development work was envisaged. The project was planned in phases with the outcomes of each phase analysed prior to proceeding to the next stage. In the end, the project was undertaken in two phases and lasted from October 2000 to July 2003. A report on Phase One was produced in November 2001. This report deals with the outcomes of the second phase of the project. Phase One was undertaken from October 2000 to September 2001. On-screen trial tests were completed by a group of fourth-year (first year GCSE) pupils in seven schools in Northern Ireland (April 2001). In parallel with the on-screen tests, a control group of pupils in the same schools completed the tests on paper with their completed scripts being scanned to facilitate computer-based processing. The tests were based on GCSE Modular Science examinations. The pilot ran successfully in four schools, two schools experienced some technical difficulties and one school encountered major difficulties using the system. Marking of the computer-based tests was a combination of computer marking and, where judgements were required, marking by trained examiners. The examiners were based in a mark centre and marked online. The focus within this first phase was on the technical feasibility of offering computer-based GCSE type tests and on identifying issues arising. A full evaluation report on Phase One of the project was produced in November 2001 and is available from either CCEA or Edexcel.
15

Department of Education Northern Ireland (1997): A Strategy for Education Technology in Northern Ireland, page 14.

Paperless Examinations Project Phase 2 Report

1.2 Phase Two

Phase Two of the project commenced in October 2001 and ended in July 2003. This phase built on the outcomes from Phase One with work being undertaken to resolve the issues that arose in the first development phase16. It expanded on the scope of Phase One with pilot computer-based GCSE-type tests offered in Science (Biology, Chemistry and Physics) and Geography. A major element in this phase was to investigate the potential for enhancing the assessment process. Since the questions were enhanced through the use of ICT, it was not possible to offer equivalent paper-based tests to be taken by a control group. The number of pilot schools was increased to twenty located across Northern Ireland and England. In addition, through a parallel development, computer-based Adult Basic/Key Skills Tests were piloted in UfI centres, FE colleges, Army and training providers in England, Germany and Northern Ireland. In support of the requirements to develop computer-based assessment, the delivery system was designed to provide for: candidate registration; tests to be delivered to centres; an option to use the Basic/Key Skills Tests live over the Internet; candidates to complete the tests onscreen; onscreen marking of the tests within a mark centre; and, analysis and reports of test results.

1.2.1 Aims
To provide a well-argued case for the use of ICT within high stakes examinations that would improve the processes associated with candidate registration, test development and delivery, marking, and reporting and analysis of results. To develop assessment that is fit for purpose where large-scale assessment is computer-based. This would include moving and manipulating large amounts of data as required by the delivery of multimedia questions and the capturing and marking of complex constructed responses. To pilot live computer-based Basic/Key Skills Tests and investigate implications for rollout of onscreen skills test.

1.2.2 Objectives
To develop a robust, viable and secure computer-based assessment delivery system that would ensure accessibility, authenticity and control, and would enable CCEA and Edexcel to plan the introduction of computer-based assessment on a large scale. To conduct six computer-based 60 minutes tests adhering to GCSE Science specifications and two computer-based 75 minutes tests adhering to GCSE Geography specifications in twenty secondary schools across Northern Ireland and England. To develop software that will enable computer-based testing and processing for Basic/Key Skills. To pilot live computer-based 60 minutes or 75 minutes Basic/Key Skills Tests in fifteen to twenty test centres (different from GCSE test centres) located in Northern Ireland, England and Germany.
CCEA/Edexcel (2001): Paperless Examinations Project Phase I Report. Executive Summary. Available on CCEA website www.ccea.org.uk

16

Paperless Examinations Project Phase 2 Report

To produce an informed report with recommendations to plan for large-scale rollout of computerbased examinations.

1.2.3 Scope
Consultation and liaison with approximately thirty-five CCEA and Edexcel schools, colleges and other training providers17 in Northern Ireland, England and Germany with regard to compatibility and system requirements. In all cases, the centres involved needed to show a significant ICT ethos and a willingness to participate in this type of developmental work. Design and testing of the computer-based GCSE-type tests as specified by CCEA and Edexcel Science and Geography examination teams. The test items developed had to be based on GCSE or Basic/Key Skills Tests requirements, as agreed by the Regulatory Authorities (QCA, ACCAC and CCEA)18. Recruitment of CCEA and Edexcel examiners and markers and the provision of training in the design of innovative items and the use of the marking software. Review of the computer-based GCSE-type tests in collaboration with CCEA and Edexcel examination officers and representatives from the pilot schools with regard to quality assurance and relevance to the curriculum and assessment criteria requirements. Design, adaptation and development of invigilation instructions for delivery of assessment to test centres in relation to computer-based test delivery based on the BSI Code of Practice (BS 7988)19. Administration of the computer-based tests including test centre set-up, training of test centre staff with regard to the technology, registration, test requirements, test delivery to centres, delivery of the pupils responses to markers, test results processing and reporting. The system/software required complying with interoperability standards20 being developed in the area of computer-based assessment and online managed learning systems where these were in place or under development.

1.2.4 Project Context

The project undertook two different strands of piloting computer-based assessment: GCSE-type exams and Adult Basic/Key Skills Tests. Strand A: Trialling GCSE-type examinations One of the strands was the development of computer-based GCSE-type examinations that were trialled with a number of post-primary schools in England and Northern Ireland. These exams were not 'live' examinations. The chosen subjects were Science (Biology, Chemistry, Physics) and Geography. The sciences had been trialled in Phase One. They were taken forward within Phase Two to allow a better understanding to be gained of the implications and differences of entirely computer-based examining without a possible paper-based contingency as had been the process in Phase One. The second subject

17 18 19 20

Hereafter referred to as centres. Refer to glossary section. BS 7988:2002. Code of practice for the use of information technology (IT) in the delivery of assessments. SCORM: The Sharable Content Object Reference Model (SCORM) defines a Web-based learning Content Aggregation Model and Run-Time Environment for learning objects. The SCORM is a collection of specifications adapted from multiple sources to provide a comprehensive suite of e-learning capabilities that enable interoperability, accessibility and reusability of Web-based learning content. The work of the ADL Initiative to develop the SCORM is also a process to knit together disparate groups and interests. This reference model aims to coordinate emerging technologies with commercial and/or public implementations. www.adlnet.org

Paperless Examinations Project Phase 2 Report

(Geography) was chosen because questions/items in this subject would require more and longer free response answers and certain scientific methods relevant to data handling. In addition, Geography was considered to offer further opportunities for the use of multimedia within the item development. Since the trial exams were based on the GCSE specifications offered by both Awarding Bodies, it was expected that they would be useful in the schools for revision purposes prior to the live GCSE examinations. The combination of automated and professional marking was an important feature of the pilot software. Strand B: Piloting live Adult Basic/Key Skills Tests The second strand of the project was the live piloting of Adult Basic/Key Skills National Tests in centres across England, Northern Ireland, and Germany.

Background information
The current (paper-based) procedure for these tests is outlined in the following. The four Adult Basic Skills Qualifications available are: Certificate in Adult Literacy (Level One); Certificate in Adult Literacy (Level Two); Certificate in Adult Numeracy (Level One); Certificate in Adult Numeracy (Level Two).

These tests are offered by a number of accredited Awarding Bodies. The Awarding Bodies currently accredited to offer assessments for these qualifications include AQA, City & Guilds, Edexcel, LCCI, NCFE and OCR. The only form of assessment currently approved for these qualifications are the National Basic Skills tests devised for the Regulatory Authorities (QCA, ACCAC and CCEA ). These tests lead to certification for adult learners who wish to have their achievements in the National Adult Core Curricula for Literacy and Numeracy recognised. The above four tests are the same tests as those for Key Skills qualifications in Communication (Levels One or Two) and Application of Number (Levels One or Two). For the Key Skills qualification, in addition to the tests, learners are required to complete portfolios of evidence. For Key Skills, tests in Information Technology (Levels One or Two) are also available Test questions are based around everyday scenarios, each group of questions testing different aspects of reading, writing, numeracy, or IT skills. All questions are multiple-choice questions of the formula offering four answers out of which one is correct. During each test window a different version of the tests is offered. Candidates are allowed to re-sit the tests. They can be drawn from any type of Basic/Key Skills provision. The set of paper-based tests converted for computer-based assessment for this project included: Adult Literacy/Communication (Level One) Adult Literacy/Communication (Level Two) Adult Numeracy/Application of Number (Level One) Adult Numeracy/Application of Number (Level Two) Information Technology (Level One) Information Technology (Level Two)

For the Phase Two pilot it was essential that the learners were prepared for the assessment and that they were confident enough in their use of computers. The test window lasted for two weeks.

10