Environment. Technology.

Resources, Rezekne, Latvia

Proceedings of the 11th International Scientific and Practical Conference. Volume II, 74-77

Evaluating the Quality of E-learning

Material
Ilze Kazaine
Latvia University of Agriculture

Abstract. An increasing number of educational institutions in the study process uses one of the e-learning systems.
Consequently, more and more students are offered learning materials in electronic format. E-materials in distance
learning and e-learning is one of the most important elements, therefore much attention and enough time should be paid
for their development. There are a number of studies on e-learning quality, where criteria of quality are discussed in the
context of chosen e-learning environment and the process of implementation. This article examines only the quality of e-
material. The aim was to find a way to reduce the effort and time of electronic learning material quality evaluation. The
study used content analysis by summarizing the most important factors influencing the quality for teaching materials.
Based on the quality criteria mentioned in the literature and personal experience, a criterion, which affects quality of e-
learning material, were summarized and grouped. The criteria were grouped into four groups resulting from didactic,
media, usability and formal quality. Quality evaluation is performed by using one of the methods used in software
engineering - checklist. Based on the identified quality criteria a checklist was established. In order to facilitate the
evaluation process a web-based tool is offered. The tool includes a defined checklist with assessment rating scale and
three levels of impact. Evaluation of material quality is shown in the terms of percentage. After testing the tool, it could be
used for course developers, program managers or other persons involved in evaluation process of e-learning resources.

Keywords: checklist, evaluation, quality indicators, self-assessment tool.

I. INTRODUCTION approaches and the aim of studies is to establish

Any company's goal is to provide quality service unified quality criteria for e-learning [4].
or sell a quality product. According to the general There are now a number of organizations and
definitions, quality is a set of features and educational institutions, who have created their
characteristics of a product or service that relates to products for quality assessment [5]. Depending on the
its ability to satisfy certain or planned needs [1]. In target object (lecture, course, institution), these
the distance learning quality is defined as the criteria are focused either on design or usability, on
characteristic or set of properties that characterizes didactics, or to learning process. As shown by study
object, event or process compliance with certain of E.Bratengeyer and G.Schwede, the most
previously appointed requirements [2]. However, commonly occurring criteria is pedagogy, didactics
requirements are determined by two main factors: and media design, the quality of the content [5].
customers and legislation. Often, when it comes to e- Emphasizing the significance of learning material
learning quality, then the interpretation is quite content quality, exactly these quality indicators were
differing. Its definition depends on the stakeholders' analysed in this study. For the company or institution
point of view. In the distance learning quality is providing distance learning services, it is essential not
defined as the characteristic or set of characteristics only to provide service and organization, but also
of the object, phenomenon and process compliance quality of e-learning materials used in the process. Its
with certain prior requirements [3]. There are a basic requirements are resulting from learning
number of project activities and organizations studies content, which in turn is determined by the
that relate to quality of e-learning and distance corresponding educational standard. Quality
learning. Quality is mainly focused on the use of assessment has an internal dimension – self-
resources to achieve specific goals. There are a assessment and external dimension carried out by
number of recommendations on e-environment external experts. Project implementer or creator of e-
choice, process organization, accessibility of learning material can implement self-assessment by
education, adaptation of learning environment for using a checklist.
personal characteristics and needs. In order to support Checklists are being used in the case of evaluation
a common understanding of quality and several aspects of product quality or separate stages
organizations, providing e-learning services, could be of process. Them used in quality assurance of
able to provide the appropriate quality, the studies software engineering, to check process compliance,
have been conducted. Studies have analysed code standardization and error prevention, and others.
ISSN 1691-5402
© Rezekne Academy of Technologies, Rezekne 2017
http://dx.doi.org/10.17770/ 10.17770/etr2017vol2.2557
 Ilze Kazaine/ Environment. Technology. Resources, (2017), Volume II, 74-77

Checklist is a list of items required, things to be done, learning units and learning entities [14]. In this study
or points to be considered, used as a reminder [6]. attention has been brought to the quality criteria
With the help of a checklist it is possible to determine relating to content of learning units. Quality criteria
conformity of content to quality indicators. Checklist are grouped into four sections according to their
is a character or a list of activities in which the nature - criteria relating to didactics, technical
observer makes notes [7]. Checklists have been used requirements of media, usability. Individually
in numerous evaluations and have been considered as isolated is formal criteria, which have the greatest
a helpful tool for quality assessment. In many cases, impact on the overall quality of the material. Table 1
only if there is compliance with certain criteria, there shows the list of the quality indicators used in the
is a further evaluation using other methods. various E-Learning systems.
Checklists help evaluators monitor criteria during an
evaluation process. Wingate [8] noted that the most Table I
List of quality criteria
basic type of checklist can be useful in conducting an
Quality
evaluation in many aspects. Checklist can be short Criteria
Quality Subcriteria Reference
and simple list of some of the elements, but can also Formal Segmenting, text [16], [18], [19], [20], [21],
be a complex surveillance system with precise structuring [23]
definitions of expected behaviour. In any case, the Grammatical and [16], [19]
spelling errors
development of checklist should be taken into language [16], [18], [19]
account for assessment of work analysis, analysis of topicality [19]
objectives and tasks. Literature sources [16], [19]
The aim of the research was to find way to Copyright [19], [23]
minimize the effort and time required for evaluation compliance
Review of Version [19]
of the electronic learning materials. This article
Didactic Course Objectives [4], [15], [16], [19] - [21]
describes the checklist as a quality assessment tool, Course Assessments [4], [15], [16], [19], [20]
its creation and application to e-learning. Target group [19] – [21], [23]
media choice [4], [16], [20]
II. MATERIALS AND METHODS Course metadata [4], [16], [18], [19]
Monographic method has been used for this compliance with the [20], [22], [23]
curriculum
article. Information was searched in electronic Media Corporate Design [19]
databases and printed publications, published in Content Layout [4], [15], [16], [19], [22]
Europe. Broad spectrum is discussed and analysed the Graphics and Fonts [16], [19], [21]
scientific literature on quality evaluation of e-learning Multimedia [4], [15], [16], [19], [22]
material, factors influencing quality and most Design of text and [16], [18], [19], [21]
graphics
important characteristics of the period from 2004 to External content [4], [15], [18], [21], [23]
2016. resources
The study used content analysis by summarizing Search [19]
the most important factors influencing the quality of Material load time [16], [18]
Usability Navigation [16], [18], [19], [21], [23]
the e-learning materials. A checklist was created
Functionality [18] - [20]
based on the collected quality criteria which are Perception [19], [20]
mentioned in the reviewed literature (15 sources) and Accessibility [16], [19], [21]
based on the author's experience, obtained while Alternative [19], [20]
working as a teacher from 1997 to 2014 at the Latvia provision
University of Agriculture and from 2014 at the Learning tasks and [18], [21], [23]
feedback
Professional Distance Learning Centre of Latvia. To
evaluate teaching materials checklist questions (total
From an analytical point of view, the existing
86 questions) were evaluated and adjusted in
works helps to obtain a complete overview of the
conversations and discussions in the work and
various aspects, which have be considered in order to
educational environment. In September and October
provide quality of the e-learning material. With
of 2016 unstructured interviews were conducted with
critical assessment it can be concluded that the major
both teachers and distance learning students in an
attention is on the didactic aspect and the appropriate
informal atmosphere.
use of media. Less often mentioned is criteria, which
relates to the material adaptation for people with
III. RESULTS AND DISCUSSION
special needs.
In sources of information on e-learning the most
During the research a web-based self-assessment
emphasis is on following criteria: content, navigation,
tool was developed for e-learning material content
design of text and graphics, learning tasks and
quality evaluation. This tool can be used by course
feedback [9], [10]. Of course the most important
developers, program managers or other persons
element is the Content [11]-[17]. Any electronic
involved in the evaluation process of e-learning
learning material fall into three types: technical parts,

75
 Environment. Technology. Resources, Rezekne, Latvia
Proceedings of the 11th International Scientific and Practical Conference. Volume II, 74-77

resources. It is based on a checklist, which is grouped • a tool to guide a discussion between evaluators
into four sections: and material developers regarding the
• Formal quality; preferred contents of e-learning materials;
• Didactic; • a tool that can use the course developers to
• Media; evaluate the material already in the
• Usability. development phase.
Checklist is prepared according to the identified Checklist provides guidelines for course content
quality sub-criteria. The evaluation uses a rating scale for developer about whether it is necessary to
from 1 (doesn’t comply) to 5 (fully meets) and three improve the e-learning materials, allow identify
impact levels: low, medium, high (Fig. 1). After weaknesses in the material that need improvement.
evaluation of the material also quantitative Course developers themselves can use the checklist
assessment is given. It is shown in both -by all the as a monitoring and reflection tool. Checklist allows
four sections and the overall average rating. identifying a number of conditions and requirements
that needs be taken into account in the development
of e-learning materials. To improve assessment
process of materials, a web-based self-assessment
tool was created. Electronic checklist tool is intended
to help assessors to evaluate the final product - the
electronic learning materials, that is, whether it meets
the expected quality. The research should continue to
evaluate the usefulness of the developed checklist
tool.

REFERENCES
[1] Latvijas Nacionālā standartizācijas institūcija „Latvijas
standarts”, “Informācija un dokumentācija – Vārdnīca,”
(Information and documentation – Vocabulary), standarts
LVS ISO 5127:2005, 2005. (In Latvian)
[2] J. Dzelme, A. Kapenieks, M. Upmale, L. Jermolajeva, V.
Vikmane, A. Rengarte, I. Buligina, B. Žuga, Dz. Tomsons, S.
Gibže un A. Škute, Tālākizglītības kvalitātes vērtēšanas
rokasgrāmata. (Quality Assessment Guide of Distance
Fig. 1. The view of e-learning material content quality evaluation learning) Rīga: Lielvārds, 2001, 32.lpp. (In Latvian)
tool - formal quality evaluation part. [3] U.-D. Ehlers, “Quality in e-learning from a learner’s
perspective,” European Journal for Distance and Open
After evaluation of the material also quantitative Learning, [Online]. Available: http://www.eurodl.org/
assessment is given. It is shown in both - by all the materials/contrib/2004/Online_Master_COPs.html. [Accessed
January 16, 2017].
four sections and the overall average rating (Fig. 2). [4] E. Bratengeyer, A. Bubenzer, J. Jäger und G. Schwed,
eLearning Qualitäts-Evaluationstool. (E-learning quality
evaluation tool ) Books on Demand, 2015. (In German)
[5] E. Bratengeyer and G. Schwed, “eLQe: A Cool Tool for
Simply Evaluating Your E-learning Course Development,”
In: Proceedings of the Eighth International Conference on E-
Learning in the Workplace, 2016, New York, USA. [Online].
Available: https://www.icelw.org/proceedings/2016/ICELW
2016/Papers/Bratengeyer_Schwed.pdf [Accessed February 6,
2017].
[6] Oxford Dictionary of English, Oxford University Press:
Oxford, 2010, 2112 p.
[7] A. Geske un A Grīnfelds, Testu teorijas elementi izglītībā.
Fig. 2. The view of e-learning material content quality evaluation (Test theory in education) LU projekta „Profesionālajā
tool – summary of quality. izglītībā iesaistīto vispārizglītojošo mācību priekšmetu
pedagogu kompetences paaugstināšana” materiāls,
vienošanās Nr.2009/0274/1DP/1.2.1.1.2/09/IPIA/VIAA/003,
The developed web-based self-assessment tool is LU reģistrācijas Nr.ESS2009/88, 2010. [Online]. Available:
still in the testing stage and its activities remain to be http://profizgl.lu.lv/pluginfile.php/22519/mod_resource/conte
tested in real work environment. nt/0/AGrinfelds_AGeske/Testi_izglitibaa_final2.pdf
[Accessed January 16, 2017]. (In Latvian)
[8] L. A. Wingate, “The Evaluation Checklist Project: The Inside
IV. CONCLUSIONS Scoop on Content, Process, Policies, Impact, and
The main quality criteria have been identified and Challenges,” Nov 4, 2002, [Online]. Available:
grouped according to their effect. Mainly they are https://wmich.edu/sites/default/files/attachments/u350/2014/i
based on the technical requirements, but also cover nsidescoop.pdf. [Accessed February 6, 2017].
[9] H. M. Niegemann, S. Domagk, S. Hessel, A. Hein, M. Hupfer
didactical aspects. Based on these criteria has been und A. Zobel, Kompendium multimediales Lernen.
created a checklist. E-learning material assessment (Compendium of multimedial learning) Heidelberg: Springer,
checklist has two main roles: 2008, 81 p. (In German)

76
 Ilze Kazaine/ Environment. Technology. Resources, (2017), Volume II, 74-77

[10] G. D. Rey, E-Learning: Theorien, Gestaltungsempfehlungen facdev/guidebk/online/web-elem.htm. [Accessed January 16,

und Forschung. (E-learning: theories, design 2017].
recommendations and research) Hogrefe AG: Bern, 2009. (In [19] A. Sperl, „Bewertung der Qualität von Lehrmaterialien,“
German) (Assessment of the quality of e-learning materials) Dec.8,
[11] Swedish National Agency for Higher Education “E-learning 2015. [Online]. Available: https://alexandersperl.wordpress.
quality. Aspects and criteria for evaluation of e-learning in com/2015/12/08/bewertung-der-qualitat-von-lehrmaterialien/
higher education,” Report 2008:11R, 2008. [Accessed February 6, 2017]. (In German)
[12] A. Usoro and G. Majewski, “Measuring Quality e-Learning [20] U. Grabe, L. Ionica, K. Kunze, P. Schneider und A. Schulz,
in Higher Education,” International Journal of Global „Entwicklung eines Bewertungsverfahrens zur Vergabe des
Management Studies, vol.1, pp. 1-32, 2009. @ward Preis für multimediales Lehren und Lernen an der
[13] E. Bratengeyer, “Entwicklung eines Web-basierten Tools zur Universität Halle,“ (Development of an evaluation procedure
eLearning-Qualitätsevaluation” (Development of a web- for awarding the @ward Prize for multimedia teaching and
based tool for eLearning quality evaluation) Hamburger learning at the University of Halle) Hamburger eLearning-
eLearning-Magazin, no. 17, S. 28-31, Juli 2015. [Online]. Magazin, no.17, S. 37-39, Juli 2015. [Online]. Available:
Available: https://www.uni-hamburg.de/elearning/ https://www.uni-hamburg.de/elearning/hamburger-elearning-
hamburger-elearning-magazin-14.pdf. [Accessed February 6, magazin-14.pdf. [Accessed February 6, 2017]. (In German)
2017]. (In German) [21] C. Schoor and H. Körndle, “Checklist for a Didactically
[14] D. Dinevski, J. Jakončič, M. Lokar and B. Žnidaršič, A model Sound Design of eLearning Content,” no. 29, June 2012.
for quality assessment of electronic learning material: [Online]. Available: https://www.scribd.com/document/
Proceedings of the ITI 2010, 32nd International Conference 99490612/Cornelia-Schoor-Elearning-Papers-2012-checklist-
on Information Technology Interfaces, pp. 343-348, 2010. for-a-Didactically-Sound-Design-of-Elearning-Content.
[15] C.-Ch. Fey, Kostenfreie Online-Lehrmittel: eine kritische [Accessed February 6, 2017].
Qualitätsanalyse. (Free online teaching material: a critical [22] R. C. Clark and C. Lyons, Graphics for Learning: Proven
quality analysis) Bad Heilbrunn: Klinkhardt, 2015, 320 S. (In Guidelines for Planning, Designing, and Evaluating Visuals
German) in Training Materials, 2nd Edition, San Francisco, CA:
[16] C. Pappas, “The Ultimate eLearning Course Design Pfeiffer, 2010, 420 p.
Checklist,” April 10, 2014. [Online]. Available: [23] Swedish Civil Contingencies Agency, “Developing Training
http://elearningindustry.com/the-ultimate-elearning-course- Material Guide,” 2012. [Online]. Available: https://www.
design-checklist. [Accessed February 6, 2017]. msb.se/RibData/Filer/pdf/26433.pdf. [Accessed February 6,
[17] Z. Krajcso, “Classification and quality criteria for Open 2017].
Educational Resources in the field of foreign language [24] E. J. Davidson, Evaluation Methodology Basics The Nuts and
learning,” Journal of Language and Cultural Education, vol. Bolts of Sound Evaluation Thousand Oaks, CA: SAGE
4, Iss. 1, p. 48–59, 2016. Publications, 2005, 280 p.
[18] D. Madden, “17 elements of good online courses,” Aug 3,
1999. [Online]. Available: https://www.honolulu.hawaii.edu/

77