A Learning Analytics Tool for Monitoring and

Improving Students' Learning Process

Yusep Rosmansyah1, Nunung Kartikasari2, Aciek Ida Wuryandari3
School of Electrical Engineering and Informatics
Bandung Institute of Technology
Bandung, Indonesia
1
yusep@stei.itb.ac.id, 2nu2ng.kartikasari@gmail.com, 3aciek@lskk.ee.itb.ac.id

Abstract— A learning analytics tool that facilitates teachers in

monitoring students’ learning process and enables them to early
recognize the signs of failure was developed. The tool was added
paperID_99_iceei2017.pdf
to an e-learning system based on Moodle operated by a primary an d interaction with teachers or
school. Two parallel classes (i.e., an experimental group and a fellow students, extracurricular activities, and so forth.
control group) of students were involved in the research for Learning analytics simply focuses on improving the
duration of one semester. The statistical test results showed that performance of students so that the data collected might be
better learning improvement occurred to the experimental group used by teachers or instructors and heads of educational units
with an increase in the average marks by 5.3 (out of 100 points). in fixing or revising ecosystems or components of learning
The use of learning analytics has helped improve the effectiveness environment in the surrounding areas, such as curriculum,
of the students' learning process. educational programs, instructional design, and so on [4].
Keywords—Learning analytics, e-learning
B. Domains of Competencies of the 21st Century Learning
I. INTRODUCTION According to Pellegrino et al. [5], there are three domains
in developing competencies required by the current students
E-learning (electronic learning) is a type of learning that living in the 21st century, namely:
allows the delivery of teaching materials to students using the
computer, multimedia, and Internet technologies. E-learning 1. The cognitive domain that includes expertise in critical
offers various features that can improve availability, flexibility, thinking, information literacy, logical thinking,
effectiveness, and efficiency of a learning process. In argumentation, and innovation.
particular, it can have a special feature to monitor the learning 2. The intrapersonal domain that covers self-management
activities of students, known as learning analytics. In the which includes competencies such as flexibility,
business world, learning analytics is similar to customer initiative, appreciation of diversity, and metacognition
analytics that becomes an important part of a company's (the ability to reflect on one’s own learning and make
marketing strategy. Similarly, learning analytics becomes a adjustments accordingly).
potential tool to promote effective learning, since it provides
tools, technologies, and platforms to empower teachers to 3. The interpersonal domain that includes competencies
involve, inspire, and prepare students to achieve success in the such as communication, cooperation, collaboration,
learning process [1]. responsibility and conflict resolution.
Referring to [5], the study of learning analytics reported
II. LITERATURE STUDY in this paper was limited to the cognitive and intrapersonal
domains.
A. Learning Analytics
Learning analytics can be defined as the process of C. Related Work
measuring and collecting data on students and learning with the Huang et al. [7] developed a real-time learning analytics
aim of improving teaching and learning processes through the based on a database log. This represents a sophisticated
analysis of data [2]. Learning analytics is at the level of learning analytic system that was too complex to be operated
understanding of the overall learning system (holistic) to by teachers of primary schools in Bandung areas. A similar
support the decision-making process. Another definition was system was developed by Dunbar at al. [8], where data log was
proposed by Arnold [3] which is data interpretation activity collected continuously so that the analysis can be applied as
related to students that is acquired during the academic activity further as for revising the curriculum. In this research, a
process to predict the performance and detect potential simple-to-use learning analytics tool by ordinary Indonesia
problems that may arise in the future. There is also a wide teacher was set as the main objective.
range of data explicitly drawn from diverse sources such as:
test results, discussion records,

978-1-5386-0475-5/17/$31.00 ©2017 IEEE

Authorized licensed use limited to: b-on: Instituto Politecnico de Viana do Castelo. Downloaded on May 10,2020 at 11:41:41 UTC from IEEE Xplore. Restrictions apply.
 III. DESIGN AND DEVELOPMENT students learned while its teacher used the proposed learning
Overall research project followed the method described in analytics tool.
[9]. Starting with the need analysis, the following are The statistical testing proceeded as follow: first, each group
functional purposes of the system, which can be: was given the same questions (pre-test) to find out the average
1. Saving time and reducing operational complexity for value between groups. Pre-test results for each group can be
teachers. seen in Fig. 6. All scores range from 0 to 100.

2. Helping teachers to monitor the progress of the

learning process of students.
3. Providing additional assistance to students who need
more attention and learning support.
4. Accelerating the students' learning process.

Based on the conducted analysis, a system architecture was

needed, covering aspects of business processes and system of
interface design. In short, adopting the method described in [6],
in this paper only the main use case diagram of the developed
learning analytics is shown, such as in Fig. 2.

Inspect Individual Student

Progress Fig. 3 Managing group of students.

Manage Group of Students

based on Progress

Treat Group of Students

Accordingly

Manage Schedule of
Learning Ac vi es

Fig. 2 Use case diagram

IV. IMPLEMENTATION AND EVALUATION

Implementation of the learning analytics was performed by
modifying the Moodle web application. Fig. 3-5 show Fig. 4 Individual student progress.
screenshots of main menus available to a teacher. When all
stages of development and testing had been done and the
learning analytics tool was ready to use, it was time to apply
the tool to students. Two classes of 5th grade students in UPI
Primary School, Bandung, Indonesia, were chosen as
respondents. The cognitive domain of [5] was elaborated to
Science subject. The proposed tool was used for a period of
one semester. The respondents were from middle class of
economy.
The main objective of this paper is the students'
improvement in learning outcome, which will be evaluated by
means of statistical analysis. Meanwhile, other objectives such
as time saving, complexity reduction for teacher in monitoring
the students' progress, and pinpointing certain students who
need more attention, were evaluated by way of interview, in
which both guardian teachers of the two classes confirmed
positively. Fig.5 Managing treatments for students in different groups.
Statistical testing was conducted using a sample of 53
students coming from 2 parallel clases: control group (27
students of Class 5A) and experimental group (26 students of
Class 5B). The control group was the students attending the
learning process normally. Meanwhile, the experimental group

Authorized licensed use limited to: b-on: Instituto Politecnico de Viana do Castelo. Downloaded on May 10,2020 at 11:41:41 UTC from IEEE Xplore. Restrictions apply.
 groups of data to be tested are homogenous (have the same
variances) and have normal distribution. To determine the
variance similarity (homogeneity) in an unpaired groups of
data, the Levene test will be performed. A criterium for groups
of data to be classified as having same variance is if sig. > 0.05,
and otherwise. If the variance is the same, then the t-test can be
performed (by using the Equal Variance Assumed).
To test the normality, Shapiro-Wilk test will be conducted,
since the number of respondents from each group is less than
30.
When the homogeneity and normality tests have been
determined, decision-making criteria for the t-test results are as
follows:
Ho: There is no significant improvement difference of learning
progress between the control group and the experimental
group.

Fig.6 Pre-test results of control and experimental groups. Ha: There is a significant improvement difference of learning
progress between control group and the experimental
group.
Average scores of 74.2 and 75.3, respectively for the
control and experimental group, it can be seen that both groups The deciding factor is the result of the t-test, as follow:
have very similar competence at the beginning of semester. If t-test's sig. (2-tailed) > 0.05
After both groups have gone through the learning process, a
post-test was given to determine the effectivity of the students' then Ho is accepted and Ha rejected
learning progress, with and without using the proposed tool.
else Ho is rejected and Ha accepted.
The results of the average value of the post-test of the two
groups can were 77.6 and 83.8, respectively, as depicted in Fig To make this hypothesis testing simpler, a pseudocode can be
7. presented as follow:
If (t-test's sig. (2-tailed) > 0.05)
then (there is NO significant improvement)
else (there is SIGNIFICANT improvement).

B. Results and Analysis

During the pre-test, the result of Shapiro-Wilk tests showed
values of sig. of 0.616 for control group and 0.100 for
experimental group. The corresponding values for the post-test
were 0.228 and 0.061. Since all values are > 0.05, it was
concluded that all data were considered to be normally
distributed. The remaining tests are homogeneity and t-tests.
Referring to Table I, it is seen that the results of Levene test
in the pre-test is sig. = 0.826, so it can be concluded that
variances of the control and experimental groups are
considered the same. Based on the decision-making criterium
Fig.7 Post-test results of control and experimental groups. and looking at the results of t-test calculation in Table III, it
was shown that the sig.(2-tailed) = 0.763 (which is > 0.05). It
means that Ho is accepted (and Ha is rejected), which means
From the average scores of pre-test and post-test, it can be
there is no significant difference of scores between control
seen that both groups improved in terms of scores. The average
group and the experimental group during the pre-test.
scores of the control group was up (77.6 - 74.2) = 3.4 points,
and it was up (83.8 - 75.3) = 8.6 for the experimental group. Looking at the post-test results in Table II, it can be seen
There was a difference of 5.1 points between the two groups. It that the Levene test to determine the variance of the data group
can be concluded that there was an increased competence of is sig. = 0.181 (which is > 0.05) so that it can be concluded that
the students in the experimental group, which used the data group variance is categorized the same. Based on the
proposed application. decision-making criterium and the results of t-test calculation
in Table IV, it shows that the value of t-test is sig.(2-tailed) =
A. Hypothesis Testing 0.440 (which is < 0.05), so it can be concluded that Ho is
Following [10], hypothesis testing was conducted using rejected and Ha is accepted. This means that there are
unpaired t-test (independent t-test). T-test can be conducted if significant differences of results between control group and
experimental group. In other word, statistical test

Authorized licensed use limited to: b-on: Instituto Politecnico de Viana do Castelo. Downloaded on May 10,2020 at 11:41:41 UTC from IEEE Xplore. Restrictions apply.
 TABLE I
LEVENE TEST RESULTS OF PRE-TEST

Levene's Test for

Equality of
Variances t-test for Equality of Means
95% Confidence
Sig. Mean Interval of the
(2- Differe Std. Error Difference
F Sig. t Df tailed) nce Difference Lower Upper
Equal
variances .049 0.826 .304 51 .763 .936 3.081 -5.250 7.122
assumed
Equal
variances .303 50.148 .763 .936 3.087 -5.263 7.135
not assumed

TABLE II
LEVENE TEST RESULTS OF POST-TEST

Levene's Test for

Equality of Variances t-test for Equality of Means
Sig. 95% Confidence Interval
(2- Mean of the Difference
taile Differe Std. Error
F Sig. T df d) nce Difference Lower Upper
Nilai Equal variances
assumed
1.836 0.181 2.069 51 .044 6.365 3.076 .189 12.540

Equal variances not

2.059 47.254 .045 6.365 3.091 .148 12.581
assumed
____________________________________________________________________

2. From these tests, it indicated that the group using

TABLE III
the application has higher learning improvement
INDEPENDENT T-TEST RESULTS OF PRE-TEST than that who did not.
Equal 3. From the post-test results, it can be concluded that
Equal Variances
Variances
Assumed
Not Assumed there is a significant difference of learning progress
F .049 between the experimental group and the control
Sig. .826 group.
t .304 .303
df 51 50.148
Sig.(2-tailed) 0.763 .763 TABLE IV
Mean Difference .936 .936 INDEPENDENT T-TEST RESULTS OF POST-TEST
Std. Error Difference 3.081 3.087
95% interval of the confidence Equal
Lower -5.250 -5.263 Equal Variances
Variances
Upper 7.122 7.135 Not Assumed
Assumed
F 1.836
Sig. .181
t 2.069 2.059
Based on the obtained resuls, some findings can be df 51 47.254
discussed as follows: Sig.(2-tailed) 0.440 .45
1. In this research, the application of the proposed Mean Difference 6.365 6.365
Std. Error Difference 3.076 3.091
learning analytics has helped improve the learning 95% interval of the confidence
of cognitive and intrapersonal domains of the 21st Lower .189 .148
century skills. Upper 12.540 12.581

Authorized licensed use limited to: b-on: Instituto Politecnico de Viana do Castelo. Downloaded on May 10,2020 at 11:41:41 UTC from IEEE Xplore. Restrictions apply.
 REFERENCES

V. CONCLUSION [1] Elias, Tanya. Learning analytics: Definitions, processes and

potential. 2011. (Accessed from
Based on the research design and implementation in the http://learninganalytics.net/DefinitionsProcessesPotential.pdf,
field, some conclusions and future work can be expressed as December 2016.)
follows: [2] Long, P. & Siemens, G. Penetrating the Fog: Analytics in Learning
and Education. Educause Review, 46(5), 31-40, 2011.
1. A simple-to-use learning analytics tool has been planned,
[3] Arnold, K. “Signals: Applying Academic Analytics". 2010. Accessed
designed and implemented for a semester to 2 classes of from: http://er.educause.edu/articles/2010/3/signals-applying-
the 5th grade students. Tests and evaluation of the academic-analytics)
impacts have indicated that there is a significant [4] Macfayden, L. P., and S. Dawson. “Mining LMS Data to Develop an
improvement in terms of scores (5.3 points). This ‘Early Warning’ System for Educators: A Proof of Concept.”
indicated that cognitive and intrapersonal aspects of Computers & Education 54 (2): 588–599. 2010.
learning have been improved. [5] Pellegrino, J.W. & Hilton, M.L.Education for Life and work:
Developing Transferable Knowledge and Skills in the 21st
2. The proposed learning analytic tool has helped teachers Century.Washington, DC: National Academies Press., Eds.2013.
to treat the "slowing-down" students to catch up with the [6] A. Dennis, B. H. Wixom, and D. Tegarden, Systems analysis and
rest. It saved teachers' time and reduce complexity design: An object-oriented approach with UML. John Wiley & Sons,
compared to traditional method in monitoring the 2015.
students' learning progress. [7] L. H. Huang, C. R. Dow, Y. H. Li, P. Hsuan and W. L. Koo, "A
Prompt Response Monitoring System for the Guiding Support of
3. In the subsequent research, better improvement of Learning Activities," Information Technology: New Generations
learning effectivity is expected. This may require the (ITNG), 2010 Seventh International Conference on, Las Vegas, NV,
2010, pp. 1080-1085.doi: 10.1109/ITNG.2010.74
development of learning analytics using predictive and
[8] Dunbar, Robert L.; Dinger, Molly J.; PRAT-RESINA, Xavier.
prescriptive analysis. Both of these analyses may assist "Connecting Analytics and Curriculum Design: Process and
teachers in predicting things that will happen in the Outcomes of Building a Tool to Browse Data Relevant to Course
future based on the patterns available at present and help Designers." Journal of Learning Analytics, [S.l.], v. 1, n. 3, p. 223-
in the decision making process to determine the best 243, Aug. 2014. ISSN 1929-7750
treatment for the students. [9] L. T. Blessing and A. Chakrabarti, DRM: A Design Reseach
Methodology. Springer, 2009
[10] Fraenkel, Jack R, and Norman E. Wallen. How to Design and
Evaluate Research in Education. 7th ed. Boston: McGraw-Hill, 2009.

Authorized licensed use limited to: b-on: Instituto Politecnico de Viana do Castelo. Downloaded on May 10,2020 at 11:41:41 UTC from IEEE Xplore. Restrictions apply.