BCTt

Beginners Computational Thinking Test

1
María Zapata Cáceres
Computer Science Department URJC
@maria__z_c

Estefanía Martín Barroso

Computer Science Department URJC
@EstefaniaURJC

Marcos Román González

Faculty of Education UNED
@elpsycomago

2
 Computational Thinking (CT)
BROAD
Many definitions DEBATE Many breakdowns

Human problem-solving Abstraction

process that requires abstract
Decomposition
thinking
Generalization
Thinking skills that precede
Iteration
programming, and are applied
in understanding a problem Debugging
and formulating a solution like
a computer scientist Algorithms

Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017a). Demystifying computational thinking.Educational Research Review, 22,
142-158 doi://doi.org/10.1016/j.edurev.2017.09.003
 Computational Thinking (CT)

CONSENSUS

Fundamental skill required to adapt to the future

Should be taught at schools

Programming exposes students to CT

 Computational Thinking (CT)

Many frameworks

CT Concepts
3D
Framework
CT Practices
(Brenan &
Resnick*)
CT Perspectives

* K. Brennan, M. Resnick and MIT Media Lab, "New frameworks for studying and assessing the development of computational thinking," American Educational Research
Association Meeting, Vancouver, BC, Canada, 2012.
5
Questions

When How
to teach? to teach?

How What
to assess? to teach?
Questions

When How
to teach? to teach?

How What
to assess? to teach?
 Assessment

When to How to
teach? teach?

Traditional
How to What to
assess? teach? test

Portfolio Focused on middle

school grades and
Types of Survey specific
programming
assessment Interview environments

8
 Computational Thinking Test (CTt)

Stand-alone assessment instrument

Computational
Thinking Test
Reliability and criterion validity,
(CTt) * psychometric approach

Aimed at
Aligned with the international standards
10 to 16 years old

CT concepts
3D CT practices
Framework
CT perspectives

* M. Román-González, J. Pérez-González and C. Jiménez-Fernández, "Which cognitive abilities underlie computational thinking? Criterion validity of the Computational 9
Thinking Test," Computers in Human Behavior, vol. 72, pp. 678-691, 2017.
 Beginners Computational Thinking Test (BCTt)

Based on CTt

Stand-alone
assessment instrument
Beginners
Computational 5 to 10 years old
Thinking Test
Form / content
(BCTt) adaptation

Substantial
improvements
10
 v.1 BCTt v.1 Design
25 items long 40 minutes

3 alternative responses 3D Framework computational

concepts
Computational concepts in BCTt
Test Loops Conditionals
1.
items 2. Simple 3. Nested 4. IF- 5. If-then- 6.
Sequences
loop loop then else While
1-6
7 - 11
12 - 18
19 - 20
21 - 22
23 - 25
11
v.1
BCTt v.1 Design
Self explanatory symbols Least possible text

Emotional connection

Canvas and maze type

12
v.1 BCTt v.1 Design

13
If-else example

A B C
If-else example

A B C
If-else example

A B C
If-else example

A B C
If-else example

A B C
If-else example

A B C
If-else example

A B C
v.1 BCTt Expert Judgement procedure & results
45 experts 66 items form

Content validation

Item difficulty level

Item relevance to measure CT

Test length adequacy

Graphic interface adequacy

Improvements adequacy: e.g. transitions

Other / suggestions / comments

 v.1 BCTt Expert Judgement procedure & results
5,00
4,50
Test length
4,00
3,50
3,00 adequate
2,50
2,00 y = 0,1063x + 1,4239
too long
R² = 0,8883
1,50
1,00
too short
0,50
0,00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Adequacy to evaluate CT
BCTt item difficulty perceived by experts
very good
5,00 good
4,50
4,00 intermediate
3,50
3,00 bad
2,50
2,00
1,50
very bad
y = 0,0786x + 3,6857
1,00
R² = 0,5818
0,50
0,00
0 1 2 3 4 5 6 7

83% → Transitions
BCTt computational concept relevance to
measure CT, perceived by experts are positive 22
 v.1 BCTt Expert Judgement procedure & results
Comments / suggestions

“the allowed paths are

“transitions are easily associated clear with transitions,
to arrows in the answers”
because it excludes
diagonal movements”

“In the design without transitions,

“It is not clear
doubts are generated about when
a character reaches another “the test is if two chicks
(either when it reaches the TOO HARD” can move
previous square or when it reaches together after
the other character square?)”. meeting”
23
 v.2 BCTt v.2 Design
Form and content modifications

Oral
explanation

4 alternative
responses

If-else
reformulation

If-then-else
reformulation

24
v.2 BCTt v.2 Design
Colour blindness adaptation

25
 v.2 BCTt Administration: participants and procedure

299 Primary School Students Action protocol

5 to 12 years old Test printed in paper form

BCTt transitions

BCTt variation No-transitions

School Educational stage Grades Students ages

Colegio Público Carlos Ruiz 1st 1st and 2nd 5-8
Colegio Los Escolapios 2nd 3rd and 4th 7 -10
CEIP León Felipe 3rd 5th and 6th 9 -12
 v.2 BCTt Administration: participants and procedure
Educational BCTt
Grade Identifier BCTt
stage variation
1 A A1: n=52
1st
2 B B1: n=18 B2: n=18
4 C C1: n=54
2nd
4 D D1: n=28 D2: n=28
5 E E1: n=51
3rd
6 F F1: n=25 F2: n=25
B2, D2, F2 A1, B1, C1, D1, E1, F1 D1
BCTt
BCTt BCTt
variation

Procedure

Time 1 Time 2: 5 weeks later 27

 v.2 BCTt Administration: results

Student’s Significant difference in test scores

Transitions
t-test (p=0.005< 0.01) in lower grades

Sample Entire sample A1 B1 C1 E1 F1

Grade 1-6 1 2 4 5 6
Descriptive
statistics N 200 52 18 54 51 25
Mean 19.92 16.52 16.78 21.57 21.84 21.72
Median 20.00 16.00 18.00 23.00 23.00 22.00
Std. Deviation 3.79 3.31 2.49 3.044 2.61 2.62
Variance 14.36 10.96 6.183 9.268 6.815 6.88
Minimum 8.00 8.00 11.00 14.00 13.00 15.00
Maximum 25.00 24.00 20.00 25.00 25.00 25.00
25 17.00 14.00 15.75 19.00 20.00 19.50
Percentiles 50 20.00 16.00 18.00 23.00 23.00 22.00
75 23.00 19.00 18.00 24.00 24.00 24.00
28
 v.2 BCTt Administration: results
Item analysis
5,00 1,2
4,50
4,00 1
3,50
3,00 0,8
2,50
2,00 0,6
1,50
1,00 0,4

0,50
0,2 y = -0,0155x + 1,0083
0,00
R² = 0,6344
Sequences Simple loop Nested loop If-then If-then-else While
0
1st 2nd 4th 5th 6th 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Computational concept by grade Item difficulty index for each BCTt item
1,2
y = -0,0173x + 0,9249
1 R² = 0,3904

0,8

0,6

0,4

0,2

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Item difficulty index 29

 v.2 BCTt Administration: results
Sample Reliability Statistics Item Statistics
Cr. 's Alpha
Based on
N of Cronbach Stand.
N Items 's Alpha Items Mean Min. Max. Variance
200 25 0.824 0.829 0.807 0.576 0.976 0.021
Subsamples Reliability Statistics Item Statistics
Cr. 's Alpha Based on
Ed. stage Grade Id. n Cronbach's Alpha Stand. Items Mean Variance
1st 1 A1 52 0.833 0.838 0.742 0.041
1st 2 B1 18 0.793 0.801 0.630 0.042
2nd 4 C1 54 0.771 0.735 0.837 0.022
3rd 5 E1 51 0.660 0.683 0.863 0.012
3rd 6 F1 25 0.657 0.648 0.844 0.015

Task and re-task method

very strong significant correlation
(D1 subsample)
Non-parametric Spearman’s test (rs=0.93; p<0.01).

30
 Conclusions
✓BCTt is adequate for the assessment of CT in Primary School

✓Transitions between maze squares are a relevant improvement

for young students

✓BCTt seems to be aimed at 1st to 4th grades (5 to 10 years old)

✓Reliability is high and higher in younger students

✓Recommended to use in parallel with other tools → system of

assessments
CT concepts
3D CT practices
Framework
CT perspectives 31
 Conclusions
✓It could be used as a pre-test / post-test instrument

Blue Ant Code (Android and IOs)

 Conclusions

✓BCTt lower limit

✓Other countries: Portugal, Chezch republic, Germany,

France, …

✓Populations

✓BCTt new version → Cornell University NY

33
Thank you very much for your time!

maria.zapata@urjc.es
@maria__z_c
34