Home Search Collections Journals About Contact us My IOPscience

A diagnosis of the mathematical and scientific reasoning ability of first-year physics

undergraduates

This content has been downloaded from IOPscience. Please scroll down to see the full text.

1991 Phys. Educ. 26 359

(http://iopscience.iop.org/0031-9120/26/6/005)

View the table of contents for this issue, or go to the journal homepage for more

Download details:

IP Address: 193.255.248.150
This content was downloaded on 01/02/2015 at 10:36

Please note that terms and conditions apply.

 P o p Educ 26(1991).Prinl~dinIheUK

A diagnosis of the mathematical and

scienjf ic reasoning ability of f irst-year
physics undergraduates

Rikus Saayman

The strudure snd results of a universlly physks Structure

enlry test with 75 multlpie-cholca items are din-
The diagnosis contains:
cusaed. It measures studants axpertlse wilh (i) six subscales or criteria of mathematical
mathemstlcai 10018 and formal logical cognitive
skills required for university study of physics:
operations required lor the sh@3yof physics, and numbers, algebraic expressions, ratios and geo-
senen as a useM am lo evalustlng the academic metrical scaling, equations and formulae, func-
potential of a parllcuiar ciaan, points out specific
tions and graphs, and trigonometry and vectors;
ShOriCorningS Mbe rectifiedand advlses individual
and
abxlents on their study choka.
(ii) six subscales or criteria of formal logical
cognitive operations necessary for scientific
During the past four years a university physics
research: proportionalities, control of variables,
probabilities, combinations, hypotheses and logi-
entry test was developed and taken by a total of
cal analysis and synthesis.
3500 students at the Department of Physics,
University of Stellenbosch, South Africa. Each criterion contains two to eleven items of
varying degrees of difficulty, a total of 75 items for
the complete test and 12 criteria. A few of the
maths questions and most of the science logic
problems are similar to those developed at USA
Aims
universities by Renner and Paske (1977) (Okla-
The diagnostic test was designed with the follow- homa), and Griffith and Weiner (1982) (Pacific).
ing in mind The mathematical items require essential school
(i) to assess the average level of mathematical maths tuition to be acquired by all the test persons.
skills and scientific reasoning of a particular The science logic items are independent of physical
science knowledge and based on the test persons
first-year physics class in conjunction with their
everyday experience, for example recipes, plant
school matriculation grades in maths and physical
growth, raffle tickets, shopping centres, traffic
science;
and crime cases. Apart from a short definite
(ii) to indicate specific shortcomings in these
prerequisites in order to choose and plan remedial answer the science logic questions also require a
clear explanation.
action;
(iii) to aid individual students in their subject
choice and advise them on their prospects of study
success. Test admlnistratlon
The diagnosis was group executed for 2-2% hours
on the first six consecutive afternoons of the
1987-90 academic years by an annual average of
RlLus Saayman is a Senior Lecturer in Physics at the 875 first-year physics students enrolled in five
University of Stellenbosch. South Africa, where he also
obtained his PhD and BEd. He has a particular interest
differentiated courses at the University of Stellen-
in stellar structureand evolution, and conceptual bosch, South Africa: continuation course for
development in mechanics BSc, ancillary courses for engineering (Eng) and
Table 1. Analysis of diagnostic university entry test on mathematical skills and scientific reasoning of first-year
physics students. The number of students for each course is shown in parentheses.

Percentage student failures

Item Mathematical skill/ BSc BioY Total

no scientific reasoning (305) (444) (2631)
-
Numbers
1. Scientific notation 3 5 10 6
2. Significant figures 74 59 16 75 72
3. Order of magnitude 61 61 75 12 65
4. Estimation 60 57 63 67 67 62
5. Signsand brackets 15 17 15 22 21 18
Average 43 40 46 50 49 45
Algebraic expressions
6. Product 20
0 2 I 4 3 2
7. Quadration 20 15 33 30 23
8. Mean factor II 6 5 12 8 8
9. Simplification of fractions 10
59 56 54 74 81 64
IO. Powers I1 10 20 18 14
II. Rootforms 21 19 17 31 29 23
12. Logarithms 14 18
9 I1 34 33 19
Average 19 16 30 29 22
Ratios and geometrical scaling
13-14. Ratios I2 10 12 18 22 14
15-16. Percentages 16 20 17 28 31 22
17-18. Similar triangles I1 7 II 16 20 12
19-21. Areaofa triangle 25 I5 27 36 44 28
22-23. Area o f a cube 47 49 56 67 67 57
24. Volume o f a cube 28 21 25 42 42 30
Average 23 20 25 35 38 27
Equations and formulae
25. Logarithms 32 29 35 57 48 39
26. Reciprocals 16 20 16 27 27 21
27. Linear with one unknown 18 22 19 29 28 23
28-29. Linear with two unknowns 19 18 17 31 30 22
30. Quadratic with one unknown 13 11 I1 25 17 15
31-32. Linear and quadratic with two
unknowns 59 68 59 75 75 67
33. Change of subject 23 19 22 47 42 29
34. Elimination of a parameter 44 34 42 61 59 46
Average 28 28 28 44 41 33
Functions and graphs
35. Function value by substitution 19 18 16 26 22 20
36. Function conversion 16 14 10 32 21 18
37. Vertical cut otT of straight line 20 19 20 31 32 24
38. Gradient of straight line 17 10 14 33 30 19
39. Intersections of two graphs 39 41 39 54 49 44
40. Derivative o f a polynomial I8 16 9 31 16 18
41. Extreme value 37 31 33 54 50 39
42. Types of graph 70 75 70 80 79 75
Average 30 28 26 43 31 32
Trigonometry and vectors
43. Pythagoras and trig. function 45 37 37 56 48 43
44. Trig. function values 54 54 55 75 73 61
45. Supplementary angles 39 34 39 56 53 43

360
Table 1. (continued)

Percentage student failures

Item Mathematical skill/ BSc Eng Med BioY Bios Total
no scientific reasoning (305) (790) (629) (444) (463) (2631)

46. Cosine formula application 36 31 31 52 58 40

47. Sine formula application 47 47 49 68 69 55
48. Vector components 32 32 28 53 49 38
49. Vector resultant 24 19 23 38 32 26
Average 40 36 37 57 55 44
Proportionalities
50-51. Recipequantities 19 21 19 30 35 24
55-56. Measuring cylinders 29 31 35 47 49 38
Average 24 26 27 39 42 31
Control of variables
52-53. Gas production 46
56 54 M) 66 69 61
64. Plant growth 45 52 61 61 51
72-73. Bending rods 20 18 21 28 28 23
Average 41 39 44 52 53 45
Probabilities
54. Picking items in the dark 26 24 26 35 40 30
59-61. Carnival raffletickets 42 35 42 52 55 44
66-69. Parking garages 29 24 27 38 36 30
Average 32 28 32 42 44 35
Combinations
57-58. Shopsarrangement 62 54 67 73 60 62
62-63. Crabs and algae 80 80 75
82 85 88 84
Average 71 67 79 74 73
Hypothew
65-69. Detective case 53 60 57 69 65 59
70-71. Playing cards 71 68 74 79 82 74
Average 62 64 65 74 74 67
Logical analysis and synthesis
74. Age 20 19 24 40 40 28
75. Speed 56 44 51 71 73 56
Average 38 32 38 56 56 42
1-75. Total test average 66 68 65 53 54 62
Standard deviation 13 II I2 13 II 12

medicine (Med), year course for the biological person also his or her own additional other answer
sciences (BioY), for example agriculture, forestry, with proof.
dietetics and food science, and semester course for Student results on each test item are explained in
biologists (Bios). initially (1987) the test format table I and summarized for all twelve subscales in
was open-ended with time-consuming manual figure 1. Only comparable results from multiple-
marking and error analysis to develop distractors choice answer test types (1988-90) are listed. The
for multiple-choice answer types, optical mark correct solutions were displayed on the bulletin
reading and computer analysis since 1988. The board, discussed during a tutorial session with the
answer choice of each question allows the test BSc group and followed up by hand-outs of

361
 Rntios ond geometrical r r o l i n g ,

I I '. Functions ond grophr ,

I I
I Equntionr ond formulae

IbI

0 20 LO 60 80
Student foilureri%

Figure 1. Major shorlwmings i n (a) mathematicalskills and (b)scientific

reasoningof firstyear physics students in 0%. Engineering, Medical
(BEM) and Biological (WO)courses at the University of Stellenbosch.
South Africa, 1988-W.

remedial exercises and homework. BSc students significant figures (no tuition at school level),
who got less than 60% in their tests and school rounding off to order of magnitude (above-
grading were advised to seriously reconsider their average students did 8% better) and inability to
choice of the 'continuation physics' course. estimate the width of the lecture hall within two
metres correctly. Omitting the above-mentioned
item 2 reduces failures to 38% and shifts the most
Mathematical skill analysis
problematic skill to trigonometry and vectors on
From table 1 it follows that the highest failure which more than half of the students (although
(45%) is in the subscale on numbers which is 15% less of the above-average students) failed item
mainly attributed to lack of understanding of 44 on trigonometric function values and appli-

362
cation of the sine formula supplied earlier. About and ratios) as well as 5- 19% in the original first five
a third of the freshmen do not match the two science logic subscales. The correlation between
criteria on equations and graphs where items the 1987 (only BSc group) and 1988-90 (all
31-32, requiring the simultaneous solution of a courses) subscale totals is a good r=0.82, signifi-
Linear and quadratic equation needed in the appli- cant at thep<O.Ol level. That between the correct
cation of momentum and kinetic energy conser- answer and correct reason choice for ten items of
vation during collisions, and item 42, recognising the scientific reasoning part of the test is an even
the Cartesian equation for a circle, are respectively better r=0.93, significant atp<0.001, with mentio-
the major weighting factors. Among the remaining nable discrepancies, namely correct answer-wrong
two less problematic skills the major obstacles are reason combinations (probably guessed) only in
items 22-23 which require the change in cross- the case of item pairs 50-51, 70-71 and 72-73,
sectional and surface area of a cube when all three which decrease their relevant subscale failures by
of its dimensions are scaled up by a factor of two only 6, 4 and 3% respectively. The test results of
and simplification of a not so straightforward the 1988-90 first-year intakes are very consistent,
algebraic fraction in item 9, but the above-average the deviations from the averages quoted on the
students succeeded substantially better by 13% tables and displayed in the figure varying only by
than the total group. a few per cent for the twelve criteria but up to a
14% maximum for some individual items.
Scientific reasoning analysis Quantitively the test reliability is highly esti-
From table 1 it follows that combinatorial and mated according to the Kuder-Richardson
formula (Gronlund 1982, p 134):
hypothetical reasoning are the formal cognitive
operations which are the least realized (only !4
and M of universum), for example only 16% of
-
X(J-X) 6375-62)
students could correctly determine the 15 different R=l--- -I- =0.925
possible combinations, avoiding permutation of IS2 75( 12)*
previous listings, of finding four different algae in
the stomach contents of crabs (items 62-63), and
48% including 19% of the above-average group with I = number of test items (the more the better),
chose the distractors in items 70-71 which indi- x=average test score and s = standard deviation.
cate ignorance of the truth that when supplied Personal factors like physiological and emotional
information falls outside the scope of a hypothesis state, fatigue during the university orientation
or rule it neither confirms nor disproves it. Student week, intervening non-academic years between
performance on control of variables and logical school leaving and university entry, attitude and
analysis and synthesis are comparable with that of cognitive style-for example motivation to read
the two most inadequate maths skills. For example and react carefully on each question-will play a
in items 52-53,49% of test persons but 1 I % less of role in this like any other group-administered
above-average group lacked insight into the need written diagnosis. The results applicable on indi-
to hold other variables constant to compare fairly vidual students should therefore be interpreted
the relation between two of interest, in this case the carefully and preferably supplemented by oral
effect of temperature on the amount of carbon interviews.
dioxide used by plants. In item 75, 56% of the
total, but 13% less of the above-average group, Comparison with academic achievements
were unable to analyse and synthesize a word- Table 2 compares the diagnostic test results of the
formulated kinematics problem specifying the total student sample with school grades in mathe-
relation between the speed and travel time of two matics and physical science as well as the uni-
vehicles and requiring the construction and solu- versity physics examination after the first semester
tion of an algebraic equation. Only about a third of tuition, considering average scores, standard
of the test persons failed on proportional and deviations which include -V, of scores within a
probabilistic reasoning. normal distribution and classification into three
subgroups of achievers. The average scores and
Test reliability standard deviations, ,T&s of the two school grades
and diagnostic test are in good agreement but
Changing the answer format from open-ended to there is a drastic decrease (up to 21%) in the
multiple-choice decreased the difficulty level from average university physics performance of non-
5-9% in four of the six maths subscales and biological students, while the distribution of scores
increased it slightly in the remaining two (numbers widens from 3=12 to 15%. Regarding top

363
Table 2. The academic'achievements (in percentages) of first-year physics students. The number of students lor
each course is shown in parentheses.

Test BSc Eng Med BioY Bios Total

Measure (305) (790) (629) (444) (463) (2631)

School gradet in mathematics

Average score 68 70 68 57 58 65
Standard deviation 12 12 12 13 12 I2
A-grades ( 2 00%) 17 24 17 4 2 15
BC-grades (60-79%) 53 52 55 37 36 47
DEF-grades ( < 60%) 30 24 29 59 61 30

School gradet in physical science

Average score 60 71 70 60 50 66
Standard score 13 25
13 12 12 II 12
A-grades ( > 80%) 15 20 5 2 15
BC-grades (61-79%) 50 56 60 36 38 51
DEF-grades (< 60%)$ 27 19 20 59 60 34

Diagnostic university entry test

Average score 66 60 53 54 62
Standard deviation 13 II 12
65 13 II 12
Formal thinkers ( 2 8 0 % ) 14 15 II 3 I 9
Transitional thinkers (60-79%) 59 65 59 31 33 52
Concrete thinkers (< 60%) 21 20 30 66 66 39

First university examination

Average score 52 50 57 55 54 53
Standard deviation I6 15 I5 16 15 15
Distinctions ( 275%) 6 6 12 10 0 0
Passes (40-4770) 60 64 68 57 61 63
Failures (<a%)$ 34 30 20 33 31 29

t.Standard grades were classified one interval (10%) lower than Higher grader.
In t h e percentages those studena(<B%)whodid no1 lake thesubject at shoal level arealso included.
In t h e e percentages those students who dropped out ofthe course, did not qualify or look the examination are also included

Table 3. Correlafion coefficients ( x 1W)between different test measures of the academic potential of first-year
physics students. The number of students lor each course is shown in parentheses.

0% Eng Med BioY Bios Total

(305) (790) (629) (444) (463) (2631)

School Maths+ School Science 54 59 59 59 30 55

School Maths+ Phys. Exam 43 50 40 47 35 43
School Science+Phys. Exam 55 51 59 53 39 52
Maths-cum-Science+ Phys. Exam 57 57 61 56 45 56.

School Maths+ Diagn. Test 40 45 52 40 33 44

School Science+Diagn. Test 46 40 56 49 32 45
+
Maths-cum-Science Diagn. Test 49 48 61 54 39 51
Phys. Exam+ Diagn. Test 43 44 53 43 33 44

achievers, the total 9% formal thinkers and 8% different tuition methodology, quicker pace, lack
physics distinctions seem to indicate an over- of parental and educational supervision and pres-
estimation of A-grades (12%) at school level. sures, emotional and social environmental influ-
Contributing factors to the two previously men- ences, etc. About halfof the total first-year physics
tioned results could be higher academic standards, intake can be classified as promising students with

364
first-class school passes and diagnosed as trans- average >IO% lower than the first mentioned
itional thinkers. It is doubtful whether two-thirds group (see table 2). Regarding the different diag-
of the biological classes and a quarter of the others nostic subscales, these two subgroups are repre-
with <60% school passes and still operating sented by two separate bar graphs in figure 1, from
on the concrete cognitive level will be successful which it can be deduced that the difference in
physics students. percentage failure widens from -7% in thecaseof
With the exception of the course for biologists numbers, to 21% for analysis and synthesis. The
(Bios) the Pearson product-moment correlations Bio students feature very few school A-grades and
between individual student score pairs listed in formal thinkers, only about half the percentage of
table 3 are fairly reasonable, varying between 0.40 transitional operators (-% of group) but double
and 0.61. The correspondence of the university (- 45 of group) the percentage of concrete thinkers
physics exam results with the school grades is compared with the BEM students, which however
slightly better than with the diagnostic results, does not affect their performance in their physics
which is to be expected since conventional closed- exam much due to lesser maths and science logic
hook time-restricted exams reward subject know- requirements of their non-calculus course (see
ledge and skills rather than science logic and table 2).
reasoning. As a prognosis tool the diagnostic test
proves to he a fair (within 15%) indicator of first
university exam performance in two-thirds of each
physics class. Prognosis beyond first-year physics
All average first-year scores of BSc students
continuing their study of physics listed in table 4
Differentiation between physics courses
are - I % lower than that of their total class (see the
The overall performances of the continuation first column of table 2) hut increase from 2 to 14%
BSc, Engineering and Medical students (jointly with seniority of physics course. The listed cor-
indicated as BEM) in the relevant two school sub- relation coefficients horizontally indicate a slight
jects and diagnostic test are very similar, while weakening between first-year university and school
those in the two biological courses (indicated as grading, hut a drastic strengthening (almost
Bio) correlate well with one another hut are on doubling) between first-year university and diag-

Table 4. Comparison of the academic achievements of BSc students continuing

their study of physics. The number of students is shown in parentheses.
Physics I Physics I1 Physics 111

Average swre ( O h )
School Maths and Science 72 74
Diagnostic test 70 69
First-year grading 63 65
Second-year grading 61 63
Third-year grading - 57

Correlation coefficient ( x 100) between

School t First-year grade 49 56
Schhol t Second-year grade 51 31
School t Third-year grade ~

17
School t Diagnostic test 76 57
Diagnostic test + First-year grade 50 58
Diagnostic test t Second-year gdde 35 38
Diagnostic test +Third-year grade - I1
First t Second-year grades 75 64
First t Third-year grades - 43
Second + Third-year grades - 19

365
 Table 5. The drain of 'top' students away from physics at the University of Stellenbosch. South
Africa. 1987-90.

Students Physics I Physics 11

N % ofclasst N % ofclasst Yo ofPhys. I

Maths A-grade matrics 72 16 20 21 28

Science A-grade matrics 62 14 17 23 21
Formal thinkers 54 12 15 20 28
University distinctions 38 8 I6f 22 42
School A-grades+ University
distinctions 33 7 18: 24 55
Formal thinkers + University
distinctions 21 5 I It 15 52

t Excluding students repeating the course.

Distinctions in Physics 1 course.

nostiq results with seniority. Regarding majoring Conclusion

in physics, vertical comparison in the last column
of table 4 reveals a drastic reduction in cor- A useful diagnostic tool was developed, fulfilling
respondence between exam results and both school the aims to assess the mathematical skills and
'
grading and diagnostic tesling. The university scientific reasoning of first-year physics under-
physics exam results of the three academic years graduates; it indicates particular pre-tuition short-
are in better agreement with each other than with comings and, together with school grading, sup-
either the school grades or the university entry plies academic information to advise individual
diagnosis. students on their ability to be successful in their
A worrying phenomenon is the deflation and course.
drain away of 'top' student potential as indicated
in table 5. Approximately only one out of four
school A-grades in maths and physical science as Acknowledgment
well as diagnosed formal thinkers continue their The author is grateful to Professors Griffith and
studies in physics from the first to second academic Renner for making available their test questions.
years and three out of four of these also major in
physics afterwards. Major reasons for termination
identified from student opinion polls by the author References
are lack of interest or career choice, which forced
them to take Physics I only as an ancillary subject, Griffith W T and Weiner E 1982 Developmenr o / a
or unsatifactory university achievements. Fewer wrirren r e ~offormal
t logicol operations used in science,
than half of the school A-grades and diagnosed unpublished manuscript, Department of Physics,
Pacific University, Forest Grove, OR 971 16, USA
formal cognitive operators pass their first-year Gronlund N E 1982 Constructing Achievemenr Tests
physics course (two examinations) with distinction 3rd edn (Englewood Cliffs, NI: Prentice-Hall)
and slightly more than half of these opt for Renner J W and Paske W C I977 Comparing two forms
Physics 11. of instruction in college physics Am. J . Phys. 45 85 I

366