Prof DR Muhamad Jantan - Malaysia
Prof DR Muhamad Jantan - Malaysia
Prof DR Muhamad Jantan - Malaysia
Malaysian Experience
with Rating
Muhamad Jantan
Director, Institutional Development Division,
UNIVERSITI SAINS MALAYSIA
PENANG
AGENDA
Introductory Remarks
Malaysia – Socio-Economic Profiles
Current Higher Education Scenario
Rating Exercise & Instruments Currently
in Use
SETARA – An Instrument to Assess
Quality of Learning & Teaching
Lesson learnt from SETARA exercise
MALAYSIA
KUALA LUMPUR
Socio-Cultural Profile
• The Country: a federation of 13 states
• Land Mass: 329,847 sq. km. (127,355 sq mi)
• Population: 28.31 million people;
• Ethnic Groups: 50.4% Malays; 23.7% Chinese;
7.1% Indian
• Religion: Islam (60.4%), Buddhism (19.2%),
Hinduism (6.3%), Christianity (9.1%)
Economic Profile
25000 60.00
Number of Faculty Percent of Faculty
50.00
20000
40.00
15000
30.00
10000
20.00
5000
10.00
0 0.00
Professor Associate Senior Lecturer Total Professor Associate Senior Lecturer
Professor Lecturer Professor Lecturer
4 POLYRATE 2011 Assess the Quality of Teaching and Learning of Polytechnic Division
Polytechnics offering TVET Diploma of MoHE
4/4/2013 12
The SETARA Framework
RATING OF INSTITUTIONS
TOTAL CUT-OFF
SCORE VALUES
DOMAINS
DATA INSTRUMENT
INDICATORS
INSTITUTIONS EMPLOYERS
TRACER STUDY
© mjantan@usm.my 13
Principles used in Developing the
SETARA Instrument
• Validity and Reliability: Data used must be valid and reliable
• Relevant and comparable : internationally and locally
(previous SETARA)
• Parsimony: minimal indicators to avoid data fatigue;
• Data reuse: use of readily available databases (when
appropriate, e.g. MyMOHES, Tracer Study, COPIA)
• Scoring System: Be readily incorporated into the other rating
scoring system
• Verifiability: Be subject to verification in terms of methods
and data (Independent Verification Committee)
• Normalization & Equalization: For missing data
14
Generic Framework
INPUT
Talent PROCESS
• Faculty • Curriculum Design OUTPUT
• Student • Delivery/Pedagogy • Employability
Resources • Assessment • Graduate
Satisfaction
• Physical • Accreditation
• Employers’
• Non-physical • Monitoring
Satisfaction
Governance • Ancillary activities
• Generic Student
(Counseling,
• Autonomy Attributes
Internship/Outreach,
(conducive Exchange program)
institutional climate)
Instrument hierarchy
4/4/2013
15
International
Benchmarking
• International Initiatives
–Compared against Chalmers’
Framework (for Australia), UK
and THES-QS
• ASEAN Comparison
–Countries of Indonesia and
Thailand
4/4/2013 16
International
Benchmarking
• Domains & Dimensions
– Are we missing any significant domains and dimensions?
– Are there domains unique in Indonesian Higher
education?
• Items
– Are the items reflective of the domains/dimensions?
• Weight
– Are the weights assigned in the instrument appropriate?
– Would the weights be different from that in Indonesia?
• Benchmark Figures
– Are the benchmark figures appropriate?
– Would the same benchmark figures apply in
Indonesia/Thailand?
17
International Benchmarking:
ASEAN Comparisons
• Do you think such a rating exercise can be
undertaken for Indonesian/Thai HEIs?
• If yes,
– Would the SETARA Instrument be
appropriate? What modifications do you
think is necessary?
– If the SETARA instrument is used, which
HEIs in Indonesia/Thailand will be in the
TOP 5?
• If No, what would be the objections/
obstacles?
4/4/2013 18
International Benchmarking:
ASEAN Comparisons
• Outcome
– Likelihood of Using the SETARA Instrument:
They are already doing something similar
particularly in Indonesia; but felt that SETARA is
more comprehensive for T&L Quality
– Domains in the Instrument: Consensus being
they are agreeable; Thailand may want to
include Socio-cultural heritage issues
– Weights: They may attribute some variations of
the weights
4/4/2013 19
Chalmers 2008
(Australia)
Outcome indicator Sub indicator
Graduate satisfaction Overall satisfaction
Good teaching satisfaction
Generic skills satisfaction
Employer satisfaction
Stakeholder satisfaction
Learning outcomes Motivation for lifelong learning
Student achievement scores
Student involvement and
engagement
Student participation
Student literacy level
Graduate competencies
4/4/2013 20
Chalmers (cont.)
11-12/1/2010 21
Chalmers (cont.)
Level INPUT OUTPUT OUTCOME PROCESS
Department/ Enrolment rate Retention rate Stakeholder Accommodation for
Program Student/staff Citation/publicatio satisfaction/ student diversity
ratio n rate of engagement Student centred
Teaching research Value of graduates approach
experience/ Quality of Use of current
qualification research research in
s informing
Explicit learning teaching and
outcomes curriculum
content
Specific, continuous
and timely
feedback
Community
engagement
/partnership
11-12/1/2010 22
International benchmarking:
International Initiatives
THES-QS
• THES-QS: currently using
– Student-staff ratio
– International student %
– Faculty quality (research)
– Employers’ survey
– Collecting data on
• Employability
• Students’ satisfaction
• Exchange student rate
• Expenditure
11-12/1/2010 23
Comparison with Others:
Source: HEFCE – Higher Education Funding
Council England (2008)
11-12/1/2010 24
Stakeholders Meeting
HEP
• Primary Purpose
– Update the Stakeholders (HEP) of the results of the
International Benchmarking,
– Changes in the Instrument - inclusion of GSA as one
domain in the OUTPUT, with a Weight of – 10%, at the
expense of Inputs
– Other Changes – reduction in data requirement, only
2009 data, removal of redundant indicators, other
sources of data
– Request for Institutional Data
4/4/2013 25
Data Collection
Procedures
• Four Sources of Data
– KPT – Tracer Study
– JPT – My3S for GSA
– MQA – APA scores
– Employers ‘Survey
– HEIs – Some basic Institution data regarding staff, students, finance
• Issues with Data Collection
– Incomplete – particularly in the OUTPUT, tracer study, employers
survey and My3S
– Timeliness
4/4/2013 26
Verification
• Verification Team:
– Independent committee to verify (a) the procedures, and (b)
data
• Verification of Procedures
– Consensus – approval of the procedures and system used to
rate the HEIs
• Verification of Data
– Some minor changes requested by HEIs
– A number of HEIs changing their data upon request for
supporting documents
4/4/2013 27
Dealing with Missing
Data
• Predictive Approach
– Correlation
– Regression
• Normalization & Equalization
– Against the Total
– Within Domain
4/4/2013 28
Impact of SETARA
4/4/2013 29
Lesson Learned
4/4/2013 30
Way Forward
• SETARA 2011
– Aggregating with Research and Services ?
– Rating philosophy
– Maintenance of Instrument
– Improved Mechanisms
• Greater transparency particularly policy implications
• Enhanced employers survey & Tracer Study
• My3S validity accuracy
• APA mechanisms – reducing audit fatigue
• D’SETARA
– Discipline based – pilot for Healthcare; Engineering; and
Hospitality
• Information based system
4/4/2013 31
THANK YOU
4/4/2013 32