J Autism Dev Disord (2015) 45:1658–1672

DOI 10.1007/s10803-014-2323-3

ORIGINAL PAPER

The Screening Accuracy of the Parent and Teacher-Reported

Social Responsiveness Scale (SRS): Comparison with the 3Di
and ADOS
Jorieke Duvekot • Jan van der Ende •
Frank C. Verhulst • Kirstin Greaves-Lord

Published online: 28 November 2014

Ó Springer Science+Business Media New York 2014

Abstract The screening accuracy of the parent and tea- children with ASD, along with the symptom overlap and
cher-reported Social Responsiveness Scale (SRS) was co-occurrence of ASD with other disorders (Lai et al.
compared with an autism spectrum disorder (ASD) clas- 2014). The diagnostic assessment for ASD has been
sification according to (1) the Developmental, Dimen- advanced by the development of reliable and valid stan-
sional, and Diagnostic Interview (3Di), (2) the Autism dardized diagnostic instruments (Ozonoff et al. 2005; Fil-
Diagnostic Observation Schedule (ADOS), (3) both the 3Di ipek et al. 1999). The current gold standard procedure for
and ADOS, in 186 children referred to six mental health diagnosing ASD includes a standardized interview with
centers. The parent report showed excellent correspon- parents, e.g. the Autism Diagnostic Interview-Revised
dence to an ASD classification according to the 3Di and (ADI-R; Lord et al. 1994) or the Developmental, Dimen-
both the 3Di and ADOS. The teacher report added signif- sional and Diagnostic Interview (3Di; Skuse et al. 2004),
icantly to the screening accuracy over and above the parent and a standardized clinical observation of the child, e.g. the
report when compared with the ADOS classification. Autism Diagnostic Observation Schedule (ADOS; Lord
Findings support the screening utility of the parent-repor- et al. 2012). However, the use of these instruments is time-
ted SRS among clinically referred children and indicate consuming and expensive, and requires trained experts,
that different informants may provide unique information making it necessary to carefully identify children who
relevant for ASD assessment. require in-depth diagnostic assessment using these stan-
dardized diagnostic instruments. To help clinicians make
Keywords Social Responsiveness Scale (SRS)
Autism more informed decisions about which children need in-
spectrum disorder (ASD)
Screening
Multi-informant depth diagnostic assessment for ASD, several screening
questionnaires for ASD have been developed that are rel-
atively quick and easy to administer.
Introduction One ASD-specific screening questionnaire that is widely
used in clinical practice as well as in research is the Social
Diagnosing autism spectrum disorder (ASD) is a complex Responsiveness Scale (SRS; Constantino and Gruber
process due to the variability in the clinical presentation of 2012). The SRS consists of 65 items that can be scored in
15 min by a parent or teacher. One of the advantages of the
SRS is that it has been designed to assess social impairment
J. Duvekot
J. van der Ende
F. C. Verhulst
associated with ASD as rated by multiple informants on a
K. Greaves-Lord (&)
Likert response scale. Therefore, the SRS is considered
Department of Child and Adolescent Psychiatry/Psychology,
Erasmus MC-Sophia Children’s Hospital, Wytemaweg 8, suitable to capture autistic characteristics in varying
3015 CN Rotterdam, The Netherlands degrees and to provide a severity index of autistic social
e-mail: k.greaves-lord@erasmusmc.nl impairment (Constantino and Gruber 2012). Although the
SRS has also been validated for use in the general popu-
J. Duvekot
K. Greaves-Lord
Yulius Academy, Yulius Mental Health, P.O. Box 753, lation, it is more generally applied as a screening instru-
3300 AT Dordrecht, The Netherlands ment in high-risk populations, i.e. clinically referred

123
J Autism Dev Disord (2015) 45:1658–1672 1659

individuals. Several studies have supported the ability of demonstrated acceptable screening accuracy, the parent
the parent-reported SRS to discriminate between children report was more accurate in identifying children with ASD
with ASD and those with other psychiatric disorders (e.g. than the teacher report (Kamio et al. 2013; Schanding et al.
Kamio et al. 2013; Bölte et al. 2011; Charman et al. 2007; 2012). Although these studies provide some indication that
Constantino and Gruber 2005). combining the parent and teacher-reported SRS may
Although using multiple informants is considered improve the identification of children with ASD, the gen-
important in the assessment of ASD (Kim and Lord 2012; eralizability of these results to clinical practice is limited
Ozonoff et al. 2005) and in child psychiatry in general by sample characteristics: a research sample of children
(Kazdin 2005), little is known about the contribution of who had already been diagnosed with ASD (Constantino
information obtained from teachers over and above the et al. 2007; Schanding et al. 2012; Kamio et al. 2013), a
information obtained from parents in the assessment of comparison group of typically developing children (Fo-
ASD. It is well established that different informants often mbonne et al. 2012), and a small sample size (Aldridge
do not agree in their ratings of child behavior, since parents et al. 2012). Therefore, more research is needed to estimate
and teachers see the child in different contexts and have the utility of the teacher-reported SRS in addition to the
different perspectives (van der Ende et al. 2012; Achen- parent-reported SRS in children who are consecutively
bach et al. 1987; De Los Reyes and Kazdin 2005). referred for various mental health problems, which more
Therefore, parents and teachers are considered to provide closely represents the population in which ASD screening
unique and complementary information. Parents are gen- instruments are commonly used.
erally considered important information sources of their The present study aimed to extend previous findings by
child’s behavior, as they experience how their child investigating the screening accuracy of the parent and
behaves in various circumstances and develops over time teacher-reported SRS in children aged 4–10 years who had
(e.g. Richters 1992). For the assessment of some childhood been consecutively referred for a variety of mental health
disorders, such as ADHD, information from teachers in problems (e.g. behavioral, emotional or developmental
addition to parents is considered a necessary component, problems) to one of six mental health care centers,
because the symptoms, such as inattentiveness and hyper- including secondary and tertiary mental health care ser-
activity, should be present in multiple settings and may be vices. We examined whether the SRS is able to identify
particularly visible and disrupting in the school setting children who have a high likelihood (‘high risk’) of
(Pelham et al. 2005). For the assessment of ASD, teachers receiving an ASD classification according to two widely
may also be a valuable source of information about the used standardized diagnostic instruments: the 3Di and the
social functioning of the child as they regularly observe ADOS. Since evaluation using these instruments is valu-
how the child interacts with other children in the school able, it would be useful if the SRS can assist in targeting a
setting. Moreover, teachers have the expertise and oppor- high-risk group who need further diagnostic evaluation and
tunity to compare the behavior of the child with that of preventing unnecessary diagnostic evaluations for children
many other children, which may allow them to better dis- with a low risk of being classified as ASD according to
tinguish between typical and atypical behavior (Constan- these instruments. Although we acknowledge that a clinical
tino et al. 2007). diagnosis of ASD also includes a clinical judgment, we did
Despite the potential additional value of information not take this into account given its limited objectivity and
obtained from teachers in the assessment of ASD, only few larger variability across centers and clinicians (Lord et al.
studies have examined the screening accuracy of the tea- 2012). The first aim of this study was to examine the
cher-reported SRS. Constantino et al. (2007) found that screening accuracy of the parent report alone in predicting
when both a parent and a teacher rated the child as having a ASD classifications according to the 3Di and ADOS.
SRS T-score of 60 or higher, the likelihood that the child Consistent with previous studies, we expected good cor-
had an ASD diagnosis was very high (96.8 %). Another respondence of the parent-reported SRS scores to the ASD
study that contrasted children with ASD and typically classifications according to one or both of these diagnostic
developing children found that combined use of the parent instruments. Our second aim was to examine the additional
and teacher report improved the screening accuracy of the contribution of the teacher-reported SRS over and above
SRS, but the improvement was very small compared with the parent report in the identification of children who are
the use of the parent report alone (Fombonne et al. 2012). classified as possibly having ASD according to the 3Di, the
In a small sample (n = 48) of children who were referred ADOS, and both the 3Di and ADOS. As suggested in
to an ASD-specific clinic, the teacher-reported SRS cor- previous studies, we hypothesized that using both infor-
responded better to an ASD diagnosis than did the parent mants would improve the utility of the SRS to identify
report (Aldridge et al. 2012). In contrast, other studies children with possible ASD according to these diagnostic
reported that, although the teacher-reported SRS instruments (e.g. Constantino et al. 2007).

123
1660 J Autism Dev Disord (2015) 45:1658–1672

Methods centers covered rural as well as urban areas. Of the study

population, 87 % of the children were referred to second-
Study Design ary mental health care services, 8 % to tertiary ASD-spe-
cialized mental health care services, and 5 % to other
The present study was part of the ‘‘Social Spectrum Study’’, a tertiary mental health care services. The children were
prospective multicenter study investigating interrelation- referred for a variety of mental health problems, including
ships between ASD and behavioral characteristics and ASD, internalizing/externalizing disorders, and more gen-
family factors in clinically referred children at risk for ASD eral developmental/learning difficulties. Thus, the target
and their family members. The study was approved by the population was representative of children who are referred
local medical ethics committee and the participating mental to mental health care, which we consider a good reflection
health care centers (MEC-2011-078). of the population in which the SRS is used to help to
The target population of the present study was children determine which children are in need for ASD-specific
aged 4–10 years old who were consecutively referred to diagnostic assessment.
one of six participating mental health care centers in the We used a two-phase sampling design (e.g., Dunn et al.
south-west of the Netherlands during an interval of 1999), in which we oversampled the children who screened
6 months at each site, within a period from April 2011 to positive on the parent-reported SRS to participate in an
July 2012 (see Fig. 1). The participating mental health care ASD-specific diagnostic assessment. First, as part of the

Fig. 1 Participant flow chart

123
J Autism Dev Disord (2015) 45:1658–1672 1661

routine procedure for clinical evaluation, the SRS was 393 = 34 %) than the proportion of screen-negative chil-
completed by parents and teachers before the first intake dren (52/203: 26 %; v2(1) = 4.49, p = .03). The screen-
appointment at the mental health care center. Second, we positive children who were included were younger [7.0 vs
selected all children with a positive screen based on the 7.5, t(391) = 2.2, p = .03], but did not significantly differ
parent-reported SRS (cut-off: total raw score C75) and a with respect to gender [v2(1) = 1.88, p = .17], parent-
random selection of children with a negative screen result reported SRS scores [t(391) = -1.67, p = .10], and tea-
(total raw score \75 on the parent-reported SRS) and cher-reported SRS scores [t(344) = -1.04, p = .30].
invited these children and their families to participate in There were no significant differences between the screen-
further assessments. The assessments included a standard- negative children who were included and those who were
ized parent interview (3Di), a standardized observation of not included with regard to gender [v2(1) = 1.38, p = .24],
the child (ADOS), and a standardized test to assess intel- age [t(201) = -1.66, p = .10], parent-reported SRS scores
ligence quotient (IQ). During this phase, written consent [t(201) = -.22, p = .83], and teacher-reported SRS scores
was obtained for all of these assessments. [t(179) = -.83, p = .41].

Sample Measures

Figure 1 shows the flow of the participants through the dif- Screening
ferent phases of the study. The SRS was sent out to 4,344
children in total. The response rate for the parent-reported The Social Responsiveness Scale (SRS) is a questionnaire
SRS ranged from 40 to 81 % across centers, with an overall that assesses the severity of social impairment related to
response rate of 53 % (n = 2,322). Only children aged ASD (Constantino and Gruber 2012, 2005). The child
4–10 years old were eligible for the present study. Of these version for children aged 4–18 years old contains 65 items
children, we received 1,182 completed parent reports (mean that are scored on 4-point scale from 0 (not true) to 3
age 7.2, SD = 1.9; 68 % male). Of 1,018 (86 %) of the (almost always true) by parents or teachers who have
children for whom a parent completed the SRS, a teacher also experience with the child in everyday social settings. The
completed the SRS. Of the 1,182 children with a completed total sum score of 65 items, which can range from 0 to 195,
parent-reported SRS, 393 (33 %) screened positive (total is used for screening purposes (Constantino and Gruber
raw score C75 on the parent report) and 789 (67 %) screened 2005). A higher total score reflects more social impairment.
negative (total raw score \75 on parent report). The mean The total score can be converted to a T-score, based on
age of the children who screened positive did not differ norms for gender and rater type, but to increase compara-
significantly from that of the children with a negative screen bility between research studies it is recommended to use
[t(730.64) = -1.06, p = .29], nor did the gender proportion the raw total score for research (Constantino and Gruber
[v2(1) = 2.55, p = .11]. 2005). Moreover, T-scores for the Dutch version of the
Based on the scores on the parent-reported SRS, we teacher report of the SRS do not yet exist. Therefore, in the
selected 596 children aged 4–10 years for further assess- present study we used total raw scores for both the parent-
ments, including the 3Di, ADOS, and IQ assessment: all reported and the teacher-reported SRS. Since we used total
393 children who screened positive and a random selection raw scores and the questions of the child version are the
of 203 children who screened negative (26 %). The same for both the original SRS (Constantino and Gruber
assessments took place at an average of 10 months 2005) and the SRS-2 (Constantino and Gruber 2012), the
(SD = 4) after the SRS was completed. The final sample findings of our study are applicable to both the original
contained 186 children (134 screen-positives and 52 SRS and the SRS-2. In the present study, the total raw cut-
screen-negatives on the parent-reported SRS) for whom a off score of 75 on the parent-reported SRS was chosen to
teacher-reported SRS, the 3Di, and the ADOS were screen for ASD, which was found to differentiate between
available. The final sample (n = 186) was weighted in children with ASD and children with other psychiatric
order to represent the total eligible sample (n = 1,182; see disorders with a sensitivity of .85 and a specificity of .75
the ‘‘Statistical Analysis’’ section for a description of the (Constantino and Gruber 2005).
weighing procedure). Consistent with validation studies in other countries, the
To examine possible selective attrition between the Dutch version of the parent-reported SRS demonstrated
screening phase and follow-up assessment phase, we high internal consistency (Cronbach’s alphas ranged from
compared the selected children who were included in the .92 to .95), good convergent validity (r = .63 with the
final sample (n = 186) with the selected children who were ADI-R) and was able to differentiate between children with
not (n = 410). A larger proportion of the screen-positive ASD and children from the general population (Roeyers
children was included in the final sample (134/ et al. 2011). Previous studies have shown moderate to good

123
1662 J Autism Dev Disord (2015) 45:1658–1672

agreement between the parent and the teacher-reported collapsed into ASD versus non-ASD. Preliminary findings
SRS scores (r = .24–.82; Constantino et al. 2003; from our data demonstrated a fairly good sensitivity (.75)
Schanding et al. 2012; Constantino et al. 2007; Kamio et al. and specificity (.74) for the Dutch version of the 3Di with
2013; Fombonne et al. 2012; Constantino et al. 2000; respect to a DSM-IV-TR clinical ASD diagnosis.
Kanne et al. 2009; Reszka et al. 2014). In the current study, The Autism Diagnostic Observation Schedule (ADOS;
the correlations between the parent and teacher-reported Lord et al. 1999, 2012) is a semi-structured and standard-
SRS scores were r = .28 (p \ .01, n = 186) in the final ized observation of the child’s social interaction, play/
sample and r = .29 (p \ .001, n = 1,018) in the total imaginative use of materials, and restricted and repetitive
screened sample. The Cronbach’s alphas in the total behaviors that is used as part of the diagnostic assessment
screened sample for the total scale were .95 for the parent of ASD. The ADOS has different modules that can be used
report as well as for the teacher report. for individuals with different levels of expressive language.
In the present study, we used Module 1 (n = 2), Module 2
Assessment (n = 22) and Module 3 (n = 162). Children were classified
as having ASD (Autism ? ASD combined) or not
The Developmental, Dimensional and Diagnostic Inter- according to the ADOS using the revised algorithms as
view (3Di; Skuse et al. 2004) is a standardized, comput- described in the second edition of the ADOS manual
erized parent interview during which parents are asked (ADOS-2; de Bildt et al. 2013; Lord et al. 2012). This
about their child’s current and past social communication revised algorithm has been found to increase comparability
and interaction, as well as about restricted, repetitive between modules and to improve diagnostic validity
behaviors or interests that are characteristic of children (Gotham et al. 2007, 2008). The validity of the revised
with ASD. The 3Di has been designed according to the algorithms has been confirmed in Dutch samples (de Bildt
current conceptualization of ASD as a dimensional disor- et al. 2009; Oosterling et al. 2010).
der that is often present in individuals with normal IQ The 3Di and ADOS were always administered by two
levels. The 3Di covers the same ASD symptoms as the different clinicians or researchers who had met research
ADI-R and reflects the classification algorithm of the ADI- requirements of standardized administration and scoring
R, but the structure of the interview is different. The ADI-R reliability. All researchers were blind to the SRS scores
requires the interviewer to integrate information from when they performed the 3Di (n = 170, 91 %) and the
several questions into one summary score for a particular ADOS (n = 146, 79 %). In a minority of cases, a clinician
characteristic, e.g. the range of facial expressions to com- had performed the 3Di (n = 16, 9 %) or the ADOS
municate (Lord et al. 1994). In contrast, the 3Di uses short (n = 40, 21 %) as part of the routine clinical evaluation. In
focused questions (e.g. separate questions for looking sad, these cases, we cannot guarantee that the clinician per-
guilty, embarrassed) that are each individually scored and forming the diagnostic assessment was blind to the SRS
combined using a computer algorithm (Skuse et al. 2004). scores. However, a priori analyses showed that the agree-
In this way, the interview structure of the 3Di has been ment between the SRS and the diagnostic assessments was
designed to reduce the influence of the subjectivity of the not higher in the cases where a clinician performed the
interviewer, which intends to enhance the reliability of its assessment than in the cases where it was performed by a
scoring and eases its administration (Skuse et al. 2004). researcher (details available upon request). Therefore, we
While the original complete interview of the 3Di contains a regard it unlikely that a possible lack of blinding to the SRS
122-item ASD module, more recently a shorter 53-item scores in this small subsample biased the results.
ASD module became available that showed good agree- We compared the SRS with three ASD classifications
ment with the original ASD module (Santosh et al. 2009). according to these standardized diagnostic instruments,
Both versions demonstrated good agreement with the ADI- used separately and combined: (1) an ASD classification
R (Skuse et al. 2004; Santosh et al. 2009). The short ver- according to the 3Di, (2) an ASD classification according
sion has also demonstrated good ability to differentiate to the ADOS, and (3) an ASD classification according to
between children diagnosed with ASD and typically both the 3Di and ADOS. Thus, the first two classifications
developing children (Chuthapisith et al. 2012). The present reflect whether the child was classified as having ASD
study used the short ASD module of the 3Di. A computer when considering a single instrument, whereas the third—
algorithm produces scores on the scales social reciprocity, more stringent—classification reflects whether a child
verbal and non-verbal communication, and restricted/ meets the criteria for an ASD classification according to
repetitive behaviors, which we summed up to a total score both instruments. As shown in Table 1, more children were
in the current study. In addition, the algorithm produces a classified as having ASD according to the ADOS (35 %)
DSM-IV-TR classification of autistic disorder, Asperger’s than according to the 3Di (23 %). Only 11 % were clas-
syndrome, atypical autism versus non-ASD, which we sified as having ASD according to both instruments. The

123
J Autism Dev Disord (2015) 45:1658–1672 1663

score. The 3Di total score is the sum of the scores on the scales reciprocal social interaction, communication, and repetitive and stereotyped behavior. The total weighted sample is split into
Table presents means and standard deviations in parentheses unless otherwise noted. The total raw SRS scores are presented. The ADOS severity reflects the ADOS-2 total calibrated severity
overall percent agreement between the 3Di and ADOS

\.001
\.001
\.001
\.001
.32
\.01
\.01
.54
Classification according to both the 3Di and ADOS classification was 63 %, with a kappa of .20 (p = .005),
p indicating only a slight agreement (Cicchetti 2001). In
comparison, the ADI-R and ADOS have also been reported

2.8
2.8

5.4
13.3

14.0
11.3
.6

.6
to show poor to moderate agreement (e.g. de Bildt et al.
t/v2

2004; Ventola et al. 2006; Le Couteur et al. 2008; Kim and

Lord 2012).
1,053a (89 %)
741a (70 %)

97.8 (15.2)
57.2 (27.8)
61.0 (26.5)
15.1 (10.0)
Intelligence (IQ) was assessed using various tests: in
7.5 (1.9)
8.4 (2.0)

2.9 (2.2)
Non-ASD

50 % of the children with the Wechsler Intelligence Scale

for Children, third Dutch edition (WISC-III-NL; Kort et al.
2005), in 28 % with the Wechsler Abbreviated Scale of
Intelligence (WASI; Axelrod 2002), in 15 % with the
Wechsler Preschool and Primary Scale of Intelligence,
129a (11 %)
101 (78 %)

SRS Social Responsiveness Scale, 3Di Developmental, Dimensional and Diagnostic Interview, ADOS Autism Diagnostic Observation Schedule
95.7 (19.5)
105.1 (19.1)
87.9 (28.4)
6.5 (1.9)
7.5 (1.9)

34.9 (7.4)
6.9 (1.7)
third Dutch edition (WPPSI-III-NL; Hendriksen and Hurks
2009), in 5 % with the Snijders-Oomen Nonverbal Intel-
a
ASD

ligence test (SON-R; Tellegen et al. 1998), and in 2 %

using other intelligence tests. Total IQ score was available
for 174 children (94 % of the final sample).
\.001
\.001
.13
\.01
\.01
.04
.05
\.01
p

Statistical Analyses
All frequencies shown in the table are weighted; please see the ‘‘Statistical Analysis’’ section for more information
Classification according to the ADOS

4.1
2.6
2.7
2.1
2.0
3.4
4.1
17.2
2
t/v

For descriptive purposes, we tested whether children clas-

sified as ‘ASD’ and ‘non-ASD’ differed in several demo-
766a (65 %)
508 (66 %)

99.9 (14.8)
58.5 (27.4)
58.1 (26.0)
14.6 (10.7)
7.7 (1.9)
8.7 (2.0)

graphic and ASD characteristics using t tests for continuous

Non-ASD

1.7 (.9)

variables and v2 tests for categorical variables. In addition,

a

we tested whether parents and teachers rated boys and girls

ASD and non-ASD groups according to: (1) the 3Di; (2) the ADOS; (3) both the 3Di and ADOS

differently using t tests.

416a (35 %)
334 (80 %)

In order to investigate our first aim, the screening

93.0 (17.6)
69.7 (29.1)
74.7 (29.3)
22.3 (11.4)
6.8 (1.8)
7.7 (1.9)

6.4 (1.8)

accuracy of the parent-reported SRS, we calculated sensi-

tivity, specificity, positive predictive values (PPV), nega-
ASD

tive predictive values (NPV) for the parent-reported SRS

using the total raw cut-off score of 75 in relation to three
\.001
\.001
\.001
.13
.33
.47
.81

\.01

ASD classification groups: (1) ASD classification accord-

p

ing to the 3Di, (2) ASD classification according to the

ADOS, and (3) ASD classification according to both of
.7
.2
1.9
1.0

3.8

3.3
10.5

15.3
2
t/v

these instruments.
Classification according to the 3Di

In order to investigate our second aim, whether the

906a (77 %)
622 (69 %)

teacher-reported SRS added to the prediction of ASD

97.8 (15.4)
52.2 (27.3)
60.0 (27.1)
Table 1 Descriptive statistics of the weighted sample

7.5 (1.9)
8.4 (2.0)

12.8 (7.1)
3.0 (2.5)
Non-ASD

classifications over and above the parent report, firstly we

a

performed logistic regression analyses predicting the three

ASD classifications with only the parent-reported total SRS
score in the model in a first step and then the teacher-
276a (23 %)
220 (80 %)

97.0 (17.6)
95.8 (19.2)
77.0 (28.5)

reported total SRS score added to the model in a second

7.2 (2.0)
8.2 (2.0)

32.1 (7.0)
4.5 (2.7)

step. Subsequently, we used the predicted probabilities that

ASD

were the result from the logistic regression analyses as

input in a receiver operating characteristics (ROC) ana-
Age assessment (years)

lysis. The area under the curve (AUC) of the ROC analysis
Age screening (years)
Gender: n (%) male

reflects the probability that a randomly selected individual

Teacher report SRS
Parent report SRS

with the disorder has a higher score than a randomly

ADOS severity
3Di total score

selected individual without the disorder. It provides an

indication of the overall screening accuracy irrespective of
Total IQ
n (%)

a specific cut-off point and is therefore a good measure to

compare different tests (Hunink and Glasziou 2001). We
a

123
1664 J Autism Dev Disord (2015) 45:1658–1672

used the method of Hanley and McNeil (1983) to test t(185) = 3.45, p = .001] on the SRS, whereas no signifi-
whether the AUCs of the combined parent and teacher cant differences were found in the parent-reported SRS
report were higher than those of the parent report alone, scores between boys (M = 63.79) and girls [M = 59.00,
while taking into account that the AUCs were correlated t(185) = .80, p = .43]. Similar gender differences were
because they were based on the same sample. found in the non-ASD and ASD-groups (not presented).
The estimation of the weighted AUCs of the ROC
curves and accompanying standard errors were calculated Aim 1: Screening Accuracy of the Parent-Reported
using bootstrap analyses in R (R Core Team 2014). All SRS
other analyses were conducted using the complex samples
module in SPSS 20 (IBM Corporation 2011). This module Table 2 shows the sensitivity, specificity, PPV, and NPV of
uses inverse probability weighting in order to compute the parent-reported SRS for the recommended cut-off total
correct population estimates and standard errors for com- score of 75 in relation to three ASD classifications: (1) an
plex designs that include unequal sampling probabilities ASD classification according to the 3Di, (2) an ASD
and differential response rates. Not correcting for the dif- classification according to the ADOS, and (3) an ASD
ferential sampling probabilities and response rates could classification according to both of these instruments.
yield biased screening accuracy estimates because of a In relation to the ASD classification according to the
verification bias (Hunink et al. 1990; Begg and Greenes 3Di, the cut-off of 75 on the parent-reported SRS resulted
1983). In order to correct for a verification bias in the in good sensitivity (.85) and specificity (.83). The NPV was
present study, we weighted each case with the inverse of very high (.95), indicating that the probability was very low
the probability that the case was included in the final that a child scoring below the cut-off of 75 was classified as
dataset (inverse probability weighting [IPW]; Seaman and having ASD according to the 3Di. The PPV, the probability
White 2013). First, we calculated the probability that a that a child scoring at or above 75 was classified as having
child from the total screened sample was selected (p1): for ASD, was .60.
the children with a positive screen this probability was In relation to the ASD classification according to the
100 % and for the children with a negative screen the ADOS, the specificity was moderate (.73) and sensitivity
probability was on average 26 %. Then, we conducted was poor (.45). Thus, the parent-reported SRS did not
logistic regression analysis to predict the probability that a capture a substantial proportion (55 %) of the children who
selected child was included in the final dataset (n = 186) were classified as having ASD according to the ADOS and
using the parent-reported SRS total score as predictor (age there were also a considerable proportion of children
and gender were not significant predictors). The predicted (27 %) who did not meet the ADOS cut-off for ASD.
probability of this analysis (p2) was multiplied with the In relation to the more stringent ASD classification—an
selection probability (p1) to calculate the final inclusion ASD classification according to both the 3Di and ADOS—
probability. Finally, each case in the final sample was the cut-off of 75 on the parent-reported SRS identified all
weighted by the inverse of this inclusion probability [1/ children with an ASD classification (sensitivity 100 %),
(p1*p2)], so the estimates would reflect those of the total but also 25 % of the children who were not classified as
screened sample (n = 1,182). having ASD according to both instruments (false posi-
An alpha level of .05 was used for all statistical analy- tives). Since there were no false negatives using this
ses. The screening accuracy indices (sensitivity, specificity, classification method, i.e. no children classified as having
PPV, NPV, and AUC) were interpreted according to the ASD according to both the 3Di and ADOS that scored
following guidelines: 90–100 % = excellent; 80–89 % = below the cut-off of 75 on the parent-reported SRS, we
good; 70–79 % = fair; and \70 % = poor (Cicchetti et al. were not able to calculate confidence intervals for the
1995). sensitivity and the NPV. Because of the low prevalence of
this stringent ASD classification (11 %), the PPV was
relatively low (.33).
Results
Aim 2: Contribution of the Teacher-Reported SRS
Sample Characteristics
The results of the logistic regressions that tested whether
Table 1 presents the demographic and ASD characteristics the teacher-reported SRS scores significantly added to the
of the weighted sample split into ‘ASD’ and ‘non-ASD’ prediction of an ASD classification over and above the
groups according to (1) the 3Di, (2) the ADOS, and (3) parent-reported SRS scores, are shown in Table 3. The
both the 3Di and ADOS. In addition, teachers scored boys teacher report did not significantly add to prediction of an
(M = 68.80) significantly higher than girls [M = 51.95, ASD classification according to the 3Di (Nagelkerke’s

123
J Autism Dev Disord (2015) 45:1658–1672 1665

pseudo DR2 = 1 %, Dv2(1) = 1.78, p = .16). However, 95 % CI .86–.97). Although the combined use of parent
the teacher report showed a significant independent con- and teacher report slightly increased the discriminative
tribution over and above the parent report to the prediction ability (1 %) with regard to an ASD classification
of an ASD classification according to the ADOS (Nage- according to both the 3Di and ADOS (AUC com-
lkerke’s pseudo DR2 = 8 %, Dv2(1) = 6.83, p = .01). In bined = .93, 95 % CI .89–.97), this increase was not sig-
contrast to the teacher report, the parent report did not nificant as compared with the parent report alone
significantly predict the ADOS classification. In addition, (p = .30).
the teacher report significantly added to the prediction of an Because of the gender differences in teacher-reported
ASD classification according to both the 3Di and ADOS SRS scores, we also explored whether our results regarding
over and above the parent (Nagelkerke’s pseudo the added value of the teacher report would be similar
DR2 = 6 %, Dv2(1) = 3.70, p \ .001). In this model, both when stratifying for gender. Results regarding the added
parent and teacher report had a significant independent value of the teacher report appeared similarly for boys and
contribution. girls. However, the ability of the SRS to discriminate
To determine the overall screening accuracy of the between children with and without an ASD classification
parent-reported SRS alone and in combination with the according to the ADOS was especially poor in girls (AUC
teacher report, we performed ROC analyses using the parent = .50; AUC combined = .61), compared to boys
predicted probabilities of the logistic regression analyses. (AUC parent = .60; AUC combined = .67). These results
Figure 2a–c show the results of the ROC analyses com- must be interpreted with caution as our sample contained
pared with an ASD classification according to (a) the 3Di, relatively few girls (unweighted n = 57).
(b) the ADOS, and (c) both the 3Di and ADOS. As illus-
trated in these figures, the parent-reported SRS was very
good in discriminating between children who were classi- Discussion
fied as having an ASD and those who were not according to
the 3Di (AUC parent = .91, 95 % CI .85–.96). Combining The present multicenter study investigated the screening
parent and teacher report did not improve the screening accuracy of the parent-reported SRS, alone as well as in
accuracy of the SRS compared with an ASD classification combination with the teacher-reported SRS, in comparison
according to the 3Di (AUC combined = .91, 95 % CI .86– with an ASD classification according to commonly used
.95). The parent report did not discriminate well between ASD diagnostic instruments: the 3Di (parent interview) and
children who were classified as having ASD according to the ADOS (clinical observation). The parent-reported SRS
the ADOS and those who were not (AUC parent = .59, showed an excellent screening accuracy with regard to an
95 % CI .47–.71). Combining the parent and teacher- ASD classification according to the 3Di and according to
reported SRS significantly increased the discriminative both the 3Di and ADOS. The ability of the parent-reported
ability with regard to an ASD classification according to SRS to identify children who were classified as having
the ADOS compared with the parent report alone (AUC ASD according to the ADOS was poor. Combining the
combined = .68, 95 % CI .60–.76, p = .049), although the parent report with the teacher-reported SRS significantly
correspondence to a classification according to the ADOS improved the ability to discriminate between children who
was still low. The parent-reported SRS showed an excellent met cut-off scores indicating possible ASD according to
screening accuracy with regard to an ASD classification the ADOS and those who did not.
according to both the 3Di and ADOS (AUC parent = .92, The estimates of sensitivity and specificity for the par-
ent-reported SRS with regard to an ASD classification
Table 2 Screening accuracy indices for the SRS parent report (total
according to the 3Di and both the 3Di and ADOS are
raw cut-off score of 75) similar to screening accuracy estimates of the original
validation study conducted by Constantino and Gruber
ASD Sensitivity Specificity PPV NPV
classification (95 % CI) (95 % CI) (95 % CI) (95 % CI) (2005) and those of other validation studies (Bölte et al.
according to 2011; Charman et al. 2007). In line with these previous
studies, we found a good to excellent sensitivity and a fair
3Di .85 (.63–.95) .83 (.76–.87) .60 (.51–.67) .95 (.85–.99)
to good specificity for the total raw cut-off score of 75 on
ADOS .45 (.31–.59) .73 (.65–.80) .48 (.39–.56) .71 (.57–.82)
the parent report. Although this cut-off score on the parent-
Both 3Di 1.00 .75 (.68–.81) .33 (.26–.41) 1.00
and ADOS reported SRS may not identify all children who receive an
ASD classification according to a clinical observation (i.e.
SRS Social Responsiveness Scale, 3Di Developmental, Dimensional
and Diagnostic Interview, ADOS Autism Diagnostic Observation ADOS), it identifies most children who are classified as
Schedule, PPV positive predictive value, NPV negative predictive having ASD according to a parent interview (i.e. 3Di) and
value, CI confidence interval those who meet stringent ASD classification criteria (i.e.

123
1666 J Autism Dev Disord (2015) 45:1658–1672

Table 3 Logistic regression of parent and teacher-reported SRS the population screened or the purpose of screening
scores predicting ASD classifications (Constantino and Gruber 2012). In clinical practice, the
OR 95 % CI R2 Model DR2 Dv2 SRS is generally used to indicate children for further
v2 diagnostic evaluation, requiring high sensitivity. In our
study, the total raw cut-off score of 75 served this purpose
ASD classification according to the 3Di
well. However, the aim to identify as much ASD cases as
Step 1 .54 47.16** - -
possible will go at the expense of the specificity, i.e. also a
Parent-reported 8.0** 4.4–14.5
SRS considerable proportion of non-ASD cases will be identi-
Step 2 .55 48.94** .01 1.78 fied and thus selected for further diagnostic assessment
Parent-reported 7.3** 4.1–13.1
with the accompanying costs and burden on the family.
SRS Choosing a higher cut-off may be preferred when it is
Teacher- 1.3 0.9–2.0 important to further minimize the number of false posi-
reported SRS tives, i.e. children with non-ASD who are incorrectly
ASD classification according to the ADOS identified at risk for ASD. For instance, when selecting
Step 1 .05 3.68 - - cases for biological studies, it is required that all cases meet
Parent-reported 1.5 1.0–2.2 stringent ASD criteria and the costs of incorrectly includ-
SRS ing non-ASD cases are relatively higher than those of
Step 2 .13 10.51* .08 6.83* missing some children with ASD. Thus, clinicians and
Parent-reported 1.3 0.8–1.9 researchers should be aware of the trade-off between
SRS maximizing the identification of children at risk (i.e. opti-
Teacher- 1.8* 1.1–2.9 mal sensitivity) versus minimizing the number of children
reported SRS
targeted to receive further assessments (i.e. optimal speci-
ASD classification according to both the 3Di and ADOS
ficity) when selecting the cut-off that best serves their
Step 1 .47 78.27** - -
particular purpose or the population screened (Charman
Parent-reported 7.5** 4.8–11.7
SRS
and Gotham 2013).
We found support for the contribution of the teacher-
Step 2 .53 81.97** .06 3.70**
reported SRS when screening for ASD in relation to an
Parent-reported 7.1** 4.4–11.3
SRS ASD classification according to the ADOS. When com-
Teacher- 2.2** 1.5–3.1 pared with an ASD classification according to the 3Di or
reported SRS both the 3Di and ADOS, the parent report alone already
showed excellent screening accuracy, leaving little room
Parent and teacher-reported SRS scores were entered as continuous
predictor variables in different steps: (1) SRS parent report, (2) SRS for improvement. Thus, the screening accuracy of the
teacher report. Odds ratios are expressed in the change of odds per parent and teacher-reported SRS differed depending on the
standard deviation change: SD = 29.11 for the parent-reported SRS ASD classification method used: the 3Di, the ADOS, or
and SD = 27.07 for the teacher-reported SRS. The pseudo Nage-
both the 3Di and ADOS. One factor that may be important
lkerke’s R2 is reported. OR odds ratio, CI confidence interval, SRS
Social Responsiveness Scale, 3Di Developmental, Dimensional and in this respect is the source of information. It is perhaps not
Diagnostic Interview, ADOS Autism Diagnostic Observation surprising that the parent-reported SRS showed high
Schedule agreement with the 3Di classification, as for both measures
* p \ .05, ** p \ .001 the parent is the main source of information (i.e. shared
method variance). However, an important difference
between both measures is that during the 3Di the infor-
classification according to both the 3Di and ADOS). Since mation from the parent is obtained, interpreted, and scored
in our sample more children had a non-ASD classification by a trained expert, while the parent-reported SRS purely
than an ASD classification (prevalence of ASD classifica- reflects the parent’s perspective. In addition, when an ASD
tions varied from 11 to 35 % dependent on the diagnostic classification according to both a parent interview and
instruments used), the NPVs were higher than the PPVs. clinical observation was used as comparison, the parent-
This indicates that, in diverse clinically referred popula- reported SRS also showed a high screening accuracy. Thus,
tions where the overall prevalence of ASD is relatively despite the shared method variance, these results indicate
low, the parent-reported SRS is especially effective in that the parent-reported SRS, which is relatively short and
correctly identifying children who do not need further easy to administer, is able to differentiate between children
ASD-specific diagnostic assessment. who meet cut-off scores indicating possible ASD according
There is no single cut-off for the SRS that can be used in to a more elaborate parent interview and to both a parent
all circumstances; the most optimal cut-off may vary with interview and clinical observation and those who do not.

123
J Autism Dev Disord (2015) 45:1658–1672 1667

Fig. 2 a ROC curve of the

parent-reported SRS alone and
the combined use of the parent
and teacher-reported SRS
compared with an ASD
classification according to the
3Di. b ROC curve of the parent-
reported SRS alone and the
combined use of the parent and
teacher-reported SRS compared
with an ASD classification
according to the ADOS. c ROC
curve of the parent-reported
SRS alone and the combined
use of the parent and teacher-
reported SRS compared with an
ASD classification according to
both the 3Di and ADOS

The finding that the teacher-reported SRS improved the clinicians and teachers both observe regularly more chil-
prediction of an ASD classification according to the ADOS dren than parents do, and thus are expected to have more
over and above the parent report alone is consistent with knowledge of how the child behaves in comparison with
previous studies that have shown that teacher reports cor- peers (Ferdinand et al. 2003).
responded better to the ADOS than parent reports to the Even when combining parent and teacher-reported SRS,
ADOS (Schanding et al. 2012; Reszka et al. 2014; de Bildt the screening accuracy with regard to the ADOS classifi-
et al. 2003). A possible explanation is that children with cation was not that high. This finding is consistent with
ASD behave more similarly in school and research or previous studies showing poor agreement between
clinical contexts than at home (Schanding et al. 2012; screening questionnaires, particularly parent-reported, and
Reszka et al. 2014; de Bildt et al. 2003). This may be the ADOS (de Bildt et al. 2009; Bishop and Baird 2001;
attributed to the fact that teachers and clinicians observe Sikora et al. 2008). The lack of correspondence of parent
the child in relatively structured settings, whereas parents and teacher ratings with clinical observations may also be
see the child across a variety of unstructured settings explained by contextual factors and different perspectives.
(Szatmari et al. 1994; Koning and Magill-Evans 2001). The Clinicians have been trained extensively to recognize
possible influence of the environmental context on the autistic behaviors and have considerable knowledge on the
expression of behavior in children with ASD was also typical as well as atypical development of children,
indicated in the study by Kanne et al. (2009). However, whereas parents and teachers may have more opportunities
even when parents and teachers rated behavior problems of to observe all kinds of behaviors in everyday life that might
children with ASD in the same setting, large discrepancies not always be shown during relatively short one-to-one test
were found between their ratings for individual children situations. Thus, all these different perspectives—parents,
(Reed and Osborne 2013). Besides the role of environ- teachers, and clinicians—are needed to form a more
mental context, the better agreement between teachers and complete understanding of the child’s autistic symptoms.
clinicians than between parents and clinicians could also We found that teachers rated girls lower than boys,
reflect the perspective of the raters, i.e. teachers and cli- although this did not seem to affect the results regarding
nicians may observe and rate autistic behavior more simi- the added value of the teacher-reported SRS. Parents also
larly than parents and clinicians (Reszka et al. 2014; tended to rate girls lower than boys, but this difference was
Schanding et al. 2012). This could be due to the fact less pronounced and not significant. Similar gender

123
1668 J Autism Dev Disord (2015) 45:1658–1672

differences in teacher-reported SRS scores have been be a useful instrument to identify cases with SCD. Since
reported for the US norm data (Constantino and Gruber, there are yet no gold standard procedures and instruments
2012) and in a Japanese study (Kamio et al. 2013). These for the diagnosis of SCD (Gibson et al. 2013), this potential
findings are also consistent with other studies that found of the SRS was not investigated in the current study.
that particularly teachers rated lower levels of ASD However, it would be an interesting avenue for future
symptoms in girls than boys (Mandy et al. 2012; Posserud research.
et al. 2006). This could reflect a better adaptation of girls in A limitation of the current study is that not all children
the school setting. Alternatively, it has been raised that screened participated in the diagnostic assessments. In an
girls with ASD may present with different or more subtle ideal situation all screened children would have undergone
difficulties than boys with ASD, which are less easily diagnostic assessments, but this was not feasible due to
recognized by clinicians (Dworzynski et al. 2012) and even time and financial constraints. Another limitation is that we
less so by teachers (Hiller et al. 2014). Teachers also only used the parent-reported SRS to select children for
reported lower levels of behavioral problems in girls with further assessments, which may have influenced the results
ASD (Mandy et al. 2012), suggesting that girls with ASD regarding the teacher-reported SRS. However, we did not
may show less disruptive behavior than boys with ASD in only select children who screened positive on the parent-
the school environment. In addition, girls with ASD appear reported SRS, but also an additional random sample of
to be less overtly rejected by peers than boys with ASD consecutively referred children who screened negative.
(Dean et al. 2014). This could all contribute to girls with Since the selected screen-negative children had a similar
ASD being overlooked at school. It would be interesting mean and standard deviation on the teacher report as the
for future research to study how ASD may present differ- screen-negative children who were not selected, we were
ently in girls versus boys across different contexts using also able to estimate the screening accuracy of the teacher-
observational measures and how the identification of ASD reported SRS. Moreover, because the characteristics of the
in girls by teachers may be improved. total screened sample from which we selected were known,
A problem inherent in research regarding screening we could estimate the screening accuracy of the SRS for
accuracy is choosing the reference standard to which the the total screened sample using an inverse weighting pro-
screener is compared, as a single and error-free test often cedure and thereby correcting for a possible verification
does not exist (Reitsma et al. 2009). Although the com- bias. This methodological approach thus helped overcom-
monly accepted gold standard is a diagnosis of ASD ing our practical design limitations.
determined by a multidisciplinary team using clinical A strength of this study is that we included a broad
judgment and standardized diagnostic instruments (Falk- variety of children who had been consecutively referred
mer et al. 2013), scores on standardized assessment for mental health care, representing the population in
instrument have been found to be more consistent across which the SRS is most likely to be used. Previous studies
centers than clinical judgment (Lord et al. 2012). Since it is have investigated the utility of the teacher-reported SRS in
important to use a reliable and replicable reference stan- research samples of children who had already been diag-
dard in diagnostic research (Reitsma et al. 2009), we nosed with ASD before the start of the study (Constantino
evaluated the screening accuracy of the SRS against ASD et al. 2007; Schanding et al. 2012; Kamio et al. 2013); in a
classifications according to commonly used and well-vali- general population sample (Fombonne et al. 2012); or in a
dated standardized diagnostic instruments. Certainly, from very small and specific sample (Aldridge et al. 2012). In
a clinical perspective, a diagnosis should not be based these case–control design studies, the size of the popula-
solely on the classification according to diagnostic instru- tion from which is sampled and the predicted values for
ments, but needs to incorporate a clinical judgment in ASD by the SRS scores in the source population are often
which all information is taken into account. However, the not known. Consequently, correction for a possible veri-
SRS is mostly used as a first step in the diagnostic process fication bias is not possible, which could have led to
to decide which children need to be further evaluated using biased screening accuracy estimates in these studies
standardized diagnostic instruments, such as the ADOS and (Whiting et al. 2013; Begg and Greenes 1983). To our
ADI-R/3Di. Therefore, we also consider it clinically rele- knowledge, this is the first study to examine the contri-
vant to compare the results of the SRS against the out- bution of the teacher-reported SRS in consecutively
comes on these standardized diagnostic instruments. referred children, which is important for the external
In the DSM-5, the new category of social (pragmatic) validity of our findings.
communication disorder (SCD) has been introduced for An important implication of our findings is that the
individuals who have significant problems in the social use choice of using parent report alone or in combination with
of language and non-verbal communication, but who fall teacher report depends on the purpose of screening. Since
outside the autism spectrum. Potentially, the SRS may also the parent-reported SRS already show a good screening

123
J Autism Dev Disord (2015) 45:1658–1672 1669

accuracy and acquiring teacher reports can be difficult and informants in the overall diagnostic process, since they all
time-consuming, it may be more cost-effective to collect contribute distinct information from unique contexts and
only parent reports in clinical practice. In addition, in a perspectives. More research is needed before a firm con-
research context, when the aim is to select only children clusion can be drawn about what unique information the
with a stringent ASD classification (i.e. fulfilling ASD teacher information adds in different stages of the assessment
criteria according to both a parent interview and child of ASD (i.e. screening versus diagnostic assessment) and
observation), one may choose to use only the parent report how information from different informants should be com-
to efficiently identify children who have a high likelihood bined or integrated. More specifically, future studies could
of receiving an ASD classification according to both investigate whether the contributions of parents and teachers
diagnostic instruments. However, using the parent report differ depending on characteristics of the child or rater.
alone may not identify all children with potential ASD,
specifically those who are classified as having ASD Acknowledgments We gratefully acknowledge the contribution of
all graduate students, PhD-students, and research assistants involved
according to the ADOS. The ADOS is widely used in in the study, as well as the clinical professionals, management and
research as well as clinical practice and such information administrative staff of the participating mental health care centers:
from clinical child observation is an important source of Emergis, Erasmus MC-Sophia Children’s Hospital, GGZ WNB,
information in the diagnostic evaluation of ASD (Risi et al. Lucertis, Riagg Rijnmond, Yulius. We thank all children and parents
who participated in the study. This research was supported by a Grant
2006; Corsello et al. 2013). Therefore, it may be recom- from the Sophia Foundation for Scientific Research (SSWO; project
mendable to use the parent report in combination with the number 958).
teacher report in research and specialized ASD settings
when identification of all potential ASD cases using a
broader ASD definition is important. Furthermore, it should
References
be kept in mind that information from parents cannot
substitute the unique information from the experiences of Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987).
the teacher with the child at school. It may be helpful to Child/adolescent behavioral and emotional problems: Implica-
stress this to parents in order to increase the chances that tions of cross-informant correlations for situational specificity.
teachers will complete and return the SRS. A model pro- Psychological Bulletin, 101(2), 213–232. doi:10.1037//0033-
2909.101.2.213.
cess has been developed to involve teachers in the identi- Aldridge, F. J., Gibbs, V. M., Schmidhofer, K., & Williams, M.
fication of ASD, but further research is needed to evaluate (2012). Investigating the clinical usefulness of the Social
the cost-effectiveness of this approach (Noland and Gabr- Responsiveness Scale (SRS) in a tertiary level, autism spectrum
iels 2004). disorder specific assessment clinic. Journal of Autism and
Developmental Disorders, 42(2), 294–300. doi:10.1007/s10803-
To conclude, this multicenter study confirms the utility 011-1242-9.
of the SRS as a screening tool to identify children who Axelrod, B. N. (2002). Validity of the Wechsler Abbreviated Scale of
need ASD-specific diagnostic assessment among children Intelligence and other very short forms of estimating intellectual
who are referred for various mental health problems. functioning. Assessment, 9(1), 17–23. doi:10.1177/107319110
2009001003.
Careful consideration should be given to the cut-off as well Begg, C. B., & Greenes, R. A. (1983). Assessment of diagnostic tests
as the selection of an informant when using the SRS to when disease verification is subject to selection bias. Biometrics,
screen for ASD, as the parent and teacher report show 39(1), 207–215. doi:10.2307/2530820.
differential relationships to the different ASD diagnostic Bishop, D. V. M., & Baird, G. (2001). Parent and teacher report of
pragmatic aspects of communication: Use of the children’s
instruments. It is important to stress that an outcome on the communication checklist in a clinical setting. Developmental
SRS, or any instrument, does not equal a clinical diagnosis Medicine and Child Neurology, 43(12), 809–818. doi:10.1017/
of ASD, which can only be made by a multidisciplinary S0012162201001475.
team after an extensive diagnostic procedure, preferably Bölte, S., Westerwald, E., Holtmann, M., Freitag, C., & Poustka, F.
(2011). Autistic traits and autism spectrum disorders: The clinical
including standardized diagnostic instruments such as the validity of two measures presuming a continuum of social
ADOS or ADI-R/3Di (Falkmer et al. 2013). The SRS communication skills. Journal of Autism and Developmental
scores can be used to provide a standardized and quantified Disorders, 41(1), 66–72. doi:10.1007/s10803-010-1024-9.
indication of whether a child needs further diagnostic Charman, T., Baird, G., Simonoff, E., Loucas, T., Chandler, S.,
Meldrum, D., et al. (2007). Efficacy of three screening instru-
assessments. In addition, the SRS scores may be valuable ments in the identification of autistic-spectrum disorders. British
as part of the entire diagnostic process by complementing Journal of Psychiatry, 191, 554–559. doi:10.1192/bjp.bp.107.
other information in acquiring a comprehensive view of the 040196.
ASD characteristics of the child. The limited agreement Charman, T., & Gotham, K. (2013). Measurement issues: Screening
and diagnostic instruments for autism spectrum disorders—
between instruments and informants—parents, teachers, Lessons from research and practise. Child and Adolescent
and clinicians—highlights the importance of using multiple Mental Health, 18(1), 52–63. doi:10.1111/j.1475-3588.2012.
instruments that collect information from multiple 00664.x.

123
1670 J Autism Dev Disord (2015) 45:1658–1672

Chuthapisith, J., Taycharpipranai, P., Ruangdaraganon, N., Warring- ADOS revised algorithm: The applicability in 558 Dutch
ton, R., & Skuse, D. (2012). Translation and validation of the children and adolescents. Journal of Autism and Developmental
developmental, dimensional and diagnostic interview (3Di) for Disorders, 39(9), 1350–1358. doi:10.1007/s10803-009-0749-9.
diagnosis of autism spectrum disorder in Thai children. Autism, De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in
16(4), 350–356. doi:10.1177/1362361311433770. the assessment of childhood psychopathology: A critical review,
Cicchetti, D. V. (2001). Methodological commentary the precision of theoretical framework, and recommendations for further study.
reliability and validity estimates re-visited: Distinguishing Psychological Bulletin, 131(4), 483–509. doi:10.1037/0033-
between clinical and statistical significance of sample size 2909.131.4.483.
requirements. Journal of Clinical and Experimental Neuropsy- Dean, M., Kasari, C., Shih, W., Frankel, F., Whitney, R., Landa, R.,
chology, 23(5), 695–700. doi:10.1076/jcen.23.5.695.1249. et al. (2014). The peer relationships of girls with ASD at school:
Cicchetti, D. V., Volkmar, F., Klin, A., & Showalter, D. (1995). Comparison to boys and girls with and without ASD. Journal of
Diagnosing autism using ICD-10 criteria: A comparison of neural Child Psychology and Psychiatry and Allied Disciplines, 55(11),
networks and standard multivariate procedures. Child Neuropsy- 1218–1225. doi:10.1111/jcpp.12242.
chology, 1(1), 26–37. doi:10.1080/09297049508401340. Dunn, G., Pickles, A., Tansella, M., & Vazquez-Barquero, J. L.
Constantino, J. N., Davis, S. A., Todd, R. D., Schindler, M. K., Gross, (1999). Two-phase epidemiological surveys in psychiatric
M. M., Brophy, S. L., et al. (2003). Validation of a brief research. British Journal of Psychiatry, 174, 95–100. doi:10.
quantitative measure of autistic traits: Comparison of the social 1192/Bjp.174.2.95.
responsiveness scale with the autism diagnostic interview- Dworzynski, K., Ronald, A., Bolton, P., & Happe, F. (2012). How
revised. Journal of Autism and Developmental Disorders, different are girls and boys above and below the diagnostic
33(4), 427–433. doi:10.1023/A:1025014929212. threshold for autism spectrum disorders? Journal of the Amer-
Constantino, J. N., & Gruber, C. P. (2005). Social Responsiveness ican Academy of Child and Adolescent Psychiatry, 51(8),
Scale (SRS). Los Angeles, CA: Western Psychological Services. 788–797. doi:10.1016/j.jaac.2012.05.018.
Constantino, J. N., & Gruber, C. P. (2012). Social Responsiveness Falkmer, T., Anderson, K., Falkmer, M., & Horlin, C. (2013).
Scale, second edition (SRS-2). Los Angeles, CA: Western Diagnostic procedures in autism spectrum disorders: A system-
Psychological Services. atic literature review. European Child and Adolescent Psychi-
Constantino, J. N., LaVesser, P. D., Zhang, Y., Abbacchi, A. M., atry, 22(6), 329–340. doi:10.1007/s00787-013-0375-0.
Gray, T., & Todd, R. D. (2007). Rapid quantitative assessment of Ferdinand, R. F., Hoogerheide, K. N., van der Ende, J., Visser, J. H.,
autistic social impairment by classroom teachers. Journal of the Koot, H. M., Kasius, M. C., et al. (2003). The role of the
American Academy of Child and Adolescent Psychiatry, 46(12), clinician: Three-year predictive value of parents’, teachers’, and
1668–1676. doi:10.1097/chi.0b013e318157cb23. clinicians’ judgment of childhood psychopathology. Journal of
Constantino, J. N., Przybeck, T., Friesen, D., & Todd, R. D. (2000). Child Psychology and Psychiatry, 44(6), 867–876. doi:10.1111/
Reciprocal social behavior in children with and without perva- 1469-7610.00171.
sive developmental disorders. Journal of Developmental and Filipek, P. A., Accardo, P. J., Baranek, G. T., Cook, E. H., Dawson,
Behavioral Pediatrics, 21(1), 2–11. doi:10.1097/00004703- G., Gordon, B., et al. (1999). The screening and diagnosis of
200002000-00001. autistic spectrum disorders. Journal of Autism and Developmen-
Corsello, C. M., Akshoomoff, N., & Stahmer, A. C. (2013). Diagnosis tal Disorders, 29(6), 439–484. doi:10.1023/A:1021943802493.
of autism spectrum disorders in 2-year-olds: a study of Fombonne, E., Marcin, C., Bruno, R., Tinoco, C. M., & Marquez, C.
community practice. Journal of Child Psychology and Psychi- D. (2012). Screening for Autism in Mexico. Autism Research,
atry, 54(2), 178–185. doi:10.1111/j.1469-7610.2012.02607.x. 5(3), 180–189. doi:10.1002/Aur.1235.
de Bildt, A., Greaves-Lord, K., & De Jonge, M. (2013). ADOS-2: Gibson, J., Adams, C., Lockton, E., & Green, J. (2013). Social
Autisme diagnostisch observatieschema. Handleiding. Amster- communication disorder outside autism? A diagnostic classifi-
dam: Hogrefe. cation approach to delineating pragmatic language impairment,
de Bildt, A., Mulder, E. J., Hoekstra, P. J., van Lang, N. D., Minderaa, high functioning autism and specific language impairment.
R. B., & Hartman, C. A. (2009a). Validity of the Children’s Journal of Child Psychology and Psychiatry and Allied Disci-
Social Behavior Questionnaire (CSBQ) in children with intel- plines, 54(11), 1186–1197. doi:10.1111/jcpp.12079.
lectual disability: Comparing the CSBQ with ADI-R, ADOS, Gotham, K., Risi, S., Dawson, G., Tager-Flusberg, H., Joseph, R.,
and clinical DSM-IV-TR classification. Journal of Autism and Carter, A., et al. (2008). A replication of the autism diagnostic
Developmental Disorders, 39(10), 1464–1470. doi:10.1007/ observation schedule (ADOS) revised algorithms. Journal of the
s10803-009-0764-x. American Academy of Child and Adolescent Psychiatry, 47(6),
de Bildt, A., Sytema, S., Ketelaars, C., Kraijer, D., Mulder, E., 642–651. doi:10.1097/Chi.0b013e31816bffb7.
Volkmar, F., et al. (2004). Interrelationship between autism Gotham, K., Risi, S., Pickles, A., & Lord, C. (2007). The autism
diagnostic observation schedule-generic (ADOS-G), autism diagnostic observation schedule: Revised algorithms for improved
diagnostic interview-revised (ADI-R), and the diagnostic and diagnostic validity. Journal of Autism and Developmental Disor-
statistical manual of mental disorders (DSM-IV-TR) classifica- ders, 37(4), 613–627. doi:10.1007/s10803-006-0280-1.
tion in children and adolescents with mental retardation. Journal Hanley, J. A., & Mcneil, B. J. (1983). A method of comparing the
of Autism and Developmental Disorders, 34(2), 129–137. doi:10. areas under receiver operating characteristic curves derived from
1023/B:JADD.0000022604.22374.5f. the same cases. Radiology, 148(3), 839–843. doi:10.1148/
de Bildt, A., Sytema, S., Ketelaars, C., Kraijer, D., Volkmar, F., & radiology.148.3.6878708.
Minderaa, R. (2003). Measuring pervasive developmental Hendriksen, J., & Hurks, P. (2009). Wechsler preschool and Primary
disorders in children and adolescents with mental retardation: Scale of intelligence—Third. Amsterdam: Pearson.
A comparison of two screening instruments used in a study of Hiller, R. M., Young, R. L., & Weber, N. (2014). Sex differences in
the total mentally retarded population from a designated area. autism spectrum disorder based on DSM-5 criteria: Evidence
Journal of Autism and Developmental Disorders, 33(6), from clinician and teacher reporting. Journal of Abnormal Child
595–605. doi:10.1023/B:JADD.0000005997.92287.a3. Psychology, 42(8), 1381–1393. doi:10.1007/s10802-014-9881-x.
de Bildt, A., Sytema, S., van Lang, N. D., Minderaa, R. B., van Hunink, M. G., & Glasziou, P. (2001). Decision making in health and
Engeland, H., & de Jonge, M. V. (2009b). Evaluation of the medicine. Cambridge: Cambridge University Press.

123
J Autism Dev Disord (2015) 45:1658–1672 1671

Hunink, M. G., Richardson, D. K., Doubilet, P. M., & Begg, C. B. Noland, R. M., & Gabriels, R. L. (2004). Screening and identifying
(1990). Testing for fetal pulmonary maturity: ROC analysis children with autism spectrum disorders in the public school
involving covariates, verification bias, and combination testing. system: The development of a model process. Journal of Autism
Medical Decision Making, 10(3), 201–211. doi:10.1177/ and Developmental Disorders, 34(3), 265–277. doi:10.1023/B:
0272989x9001000307. JADD.0000029549.84385.44.
IBM Corporation (2011). IBM SPSS statistics for windows (20.0 ed.). Oosterling, I., Roos, S., de Bildt, A., Rommelse, N., de Jonge, M.,
Armonk, NY: IBM Corp. Visser, J., et al. (2010). Improved diagnostic validity of the
Kamio, Y., Inada, N., Moriwaki, A., Kuroda, M., Koyama, T., Tsujii, ADOS revised algorithms: A replication study in an independent
H., et al. (2013a). Quantitative autistic traits ascertained in a sample. Journal of Autism and Developmental Disorders, 40(6),
national survey of 22 529 Japanese school children. Acta 689–703. doi:10.1007/s10803-009-0915-0.
Psychiatrica Scandinavica, 128(1), 45–53. doi:10.1111/acps. Ozonoff, S., Goodlin-Jones, B. L., & Solomon, M. (2005). Evidence-
12034. based assessment of autism spectrum disorders in children and
Kamio, Y., Moriwaki, A., & Inada, N. (2013b). Utility of teacher- adolescents. Journal of Clinical Child and Adolescent Psychol-
report assessments of autistic severity in Japanese school ogy, 34(3), 523–540. doi:10.1207/s15374424jccp3403_8.
children. Autism Research and Treatment, 2013, 373240. Pelham, W. E., Fabiano, G. A., & Massetti, G. M. (2005). Evidence-
doi:10.1155/2013/373240. based assessment of attention deficit hyperactivity disorder in
Kanne, S. M., Abbacchi, A. M., & Constantino, J. N. (2009). Multi- children and adolescents. Journal of Clinical Child and Adoles-
informant ratings of psychiatric symptom severity in children cent Psychology, 34(3), 449–476. doi:10.1207/s15374424jccp
with autism spectrum disorders: the importance of environmental 3403_5.
context. Journal of Autism and Developmental Disorders, 39(6), Posserud, M. B., Lundervold, A. J., & Gillberg, C. (2006). Autistic
856–864. doi:10.1007/s10803-009-0694-7. features in a total population of 7–9-year-old children assessed
Kazdin, A. E. (2005). Evidence-based assessment for children and by the ASSQ (Autism Spectrum Screening Questionnaire).
adolescents: Issues in measurement development and clinical Journal of Child Psychology and Psychiatry and Allied Disci-
application. Journal of Clinical Child and Adolescent Psychol- plines, 47(2), 167–175. doi:10.1111/j.1469-7610.2005.01462.x.
ogy, 34(3), 548–558. doi:10.1207/s15374424jccp3403_10. R Core Team. (2014). R: A language and environment for statistical
Kim, S. H., & Lord, C. (2012). Combining information from multiple computing. Vienna: R Foundation for Statistical Computing.
sources for the diagnosis of autism spectrum disorders for Reed, P., & Osborne, L. A. (2013). The role of parenting stress in
toddlers and young preschoolers from 12 to 47 months of age. discrepancies between parent and teacher ratings of behavior
Journal of Child Psychology and Psychiatry, 53(2), 143–151. problems in young children with autism spectrum disorder.
doi:10.1111/j.1469-7610.2011.02458.x. Journal of Autism and Developmental Disorders, 43(2),
Koning, C., & Magill-Evans, J. (2001). Social and language skills in 471–477. doi:10.1007/s10803-012-1594-9.
adolescent boys with Asperger syndrome. Autism, 5(1), 23–36. Reitsma, J. B., Rutjes, A. W., Khan, K. S., Coomarasamy, A., &
doi:10.1177/1362361301005001003. Bossuyt, P. M. (2009). A review of solutions for diagnostic
Kort, W., Schittekatte, M., Dekker, P. H., Verhaeghe, P., Compaan, E. accuracy studies with an imperfect or missing reference
L., Bosmans, M., & Vermeir, G. (2005). Wechsler intelligence standard. Journal of Clinical Epidemiology, 62(8), 797–806.
scale for children-III. Nederlandstalige Uitgave. Amsterdam: doi:10.1016/j.jclinepi.2009.02.005.
Pearson. Reszka, S. S., Boyd, B. A., McBee, M., Hume, K. A., & Odom, S. L.
Lai, M. C., Lombardo, M. V., & Baron-Cohen, S. (2014). Autism. Lancet, (2014). Brief report: Concurrent validity of autism symptom
383(9920), 896–910. doi:10.1016/S0140-6736(13)61539-1. severity measures. Journal of Autism and Developmental
Le Couteur, A., Haden, G., Hammal, D., & McConachie, H. (2008). Disorders, 44(2), 466–470. doi:10.1007/s10803-013-1879-7.
Diagnosing autism spectrum disorders in pre-school children Richters, J. E. (1992). Depressed mothers as informants about their
using two standardised assessment instruments: The ADI-R and children: a critical review of the evidence for distortion.
the ADOS. Journal of Autism and Developmental Disorders, Psychological Bulletin, 112(3), 485–499. doi:10.1037/0033-
38(2), 362–372. doi:10.1007/s10803-007-0403-3. 2909.112.3.485.
Lord, C., Petkova, E., Hus, V., Gan, W., Lu, F., Martin, D. M., et al. Risi, S., Lord, C., Gotham, K., Corsello, C., Chrysler, C., Szatmari, P.,
(2012a). A multisite study of the clinical diagnosis of different et al. (2006). Combining information from multiple sources in
autism spectrum disorders. Archives of General Psychiatry, the diagnosis of autism spectrum disorders. Journal of the
69(3), 306–313. doi:10.1001/archgenpsychiatry.2011.148. American Academy of Child and Adolescent Psychiatry, 45(9),
Lord, C., Rutter, M., DiLavore, P. C., & Risi, S. (1999). Autism 1094–1103. doi:10.1097/01.chi.0000227880.42780.0e.
diagnostic observation schedule (ADOS). Manual. Western Roeyers, H., Thys, M., Druart, C., De Schryver, M., & Schittekate, M.
Psychological Services: Los Angeles. (2011). SRS Screeningslijst voor autismespectrumstoornissen.
Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, Handleiding. Amsterdam: Hogrefe.
S. L. (2012). Autism diagnostic observation schedule, Second Santosh, P. J., Mandy, W. P. L., Puura, K., Kaartinen, M.,
Edition (ADOS-2). Manual (Part I). Los Angeles: Western Warrington, R., & Skuse, D. H. (2009). The construction and
Psychological Services. validation of a short form of the developmental, diagnostic and
Lord, C., Rutter, M., & Lecouteur, A. (1994). Autism diagnostic dimensional interview. European Child and Adolescent Psychi-
interview-revised—A revised version of a diagnostic interview atry, 18(8), 521–524. doi:10.1007/s00787-009-0004-0.
for caregivers of individuals with possible pervasive develop- Schanding, G. T., Nowell, K. P., & Goin-Kochel, R. P. (2012). Utility
mental disorders. Journal of Autism and Developmental Disor- of the Social Communication Questionnaire–Current and Social
ders, 24(5), 659–685. doi:10.1007/Bf02172145. Responsiveness Scale as teacher-report screening tools for
Mandy, W., Chilvers, R., Chowdhury, U., Salter, G., Seigal, A., & autism spectrum disorders. Journal of Autism and Developmen-
Skuse, D. (2012). Sex differences in autism spectrum disorder: tal Disorders, 42(8), 1705–1716. doi:10.1007/s10803-011-
Evidence from a large sample of children and adolescents. 1412-9.
Journal of Autism and Developmental Disorders, 42(7), Seaman, S. R., & White, I. R. (2013). Review of inverse probability
1304–1313. doi:10.1007/s10803-011-1356-0. weighting for dealing with missing data. Statistical Methods in

123
1672 J Autism Dev Disord (2015) 45:1658–1672

Medical Research, 22(3), 278–295. doi:10.1177/0962280210395 2‘-7 Handleiding and verantwoordig [Snijders-Oomen Non-
740. verbal Intelligence Test. SON-R 2‘-7 Manual and Research
Sikora, D. M., Hall, T. A., Hartley, S. L., Gerrard-Morris, A. E., & Report]. Lisse: Swets & Zeitlinger, B.V.
Cagle, S. (2008). Does parent report of behavior differ across van der Ende, J., Verhulst, F. C., & Tiemeier, H. (2012). Agreement
ADOS-G classifications: Analysis of scores from the CBCL and of informants on emotional and behavioral problems from
GARS. Journal of Autism and Developmental Disorders, 38(3), childhood to adulthood. Psychological Assessment, 24(2),
440–448. doi:10.1007/s10803-007-0407-z. 293–300. doi:10.1037/a0025500.
Skuse, D., Warrington, R., Bishop, D., Chowdhury, U., Lau, J., Ventola, P. E., Kleinman, J., Pandey, J., Barton, M., Allen, S., Green,
Mandy, W., et al. (2004). The developmental, dimensional and J., et al. (2006). Agreement among four diagnostic instruments
diagnostic interview (3di): A novel computerized assessment for for autism spectrum disorders in toddlers. Journal of Autism and
autism spectrum disorders. Journal of the American Academy of Developmental Disorders, 36(7), 839–847. doi:10.1007/s10803-
Child and Adolescent Psychiatry, 43(5), 548–558. doi:10.1097/ 006-0128-8.
00004583-200405000-00008. Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., & Mallett, S.
Szatmari, P., Archer, L., Fisman, S., & Streiner, D. L. (1994). Parent (2013). A systematic review classifies sources of bias and
and teacher agreement in the assessment of pervasive develop- variation in diagnostic test accuracy studies. Journal of Clinical
mental disorders. Journal of Autism and Developmental Disor- Epidemiology, 66(10), 1093–1104. doi:10.1016/j.jclinepi.2013.
ders, 24(6), 703–717. doi:10.1007/Bf02172281. 05.014.
Tellegen, P. J., Winkel, M., Wijnberg-Williams, B. J., & Laros, J. A
(1998). Snijders-Oomen Niet-verbale Intelligentietest. SON-R

123