Autism spectrum disorder (ASD) is a neurodevelopmental disorder associated with impairments in social communication and restricted and repetitive behaviors [1]. According to a 2018 survey by the U.S. Centers for Disease Control and Prevention (CDC), approximately 1 in every 44 children has ASD [2]. In a Korean ASD prevalence survey conducted in 2011, the prevalence of ASD was 2.64% [3], which is similar to the estimates from the CDC. Symptoms of ASD manifest from the beginning of life, and 85% of individuals with this disorder cannot live independently when they become adults [4]. However, early intervention and intensive care are known to result in a better prognosis [5], emphasizing the importance of early screening tests.

The Childhood Autism Rating Scale (CARS) is a 15-item questionnaire developed in 1988 [6]. It is one of the most widely used tools for diagnosing autism in Korea and worldwide. CARS is particularly helpful in identifying children with autism from those with cognitive impairment in a relatively brief amount of time and thus, is especially useful in community institutions [7]. On the contrary, the tool recognized as the gold standard for diagnosing ASD is the Autism Diagnostic Observation Schedule (ADOS) [8,9]. The ADOS is a semistructured evaluation tool that comprehensively assesses autism- related symptoms by observing the participant directly over a 40–60 minute period. Although it is extremely useful to diagnose ASD, it is more expensive and requires more time to administer by qualified evaluators with sufficient clinical experience [10]; therefore, it is used in a limited number of institutions in community settings in Korea.

Previous studies have compared these tools and suggested the optimal diagnostic cutoff for the CARS using the ADOS as the reference standard [11,12]. Currently, the revised versions of both CARS and ADOS have been introduced, but a comparative study between the revised diagnostic tools is lacking.

In this study, we examined the correlation between the CARS-2 and ADOS-2 and investigated the optimal cutoff score of the CARS-2 compared with the ADOS-2. These findings can help improve the reliability of ASD diagnosis using the CARS-2, which is relatively easy to implement in community institutions compared with the ADOS-2.

Participants

This study was conducted on 237 children (aged 24–145 months) who were tested using both the standard version of CARS-2 (CARS2-ST) and the ADOS-2 in the child and adolescent psychiatry outpatient clinic at our center from January 1, 2019 to July 19, 2020. The tests were conducted by two highly trained examiners with many years of experience who had participated in our previous work using the original versions of both tests [11]. One examiner had completed a master’s degree, and the other had completed a doctoral degree in special education. Both had a license to administer the ADOS test. The ADOS-2 and CARS2-ST scores and demographic characteristics (sex and age) were retrospectively acquired from the electronic medical charts. This study was approved by the Institutional Review Board of our institution, and the need for informed consent was waived by the Institutional Review Board (IRB No: H-2006-239-1142).

Assessment tools

The CARS is an evaluation tool used for identifying autism and determining the severity of autism disorders [6]. Several studies have proven that the CARS is a useful measure because of its robust internal consistency and reliability [13]. Schopler et al. [14] developed a second version, CARS-2, in 2010, with two parts: CARS2-ST, which is equivalent to the original CARS, and a high function version (CARS2-HF) for those aged 6 years and older with intelligence quotient scores above 80 [14]. Reliability and validity tests of the CARS-2 showed a high diagnostic consistency rate with the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) [1,15,16]. The CARS-2 consists of 15 questions and is evaluated by clinical experts based on individual interviews with the primary caregiver and direct observation of children. Each question is rated on a scale of 1 (normal at the corresponding age) to 4 (severely abnormal at the corresponding age), and the scores of each question are added to obtain a total score for each patient, which ranged from 15– 60. In the case of the CARS2-ST, a total score of <30 indicates non-autism, 30–36.5 indicates mild to moderate autism, and a total score of ≥37 indicates severe autism. For the CARS2-HF, a total score of ≤27.5 indicates non-autism, a score between 28–33.5 indicates moderate autism, and a score of ≥34 indicates severe autism [14,17]. In the Korean standardization study of the CARS-2, the cutoff scores for autism screening were 30 and 26.5 for the CARS2-ST and CARS2- HF, respectively [18,19].

The ADOS is a semi-structured autism disorder diagnostic tool developed in 1999 that evaluates communication, social interaction, the imaginary use of play or objects, and homologous behavior by directly observing the individual. It is configured such that the test can be performed by selecting a module suitable for the participants according to their expressive language level and age. Each module takes approximately 40–60 minutes to complete [20]. Currently, the ADOS is considered the gold standard along with the Autism Diagnostic Interview-Revised [8]. In 2012, the ADOS-2 was published, which added new comparison scores, revised algorithms, and a Toddler Module (Module T). The revised algorithm includes social affect and restricted and repetitive behavioral domains. Additionally, the procedure was modified to be more congruent with the diagnostic criteria of the DSM-5 [1]. The ADOS-2 comprises five modules. Module T is designed for infants aged between 12 and 30 months who have difficulty using sentences. Module 1 is designed for children aged 31 months and older who do not consistently utilize phrase speech. Module 2 is designed for children of all ages who use phrase speech but are not verbally proficient. Module 3 is for children and young adolescents who speak fluently, and Module 4 is for older adolescents and adults who are fluent in language [20]. In Modules 1–4, the results of the algorithm are compared to the cutoff scores to classify the diagnoses (autism, autism spectrum, or non-spectrum). Algorithms in Module T provide ranges of concern instead of diagnostic classification scores.

We examined the correlation between total CARS2-ST and ADOS-2 scores using Pearson’s correlation analysis. We also assessed the relationship between these scores and age using the same method. We analyzed the differences in the total scores of the CARS2-ST and ADOS-2 by sex using the Mann–Whitney U test. In addition, we used a receiver operating characteristic (ROC) graph to determine the optimal CARS2-ST cutoff score for ASD diagnosis using the ADOS-2. Cohen’s kappa was calculated based on the cutoff score. SPSS version 26.0 (IBM Corp., Armonk, NY, USA) was used for all analyses. Statistical significance was determined using a 2-sided p<0.05.

Demographic characteristics

Our study included a total of 237 children who were assessed using the CARS2-ST and the ADOS-2 (mean age, 48.93±17.79 months; range, 24–145 months) (Table 1). ADOS-2 Module 1 was used for 211 children, Module 2 for 9 children, and Module T for 17 children (Table 2). Neither the CARS-2 total scores (r=-0.105, p=0.106) nor the ADOS-2 total scores (r=-0.124, p=0.057) significantly correlated with age. We found no significant difference in the total scores of the two tests between the sexes (CARS-2, p=0.203; ADOS-2, p=0.103).

Table 1

Demographic characteristics (n=237)

Characteristics	n (%)
Sex
Female	49 (20.7)
Male	188 (79.3)
Age (month)
24-48	141 (59.5)
48-72	76 (32.1)
72-96	14 (5.4)
96-120	4 (1.5)
120-144	1 (0.4)
>144	1 (0.4)

Table 2

Total scores of CARS2-ST and ADOS-2

	CARS2-ST	ADOS-2
		Module 1	Module 2	Module T	Total
n (%)	237 (100)	211 (89)	9 (3.8)	17 (7.2)	237 (100)
Total score, mean (SD)	35.07 (4.29)	17.61 (4.34)	10 (4.06)	17.88 (2.91)	17.22 (4.60)
Total score, range	17.5-54	7-25	1-14	12-22	1-25

CARS-2 ST, the standard version of Childhood Autism Rating Scale-Second Edition; ADOS-2, Autism Diagnostic Observation Schedule-Second Edition; Module T, Toddler Module

Concurrent validity

There was a significant correlation between the total scores of the CARS2-ST and ADOS-2, and the total scores of the CARS2-ST significantly correlated with those of ADOS-2 Modules 1, 2, and T (Table 3).

Table 3

Correlation between the CARS2-ST and the ADOS-2 total scores

	ADOS-2
	Module 1	Module 2	Module T	Total	Total (CSS)†
CARS-2 ST	0.855**	0.954**	0.678*	0.864**	0.843**

*p<0.01, **p<0.001, †Total CSS was calculated using only Modules 1 and 2. CARS-2 ST, the standard version of Childhood Autism Rating Scale-Second Edition; ADOS-2, Autism Diagnostic Observation Schedule-Second Edition; Module T, Toddler Module; CSS, calibrated severity score

Diagnosis and severity classification

All children categorized as non-autism according to the CARS2-ST were also categorized as non-spectrum based on their ADOS-2 Module 1 or 2 scores. In addition, according to the CARS2-ST, all children categorized as having autism based on their ADOS-2 scores were also classified as having autism. According to the ADOS-2, 30 children were considered to have autism spectrum, and 24 of them were considered to have non-autism using the CARS2-ST (χ²=179.512, p<0.001) (Table 4). According to the ADOS-2 Module T, all 17 children categorized as being within or above the mildto- moderate range of concern were considered to have autism based on the CARS-2 ST (χ²=0.944, p<0.331) (Table 4).

Table 4

Diagnosis and severity classification

	CARS2-ST
	Non-autism	Mild-tomoderate autism	Severe autism	Total	χ2(p)
ADOS-2 (modules 1 and 2)					179.512 (<0.001)
Non-spectrum	6 (2.7)	0	0	6 (2.7)
Autism spectrum	24 (10.9)	5 (2.3)	1 (0.5)	30 (13.6)
Autism	0	116 (52.7)	68 (30.9)	184 (83.6)
Total	30 (13.6)	121 (55.0)	69 (31.4)	220 (100)
ADOS-2 (Module T)					0.944 (0.331)
Little-or-no concern	0	0	0	0
Mild-to-moderate concern	0	1 (5.90)	0	1 (5.90)
Moderate-to-severe concern	0	8 (47.10)	8 (47.10)	16 (94.10)
Total	0	9 (52.90)	8 (47.10)	17 (100)

Data are presented as n (%). CARS-2 ST, the standard version of Childhood Autism Rating Scale-Second Edition; ADOS-2, Autism Diagnostic Observation Schedule-Second Edition; Module T, Toddler Module

Optimal diagnostic cutoff using the CARS-2

We investigated the diagnostic accuracy of the CARS2-ST. Regarding the diagnosis of autism using the ADOS-2 (Modules 1 and 2), a CARS2-ST score of 30 exhibited 100% sensitivity, 84% specificity, 97% positive predictive value, and 100% negative predictive value. As for the diagnosis of autism or autism spectrum using the ADOS-2 (Modules 1 and 2), a CARS2-ST score of 30 exhibited 89% sensitivity, 100% specificity, 100% positive predictive value, and 23% negative predictive value.

Using an ROC graph, we estimated the area under the curve (AUC) to identify the optimal CARS2-ST cutoff score. Regarding the diagnosis of autism using the ADOS-2 (Modules 1 and 2), the highest levels of sensitivity (98.9%) and specificity (86.1%) were found at an optimal CARS2-ST cutoff score of 30.25 (AUC=0.962, p<0.001). As for the diagnosis of autism or autism spectrum using the ADOS-2 (Modules 1 and 2), the highest sensitivity (94.9%) and specificity (100%) were found at the optimal cutoff score of 28.25 (AUC=0.977, p< 0.001) (Table 5).

Table 5

Sensitivity and specificity for each one of the CARS2-ST cutoff scores for diagnoses based on ADOS-2

CARS2-ST cut-off	ADOS-2 Modules 1 and 2
	Autism only			Autism or autism spectrum
	Sensitivity	Specificity		Sensitivity	Specificity
26.25	1.000	0.194		0.977	0.333
26.75	1.000	0.250		0.977	0.667
27.25	1.000	0.333		0.963	0.667
27.75	1.000	0.361		0.963	0.833
28.25*	1.000	0.472		0.949*	1.000*
28.75	1.000	0.556		0.935	1.000
29.25	1.000	0.639		0.921	1.000
29.75	1.000	0.833		0.888	1.000
30.25*	0.989*	0.861*		0.874	1.000
30.75	0.984	0.861		0.869	1.000

*Optimal CARS2-ST cutoff for each diagnosis and its related values. CARS2-ST, the standard version of Childhood Autism Rating Scale-Second Edition; ADOS-2, Autism Diagnostic Observation Schedule-Second Edition; Module T, Toddler Module

We investigatedthe utility of the optimal cutoff scores derived in this study as a screening test for ASD. Using the default diagnostic cutoff score of 30, 24 children identified as autism spectrum in the ADOS-2 (Modules 1 and 2) were diagnosed with non-autism by CARS2-ST (κ=0.30, p<0.001). However, if the newly derived cutoff score of 28.25 was used, 11 children identified as autism spectrum on the ADOS-2 (Modules 1 and 2) were diagnosed with non-autism in the CARS2-ST (κ=0.52, p<0.001). Furthermore, based on three diagnostic classifications of ADOS-2 Modules 1 and 2 (nonspectrum, autism spectrum, and autism) and CARS2-ST using the newly derived cutoff scores (<28.25, 28.25 to 30.25, and >30.25), Cohen’s kappa value increased (κ=0.71, p<0.001) indicating a higher level of agreement.

In this study, we examined the correlation between the revised CARS2-ST and ADOS-2. In addition, we derived the optimal diagnostic cutoff score of the CARS2-ST, which corresponds to the diagnosis of ASD based on the ADOS-2.

First, a high quantitative correlation between the CARS2- ST and ADOS-2 total scores was found, which was consistent across the different modules of the ADOS-2. CARS2-ST categorized all children diagnosed with autism using the ADOS-2 (Modules 1 and 2) as having autism. Among the 190 children who were classified as having autism on the CARS2-ST, 6 were classified as having autism spectrum and 184 were classified as having autism using the ADOS-2 (Models 1 and 2). These results are similar to those of a previous study, which found that the CARS, based on patient observation scores, had significant correlations with ADOS-2 scores [21].

Next, we explored the optimal CARS2-ST diagnostic cutoff score using the AUC and suggested a score of 30.25 when diagnosing only autism using the ADOS-2 (Modules 1 and 2). Considering that the CARS-2 score was in units of 0.5, this was similar to the default diagnostic cutoff of 30. On the contrary, based on the ADOS-2 diagnosis of autism or autism spectrum (Modules 1 and 2), the optimal cutoff score of the CARS2-ST was 28.25, which was slightly lower than 30. In our previous study comparing the original versions of the CARS and ADOS, the diagnostic cutoff scores of the CARS were suggested in two ways: 30 for detection of autism and 24.5 for detection of autism and autism spectrum on the ADOS [11]. Compared with this previous report, it is noteworthy that a smaller difference was found between the two cutoff scores using the revised versions of the tools. A possible reason for this difference is that the revised CARS-2 was divided into two versions: CARS2-ST and CARS2-HF. It has been proposed that patients with high-functioning ASD might be diagnosed as non-ASD on the CARS; therefore, lower cutoff scores have been suggested [11,22,23]. However, the revised version of the CARS includes the newly developed CARS2-HF for high-functioning ASD, and the present study was conducted with only the CARS2-ST. Accordingly, the exclusion of high-functioning participants may explain the substantially higher cutoff score obtained here for identifying the autism spectrum.

As shown above, we recommend a lower cutoff score (28.5) for detecting ASD on the CARS2-ST due to its higher sensitivity. Using this threshold, more than half of the children (i.e., 13 out of 24) misdiagnosed as non-autism using the conventional cutoff of 30 were newly screened as having autism spectrum according to the ADOS-2, making a critical difference for children in need of early intervention. These 13 children were relatively older (mean age 57.23±14.93 months; range, 32–80 months) compared with the overall group of participants, possibly suggesting that children with milder symptoms and/or fewer impairments (i.e., scoring slightly below 30 in CARS2-ST) whose clinic visits are delayed are more likely to be missed when screened with the conventional cutoff score of 30. This, in turn, may suggest the need to expand the age range of the CARS2-HF, as the current candidates for the CARS2-HF are those aged 6 years and older.

This study has some limitations. First, only children who had an outpatient visit at our institution were included and most of the participants (91.6%) were aged under 72 months. Therefore, the findings may not be applicable to other clinical settings or older populations. Second, our study was conducted with only the CARS2-ST and ADOS-2 Modules 1, 2, and T. A retrospective chart review during the study period revealed that only a few children were evaluated with the CARS2-HF or ADOS-2 Modules 3 and 4 in our clinic. Third, the CARS-2 and the ADOS-2 were administered by the same examiners on the same day, raising the possibility that one influenced the other. However, a possible advantage of conducting the two tests on the same day is that the results would not have been affected by changes in the developmental level or variability in the day-to-day performance of the child. In addition, comparing the cut-off scores of the CARS-2 and the ADOS-2 without being affected by discrepancies between examiners can be a strength of the study. Fourth, the sample size was small for ADOS-2 Modules 2 and T. Future studies with larger samples may need to be extended to all versions and modules of both tools. Regarding the sample size, the relatively small number of participants who were not autistic was another limitation. Fifth, although the results may vary depending on who provided the information about the child, information about the informant was not available during the retrospective chart review. Lastly, although ADOS is often referred to as the gold standard instrument for diagnosing ASD, the true gold standard would be a comprehensive clinical assessment by an expert clinician. In the usual clinical settings, however, not all visits are accompanied by such a comprehensive clinical assessment, and not all information is documented in the medical records, limiting our ability to identify the gold standard diagnosis of each participant.

The present study supports the validity of the CARS2-ST as a screening test compared with the ADOS-2. Our findings also demonstrated that autism diagnosis using the original CARS2-ST cutoff score was congruent with the autism diagnosis on the ADOS-2 (Modules 1 and 2) and further suggested an optimal CARS2-ST cutoff score of 28.5 to screen for a wider spectrum of the disorder. We expect that this newly derived cutoff may contribute to the early detection and intervention of children with ASD, especially in community institutions where it is difficult to utilize the ADOS-2, thus improving the prognosis of patients and reducing the burden on their families and society in the long run.

The data may be available uponrequest after approval from the Institutional Review Board.

The authors have no potential conflicts of interest to disclose.

Conceptualization: all authors. Data curation: Seong-In Ji. Formal analysis: Seong-In Ji. Investigation: all authors. Methodology: all authors. Supervision: Soon-Beom Hong. Writing—original draft: Seong-In Ji. Writing—review & editing: Seong-In Ji, Sun Ah Yoon, Soon-Beom Hong.

None