Because infants and toddlers’ development differs from that of adults and is interdependent in each area, when development delays are evident, difficulties in each area may overlap [1]. When developmental delays in one area affect others, symptoms and signs become more convoluted, and treatment becomes challenging. Another characteristic of the infant development problem is that it is a continuous series of processes and dynamics, rather than a discrete event [2]. Some infants and toddlers develop swiftly at birth but later fail to acquire additional developmental abilities and are delayed, and vice versa. Although the features of development areas are interrelated, one area’s development does not guarantee or predict the development of another. Therefore, it is vital to review a wide variety of developmental areas on a regular basis over time [3].

The Ministry of Health and Welfare, Republic of Korea launched the National Health Screening Program for Infants and Children (NHSPIC) in 2007. The Korean Developmental Screening Test for Infants and Children (K-DST) was developed under the NHSPIC by researchers directed by pediatricians to assess infants and toddlers in Korea [4]. The KDST was developed in 2014 with support from Centers for Disease Control and Prevention in Korea, and with the collaboration of experts in related fields, the Korean Pediatric Society, the Korean Society of Pediatric Rehabilitation and Developmental Medicine, the Korean Academy of Child and Adolescent Psychiatry, and the Korean Psychological Association. Although the K-DST has the advantage of being tailored to the developmental features of infants and toddlers, it has limitations in that it cannot access all comorbid disorders, particularly socioemotional difficulties that go beyond physical development. This is because when developmental delays in mental health, such as cognition, language, and sociability, are found, parents are more concerned than when physical problems, such as height and weight, are detected [5].

Excessive pathological or medical considerations, such as normal mutations, individual differences, temporary changes, and relationship problems with babies and caregivers, are insufficient in diagnosing and classifying mental health in infants and toddlers [6]. To differentiate these variations, an approach focusing on family-centered preventive interventions based on developmental, environmental, and relationship- oriented data is needed [7]. It is important to approach and evaluate the mental health of infants and toddlers, in contrast to the psychiatric diagnostic technique used in adults. For example, infant development should not only focus on the impairment of children’s functions and relationships, but also on the impairment of family functions and relationships. By assessing the environmental aspects of infants and toddlers, it can help to comprehend the overall condition. In particular, evaluation of risk variables that can have a detrimental impact on infants and toddlers allows for the prediction of unfavorable consequences [8]. In addition to risk factors, assessing a child’s competence or creating a healthy parentchild relationship might be valuable resources for establishing a coping strategy in the event of a future impairment.

In this regard, the authors developed a comprehensive and practical screening tool for neurodevelopmental delay, as well as assessing its validity.

Tool development

Multidisciplinary specialists, including child psychiatrists, child developmental psychologists, language pathologists, developmental education specialists, and statisticians, formed a team as part of the Korea Mental Health R&D Project to develop a new scale to assess neurodevelopmental delay and comorbidities in infants, toddlers, and young children under the age of seven. To establish the practical contents required for the new scale, we conducted focus group interviews (FGI) with 12 experts [5]. Based on FGI results, authors made a draft version of the new scale. We had sent a draft with Delphi questionnaires to 77 specialists and got feedback from 36 replies. The authors had monthly seminars and two interim public hearings with translational experts over the research period of 2015–2017 to verify the work’s performance.

The Infant Comprehensive Evaluation for Neurodevelopmental Delay (ICEND) developed two versions, one for children aged 12–36 months and the other for children 37–71 months (Table 1). The ICEND comprises three parts. Part 1 includes risk factors, such as prematurity, low birth weight, domestic violence with child abuse, maternal depression, and maternal alcohol abuse. Competence and parent-child relations are two resilience components discussed in Part 2. Part 3 presents clinical symptoms and signs with eight subscales that include cognition, language, autism spectrum disorder, attention-deficit/hyperactivity disorder, internalizing symptoms, externalizing symptoms, trauma, attachment issues, maladjustment, and others. Part 1 and Part 2 each feature 41 questionnaires for both age groups, which caretakers can report. Part 3 contains 98 questionnaires for children aged 12– 36 months and 114 for children aged 37–71 months. Part 3 questionnaires will be graded by raters who were trained for the execution of the ICEND.

Table 1

Composition and number of questions of ICEND depending on the age of the subjects and raters

Rater	Part and subscales	12-36 months	37-71 months
Caretaker	Part 1. Risk factors
	Prematurity and low birth weight	10	10
	Domestic violence	7	7
	Maternal depression	10	10
	Maternal alcohol abuse	4	4
	Part 2. Resilience factors
	Competency	5	5
	Parent-child relationship	5	5
	Subtotal (Part 1+Part 2)	41	41
Trained raters for ICEND	Part 3. Clinical subscales
	Cognition	15	15
	Language	16	16
	Autism spectrum disorder	13	13
	Movement symptoms	0	8
	Attention-deficit/hyperactivity disorder	5	10
	Internalizing symptoms	14	14
	Externalizing symptoms	9	12
	Post-traumatic stress disorder	5	5
	Attachment issues	5	5
	Maladjustment and other behaviors	16	16
	Subtotal	98	114
Total		139	155

ICEND, Infant Comprehensive Evaluation for Neurodevelopmental Delay

Study participants

Subjects were recruited from a variety of institutions (including hospitals such as Kyung Hee University Hospital, Myongji Hospital, Seoul Metropolitan Children’s Hospital, and Sewon Infant Child Development Center) between February 2017 and March 2019. Those whose primary caregiver(s) voluntarily signed the consent form were included, once the study was explained to them. A total of 296 people participated in this research. The individuals’ primary caregiver(s) completed Part A, and the researcher assessed Part B with them. Caregivers also completed Korean version of the Child Behavior Checklist (K-CBCL), Korean version of Strengths and Difficulties Questionnaire (SDQ-K), Korean version of Infant-Toddler Social Emotional Assessment (ITSEA-K), and K-DST for the validity test with ICEND. Correlation between ICEND and ITSEA-K was investigated with only the aged 12–36 month.

Measures

Korean version of the Child Behavior Checklist for ages 1.5–5 (K-CBCL 1.5–5)

The CBCL 1.5–5 [9] is a 100-item assessment tool completed by a child’s primary caregiver. It evaluates seven subareas: emotional responsiveness, anxiety/depression, somatic complaints, withdrawal behavior, sleep problems, attention problems, and aggressive behavior, as well as two factor scores of internalizing and externalizing problems, and a cumulative behavioral problems score, based on the frequency of behaviors over the past two months. The Korean version has been translated and standardized [10]. In terms of Cronbach’s α, the internal consistency and reliability of the K-CBCL 1.5–5 between syndrome indices and total score internalizing and externalizing was 0.56–0.94, while the Cronbach’s α for the total score encompassing all items was 0.94.

Korean version of Strengths and Difficulties Questionnaire (SDQ-K)

The SDQ, which Goodman developed in 1997 [11], was standardized in Korean by Ahn et al. in 2002 [12], and its reliability and validity were verified. It comprises four difficulty subscales (emotional symptoms, conduct problems, hyperactivity- inattention, and peer problems) each with five items, and one strength subscale (prosocial behavior). It is graded on a 3-point scale, with subscale scores determined by adding the scores on relevant items (range 0–10). The total difficulty score can be calculated by adding the scores from the four subscales of difficulty (range 0–40).

Korean version of Infant-Toddler Social Emotional Assessment (ITSEA-K)

The ITSEA is a 169-item assessment tool with four behavioral domains: externalizing, internalizing, dysregulation, and competence [13]. The ITSEA items were evaluated on a scale of 0 (not true/rarely), 1 (somewhat true/occasionally), and 2 (very true/often). It required approximately 30 minutes to complete. The Korean version was found to be reliable and valid [14].

Korean Developmental Screening Test for Infants and Children (K-DST)

The K-DST is a Korean-developed nationwide screening instrument with good reliability and validity [4]. There are eight questions in each of the six domains: gross motor, fine motor, cognition, language, sociality, and self-care. Each question is scored on a scale of 0–3, with a maximum attainable score of 24 points for each domain. It is completed by caregivers and is indicated for infants and children aged 4–71 months.

Statistical analysis

SPSS 23.0 (IBM Corp., Armonk, NY, USA) was used to conduct the analyses. The internal consistency of the ICEND scale was assessed using Cronbach’s alpha. An alpha of 0.70 or higher is considered acceptable. Concurrent validity was evaluated using Pearson correlations between ICEND and CBCL scale scores, ICEND and SDQ, and ICEND and K-DST. Concurrent validity is hypothesized to be expressed in large positive correlations between the ICEND scales and the CBCL, SDQ, and K-SDT. We looked for the best cut-off point with the maximum area under the curve (AUC) and sensitivity/ specificity.

Research ethics

The Institutional Review Boards at the institutes from which participants were recruited, including Kyung Hee University Hospital, approved the study’s protocol (KHU 2016-11-071). The study was conducted in compliance with the Helsinki II Declaration’s ethical requirements for informed consent, voluntariness, and anonymity.

Demographic characteristics

A total of 108 people from the 12–36 months age group and 188 people from the 37–71 months group participated in the study (Table 2). The proportion of high-risk patients in the 12–36 months group was 13% (n=14), whereas it was 23.1% (n=43) in the 37–71 months group (Supplementary Table 1 in the online-only Data Supplement).

Table 2

The demographic characteristics of participants and caregivers depending on the age

Characteristics	12-36 months (n=108)	37-71 months (n=188)
Sex
Male	60 (55.6)	96 (51.1)
Female	48 (44.4)	92 (48.9)
Parental status
Natural pregnancy	91 (91.9)	169 (91.4)
Vitro baby	8 (8.1)	14 (7.6)
Adoption	0 (0)	2 (1.1)

Data are presented as n (%).

Validity

The correlations between CBCL, SDQ, K-DST, and ITSEA (only 12–36 months) were investigated to determine the ICEND’s concurrent validity. The results are presented in Tables 3 and 4. Although several subdomains of ICEND (P1- A, P1-B, P1-D, P3-G, and P3-H of 12–36 months, and P1-A and P2-B of 37–71 months) did not demonstrate statistical significance, the overall correlations between all measures were significant. Person coefficients of 0.31 (P1-A), 0.29 (P2- A), 0.30 (P3-A), 0.21 (P3-B), 0.24 (P3-F), 0.26 (P3-G), and 0.24 (P3-J) were found to be positive correlations between the ICEND 12–36 months scale and the CBCL. Person coefficients of 0.24 (P1-C), 0.35 (P3-A), 0.47 (P3-B), 0.39 (P3-C), 0.19 (P3- D), 0.41 (P3-E), 0.51 (P3-F), 0.38 (P3-G), and 0.41 (P3-J) were found to be positive correlations between the ICEND 37–71 months scale and the CBCL. Similar patterns of correlations between ICEND and SDQ or ICEND and K-DST were found for the 12–36 months and 37–71 months subgroups.

Table 3

The validity of ICEND for the aged 12-36 months

	P1-A	P1-B	P1-C	P1-D	P2-A	P2-B	P3-A	P3-B	P3-C	P3-E	P3-F	P3-G	P3-H	P3-I	P3-J
CBCL	-0.00	0.11	0.31**	-0.02	0.29**	0.10	0.30**	0.21*	0.16	0.18	0.24*	0.26*	-0.06	0.08	0.24*
SDQ (2-4 years old)	0.24	0.10	0.23	0.04	-0.12	-0.13	-0.23	-0.21	-0.22	-0.09	0.01	0.14		-0.20	-0.30*
ITSEA			-0.33	-0.20	-0.25	-0.18	-0.40	-0.68	-0.19	-0.31	-0.34	-0.01		-0.17	-0.30
K-DST (12-13 months)
K-DST onths)
K-DST (16-17 months)
K-DST (18-19 months)			0.34	0.09	-0.19	0.39	0.29	0.54	0.40	0.02	-0.61	0.11		0.33	-0.54
K-DST (20-21 months)			-0.91	0.97	-0.09		-0.91	0.82		-1.00*	-0.91	-0.85		1.00	-0.99
K-DST (22-23 months)		1.00	-0.11	0.03	-0.85***	-0.67*	-0.78**	-0.92***	-0.67*	-0.70*	-0.29	-0.41		-0.54	-0.26
K-DST (24-26 months)		-0.07	-0.86**	-0.47	-0.92***	-0.09	-0.92***	-0.53	-0.77*	-0.77*	-0.32	-0.53	-0.77	-0.22	-0.36
K-DST (27-29 months)	-0.87		-0.19	0.42	-0.76*	-0.14	-0.66*	-0.80**	-0.54	-0.68*	-0.33	-0.12		-0.54	-0.15
K-DST (30-32 months)		-0.18	-0.47	0.53	-0.33	-0.58	-0.59	-0.61*	-0.54	-0.60	-0.50	-0.10		-0.06	-0.54
K-DST (33-35 months)	-0.40	0.26	0.16	-0.29	-0.88***	-0.44	-0.74**	-0.86***	-0.65*	-0.34	-0.64*	-0.26		0.22	-0.62*
K-DST (36-41 months)			0.83	-1.00	-0.99*	-0.90	-0.99*	-0.99*	-0.99*	-0.83	0.56	-0.01		-1.00	-0.83

*p<0.05; **p<0.01; ***p<0.001. ICEND, Infant Comprehensive Evaluation Neurodevelopmental Delay; CBCL, Child Behavior Checklist; SDQ, Strengths and Difficulties Questionnaire; ITSEA, Infant-Toddler Social Emotional Assessment; K-DST, Korean Developmental Screening Test for Infants and Children; P1-A, prematurity and low birth weight; P1-B, domestic violence; P1-C, maternal depression; P1-D, maternal alcohol abuse; P2-A, competency; P2-B, parent-child relationship; P3-A, cognition; P3-B, language; P3-C, autism spectrum disorder; P3-E, attention-deficit/hyperactivity disorder; P3-F, internalizing symptoms; P3-G, externalizing symptoms; P3-H, post-traumatic stress disorder; P3-I, attachment issues; P3-J, maladjustment and other behaviors

Table 4

The validity of ICEND for the aged 37-71 months

	P1-A		P1-B	P1-C	P1-D	P2-A	P2-B	P3-A	P3-B	P3-C	P3-D	P3-E	P3-F	P3-G		P3-H	P3-I	P3-J
CBCL	-0.25		-0.19	0.24**	-0.03	-0.47***	-0.40***	0.35***	0.47***	0.39***	0.19*	0.41***	0.51***	0.38***		0.003	0.02	0.41***
SDQ (4-17 years old)	0.24		-0.02	-0.36***	0.004	-0.27**	-0.07	-0.39***	-0.38***	-0.17*	-0.07	-0.13	0.29***	0.16		0.10	-0.02	-0.07
K-DST (36-41 months)	-0.13		0.19	-0.19	-0.12	-0.45	-0.15	-0.52*	-0.56*	-0.28	-0.17	-0.24	-0.19	0.24			0.12	-0.35
K-DST (42-47 months)	-0.05		0.13	-0.06	-0.03	-0.51*	-0.66**	-0.81***	-0.89***	-0.74**	-0.58**	-0.55*	-0.17	-0.21		-0.49	-0.25	-0.42
K-DST (48-53 months)	0.67*		0.47	0.10	0.03	-0.55**	-0.55**	-0.87***	-0.92***	-0.70***	-0.38*	-0.71***	-0.39*	-0.16		-0.22	0.15	-0.45*
K-DST (54-59 months)	-0.28		0.47	-0.15	-0.82**	-0.62*	-0.52	-0.92***	-0.94***	-0.49	-0.32	-0.56	-0.03	-0.09			0.47	-0.30
K-DST (60-65 months)	-0.14		-0.01	0.08	0.04	0.63**	-0.12	-0.82***	-0.86***	-0.72***	-0.82***	-0.54**	-0.52*	-0.07		0.13	-0.10*	-0.13*
K-DST (66-71 months)	-0.26		-0.18	-0.37	-0.22	-0.64**	-0.60**	-0.86***	-0.88***	-0.72***	-0.47*	-0.57*	-0.64**	-0.16		-0.58**	-0.25**	-0.56**

*p<0.05; **p<0.01; ***p<0.001. ICEND, Infant Comprehensive Evaluation Neurodevelopmental Delay; CBCL, Child Behavior Checklist; SDQ, Strengths and Difficulties Questionnaire; K-DST, Korean Developmental Screening Test for Infants and Children; P1-A, prematurity and low birth weight; P1-B, domestic violence; P1-C, maternal depression; P1-D, maternal alcohol abuse; P2-A, competency; P2-B, parent-child relationship; P3-A, cognition; P3-B, language; P3-C, autism spectrum disorder; P3-D, movement disorder; P3-E, attention-deficit/ hyperactivity disorder; P3-F, internalizing symptoms; P3-G, externalizing symptoms; P3-H, post-traumatic stress disorder, P3-I, attachment issues; P3-J, maladjustment and other behaviors

Reliability

Domain internal consistencies were calculated for each age group 12–36 months and 37–71 months (Table 5). The Cronbach’s alpha correlation coefficients of ICEND 12–36 months varied from 0.533 to 0.928 except for P1-A -0.271, while those of ICEND 37–71 months ranged from 0.334 to 0.951 (Table 5), and were well within Nunnally and Bernstein’s acceptable range [15].

Table 5

The reliability of ICEND

Ages	Part	Subscales	Cronbach’s α
12-36 months	Risk factors	Prematurity	-0.271
		Domestic violence	0.662
		Maternal depression	0.796
		Maternal alcohol abuse	0.451
	Resilience factors	Competency	0.831
		Parent-child relationship	0.662
	Clinical subscales	Cognition	0.864
		Language	0.917
		ASD	0.770
		ADHD	0.698
		Internalizing symptoms	0.778
		Externalizing symptoms	0.771
		PTSD	0.928
		Attachment issues	0.533
		MAL	0.568
37-71 months	Risk factors	Prematurity	0.114
		Domestic violence	0.720
		Maternal depression	0.798
		Maternal alcohol abuse	0.546
	Resilience factors	Competency	0.872
		Parent-child relationship	0.724
	Clinical subscales	Cognition	0.896
		Language	0.951
		ASD	0.884
		Movement symptoms	0.538
		ADHD	0.786
		Internalizing symptoms	0.670
		Externalizing symptoms	0.734
		PTSD	0.453
		Attachment issues	0.334
		MAL	0.643

ICEND, Infant Comprehensive Evaluation Nneurodevelopmental Delay; ASD, autism spectrum disorder; ADHD, attention-deficit/hyperactivity disorder; PTSD, post-traumatic stress disorder; MAL, maladjustment and other behaviors

The detection ability of ICEND

As indicated in Table 6, receiver operating characteristics (ROC) analysis was used to distinguish young children with a high risk of developmental delay. The cut-off, sensitivity, and specificity for each domain are presented in Table 6. According to the results of ROC analysis, the cutoff numbers for ICEND 12–36 months ranged from 0.5 to 3.5, and ICEND 37–71 months ranged from 0.5 to 4.5. Both ICEND 12–36 months and 37–71 months demonstrated satisfactory levels of sensitivity with a range of 2.3%–81.4% and satisfactory levels of specificity with a range of 68.8%–100% using this new criterion.

Table 6

The AUC, cut-off, sensitivity and specificity results of ICEND depending on the age

Ages	Part	Subscales	AUC	Cut-off	Sensitivity	Specificity
12-36 months	Resilience factors	P2-A	0.832	3.5	0.071	0.968
		P2-B	0.802	2.5	0.500	0.883
	Clinical subscales	P3-A	0.727	2.5	0.500	0.742
		P3-B	0.857	3.5	0.786	0.772
		P3-C	0.827	1.5	0.786	0.849
		P3-E	0.792	1.5	0.714	0.753
		P3-F	0.741	1.5	0.643	0.688
		P3-G	0.780	1.5	0.714	0.731
		P3-H	0.609	0.5	0.250	0.967
		P3-I	0.738	0.5	0.727	0.708
		P3-J	0.720	2.5	0.714	0.688
37-71 months	Resilience factors	P2-A	0.737	4.5	0.023	1.000
		P2-B	0.832	3.5	0.116	0.972
	Clinical subscales	P3-A	0.737	0.5	0.721	0.699
		P3-B	0.845	4.5	0.814	0.832
		P3-C	0.837	1.5	0.744	0.874
		P3-D	0.701	1.5	0.488	0.825
		P3-E	0.762	2.5	0.744	0.699
		P3-F	0.584	1.5	0.512	0.650
		P3-G	0.570	1.5	0.419	0.636
		P3-H	0.529	0.5	0.120	0.935
		P3-I	0.550	0.5	0.120	0.935
		P3-J	0.752	3.5	0.605	0.755

AUC, area under the curve; ICEND, Infant Comprehensive Evaluation Nneurodevelopmental Delay; P2-A, competency; P2-B, parent-child relationship; P3-A, cognition; P3-B, language; P3-C, autism spectrum disorder; P3-D, movement disorder; P3-E, attentiondeficit/ hyperactivity disorder; P3-F, internalizing symptoms; P3-G, externalizing symptoms; P3-H, post-traumatic stress disorder; P3-I, attachment issues; P3-J, maladjustment and other behaviors

This study aimed to investigate the psychometric properties of the ICEND, a recently designed screening tool for developmental delay. Validity and reliability were both favorable. The corresponding domain of the ICEND yielded satisfactory results. The AUC of 0.609–0.857 in each domain revealed superior detection ability in the ROC analysis. The cut-offs for each domain effectively distinguished the high-risk group with reasonable sensitivity and specificity.

The tools, CBCL, SDQ, ITSEA, and K-DST were used to evaluate early childhood the development, which included infancy, toddlers, and preschool. Because they have distinct qualities and applicable age groups, it is difficult to assess their validity correlations directly. While CBCL is a caregiver rating scale, clinical subscales of ICEND is rater-rating scale. As ICEND has clinical subscales and subscales for environmental factors, it is advantageous compared with symptomsoriented domains only of ITSEA. The K-DST is a screening tool for the assessment of the development for the infant, toddler, and young child, mostly physical health. The ICEND is expected to be employed as a complementary or secondary screening tool to the K-DST, which is currently used by the NHSPIC, after its psychometric properties are confirmed. In order to apply ICEND to larger group, it is necessary to have further trials for an equal group of children across the country. ICEND is also necessary to evaluate whether it is appropriate to divide the target group into only two age groups unlike K-DST, which subdivided the target age group into 2–3 months units.

This study had several critical limitations. First, the test-retest validity was not evaluated. It should be conducted again in the near future to ensure the tool’s reliability. Second, although the age and gender distributions were matched, the sample size of 296 was insufficient for analysis.

Despite the abovementioned limitations, it is the first tool developed independently in Korea to try to distinguish highrisk groups for developmental delay, considering various mental disorders in infancy and early childhood as defined by ICD-10, DSM-5, and Diagnostic Classification of Mental Health and Developmental Disorders of Infancy and Early Childhood (DC: 0–5) [16]. This was reflected in a panel of multidisciplinary experts’ discussion of clinical subscales. It also included information on potential risk factors that could affect development, such as prematurity, child abuse, maternal depression, or alcohol abuse, as well as potential strength factors that could operate as protective factors, such as parental relationships or competency.

In summary, the ICEND exhibits good psychometric characteristics, including reliability, validity, and detection ability, according to this study. The ICEND is a valuable secondary screening tool to detect high-risk developmental delay groups in kindergarten, center, or clinical settings.

The datasets generated or analyzed during the study are not publicly available due to containing information that could compromise research participant privacy/consent but are available from the corresponding author on reasonable request.

The authors have no potential conflicts of interest to disclose.

Conceptualization: Minha Hong, Geon Ho Bahn. Data curation: Kyung- Sook Lee, Jin-Ah Park, Duk-Soo Moon. Formal analysis: Young Il Cho, Yong Woo Shin. Funding acquisition: Geon Ho Bahn. Investigation: Kyung-Sook Lee, Jin-Ah Park, Duk-Soo Moon. Methodology: Young Il Cho, Yong Woo Shin. Project administration: Ji-Yeon Kang, Seongwoo Cho, Ram Hwangbo, Seung Yup Lee. Writing—original draft: Minha Hong, Geon Ho Bahn. Writing—review & editing: Minha Hong, Geon Ho Bahn.

The Republic of Korea’s Korea Mental Health R&D Project funded this research (Grant No. HM15C1084). We also thank Dr. Kyung Hoon Suh for his contribution to project management.