Clinical Utility of the Minnesota Multiphasic Personality Inventory-Adolescent Restructured Form in the Assessment of Internalizing and Externalizing Disorders in Adolescents: A Preliminary Approach

Hye Ji Yun; Eun Hee Park; Hyun Ju Hong

doi:10.5765/jkacap.230039

J Korean Acad Child Adolesc Psychiatry 2023; 34(4): 268-274

Published online October 1, 2023

Hye Ji Yun, Eun Hee Park, and Hyun Ju Hong

Department of Psychiatry, Hallym University Sacred Heart Hospital, Hallym University College of Medicine, Anyang, Korea

Correspondence to: Eun Hee Park, Department of Psychiatry, Hallym University Sacred Heart Hospital, Hallym University College of Medicine, 170 Gwanpyeong-ro, Dongan-gu, Anyang 14068, Korea
Tel: +82-31-380-1854, Fax: +82-31-381-3753, E-mail: psypeh@hanmail.net

Received June 13, 2023; Revised August 20, 2023; Accepted August 26, 2023.

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (https://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract: Objectives: This study investigated whether the Minnesota Multiphasic Personality Inventory-Adolescent Restructured Form (MMPI-A-RF) can differentiate between two groups of adolescents, one diagnosed with internalizing disorders and another with externalizing disorders, and examined the clinical utility of the MMPI-A-RF by examining which subscales can significantly discriminate between these two groups.
Methods: A total of 105 adolescents aged 13–18 years completed the MMPI-A-RF (53 internalizing disorder and 52 externalizing disorder groups). Independent t-test, chi-square test (χ²), and discriminant analysis were used to examine whether MMPI-A-RF can distinguish between the two groups.
Results: Sixteen MMPI-A-RF scales best predicted differences between the groups with internalizing and externalizing disorders. Fourteen scales (Higher-Order Scale [Emotional/Internalizing Dysfunction], Restructured Clinical [RC] Scale [RC demoralization, Somatic Complaints (RC1), and Low Positive Emotions (RC2)], Personality Psychopathology Five Scale [Introversion/Low Positive Emotionality-Revised, Negative Emotionality/Neuroticism-Revised], Somatic/Cognitive Scale [Malaise, Head Pain Complaints, and Gastrointestinal Complaints], Internalizing Scale [Stress/Worry, Self-Doubt], Externalizing Scale [Negative School Attitudes], Interpersonal Scale [Social Avoidance, Shyness]) were associated with the internalizing disorder group, whereas two scales (Externalizing Scale [Conduct Problems, Negative Peer Influence]) were associated with the externalizing disorder group.
Conclusion: The MMPI-A-RF can be an efficient assessment tool for a quick diagnosis as it can classify individuals with internalizing and externalizing disorders in clinical settings that lack a variety of assessment tools for children and adolescents.; Keywords : Adolescent; MMPI-A-RF; Internalizing; Externalizing; Discriminant analysis

INTRODUCTION

Adolescence is a challenging transitional stage which involves significant physical, emotional, moral, and social development [1]. As problematic adolescent behavior is likely to continue through adulthood, identifying such behavior and providing appropriate therapeutic intervention before it worsens is necessary. Therefore, evaluating psychological and behavioral problems in adolescents is of great importance.

According to Achenbach and Edelbrock [2], adolescents exhibit problematic behavior when they experience difficulties in maintaining healthy adaptation. The internalizing and externalizing model for understanding psychiatric disorders has a long legacy in adolescent psychology [3], as explained in Achenbach’s work [4] on classifying children’s psychiatric symptoms. This theoretical framework defines internalization as the propensity to express distress inwards and externalization as the propensity to express it outwards [5]. Internalizing problems refer to overcontrolled behaviors such as anxiety, depression, physical symptoms, and social withdrawal [6,7]. Individuals experiencing internalizing problems could face more difficulties than expected in interpersonal relationships and daily life, and tend to be negatively affected by the shape of their personality [8]. In contrast, externalizing problem behavior, which can be easily identified by others, is caused by insufficient control of emotions or behaviors, including distractions, hyperactivity, aggressiveness, impulsiveness, and non-adaptive behaviors, such as harming or social rule violations [6,7].

Because these two categories encompass most of the problems experienced by individuals, internalizing and externalizing classifications have been used as better indicators of psychopathology than detailed diagnoses [9]. Although the concepts of internalizing and externalizing disorders have been used less often, this classification still has clinical value in that it helps in the formation of an overall diagnostic impression of patients before the detailed diagnostic procedure [10].

Among the various assessments developed for psychiatric diagnostic classification, the Minnesota Multiphasic Personality Inventory-Adolescent (MMPI-A) [11,12] is one of the most widely used objective and comprehensive personality assessment tools for adolescents. It is used to evaluate adolescent psychological problems through the interpretation of overall profile elevation patterns and combinations of different scales, centering around the Clinical Scales. However, the MMPI-A, comprising 478 questions, has a disadvantage in that it is timeconsuming to administer for adolescents. To overcome this limitation, the MMPI-A-Restructured Form (RF) [13] was released as a shorter and more time-efficient measure, created by selecting questions that are more core and relevant to clinical problems [14]. Moreover, it is believed that the MMPI-ARF, developed for a clear interpretation based on the hierarchical structure of psychopathology, can clearly classify the psychological characteristics of internalizing and externalizing disorders. However, as the MMPI-A-RF is newly released, only a few studies have provided empirical evidence on its clinical utility, and no studies have explored whether the MMPI-A-RF can effectively distinguish the psychological characteristics of these clinical groups. Therefore, this study aimed to examine whether the MMPI-A-RF can differentiate between internalizing and externalizing disorders. Furthermore, we examined the clinical utility of the MMPI-A-RF by examining its subscales that could significantly discriminate between these two groups.

METHODS

Participants

The study population consisted of adolescents aged between 13 and 18 years who visited or were admitted to the Department of Psychiatry at a University Hospital in South Korea. All patients were diagnosed by a board-certified child and adolescent psychiatrist according to the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition [15]. The participants were divided into two groups, one with internalizing disorders and another with externalizing disorders, according to the main symptoms based on the factor analysis of Achenbach and Edelbrock [16]. We conducted a retrospective medical record review of 115 participants from March 2014 to February 2019; however, data of only 105 participants were included in the study after excluding participants who met the following criteria: having a combination of internalizing and externalizing disorders; having a history of developmental problems such as autism spectrum disorder, social communication disorders, intellectual disabilities, schizophrenia spectrum or other psychotic disorders, bipolar or related disorders, or organic brain diseases; a full-scale intellectual quotient <70; and the validity criteria of Cannot Say items <15, Variable Response Inconsistency (VRIN), True Response Inconsistency (TRIN) T scores <80, and F (infrequency) T score <100 [14]. Of the entire group of participants, 93 were outpatients (internalizing disorders: 44, externalizing disorders: 49) and 12 were inpatients (internalizing disorders: 9, externalizing disorders: 3).

The internalizing disorder group comprised 53 participants (boys=20, girls=33), whereas the externalizing disorder group comprised 52 participants (boys=35, girls=17). Internalizing disorders included major depressive disorder, persistent depressive disorder, other specified depressive disorder, adjustment disorder with mixed anxiety and depressed mood, generalized anxiety disorder, obsessive-compulsive disorder, panic disorder, other specified anxiety disorder, and body dysmorphic disorder. Externalizing disorders included attention-deficit hyperactivity disorder (ADHD), oppositional defiant disorder, conduct disorder, and intermittent explosive disorder. This study was approved by the Hallym University Institutional Review Board (IRB) (IRB No. 2019-05-028).

Measurements

MMPI-A-RF

The MMPI-A-RF was published in 2016 in the United States [13] and in 2018 in South Korea [17]. It can be administered quickly as it comprises only 241 questions, reduced from the 478-item MMPI-A, and focuses on more critical areas [18]. The Validity, Higher-Order (H-O), Restructured Clinical (RC), and Specific Problem (SP) scales were included, each of which classified psychological problems at different levels. In addition to revising the existing VRIN, TRIN, F, L, and K scales, a Combined Response Inconsistency Measure scale was added to measure the overall response inconsistency. MMPI- A-RF uses the T score of 60 as a cutoff to identify clinically significant elevations, unlike the cutoff score of T ≥65 in MMPI-A [13].

According to Archer et al. [13], MMPI-A-RF has five major domains, among which the Somatic/Cognitive Dysfunction, Emotional Dysfunction, and Interpersonal Functioning domains are particularly closely associated with the internalizing problem. The scales related to each domain were as follows: 1) Somatic/Cognitive Dysfunction: RC1, Malaise (MLS), Gastrointestinal Complaints (GIC), Head Pain Complaints (HPC), Neurological Complaints, and Cognitive Complaints; 2) Emotional Dysfunction: Emotional/Internalizing Dysfunction (EID), Restructured Clinical Scale demoralization (RCd), Helplessness, Self-Doubt (SFD), Inefficacy, RC2, Introversion/ Low Positive Emotionality-Revised (INTR-r), RC7, Obsessions/ Compulsions, Stress/Worry (STW), Anxiety, Anger Proneness, Behavior-Restricting Fears, Specific Fears, and Negative Emotionality/Neuroticism-Revised (NEGE-r); and 3) Interpersonal Functioning: Family Problems, RC3, Interpersonal Passivity, Social Avoidance (SAV), Shyness (SHY), and Disaffiliativeness. In contrast, the Behavioral Dysfunction domain is related to externalizing disorders and contains the following scales: Behavioral/Externalizing Dysfunction (BXD), RC4, Negative School Attitudes (NSA), Antisocial Attitudes, Conduct Problems (CNP), Substance Abuse (SUB), Negative Peer Influence (NPI), RC9, Aggression (AGG), Aggression-Revised (AGGR-r), and Disconstraint- Revised (DISC-r) [13].

Data analysis

We conducted an independent t-test and chi-square test (χ²) to identify the differences in the demographic characteristics between the two groups. Discriminant analysis determines groups of new objects that are classified using functions created from existing data. Therefore, we performed discriminant analysis to determine whether the MMPI-A-RF was useful for distinguishing between the internalizing and externalizing disorder groups and to determine the scales which are the most effective predictors of distinguishing these groups. Finally, an independent t-test was conducted on the major discriminant scales derived through the discriminant analysis to determine whether these scales showed a clear difference between the internalizing and externalizing disorder groups. All statistical analyses were conducted using IBM SPSS version 27.0 (IBM Corp., Armonk, NY, USA).

RESULTS

The data on the major demographic variables of the two groups are presented in Table 1. Overall, 55 boys (52.4%) and 50 girls (47.6%) participated in this study. Their average age was 15.7 years. There was a significant difference in age (t (103)= 40.65, p<0.01) and sex (χ²=9.20, p<0.01) between the two groups. Participants in the internalizing disorder group were significantly older (mean [M]=16.8, standard deviation [SD]=0.9) than those in the externalizing disorder group (M=14.6, SD= 1.8). In terms of sex, the internalizing disorder group had a higher proportion of girls (62.3%) and the externalizing disorder group had a higher proportion of boys (67.3%).

Table 1

Demographic characteristics of internalizing disorder group and externalizing disorder group

	Internalizing group (n=53)	Externalizing group (n=52)	Total (n=105)	χ²/t
Sex
Boy	20 (37.7)	35 (67.3)	55 (52.4)	9.20**
Age (yr)	16.8±0.9	14.6±1.8	15.7±1.8	40.65**

Values are presented as mean±standard deviation or number (%). **p<0.01

We performed seven discriminant analyses, by choosing “enter independents together” in the IBM SPSS statistics software, to determine whether seven variables (the H-O, RC, Personality Psychopathology Five [PSY-5], Somatic/Cognitive, Internalizing, Externalizing, and Interpersonal scales) correctly discriminated the two groups. As shown in Table 2, discriminant analyses of the seven MMPI-A-RF scale groups revealed that all scale groups were significantly classified into two groups: the internalizing disorder group and the externalizing disorder group (p<0.01). The results of squaring the canonical correlation coefficient to examine the power of each discriminant model in the ascending order were as follows: RC Scale, 43.2%; PSY-5 Scale, 36.2%; Somatic/Cognitive Scale, 36.2%; Internalizing Scale, 33.6%; H-O scale, 28.7%; Interpersonal Scale, 24.6%; and Externalizing Scale, 17.3%.

Table 2

Discriminant function analysis by MMPI-A-RF

Scale	Eigenvalue	Canonical correlation	Wilks’ lambda	χ²	df
Higher-Order	0.41	0.54	0.71	34.50***	3
Restructured Clinical	0.76	0.66	0.57	55.59***	9
PSY-5	0.57	0.60	0.64	45.15***	5
Somatic/Cognitive	0.57	0.60	0.64	45.31***	5
Internalizing	0.51	0.58	0.66	40.41***	9
Externalizing	0.21	0.42	0.83	19.13**	6
Interpersonal	0.33	0.50	0.75	28.55***	4

**p<0.01; ***p<0.001. MMPI-A-RF, Minnesota Multiphasic Personality Inventory-Adolescent-Restructured Form; PSY-5, Personality Psychopathology Five

To specifically examine the psychological characteristics that differentiate between internalizing and externalizing disorders, the relative importance of the subscale groups belonging to the seven scale groups and the results of the classification prediction of discriminatory models were examined. To evaluate the relative importance of the predictors, we described their relative importance based on the discriminant loading of the structural matrices. Discriminant loading shows the correlation between each variable and a standardized canonical discriminant function [19]. According to Chang et al. [20], the significance criterion of discriminant loading is defined as ≥±0.30. As shown in Table 3, the 16 MMPI- A-RF scales employed in the current study had discriminant loadings >0.3, indicating a significant influence in discriminating between the two groups.

Table 3

Correlation between significant subscales and discriminant function

Scale	Function 1
	<hr/>
	Discriminant coefficients	Discriminant loading	Wilks’ lambda
Higher-Order
EID	1.04	0.74	0.82***
Restructured Clinical
RCd	0.47	0.55	0.82***
RC1	0.77	0.53	0.82***
RC2	0.16	0.41	0.89***
PSY-5
INTR-r	0.60	0.74	0.76***
NEGE-r	0.79	0.61	0.83***
Somatic/Cognitive
MLS	0.93	0.83	0.72***
HPC	0.37	0.57	0.85***
GIC	0.26	0.52	0.87***
Internalizing
STW	0.82	0.64	0.83***
SFD	0.78	0.59	0.85***
Externalizing
CNP	0.62	0.65	0.92**
NSA	-0.46	-0.54	0.94*
NPI	0.33	0.43	0.96*
Interpersonal
SAV	0.72	0.74	0.85***
SHY	0.52	0.64	0.88***

*p<0.05; **p<0.01; ***p<0.001. CNP, Conduct Problems; EID, Emotional/ Internalizing Dysfunction; GIC, Gastrointestinal Complaints; INTR-r, Introversion/Low Positive Emotionality-Revised; HPC, Head Pain Complaints; MLS, Malaise; NEGE-r, Negative Emotionality/ Neuroticism-Revised; NPI, Negative Peer Influence; NSA, Negative School Attitudes; PSY-5, Personality Psychopathology Five; RC1, Somatic Complaints; RC2, Low Positive Emotions; RCd, Demoralization; SAV, Social Avoidance; SFD, Self-Doubt; SHY, Shyness; STW, Stress/Worry

Among the H-O Scales, only EID was found to be a significant factor in distinguishing the two groups (Table 3). The relative importance of the RC Scales in discriminating between the two groups decreased in the order of RCd, RC1, and RC2. The relative importance of the PSY-5 Scales in distinguishing between the two groups decreased in the order of INTR and NEGE. The relative importance of Somatic/ Cognitive Scales in discriminating between the two groups decreased in the order of MLS, HPC, and GIC. Among the Internalizing scales, the best discriminant variables for discriminating between the two groups were STW and SFD. The relative importance of the Externalizing Scales in distinguishing the two groups decreased in the following order CNP, NSA, and NPI. Finally, the relative importance of the Interpersonal scales in discriminating between the two groups was in the order of SAV and SHY. In short, the internalizing disorder group had higher EID, R Table 3. Correlation between significant subscales and discrimi- Cd, RC1, RC2, INTR, NEGE, MLS, HPC, GIC, STW, SFD, and NSA scores but lower CNP and NPI scores compared with the externalizing disorder group.

Table 4 presents the mean scores and t-test results of the seven major discriminant scales. The results of the t-tests showed a s significant difference between the two groups for all the analyzed scales (p<0.05).

Table 4

Mean and standard deviation of scales and subscales

Scale	Internalizing group (n=53)	Externalizing group (n=52)	t
Higher-Order
EID	64.45±11.15	52.92±13.36	4.80***
Restructured Clinical
RCd	63.92±10.53	52.98±12.62	4.83***
RC1	60.04±12.80	48.94±11.34	4.70***
RC2	63.30±12.51	55.08±10.55	3.64***
PSY-5
INTR-r	63.98±11.11	52.15±10.22	5.67***
NEGE-r	63.94±11.54	52.65±13.16	4.68***
Somatic/Cognitive
MLS	67.83±12.36	52.21±12.86	6.35***
HPC	58.58±11.95	49.42±9.46	4.35***
GIC	60.32±15.45	49.81±11.14	4.01***
Internalizing
STW	60.96±9.11	51.50±11.73	4.62***
SFD	62.72±11.65	52.56±12.62	4.29***
Externalizing
CNP	49.14±9.07	55.46±12.10	-3.01**
NSA	58.60±10.02	53.15±12.07	2.52*
NPI	47.45±8.01	51.21±11.10	-1.99*
Interpersonal
SAV	62.25±12.02	52.46±11.22	4.31***
SHY	57.66±13.55	48.77±10.74	3.72***

*p<0.05; **p<0.01; ***p<0.001. CNP, Conduct Problems; EID, Emotional/Internalizing Dysfunction; GIC, Gastrointestinal Complaints; HPC, Head Pain Complaints; INTR-r, Introversion/Low Positive Emotionality-Revised; MLS, Malaise; NEGE-r, Negative Emotionality/ Neuroticism-Revised; NPI, Negative Peer Influence; NSA, Negative School Attitudes; PSY-5, Personality Psychopathology Five; RC1, Somatic Complaints; RC2, Low Positive Emotions; RCd, Demoralization; SAV, Social Avoidance; SFD, Self-Doubt; SHY, Shyness; STW, Stress/Worry

Table 5 shows the classification accuracy of the discriminant function that significantly predicted the internalizing and externalizing disorder groups. All scales showed high discriminant hit ratios, between 66.7% and 81.9%, with the Internalizing Scale having the lowest and the RC Scale having the highest values.

Table 5

Classification accuracy in discriminant analysis

Scale	Group	Predicted group membership		Total
		<hr/>
		Internalizing	Externalizing
Higher-Order	Internalizing	37 (69.8)	16 (30.2)	72.4
	Externalizing	13 (25.0)	39 (75.0)
Restructured Clinical	Internalizing	42 (79.2)	11 (20.8)	81.9
	Externalizing	8 (15.4)	44 (84.6)
PSY-5	Internalizing	37 (69.8)	16 (30.2)	76.2
	Externalizing	9 (17.3)	43 (82.7)
Somatic/Cognitive	Internalizing	41 (77.4)	12 (22.6)	77.1
	Externalizing	12 (23.1)	49 (76.9)
Internalizing	Internalizing	36 (67.9)	17 (32.1)	66.7
	Externalizing	18 (34.6)	34 (65.4)
Externalizing	Internalizing	35 (66.0)	18 (34.0)	69.5
	Externalizing	14 (26.9)	38 (73.1)
Interpersonal	Internalizing	40 (75.5)	13 (24.5)	77.1
	Externalizing	11 (21.2)	41 (78.8)

Values are presented as number (%) or percentage. PSY-5, Personality Psychopathology Five

DISCUSSION

This study aimed to determine whether the MMPI-A-RF could distinguish between internalizing and externalizing disorders in adolescent psychiatric patients, using discriminatory analysis. According to our results, the MMPI-A-RF is clinically useful in classifying internalizing and externalizing disorders using a few scales which could significantly discriminate the two group. This study identified 16 scales of the MMPI-A-RF that best predicted the differences between the two groups.

Our study partially corroborates Archer’s [18] interpretation framework of MMPI-A-RF that the H-O Scale (EID), RC Scale (RCd, RC1, and RC2), PSY-5 Scale (INTR-r and NEGE-r), Somatic/Cognitive Scale (MLS, HPC, and GIC), Internalizing Scale (STW and SFD), Externalizing Scale (NSA), and Interpersonal Scale (SAV and SHY) significantly predicted internalizing disorder. Furthermore, our results are also consistent with previous literature primarily focusing on the RC Scales of MMPI-A-RF [21], in that RCd, RC2, and RC7 significantly reflected the characteristics of internalizing disorders, such as depression and anxiety disorders.

Regarding the scales associated with externalizing disorder, our study showed higher scores on the Externalizing Scales (CNP and NPI) in the externalizing group. Among the five main domains, the externalizing problem is related to the Behavioral Dysfunction domain of the MMPI-A-RF [22]. Furthermore, in a recent MMPI-A-RF study on ADHD, which is the most typical externalizing disorder, an ADHD group reported significantly different scores on scales indicating externalizing concerns and behavioral dysfunction compared with a non-ADHD group [22]. The findings of the present study, which showed significantly higher CNP and NPI scores in the externalizing disorder group, are consistent with those of Chakranarayan et al. [22], demonstrating that the BXD, RC4, CNP, NPI, and DISC-r scales have diagnostic relevance for ADHD. In another study by Fertitta [23], the externalizing disorder group showed significantly higher scores than the internalizing disorder group on BXD, Antisocial Attitudes, CNP, SUB, AGG, DISC-r, and AGGR-r.

The discriminatory hit ratios of the Internalizing and Externalizing Scales were <70%, which were smaller than expected. There are several possible explanations for these results. First, the Internalizing and Externalizing Scales consisted of a few items, and the two scales shared overlapping attributes. Therefore, they may have limitations in clearly distinguishing the psychological characteristics of internalizing and externalizing disorders. A previous study suggested the possibility of not properly reflecting the core and unique components of the scales due to their low construct validity [24]. In the same context, one possible reason for the higher scores for NSA in the Externalizing Scale in the internalizing disorder group than in the externalizing disorder group is that NSA may not validly reflect the unique attributes of externalizing disorders. Specifically, NSA includes items that assess test anxiety, poor study habits, and academic problems, which are usually caused by avoidance or unwillingness to attend school due to the various stressors experienced in school. These problems are likely to occur in children with externalizing disorders as well as internalizing disorders. According to Park and Chong [25], children’s test anxiety or academic stress increases as their internalizing problems increase, and lethargy derived from internalizing problems hinders the facilitation of relationships with friends or teachers [26]. Finally, this may be because the externalizing disorder group reported less externalizing problem behaviors as they were concerned about the stigma effect or secondary negative feedback and tried to appear more favorable to others compared with the internalizing disorder group [27]. The previous study by Kim [28] also identified that the externalizing disorder group showed higher scores on the K and L scales, reflecting attempts to look positive by denying or underreporting their problems compared with the internalizing disorder group.

The implications of this study are as follows. The most significant result of this study is that the MMPI-A-RF can serve as a tool for classifying internalizing and externalizing disorder groups. As the MMPI-A-RF was published only in 2016 in the United States and in 2018 in South Korea, research on the MMPI-A-RF is extremely scarce. Furthermore, to the best of our knowledge, no study has examined the ability of the various MMPI-A-RF scales to distinguish between internalizing and externalizing problems. Therefore, the current study is clinically valuable in the absence of previous studies in that it could serve as a cornerstone of future studies on the MMPI-A-RF.

The possibility of distinguishing the two groups using the MMPI-A has already been demonstrated in many studies; however, the same finding with the MMPI-A-RF may have a differentiated clinical value. First, the MMPI-A-RF has a cost-effective advantage in that it can produce useful results similar to the MMPI-A but with fewer questions. Furthermore, the MMPI-A-RF contains RC and SP scales, which can provide not only information on critical symptoms of psychopathology but also a more detailed understanding of the specific problems adolescents have [13]. There are some multidimensional clinical assessment tools, such as the Child Behavior Checklist; however, the MMPI-A-RF is the only adolescent self-report assessment tool with higher-level scales, which allows a clearer examination of internalizing and externalizing disorders. Finally, to better discover the specific and explicit features of each disorder, the current study excluded the mixed disorder group, which comprised participants who showed both internalizing and externalizing symptoms. In summary, despite including a small number of questions, the MMPI-A-RF can gather considerable information. Furthermore, the RC and SP scales, which only the MMPI-A-RF has, provide more discriminatory information, as they capture the distinctive core of internalizing and externalizing problems. Therefore, the MMPI-A-RF can be used as an additional assessment tool in current clinical practice where there are few tools to assess the psychological symptoms and behavioral problems of adolescents. Adolescence is a transitional period in the developmental process just before adulthood, and the psychological and behavioral problems of adolescents are likely to be complex and fluid. Therefore, making accurate diagnoses in adolescents is difficult. The MMPI-ARF allows for quick diagnosis and prompt intervention by classifying internalizing and externalizing disorders in a large framework.

This study has some limitations and suggestions for further research. First, all the participants were patients from a single medical center. Therefore, the generalizability of the results is limited, rendering the data and conclusions drawn uncertain until this study is expanded and replicated. Second, there was a sex difference between the two groups. However, to some extent, this reflects the ecological characteristics of actual clinical settings. Previous studies demonstrating sex differences in internalizing and externalizing disorders have also reported that girls experience internalizing symptoms more than externalizing symptoms, whereas boys experience externalizing symptoms more frequently [29]. Furthermore, in this study, the psychological symptoms and behavioral problems of adolescents were only classified into internalizing and externalizing disorders. Therefore, future studies examining the major clinical characteristics of each specific clinical group are recommended.

CONCLUSION: Despite some issues that need to be addressed in the future, the findings of this study provide important new evidence that the MMPI-A-RF can distinguish internalizing and externalizing disorder groups. The MMPI-A-RF is a cost-effective assessment tool that reflects core psychopathological features using fewer questions and scales. Therefore, it can be used efficiently in clinical settings, especially for adolescents, a population for which the clinical assessment tools are insufficient. The results of this study show that MMPI-A-RF helps understand and evaluate adolescents with internalizing and externalizing disorders, providing a foundation for the use of MMPI-A-RF as an additional assessment tool for adolescents, as well as being useful in determining the direction of their treatment.

Acknowledgments: This research was supported by Maumsarang, Co. LTD.

Availability of Data and Material: The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.

Conflicts of Interest: The authors have no potential conflicts of interest to disclose.

Author Contributions: Conceptualization: all authors. Data curation: all authors. Formal analysis: Hye Ji Yun, Eun Hee Park. Investigation: Hye Ji Yun, Eun Hee Park. Writing—original draft: Hye Ji Yun. Writing—review & editing: Eun Hee Park.

Funding Statement: None

References

Sawyer SM, Azzopardi PS, Wickremarathne D, Patton GC. The age of adolescence. Lancet Child Adolesc Health 2018;2:223-228.
Achenbach TM, Edelbrock CS. Taxonomic issues in child psychopathology. In: Ollendick TH, Hersen M, editors. Handbook of child psychopathology. Boston, MA: Springer;1983. p. 65-93.
Hopwood CJ, Grilo CM. Internalizing and externalizing personality dimensions and clinical problems in adolescents. Child Psychiatry Hum Dev 2010;41:398-408.
Achenbach TM. The classification of children's psychiatric symptoms: a factor-analytic study. Psychol Monogr 1966;80:1-37.
Storr CL, Pacek LR, Martins SS. Substance use disorders and adolescent psychopathology. Public Health Rev 2012;34:1-42.
Lee M, Park B. The structural relationships among externalizing problems, internalizing problems, achievement value, and school adjustment in adolescents. J Learn Cent Curric Instr 2017;17:517-535.
Park MH. The effect and development of anger management program for enhancement of peer relationship and social skills of children with externalized behavioral problems. J Emot Behav Disord 2013;29:45-67.
Lee S, Koh I, Kim M. A study that related variables to effect preschoolers' problems behavior. Early Child Educ Res Rev 2009;13:163-183.
Albayrak-Kaymak D. Internalizing or externalizing: screening for both problem youth. Int J Adv Couns 1999;21:125-137.
Lee SA, Kim KH, Yook KH. Utility of K-CBCL and MMPI-A in evaluation and diagnosis of adolescents with internalizing or externalizing disorders. Kor J Clin Psychol 2012;31:391-399.
Butcher JN, Williams CL, Graham JR, Archer RP, Tellegen A, Ben-Porath YS, et al. Minnesota multiphasic personality inventory-adolescent (MMPI-A): manual for administration, scoring, and interpretation. Minneapolis, MN: University of Minnesota Press;1992.
Kim JS, Han K, Lim J, Lee J, Min B, Moon K. Korean MMPI-2 user manual. Seoul: Maumsarang Inc;2005.
Archer RP, Handel RW, Ben-Porath YS, Tellegen A. Minnesota multiphasic personality inventory-adolescent-restructured form (MMPI-A-RF): administration, scoring, interpretation, and technical manual. Minneapolis, MN: University of Minnesota Press;2016.
Ben-Porath YS, Tellegen A. Empirical correlates of the MMPI-2 restructured clinical (RC) scales in mental health, forensic, and nonclinical settings: an introduction. J Pers Assess 2008;90:119-121.
American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th ed. Arlington, VA: American Psychiatric Association Publishing;2013.
Achenbach TM, Edelbrock CS. The classification of child psychopathology: a review and analysis of empirical efforts. Psychol Bull 1978;85:1275-1301.
Han K, Lim J, Moon K, Yook K, Kim J. Korean MMPI-A-RF user manual. Seoul: Maumsarang Inc;2018.
Archer RP. Assessing adolescent psychopathology: MMPI-A/MMPI-A-RF. 4th ed. New York: Routledge;2016.
Betz NE. Use of discriminant analysis in counseling psychology research. J Couns Psychol 1987;34:393-403.
Chang IC, Hwang HG, Hung MC, Lin MH, Yen DC. Factors affecting the adoption of electronic signature: executives' perspective of hospital information department. Decis Support Syst 2007;44:350-359.
Hollar E. Assessing internalizing disorders with the MMPI-A-RF: clinical correlates of internalizing scales in an adolescent inpatient sample [dissertation]. New York: Pace Univ.;2019.
Chakranarayan C, Weed NC, Han K, Skeel RL, Moon K, Kim JH. Minnesota multiphasic personality inventory-adolescent-restructured form (MMPI-A-RF) characteristics of ADHD in a Korean psychiatric sample. J Clin Psychol 2021;78:913-925.
Fertitta MC. Minnesota multiphasic personality inventory-adolescent-restructured form (MMPI-A-RF) with juvenile drug court adjudicated outpatient adolescents [dissertation]. Melbourne, FL: Florida Institute of Technology;2018.
Park KY. Examining the measurement invariance of the MMPIA-RF externalizing scales across Korean and American adolescent normative samples [dissertation]. Norfolk, VA:Old Dominion Univ.;2018.
Park SY, Chong YS. Moderating effects of ego-resilience and social support in relations among academic achievement pressure, perceived academic stress and internalization problems in boys and girls. Korean J Dev Psychol 2010;23:17-32.
McGrath EP, Repetti RL. A longitudinal study of children's depressive symptoms, self-perceptions, and cognitive distortions about the self. J Abnorm Psychol 2022;111:77-87.
Shin EB, Park EH, Hong HJ. Clinical utility of the MMPI-A-RF's internalization and externalization higher-order scales: comparison with the K-CBCL's internalization and externalization scales. Korean J Psychosom Med 2022;30:119-126.
Kim JS. Clinical interpretation of Minnesota multiphasic personality inventory. Seoul: Seoul National University Press;1998.
Lee H, Ha EH, Oh KJ. Gender differences in emotional and behavioral problems of Korean adolescents. J Korean Acad Child Adolesc Psychiatry 2005;16:117-123.