Reliability and validity of a novel caregiver-assessed skills system based on the ALSO conception in children with autism spectrum disorders

Introduction

On World Autism Awareness Day in 2020, the United Nations released the theme “The Transition to Adulthood”, and advocated the public attention to challenges or obstacles during the transition to adulthood for individuals with autism spectrum disorder (ASD) to achieve the goal of “nurturing children with ASD to be independent social citizens” (1). However, the transition to adulthood for many with ASD is not a quick and easy process (2,3). Much unsatisfactory outcomes of the transition to adulthood for individuals with ASD has been reported (4,5). In China, previous literature found that less than 24% of adults with ASD in Shanghai had proper occupational skills, engaged in recreational activities, took transportation, or went shopping independently (6). In responding to the potential challenges of ASD trajectory and facilitating their transition to adulthood, Doctor Yanqing Guo from the Peking University Sixth Hospital in China proposed an innovative intervention conception, “ALSO”, to bridge early intervention and transition service to adulthood for individuals with ASD (7).

The core of ALSO emphasizes that “for children with ASD, current interventions should be guided by future needs in adulthood, and future goals in adulthood must be practiced in current interventions.” Several studies have indicated that merely addressing transitional goals at adolescence yielded less productive results than starting this process at the early childhood stage (8-10). Considering the much-needed time and resources, the ALSO conception aims to pursue transitional outcomes by enhancing early intervention. That is, ALSO sets “Occupational and independent-living skills” (O) in young adulthood as the ultimate objective of education intervention for individuals with ASD, and proposes this future objective could be achieved via comprehensive assessments and tailored education intervention on “Academic and cognitive skills” (A), “Living and life skills” (L), and “Social interaction and social regulation skills” (S) for individuals with ASD, which stands for the conception of “ALSO” (7).

To facilitate the application of ALSO in early interventions for children with ASD, a team of behavior analysts, special educators, technicians, and other experts collaboratively developed ALSOLIFE platform, which includes two layers of free services: the ALSOLIFE Skills Assessment System (ALSOLIFE Assessment) intended to provide a remote assessment tool for caregivers of children with ASD to evaluate their children’s skills based on the ALSO conception; and an aligned ALSOLIFE Individualized Education Plan system (ALSOLIFE IEP), designed for caregivers to deliver tailored education to children with ASD based on the ALSOLIFE Assessment report (11).

It should be noted that many commonly used behavior and skill assessments for children with ASD in China were imported. For example, the Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP), the Early Start Denver Model (ESDM), and the Psychoeducational Profile-3 (PEP-3) are all originated in the United States. They are generally adapted for professionals who have had years of discipline specific training (12). Given that the critical shortage of qualified professionals in China, many parents must function as both caregivers and home therapists (13). The actual situation is that the abovementioned assessments are usually too difficult for most Chinese caregivers of children with ASD to acquire professional qualifications on applying these assessments accurately and skillfully (14,15).

To address the practical needs of families of children with ASD under the status quo of ASD intervention in China, ALSOLIFE platform is envisioned to help families by offering the free, online, self-operated, technology-assisted, and behavioral intervention approaches supported systems guided by the ALSO conception. Quite different from other assessment systems for children with ASD, the operations of the ALSOLIFE Assessment are kept as user-friendly and practical as possible, free of jargons. Through the access of internet, families can conduct skill assessment at their convenience and obtain online training manual according to their child’s performance profile. Home-based therapy is made possible through this comprehensive and accessible system.

Although embedded on the needs of Chinese families of children with ASD, the validation and effectiveness of the ALSOLIFE tools is still unknown. Assessment is the basis for intervention, the psychometric evaluation of the ALSOLIFE Assessment would be an essential step to validate the ALSOLIFE systems. Therefore, this study aimed to comprehensively examining the reliability and validity of the ALSOLIFE Assessment. The findings of the present study will guide further scientific improvement of the ALSOLIFE Assessment and consequently benefit the families of children with ASD in China by providing a practical, validated, cultural fit and scientifically tested assessment tool based on the ALSO conception. We present the following article in accordance with the SURGE reporting checklist (available at http://dx.doi.org/10.21037/tp-20-319).

Methods

Ethical statement

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013) and approved by the ethics board of the Institute of Psychology of the Chinese Academy of Sciences (No. H19022). The number of participants was based on power analysis (16), which yielded an estimated sample size of 220 participants or more as being likely to provide sufficient power (16).

Participants

To maintain the ecological validity of the study, we directly recruited participants through the ALSOLIFE platform upon receiving the informed consent from the legal guardians. There were two types of recruited participants: client participants (children with ASD) and assessor participants (primary caregivers of the enrolled children with ASD). As client participants, the enrolled children aging from 1–10 years need be diagnosed with ASD by providing a diagnosis proof signed by a qualified pediatric psychiatrist. Children who had any additional diagnosis were also asked to document the comorbidity. As assessor participants, caregivers were required to be the primary caregivers who are familiar with the children’s daily behavior and have plenty of opportunities to interact with the children. The enrolled caregivers were also required a minimal education level of middle school and having access to the ALSOLIFE. As an incentive, we provided each assessor who completed the ALSOLIFE Assessment with a toy gift worthy ¥50 yuan.

Based on these criteria, we enrolled a total of 1,050 participating children with ASD from 31 provinces or municipalities in China. The children’s age ranged from 1.52 to 10.43 years (mean: 4.46±1.52), and the sex ratio was 5.40:1 (886 boys and 164 girls). Thirty-four children also reported comorbidity, including one with depression (0.1%), 14 with epilepsy (1.3%), and 19 with attention deficit hyperactivity disorder (ADHD) (1.8%). Amongst the 1,137 participating caregivers, on the other hand, 955 were mothers, 80 were fathers, and 15 were grandparents. There were 155 people with a middle school education (14.8%), 156 with a high school degree (14.9%), 614 with a bachelor’s degree (58.5%), and 125 with a master’s degree (11.9%). The more detailed demographics information could be seen in Table 1.

Table 1 Demographics of the 1,050 children with ASD in the study
Full table

Instruments

ALSOLIFE Skills Assessment System (ALSOLIFE Assessment) Scope

ALSOLIFE Assessment is an individually administered measure of skills and behaviors of children with ASD functioning between the ages of 0 to 6 years old. It comprises six skill domains, including academic, cognitive, life, living, social interaction, and social regulation skills, each with a series of subdomains (see Figure 1). Targeting the basic learning abilities, the academic skill domain includes motor imitation, tact, mathematical concepts, and speaking and writing skills. The cognitive skill domain assesses children’s cognitive functioning, including receptive labeling, sensory perception, and sample matching. The life skill domain evaluates basic life-care abilities, such as dressing, self-feeding, daily chores, and personal hygiene. The living skill domain refers to leisure related skills, such as independent game playing, painting and handcrafting skills, and exercise capacity. While the social regulation skill domain examines rule-governed behaviors, such as classroom behaviors, emotional management skills, and community life skills, the social interaction skills emphasizes the verbal communication, including interactive language, understanding social games, demonstrating prosocial behaviors, following directions, and mand. With a total of 511 assessment items across 22 subdomains, ALSOLIFE Assessment offered five difficulty levels, 2 to 6, each associated with a developmental age. For readers’ more intuitive understanding, we provide some items examples of the ALSOLIFE Assessment in the Appendix 1.

Figure 1 Composition of the ALSOLIFE Assessment for children with autism spectrum disorders. The structure of 6 skill-domains and 22 skills subdomains are showed. The counts in brackets are the numbers of items in the specific subdomain or domain.

Administration

Designed for caregivers with minimal professional training, ALSOLIFE Assessment can be conducted via direct testing and observation. Along with a question prompt, most assessing items also came with a short online video clip or detailed written description that addressed the uniform administration procedures and specific guidelines for preparing testing materials. If the caregivers experienced any difficulties, professional/technical assistance was available via parent support groups.

Upon completion of registration process via ALSOLIFE website (www.alsolife.com), the caregivers were asked to fill in the basic information of their child. The system then automatically delivered assessing items from each skill domain based on the child’s chronical age. ALSOLIFE Assessment used semi-adaptive testing to identify the ceiling and basal levels, so the scores best represented the child’s ability. If the child scored less than 25% of full scores of the specific level of the skill domain, the system provided items with one difficulty level lower while one level higher if the child scored more than 75%. Therefore, it was not necessary to complete all 511 items. Depending on the child’s ability, it took approximately from one hour to up to three hours to complete ALSOLIFE Assessment. Caregivers could choose to leave the assessment at any time and resume it later. All assessing items were scored on a triad scale: 0 points = not meeting the requirement, 0.5 point = partially meeting the requirement, and 1.0 point = fully meeting the requirement. Scores for items between the ceiling and basal levels were calculated to obtain the final score of each skill domain. The final total score of ALSOLIFE Assessment is the sum of the scores of the six skill domains and the maximum score of the ALSOLIFE Assessment is 511.

Psychoeducational Profile 3rd Edition (PEP-3)

The PEP-3 is an assessment tool targeting young children with ASD between 2–7 years old across multiple skills and behaviors. Multiple studies have established that the simplified Chinese version of the PEP-3 has good reliability and validity in children with ASD in China (17-19). The test administration includes the Performance Test and the Caregiver Report. The former consists of 10 subtests and a total of 172 items to measure communication ability, motor ability, and maladaptive behaviors. Three subsets assessing communication include cognitive verbal/preverbal (34 items), expressive language (25 items), and receptive language (19 items). Motor ability is measured through gross motor (15 items), fine motor (20 items), and visual-motor imitation skills (10 items). Maladaptive behaviors included four subtests; they are affective expression (11 items), social reciprocity (12 items), characteristic motor behaviors (15 items), and characteristic verbal behaviors (11 items). Additional three subtests are in the Caregivers Report, problem behavior (10 items), personal self-care (13 items), and adaptive behavior (15 items), which allows the primary caregivers to observe and report their children’s natural actions. It has a three-prong scale system, with 0= Fail, 1= Emerge, and 2= Pass.

VB-MAPP

The VB-MAPP is one of the most widely used criterion referenced assessment tool, direct training curriculum guide, and skill tracker to assess verbal and related skills of young children with ASD in China (12). Recent research has measured the reliability and validity of VB-MAPP, which suggests its efficacy to provide assessment and intervention for children with language delays (20,21). VB-MAPP is designed to assess 16 skill domains: mand, tact, echoic, intraverbal, listener, motor imitation, independent play, social and social play, visual perceptual and matching-to-sample, linguistic structure, group and classroom skills, and early academics, with a total of 170 milestones across three developmental levels (0–18, 18–30, and 30–48 months). Out of 170 items, 166 are scored using a triad system in which 0 = does not meet the requirement, 0.5 = meets half of the requirement, and 1 = meets the requirement in full. The remaining four items are scored using a binary system: 0 = does not meet the requirement and 1 = meets the requirement in full.

Procedures

Determination of content validity

Adopting content validity ratio (CVR) techniques described by Lawshe (22), we invited six experts from different disciplines to evaluate the content validity of ALSOLIFE Assessment (Table 2). While all invited experts had some extent of credentials and at least 5 years of experiences in the field of ASD (23), none of them were part of the ALSOLIFE Assessment development team. We asked the experts to rate each of the 511 ALSOLIFE items on a three-point Likert scale (3 = Essential, 2 = Useful, but not essential, 1 = Not necessary) based on their professional judgement of whether the item was “essential” for a certain skill subdomain.

Table 2 Demographics description of the expert members for content validity test
Full table

Criterion-related validity

Out of all participating children, 31 had taken PEP-3 assessment (age of 2.21–6.51 years old) while another 34 had taken VB-MAPP (age of 1.8–6.04 years old), all well within two months prior to participating in this study. Upon obtaining the electronic versions of the reports from the caregivers, we used their PEP-3 and VB-MAPP scores to examine the criterion-related validity of ALSOLIFE Assessment.

Reliability tests

We recruited both parents of each of the 49 participating children with ASD to extend the inter-rater reliability check. They were asked to conduct the assessment independently and simultaneously. In addition, 87 caregivers voluntarily assessed their own child two-week after the initial assessment, using the same procedure, which permits the test-retest reliability check.

Statistical analysis

SPSS 22.0 was used for the statistical analysis of the normal distribution of data, internal consistency reliability (Cronbach’s alpha), inter-rater reliability test (Intraclass correlation coefficient, ICC), test-retest reliability test (Spearman correlation coefficient, rs), and criterion validity (Spearman correlation coefficient, rs; regression coefficients, β). Mplus 7.0 was used to conduct exploratory factor analysis (EFA) and confirmatory factor analysis (CFA).

Results

Validity of ALSOLIFE assessment

Content validity

A CRV value was calculated via the rating scores from the six invited experts. Out of a total of 511, 464 items (i.e., 90% of the items) exceeded the CVR cutoff of 0.67, which evidenced content validity (24). Specifically, 442 items had a CVR value of 1.00 and 22 of 0.67. The remaining 47 items had a CVR value of less than 0.67, suggesting insufficient evidence of content validity. After reviewing each of these 47 items, we found that they were either redundant or repetitive, which suggested that the core content of these 47 items had been included in the other 464 items. To ensure the content validity of the ALSOLIFE Assessment, we omitted these 47 items and used the remaining 464 items for the following reliability and validity tests.

Construct validity

We randomly divided the score set of all participating children in two groups and analyzed the data through EFA and CFA. The EFA analysis was performed using the Mplus7.0 robust maximum likelihood method. The results showed that the model fit well when the number of factors was 6 [χ²/df = 4.18, comparative fit index (CFI) = 0.98, Tucker-Lewis Index (TLI) = 0.97, standardized root mean square residual (SRMR) = 0.003, root mean square error of approximation (RMSEA) = 0.08], and the factor loading of each item in the oblique rotation ranged from 0.612 to 0.994 (Table 3). The factor loading distribution pattern was consistent with the structural assumption of the ALSO conception.

Table 3 EFA factor loadings of the ALSOLIFE Assessment
Full table

The CFA results showed that the correlation coefficient values of the six factor pairs had r-values ranged from 0.663 to 0.874. Analysis with a medium-to-high correlation coefficient (i.e., above 0.4) could benefit from a bifactor model instead of the traditional second-order factor model or correlated traits multidimensional model (25-27). Through the competition model selection method, we also found that the bifactorial model with six group factors had the best model fit indices (Table 4). The fit indices of the bifactorial model with six group factors showed significant differences from other competing models: the six-factor traditional second-order factor model and the six-factor correlated traits multidimensional model (Table 5).

Table 4 Fit indices of the CFA competition models of the ALSOLIFE Assessment
Full table

Table 5 Significant difference tests between the bifactorial model with six group factors and other competing models of the ALSOLIFE Assessment
Full table

General factors further reflected the commonality among multiple factors. The general factors of the 22 subskills and the factor loadings and variance contribution rates of the six group factors fell within a reasonable range (27) (Figure 2), suggesting that the factor structure of the ALSOLIFE Assessment was consistent with a bifactorial model with six group factors.

Figure 2 Confirmatory factor model for the ALSOLIFE Assessment. This is a bi-factorial model with six group factors (i.e., academic skills, cognitive skills, life skills, living skills, social interaction skills, and social regulation skills) and one general factor (i.e., general learning ability across six domains). The “G” in this figure is representing the general factor. Factor loadings and variance are showed in the middle of the path or next to the arrow.

Criterion-related validity

Positive correlations were significant (r=0.340–0.900, P<0.05) between the scores of the 22 subskills in the ALSOLIFE Assessment and the scores of 16 milestones in VB-MAPP. Overall, the total ALSOLIFE Assessment score significantly positively predicted the total score of VB-MAPP milestones in the same sample of children with ASD (β=0.81, R2=0.923, P<0.001). The six factors of the ALSOLIFE Assessment and the three factors of the PEP-3 showed significant moderate positive correlations (r=0.578–0.627, P<0.01), and the scores for the 22 subskills of the ALSOLIFE Assessment and the 13 subtests of the PEP-3 all showed significant positive correlations (r=0.356–0.776, P<0.05).

Reliability of ALSOLIFE assessment

Internal consistency reliability

Internal consistency was measured with Cronbach’s alpha, which quantifies the degree of homogeneity among different items within a measure. The results showed that Cronbach’s alpha for the entire ALSOLIFE Assessment (total score data) was 0.942, well above the cutoff value of 0.7 (28). It indicated that ALSOLIFE Assessment had excellent internal consistency reliability.

Inter-rater reliability

Scores from both parents conducting ALSOLIFE Assessments simultaneously were used to test the inter-rater reliability. The results showed that the intraclass correlation coefficient (ICC) value of the ALSOLIFE Assessment total score was 0.92, while ICC values for the 22 skill subdomains of the ALSOLIFE Assessment ranged from 0.88 to 0.95 (P<0.01). Given ICC value over 0.80 representing good consistency, ALSOLIFE Assessment was consistent across different assessors.

Test-retest reliability

We evaluated test-retest reliability using scores from 87 participating children who were retested using the ALSOLIFE Assessment after two-week period. The results showed that the test-retest intraclass correlation coefficient (ICC) value of the ALSOLIFE assessment total score was 0.98, and the ICC value of 22 skill subdomains was between 0.94 and 0.99 (P<0.01). Thus, ALSOLIFE Assessment satisfied the requirements of test-retest reliability, suggesting stability as a skill measure (24,25).

Discussion

This study examined the reliability and validity of the ALSOLIFE Assessment based on the data collection from 1,050 children with ASD aged from 1 to 10 years old in mainland China. Psychometric evaluation of the ALSOLIFE Assessment demonstrated satisfactory internal consistency, test–retest reliability, inter-rater reliability, content validity, construct validity, and good criterion-related validity compared with VB-MAPP and PEP-3.

Through the CVR analysis, we designated 464 items out of the 511 original ones to ensure the content validity of the ALSOLIFE Assessment meets the psychometric requirement. Compared to VB-MAPP and PEP-3, the ALSOLIFE Assessment exhibited good criterion-related validity across three levels: total scores, six skill domains, and 22 skill subdomains. EFA results verified six factors making up the ALSOLIFE Assessment, which conforms to the structural assumptions of the ALSO conception (Table 3). The CFA competition models test indicated that the best fitting construct model of the ALSOLIFE Assessment score data is the bifactorial model with a general factor and six group factors (29). This finding revealed that a general factor could be extracted among the six skill factors of the ALSOLIFE Assessment and suggested that some common factors influenced the scores of the six skill domains. For instance, research found that a general factor was extracted from the construct validity model of the Wechsler Intelligence Scale for Children (30). A combination of two parts could potentially explained the general factor of ALSOLIFE Assessment. On one hand, general functioning (e.g., general learning ability) may influence to some extent of all six skill domains. On the other hand, the fact that all the domains are evaluated using the same assessment method contributing to a common methodological factor. As shown in Figure 2, by comparing the values of factor loadings and variance contribution rates of the general factor and six group factors (31), we can conclude that the general factor is the dominant factor, while the six group factors also possessing its unique contributions to the construct validity model of the ALSOLIFE Assessment.

In terms of its reliability, the results showed a satisfactory internal consistency reliability, inter-rater reliability, and test-retest reliability at three levels: the entirety of ALSOLIFE Assessment, its six skill domains, and its skill subdomains. It suggested ALSOLIFE Assessment was a consistent, stable, and reliable tool.

There are some limitations of this study. First, the participants were recruited directly via online platform, as self-identified families of ASD. It might potentially influence the sample representation and accuracy of the demographic information. However, we chose online recruitment due to its benefits to overcome geographical restrictions while keeping it consistent with the natural context of ALSOLIFE platform. Through such recruitment procedure, we might potentially prevent any issues with external validity (ecological validity). In responding to the issues related to self-reported information, we asked participants to provide a copy of the official medical reports from a pediatric psychiatrist. We also confirmed any unclear demographic information through an additional telephone interview.

Second, the criterion validity testing of the ALSOLIFE Assessment was limited to the use of existing VB-MAPP and PEP-3 electronic report scores, obtained two months prior to this study. The time lag between tests might hinder the accurate analysis of concurrent validity. However, such data was still useful to determine predictive validity. Since none of the recruited children were younger than 1.5 years of age, significant developmental changes over two months period may not be a concern.

Despite the limitations outlined above, the present study is the first empirical validation study of the ALSOLIFE Assessment that provides meaningful findings. Results indicated the reliability and validity of the ALSOLIFE Assessment satisfied psychometric requirements after trimming to 464 items. In comparison with PEP-3 and VB-MAPP, the ALSOLIFE Assessment is still reliable and validated in the same sample of children with ASD. The construct validity model of the ALSOLIFE Assessment score data was firstly explored and confirmed as the bifactorial model with one general factor and six group factors, which conforms to the structural assumptions of the ALSO conception.

In conclusion, it is essential to develop ALSOLIFE Assessment from the conceptually systematic framework to evidence-based assessment tools. The validated ALSOLIFE Assessment can be an accuracy tool for caregivers of children with ASD to obtain a general picture of their child’s current functioning. Along with the aligned ALSOLIFE IEP system, this free online service system could potentially benefit caregivers in the development of a home-based intervention program, particularly for those who have trouble accessing high-quality assessment and IEPs. Future research should investigate the effectiveness of ALSOLIFE IEP system and the proper alignment of both components. It is imperative to develop validated, effective, and systematic evaluation and intervention tools, embedded on the needs of families of children with ASD unique to the cultural and societal context of China.

Acknowledgments

Acknowledge English article editing service Editage Insights to polish the language.

Funding: None.

Footnote

Reporting Checklist: The authors have completed the SURGE reporting checklist. Available at http://dx.doi.org/10.21037/tp-20-319

Data Sharing Statement: Available at http://dx.doi.org/10.21037/tp-20-319

Peer Review File: Available at http://dx.doi.org/10.21037/tp-20-319

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/tp-20-319). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013) and approved by ethics board of Institute of Psychology of the Chinese Academy of Sciences (No. H10922). Informed consent was taken from all the participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. United Nations. World Autism Awareness Day 2020 on “The Transition to Adulthood” [Internet]. António G, editor. c2020 [cited 2020 Sep 17]. Available online: https://www.un.org/development/desa/disabilities/news/dspd/waad.html
2. van Schalkwyk GI, Volkmar FR. Autism Spectrum Disorders: Challenges and Opportunities for Transition to Adulthood. Child Adolesc Psychiatr Clin N Am 2017;26:329-39. [Crossref] [PubMed]
3. Anderson KA, Sosnowy C, Kuo AA, et al. Transition of individuals with autism to adulthood: a review of qualitative studies. Pediatrics 2018;141:S318-27. [Crossref] [PubMed]
4. Roux AM, Shattuck PT, Rast JE, et al. National Autism Indicators Report: Transition into Young Adulthood. Philadelphia (PA): A.J. Drexel Autism Institute, Life Course Outcomes Research Program, Drexel University; 2015.
5. Wisner-Carlson R, Uram S, Flis T. The Transition to Adulthood for Young People with Autism Spectrum Disorder. Child Adolesc Psychiatr Clin N Am 2020;29:345-58. [Crossref] [PubMed]
6. Hua HQ. An Investigation of Daily Life of the Youth with Autism and the Rehabilitation. Social Construction 2017;2:15-25.
7. Guo Y. Applied Behavior Analysis and Behavior Management for Children. Beijing: Huaxia Publishing House; 2012.
8. Meyer AT, Powell PS, Butera N, et al. Brief report: Developmental trajectories of adaptive behavior in children and adolescents with ASD. J Autism Dev Disord 2018;48:2870-8. [Crossref] [PubMed]
9. Bal VH, Kim SH, Cheong D, et al. Daily living skills in individuals with autism spectrum disorder from 2 to 21 years of age. Autism 2015;19:774-84. [Crossref] [PubMed]
10. Klinger LG, Klinger MR, Mussey JL, et al., editors. International Meeting for Autism Research; 2015 May 13-15; Salt Lake City, USA. Available online: https://insar.confex.com/insar/2015/webprogram/Paper20033.html
11. Alsolife.com [Internet]. Beijing: ALSOLIFE Assessment and IEP Systems Platform in China. 2017 [cited 2020 May 2]. Available online: https://www.alsolife.com/
12. Gould E, Dixon DR, Najdowski AC, et al. A review of assessments for determining the content of early intensive behavioral intervention programs for autism spectrum disorders. Res Autism Spectr Disord 2011;5:990-1002. [Crossref]
13. Sullivan OA, Wang C. Autism Spectrum Disorder Interventions in Mainland China: a Systematic Review. Rev J Autism Dev Disord 2020;7:263-77. [Crossref]
14. Liu Q, Hsieh WY, Chen G. A systematic review and meta-analysis of parent-mediated intervention for children and adolescents with autism spectrum disorder in mainland China, Hong Kong, and Taiwan. Autism. 2020;24:1960-79. [Crossref] [PubMed]
15. Chahin SS, Apple RW, Kuo KH, et al. Autism spectrum disorder: psychological and functional assessment, and behavioral treatment approaches. Transl Pediatr 2020;9 Suppl 1:S66-75. [Crossref] [PubMed]
16. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med 1998;17:101-10. [Crossref] [PubMed]
17. Yu S, Jia M, Yang X, et al. Validity and Reliability of the Chinese version of Psycho-Educational Profile for Children with Autism. Chinese Mental Health Journal 2015.697-702.
18. Yu L, Zhu X, Shek DTL, et al. Validation of the Simplified Chinese Psychoeducational Profile Third Edition in Mainland China. J Autism Dev Disord 2019;49:1599-612.
19. Shek DTL, Yu L. Construct validity of the Chinese version of the psycho-educational profile-3rd edition (CPEP-3). J Autism Dev Disord 2014;44:2832-43. [Crossref] [PubMed]
20. Montallana KL, Gard BM, Lotfizadeh AD, et al. Inter-Rater Agreement for the Milestones and Barriers Assessments of the Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP). J Autism Dev Disord 2019;49:2015-23. [Crossref] [PubMed]
21. Barnes CS, Mellor JR, Rehfeldt RA. Implementing the Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP): Teaching Assessment Techniques. Anal Verbal Behav 2014;30:36-47. [Crossref] [PubMed]
22. Lawshe CH. A quantitative approach to content validity. Pers Psychol 1975;28:563-75. [Crossref]
23. Usry J, Partington SW, Partington JW. Using expert panels to examine the content validity and inter-rater reliability of the ABLLS-R. J Dev Phys Disabil 2018;30:27-38. [Crossref]
24. Lynn MR. Determination and quantification of content validity. Nurs Res 1986;35:382-5. [Crossref] [PubMed]
25. Gu H, Wen Z. Reporting and interpreting multidimensional test scores: a bi-factor perspective. Psychol Dev Educ 2017;33:504-12.
26. Gu H, Wen Z, Fang J. Bi-factor models: A new measurement perspective of multidimensional constructs. Journal of Psychological Science 2014;37:973-9.
27. Reise SP, Morizot J, Hays RD. The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Qual Life Res 2007;16:19-31. [Crossref] [PubMed]
28. Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess 2003;80:99-103. [Crossref] [PubMed]
29. Canivez GL. Bifactor modeling in construct validation of multifactored tests: Implications for understanding multidimensional constructs and test interpretation. Principles and methods of test construction: Standards and recent advances. Psychological assessment—science and practice. Boston, MA, US: Hogrefe Publishing, 2016:247-71.
30. Reynolds MR, Keith TZ. Multi-group and hierarchical confirmatory factor analysis of the Wechsler Intelligence Scale for Children—Fifth Edition: What does it measure? Intelligence 2017;62:31-47. [Crossref]
31. Mao X, Xia M, Xin T. Full-information item bifactor analysis: Model, parameter estimation and application. Advances in Psychological Science 2017;26:358-67.