Reliability and validity of an instrument to assess pediatric inpatients’ experience of care in China

Introduction

Healthcare quality improvement collaborations and initiatives have been proliferated worldwide with aims to improve clinical processes and patient experience (1). Patient experience survey programs are necessary for monitoring quality of care, and they can provide meaningful and valuable information for quality improvement initiatives (2-4). Most patient experience surveys administered routinely in hospitals are restricted to adults, such as the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) in the US, the National Health Service Patient Survey Programme (NPSP) in the UK, and the Chinese Patient Experience Questionnaire (CPEQ) survey in China (5-8). Standardized questionnaires and methods used to measure children’s experience in hospitals in different cultural contexts have been less investigated historically (9,10).

Due to the lack of evidence on patient-reported experiences relating to quality of pediatric clinical processes, both the HCAHPS and NPSP have developed complementary surveys for pediatric patients in the past decade. The Children and Young People’s Patient Experience Survey (CYP)—part of the NPSP—was developed and implemented in 2014 (11) This survey is featured for its diverse patient populations sampled, which includes children aged 2 weeks to 7 years, children aged 8 to 11 years, and young people aged 12 to 15 years. However, diverse patient samples and corresponding questionnaires are accompanied by the additional burden of survey organization and administration, compared to the cost and use of a single and standardized survey instrument. Additionally, the possible difference between children’s responses and their parents’ responses also limits the application of CYP. The Child HCAHPS is another standardized survey assessing the experience of pediatric patients, and gathers data from parents and guardians; it was endorsed by the US National Quality Forum in 2015 (12). Although previous research has translated or directly adopted the Child HCAHPS in Belgium and Canada (13,14), there is still a lack of validated Chinese versions of survey instruments for pediatric patients to date. A significant advantage of the HCAHPS is that the Agency for Healthcare Research and Quality (AHRQ) has a public set of proven guidelines for translating HCAHPS surveys from English into a variety of languages in a cost-effective yet culturally competent manner. Thus, the Child HCAHPS can not only be measured and reported as much as the widely used Adult HCAHPS, but it can also be translated into other languages with ensured quality.

In China, with the implementation of the two-child policy after January 2016, the increasing demand for pediatric care delivery is likely to be challenging for both regulatory bodies and children’s hospitals (15). To address the lack of survey instruments to measure pediatric experience for quality improvement initiatives in China, the current study describes the translation of a Chinese version of the Child HCAHPS and primary results of the validation in a field test.

We present the following article in accordance with the STROBE reporting checklist (available at https://dx.doi.org/10.21037/tp-21-130).

Methods

Translating the child HCAHPS from English into Chinese

The Child HCAHPS survey was developed by the Center of Excellence for Pediatric Quality Measurement at Boston Children’s Hospital, which was funded by the AHRQ and the Centers for Medicare and Medicaid Services (CMS). It has 62 items, including 39 patient experience items that could be categorized into 18 composite and single-item measures. As a publicly available standardized survey of pediatric inpatient experience, the Child HCAHPS demonstrated good to excellent hospital-level reliability in a national field test in US hospitals (12).

Based on the guidelines and recommendations from the AHRQ for the translation of the HCAHPS surveys (16), we produced a Chinese version of the Child HCAHPS. Two simultaneous translators and two translation reviewers formed a translation team to ensure the quality and cultural appropriateness of the translated version. Both translators and translation reviewers were native speakers of Chinese and proficient in reading and writing English. Both had a medical degree from a medical college in China and at least 5 years of work experience in the health services research area. The translation process was as follows. First, we provided the translators and reviewers with the Chinese survey materials of the HCAHPS, which enabled them to check for and continue to use standardized translations for response scales and other text elements that are common across surveys. Background information on the translation task was also provided to help them better understand the Child HCAHPS. Second, we obtained two independent forward translations of the Child HCAHPS in Chinese from each translator, while the linguistic and cultural relevance of the translations were reviewed by the translation reviewers. Third, the translators and reviewers formed a committee to review and finalize the wording of the translated Chinese survey materials. Any issues or problems were resolved by discussion until a consensus was reached by the committee.

The translation team replaced two items regarding the child’s ethnicity and one item regarding the respondent’s education based on localized questions and answers in the Chinese context. For the child’s ethnicity, the top five ethnic groups in China were retained in the questionnaire, which included Han, Zhuang, Hui, Manchu, and Miao. All the other ethnic minorities were grouped as a separate option. The survey item regarding preferred language was dropped because it was unsuitable for the Chinese context. Considering alignment with the English version of the Child HCAHPS, we retained the direct Chinese translation of those dropped or replaced items in the translation version rather than applying it in the field test. For the final formatted Chinese version of the Child HCAHPS, please refer to Supplementary file 1 (available online: https://cdn.amegroups.cn/static/public/10.21037tp-21-130-1.pdf). The reconcile translation record master table is presented in Supplementary file 2 (available online: https://cdn.amegroups.cn/static/public/10.21037tp-21-130-2.xlsx).

Pilot hospitals

We conducted a field test using the Chinese version of the Child HCAHPS in seven hospitals across five provinces in China. The pilot hospitals included one National Clinical Research Center for Child Health, two regional free-standing children’s hospitals, and four general hospitals with a pediatric ward.

Patients

We obtained the discharge patient list from the hospital administrative department to identify potentially eligible patients. Children aged 17 years or younger with at least one overnight stay in the hospital and their parents/guardians were selected as per the eligibility criteria for determining whom to include in the survey.

The exclusion criteria were as follows: patients discharged without doctor permission, patients discharged to hospice care, patients who did not have parents/guardians, healthy newborns, patients discharged with a psychiatric principal diagnosis, and observation patients.

Sample size

The original English Child HCAHPS demonstrated good to excellent hospital-level reliability at 300 responses per hospital in the US. Thus, we aimed to obtain at least 300 completed questionnaires per hospital in the multi-site field test to achieve reliability, following the requirement for administering the Child HCAHPS survey based on the AHRQ’s guideline (17). Due to the limited number of pilot hospitals, we attempted to obtain as many complete questionnaires as possible; thus, we conducted a census of all the eligible discharges in the hospitals rather than drawing a random sample of them.

Survey administration

Surveys were conducted from August 1 to September 27, 2020. During eight consecutive weeks of data collection, we enrolled research assistants who visited the pediatric wards daily in each pilot hospital. The assistants invited all eligible patients and their parents to complete the survey using an online survey system on the day of discharge before they left the hospital. Respondents were asked to scan a QR code to access the electronic questionnaire using their mobile phone or the assistant’s mobile phone; all the respondents were approached individually and guaranteed confidentiality. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Participation was voluntary and verbal consent was obtained from the respondents before the survey. Ethical approval was granted by the Peking Union Medical College (3332019087).

Statistical analysis

First, we described the demographic and sociological characteristics of the children and respondents. Second, construct validity was assessed using confirmatory factor analysis (CFA). We used the comparative fit index (CFI), non-normed fit index (NNFI), and root mean square error of approximation (RMSEA) to evaluate the discrepancy between our pilot data and the hypothesized model. The higher the CFI and NNFI, the better the model fit, with a cut-off score of 0.9 representing a good model fit. For the RMSEA, a score lower than 0.1 indicated an acceptable model fit. Because only 4.7% of children were aged 13 years or older, we excluded the dimension of “Involving teens in care” in the CFA. We evaluated convergent validity by factor loading, average variance extracted (AVE), and construct reliability (CR).

Third, we conducted a hospital-level unit for quantitative reliability and validity testing. Cronbach’s alpha and corrected item-total correlation (CITC) were used to reflect hospital-level unit reliabilities for the survey’s item composites. The correlation of the measure score with the overall rating was calculated to evaluate criterion validity. Fourth, we calculated the item-to-composite correlations, internal consistency reliability, and composite-to-composite correlations for additional testing.

All data analyses were generated using SPSSAU software (QingSi Technology Ltd., Beijing, China), using a two-sided significance test (P<0.05).

Results

From August to September, 2020, we conducted a survey in seven hospitals across five provinces. Of the seven hospitals, six hospitals received at least 300 completed questionnaires and met the recommended minimum number of complete questionnaires for analysis and reporting. Two free-standing children’s hospitals and four pediatric wards within a general hospital were included in the study. The location, setting, completed surveys, and response rates from each site are listed in Table 1. The overall response rate was 63%, the highest was 70%, and the lowest was 55%.

The general characteristics of the children and respondents are presented in Table 2. Over 60% of patients were aged ≤4 years and 59% were girls. The majority of patients were of Han ethnicity (73.4%) and had excellent or very good health (68.4%). Most respondents (63.3%) were aged <35 years, and 51.2% had an education level of high school graduate or lower. Most respondents were the children’s mothers (65.8%), and the patients were most likely to be discharged on a Monday (17.9%) or Friday (17.8%).

The CFA model fit showed CFI=0.905, NNFI=0.886, and RMSEA=0.089. Table 3 shows the factor loading, AVE, and CR of the composite measures. Most of the items had factor loadings over 0.7, and five had factor loadings of 0.195 to 0.419. Three composites related to communication about medicines and attention to safety and comfort showed unsatisfactory AVE (<0.5) and CR (<0.7).

The Cronbach’s alpha coefficient on the overall level was 0.981, and the CITC for all measures exceeded 0.6, which demonstrated good to excellent hospital-level reliability of the composite and single-item measures, as presented in Table 4. Of the 17 composite or single-item measures, 13 had significant positive correlations with the overall rating of the hospital, with the strongest correlation of 0.944 for child comfort and correlations ranging from 0.419 to 0.938 for other measures.

Table 5 presents the internal consistency reliability using Cronbach’s coefficient. All the composite measures had good to excellent internal consistency reliability, ranging from 0.716 to 0.994. The item-to-composite correlation ranged from 0.510 to 0.997. The “Involving teens in care” composite had a low item-to-composite correlation, possibly due to each item reflecting distinct processes of care, although they are conceptually related. The composite-to-composite correlations ranged from 0.488 to 0.997, and at the hospital level, each composite measure had strong correlations with each other, except for “Involving teens in care,” as shown in Table 6.

According to the results of the test survey, for all the 18 composite or single-item measures, mean top-box scores ranged from 56% (“Involving teens in care”) to 87% (“Informed in Emergency Room”). The best-performing composites pertained to “Informed in Emergency Room” (87% in top box), “Communication about medicines” (86% in top box), and “Privacy with providers” (79% in top box). Meanwhile, the worst-performing composites dealt with “Involving teens in care” (56% in top box), “Quietness of hospital room” (58% in top box), and “Child comfort” (61% in top box).

Discussion

Several HCAHPS surveys implemented by the US CMS are available in Cantonese, Korean, Mandarin, Portuguese, and Russian. The Child HCAHPS has only been available in English and Spanish until now, and now the Chinese version has been presented in our research. We examined the transferability of the Child HCAHPS to the Chinese context through a pilot study conducted in seven hospitals in China. The Chinese version of the Child HCAHPS, which was produced in this study, demonstrated acceptable construct and criterion validity, as well as satisfactory reliability.

For fielding the survey, the AHRQ recommends drawing a random sample of eligible monthly discharges as the basic sampling procedure. In our trial in Chinese settings, we conducted a census and included all eligible discharges in the survey to ensure a sufficient number of responses. Compared to the US National Field Test of the Child HCAHPS (12), the overall response rate in our study (63%) was much higher than that of the US (17.1%), but lower than the overall response rate in Belgium (90.99%) (12). This may be mainly due to our face-to-face mode for data collection, which usually gains higher response rates than mail or telephone approaches. In addition, we believe that the electronic questionnaire and online survey system helped improve the participation of hard-to-reach groups in our trial, as demonstrated by the similar tablet administration in Japan and the US (18-20).

In this pilot study, we distributed surveys to parents of patients on the day they were discharged, rather than adopting the HCAHPS, which recommends data collection modes that initiate surveys after discharge by mail or telephone. This timeframe difference was determined by the culture and usual behavioral differences between China and the US. The traditional survey approach by mail is associated with difficulty in reaching the target parents or guardians in China because residents prefer to use the express delivery service rather than the post system in their daily life, while the express delivery service would be costly for massive and large-scale surveys in reality. The telephone approach was also not selected in our pilot, because unwanted or nuisance calls are a social problem in China, resulting in many complaints. In other words, the telephone approach may be unsuitable for a surveys in a Chinese setting.

Regarding the characteristics of the sample population, 85.4% of the children in the present study were less than 8 years and 14.6% were over 8 years. The low percentage of participants over 8 years may reflect the unique medical care-seeking process of pediatric patients in China. Due to the gap between supply and demand of pediatric care resources (15), the parent or guardian of a teenager may select the general hospital rather than the more crowded children’s hospital as their first healthcare utilization choice. The result suggests that patients less than 8 years were the main customers in children’s hospitals in China.

Regarding the exclusion criteria, we revised the survey so that it corresponded to the Chinese healthcare system, as recommended by the AHRQ. For example, the “No‐Publicity” patients were not adopted in the exclusion criteria list, which is recommended by the AHRQ, as there is a lack of similar regulation in Chinese hospitals. Additionally, the “Court/Law enforcement” patients were not adopted either because this patient group is designated for hospitalization only in an independent judicial hospital system. Meanwhile, some new criteria to reflect the situation in China were added in our survey; for example, we excluded “patients discharged without doctor’s permission,” which is common in China and may reflect a significant cultural difference compared with the US.

To validate the composite measure, two aspects need attention. First, two items in the “Communication about medicines” had low composite loadings; that is, the items did not fit the dimension well. According to our field study and interviews with the respondents in China, some patients and their parents were unable to distinguish between prescription medicines and over-the-counter medicines; thus, they felt confused and were unable to give accurate answers about the item related to communication about medicines. This result suggests that a modification of the item related to communication about medicines would be necessary for the Chinese setting. Similar research in the Belgian context also revealed this medicine-related, potentially cultural-specific difference (13). Second, two items in the “Attention to safety and comfort” had low composite loadings. One was “Providers told parents how to report mistakes,” and the other was “Hospital had things available that were right for child’s age.” The item measuring “Report mistakes” was desirable in terms of knowing whether the provider cares about safety issues from the perspective of the patient; however, on conceptual grounds, this item may not be suitable for organizing into the composite, as the other item in the composite mainly focuses on the patient identity check by the provider. The item “Hospital had things available that were right for child’s age” had the lowest composite loadings in all items, and this is consistent with previous research in the US and Belgium (12,14). We suggest that this item be reported as a separate item rather than organized into the current composite.

The hospital-level quantitative reliability and validity test results suggest that the Chinese version of the Child HCAHPS could be a valuable instrument for quality improvement initiatives in pediatric settings in China. Most composite indicators can be directly used to measure and distinguish performance at the hospital level. In addition, some composites should be used with caution (e.g., Involving teens in care) or be modified to make them more context-specific and fit for the domestic culture. This dedicated pediatric survey to measure and track patient experience would provide highly valued evidence for the provider (2,21).

Limitations

This research had some limitations. First, neither the translators or translation reviewers had any experience living in the US, nor were they able to gain insight into the regional variations in the usage of Chinese language in the US. As non-certified professional translators, they met the required qualifications for the translator and reviewer but did not reach the desirable qualifications recommended by the AHRQ, and this may have influenced the quality of the translation. Second, no case-mix adjustment procedures were applied in the data analysis because this research was aimed at instrument validation rather than hospital performance comparison. In light of the future large-scale application of the Chinese version of Child HCAHPS, an appropriate case-mix adjustment methodology would be necessary for further research. Third, the generalization of the findings might be limited, as seven hospitals in only five Chinese Provinces were included. In the future, a larger sample that covers hospitals from other provinces of China would produce more representative evidence for validation of the Chinese version of the Child HCAHPS and re-test reliability of the survey. Fourth, based on the limited sample size of the field test, we were unable to obtain enough responses to analyze the differences between children’s responses and their parents’ or guardians’ responses. Although including children and young people themselves as respondents in the surveys is recommended in the English country context (11), whether it is feasible in the Chinese context needs to be further examined. Additionally, underrepresented minority groups may be less likely to participate in surveys in the US (22); similarly, such groups need to be identified in China.

Conclusions

The Child HCAHPS and its Chinese edition produced by our research can be a valuable and reliable instrument for measuring the pediatric experience of hospital care in China. It provides a validated tool for assessing hospital performance. The application of this tool in hospitals can benchmark ongoing healthcare improvement initiatives in China. The standardized methodology and comparable survey results would not only be beneficial for hospital internal quality improvement work but also contribute to external coordination of cross-nation pediatric quality research worldwide.

Acknowledgments

We acknowledge the following collaborators for their help in the field study: Wei Li, MD (Yunnan Genzeal Medical Doctor Group); Ximing Yang, MD (Binchuan County People’s Hospital of Dali Prefecture), Long Wang, MD (Xiangyun County People’s Hospital of Dali Prefecture), and Tianqiao Li, MD (Ninger County People’s Hospital of Pu’er City).

Funding: This research was funded by the Special Research Fund for Central Universities, Peking Union Medical College (3332019087).

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at https://dx.doi.org/10.21037/tp-21-130

Data Sharing Statement: Available at https://dx.doi.org/10.21037/tp-21-130

Peer Review File: Available at https://dx.doi.org/10.21037/tp-21-130

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/tp-21-130). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013) and ethical approval was granted by the Peking Union Medical College (3332019087). Participation was voluntary, and verbal informed consent was obtained from the respondents before the survey.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Wells S, Tamir O, Gray J, et al. Are quality improvement collaboratives effective? A systematic review. BMJ Qual Saf 2018;27:226-40. [Crossref] [PubMed]
2. Wray J, Oldham G. Using parent-reported experience measures as quality improvement tools in paediatric cardiothoracic services: making it happen. Int J Qual Health Care 2020;32:140-8. [Crossref] [PubMed]
3. Hamedani A, Safdar B, Aaronson E, et al. Patient Experience Must Move Beyond Bad Apples. Ann Intern Med 2016;165:869-70. [Crossref] [PubMed]
4. Singleton PD. Patient experience should be recorded in health records. BMJ 2017;357:j2673. [Crossref] [PubMed]
5. Jha AK, Orav EJ, Zheng J, et al. Patients' perception of hospital care in the United States. N Engl J Med 2008;359:1921-31. [Crossref] [PubMed]
6. Delnoij DM. Measuring patient experiences in Europe: what can we learn from the experiences in the USA and England? Eur J Public Health 2009;19:354-6. [Crossref] [PubMed]
7. Hu G, Chen Y, Liu Q, et al. Patient experience of hospital care in China: major findings from the Chinese Patient Experience Questionnaire Survey (2016-2018). BMJ Open 2019;9:e031615 [Crossref] [PubMed]
8. Wray J, Hobden S, Knibbs S, et al. Hearing the voices of children and young people to develop and test a patient-reported experience measure in a specialist paediatric setting. Arch Dis Child 2018;103:272-9. [Crossref] [PubMed]
9. Caneiras C, Jácome C, Mayoralas-Alises S, et al. Patient Experience in Home Respiratory Therapies: Where We Are and Where to Go. J Clin Med 2019;8:555. [Crossref] [PubMed]
10. McNeill M, Noyek S, Engeda E, et al. Assessing the engagement of children and families in selecting patient-reported outcomes (PROs) and developing their measures: a systematic review. Qual Life Res 2021;30:983-95. [Crossref] [PubMed]
11. Hargreaves DS, Sizmur S, Pitchforth J, et al. Children and young people's versus parents' responses in an English national inpatient survey. Arch Dis Child 2018;103:486-91. [Crossref] [PubMed]
12. Toomey SL, Zaslavsky AM, Elliott MN, et al. The Development of a Pediatric Inpatient Experience of Care Measure: Child HCAHPS. Pediatrics 2015;136:360-9. [Crossref] [PubMed]
13. Kemp KA, Ahmed S, Quan H, et al. Family Experiences of Pediatric Inpatient Care in Alberta, Canada: Results From the Child HCAHPS Survey. Hosp Pediatr 2018;8:338-44. [Crossref] [PubMed]
14. Bruyneel L, Coeckelberghs E, Buyse G, et al. Validation of the Child HCAHPS survey to measure pediatric inpatient experience of care in Flanders. Eur J Pediatr 2017;176:935-45. [Crossref] [PubMed]
15. Zhang XY, Gao Y, Li CP, et al. Observed and projected trends in paediatric health resources and services in China between 2003 and 2030: a time-series study. BMJ Open 2017;7:e015000 [Crossref] [PubMed]
16. Agency for Healthcare Research and Quality Translating Surveys and Other Materials. Available online: https://www.ahrq.gov/cahps/surveys-guidance/helpful-resources/translating/index.html
17. Agency for Healthcare Research and Quality CAHPS Child Hospital Survey. Available online: https://www.ahrq.gov/cahps/surveys-guidance/hospital/about/child_hp_survey.html
18. Toomey SL, Elliott MN, Zaslavsky AM, et al. Improving Response Rates and Representation of Hard-to-Reach Groups in Family Experience Surveys. Acad Pediatr 2019;19:446-53. [Crossref] [PubMed]
19. Suzuki E, Mackenzie L, Sanson-Fisher R, et al. Acceptability of a Touch Screen Tablet Psychosocial Survey Administered to Radiation Therapy Patients in Japan. Int J Behav Med 2016;23:485-91. [Crossref] [PubMed]
20. Tieu L, Hobbs A, Sarkar U, et al. Adapting Patient Experience Data Collection Processes for Lower Literacy Patient Populations Using Tablets at the Point of Care. Med Care 2019;57 Suppl 6 Suppl 2:S140-8.
21. Feng JY, Toomey SL, Elliott MN, et al. Factors Associated With Family Experience in Pediatric Inpatient Care. Pediatrics 2020;145:e20191264 [Crossref] [PubMed]
22. Lee B, Hollenbeck-Pringle D, Goldman V, et al. Are Caregivers Who Respond to the Child HCAHPS Survey Reflective of All Hospitalized Pediatric Patients? Hosp Pediatr 2019;9:162-9. [Crossref] [PubMed]