Big splashes & ripple effects: a narrative review of the short- & long-term impact of publications supported by an NIH CTSA pediatrics program

Introduction

The aim of this review is to examine a promising new framework for analyzing the outputs of pediatric research in the context of translational advancement. The central question under review is how bibliometrics, or the study of publications as a scientific output, can be used to determine how a line of research is making an impact on the translation of pediatric science from the bench to the bedside, or from clinical research to clinical use and health improvements.

The National Center for Advancing Translational Science (NCATS) of the National Institutes for Health (NIH) supports a nationwide network of Clinical and Translational Science Award (CTSA) hubs charged with accelerating clinical and translational education, research, and community engagement to improve health (1,2). The Georgia Clinical & Translational Science Alliance (Georgia CTSA) was established in 2007 as one of the first of these hubs and consists of a cross-institutional collaboration among Emory University, Morehouse School of Medicine, the Georgia Institute of Technology, and the University of Georgia, with a collection of interconnected programs providing relevant services to investigators within those institutions. One such program is the Georgia CTSA Pediatrics Program, a longstanding effort to transform scientific discovery into solutions that improve pediatric health. The program provides training, seed funding, and infrastructure to researchers to ensure that research projects have the resources needed to move forward. Services include expert consultations, educational and training materials, clinical research space and staff, pilot and training grants, and administrative support.

Regular evaluation of programmatic outputs is essential for understanding the extent that a program is meeting its stated goals, and several qualitative and quantitative evaluation approaches of various CTSA programs have been published thus far, including assessment of supported publications and grants (3-5) and numbers of citations (5-8) accrued over several years. Previous internal evaluation of Georgia CTSA pediatric research output uncovered a body of work that is highly interdisciplinary and particularly impactful within the academic community, compared to similar work at other institutions (9). Pediatrics research is unique among clinical/translation research in its coverage of diverse fields of study and the translation of pediatrics research from bench to bedside often involves significant collaboration and scholarly discourse across disciplines and phases of translational research. Moreover, pediatrics research has a tendency to be of high interest to the broader lay community and internal evaluation showed that pediatrics research received more public/community attention than other specializations (10). Heightened public interest in children’s health may have implications for research priorities and directives. Therefore, it is useful to systematically analyze the impact of translational pediatric research broadly, across both academic and public spheres.

One useful way to understand the impact of a research program is through bibliometric portfolio analysis. Although publications are not themselves an end goal for translational endeavors, bibliometrics do describe a pivotal early stage in the process of translating new scientific discoveries to clinical use. Taking the further step of assessing how publications are shared and applied shines the light even further down the translational pipeline. Past bibliometric evaluations have typically relied upon citations within academic literature only (11-13). This approach is vulnerable to self-citation (11), and usually requires long windows of time, (i.e., at least several years), to observe slowly accumulating influence that does not generally speak to the public and translational impact of the research. Additionally, previous research has typically focused on journal-level, (e.g., journal impact factor, Eigenfactor score) (12,14,15) or investigator-level, (e.g., h-index, g-index) (11,15) impact, which are difficult to compare across diverse areas of pediatric research. Indeed, others have suggested that most current citation factors are ill-suited for comparing across scientific disciplines and that the use of multiple indicators is crucial (16,17). Traditional bibliometrics have been used to provide overviews that focus on long-term influence in particular areas of pediatric research (18-21), and among CTSA-supported programs (8,22). However, this is the first analysis to present complimentary bibliometric tools used together to understand and discuss both short-term attention and long-term influence of publications and shed light on how pediatric research is being used within both academic and public networks.

We first contextualize the research supported by the Georgia CTSA Pediatrics Program by summarizing the content of the publication portfolio that has amassed over 13 years of operations. Identifying the predominant subjects addressed by supported research aids in understanding the focus and diversity of the research. We also describe features of the portfolio that pertain to the translational spectrum. In the central analyses, we delineate both the short-term and long-term impact of the articles according to patterns that reflect ‘splashes’ or short-term, immediate, and emphatic forms of impact, and ‘ripples’ or long-term, accumulating, and enduring forms of impact. We combine established academic citation metrics with non-traditional ‘altmetrics’, or non-academic forms of public attention to publications (23), to form a comprehensive bibliometric review. To assess short-term ‘splash’ we examine bibliometric measures that reflect immediate academic exposure and potential for future citations (i.e., journal impact factor, peer faculty recommendations, and Mendeley reader downloads) as well as altmetrics that reflect early community attention and public interest/engagement (i.e., references in news stories, blog posts, and Twitter posts). To assess long-term ‘ripples’ we examine bibliometric measures of impact on the scholarly literature (i.e., total citations and Relative Citation Ratios calculated over time), as well as altmetrics that reflect downstream influence among the broader public (i.e., references in Wikipedia articles, patent applications, and public policy documents). We summarize overall and average citation impact metrics and altmetrics and provide case illustrations of several articles that have made notably big splashes or had broad ripple effects.

In sum, this article is intended to demonstrate an innovative methodological toolkit for assessing contributions to clinical and translational science that have been made possible by research supported by an NIH CTSA Pediatrics Program. Using complementary approaches, we examine the bibliometric products that have emerged from supported research and their translational impact. Results illuminate the scope and the short- and long-term impact of this research on academic literature, community discourse, technological advancement, and public health policy. We present the following article in accordance with the Narrative Review reporting checklist (available at https://tp.amegroups.com/article/view/10.21037/tp-21-506/rc).

Methods

Data collection

In fall 2019, internal records were queried for instances of program support received by pediatric researchers and other researchers collaborating on pediatric research, since the inception of the Georgia CTSA in 2007. A total of 224 Georgia CTSA investigators (most from Emory University and the affiliated Children’s Healthcare of Atlanta) and 988 instances of support were identified [predominantly (approx. 80%) clinical services and staff from the Georgia CTSA Clinical Research Centers and expert consultation support from the Informatics Program and the Biostatistics, Epidemiology, and Research Design Program]. These investigators were cross-referenced with publications queried from PubMed (24) as having formally acknowledged grant support from the Georgia CTSA (n=2,942). We selected publications that met all of the following inclusion criteria:

• Acknowledged a past or present Georgia CTSA NIH grant (UL1 TR002378, UL1 TR000454, UL1 RR025008, KL2 TR002381, KL2 TR000455, KL2 RR025009, TL1 TR002382, TL1 TR000456, TL1 RR025010) as having provided support
• Authored by at least one of 224 pediatric researchers from Georgia CTSA member institutions who received program support
• Contained clear pediatric content evidenced by pediatric keywords appearing in the journal title, article title, MESH terms, or Web of Science Research Area. Resulting publications were individually inspected for misattributed/aberrant keyword results.

Articles were excluded from analysis if they did not meet all three criteria. A total of 250 (8.5% of all Georgia CTSA-supported publications) articles met inclusion criteria and were included in further analyses (see Table 1). These publications were authored by 93 Georgia CTSA investigators who had received 630 instances of program support.

Next, in order to retrieve journal and content information, the list of publications was searched in Web of Science’s (WoS) subscription-based InCites application (25,26) in 2020; 236 indexed publications were found in WoS InCites, yielding a dataset that included the following for each article:

• Journal and Journal Impact Factor (JIF), a proprietary InCites metric, which is an unadjusted measure of typical citation rates for the journals in which each article was published. A JIF of five, for instance, means that the articles published in that journal in the past two years have been cited, on average, five times.
• Web of Science Research Area, the most granular categorization scheme for research content area available from InCites, which includes 252 subject categories across science, social science, arts and humanities; not all of which are expected to be applicable to clinical/translational pediatric research. The WoS Research Area is usually assigned based upon the content area of the journal in which the article is published. If the journal is general or multidisciplinary (e.g., New England Journal of Medicine, PlosOne, etc) then the article is assigned based upon its cited reference list and only assigned to the general category if no more specific designation can be made. It is typically not feasible to assign a journal/publication to a single category, therefore, up to six research areas may be assigned to a given journal and corresponding articles (27).

Next, in order to retrieve citation and translational feature information, the list of publications was searched in the NIH’s iCite application (28) in 2020. All 250 indexed publications were found in iCite, yielding a dataset that included the following for each article:

• The iCite Translational Features Module, which includes (I) percentages of research involving human, animal, and molecular/cellular research as designated through the triangle of biomedicine (29); (II) the Approximate Potential to Translate (APT) (30) a value between 0 and 1 reflecting the predicted percent likelihood that a paper will eventually receive a clinical citation; (III) designations as clinical articles and actual citations by clinical articles to date.
• Total academic citation count as of mid-2020.
• Relative Citation Ratio (RCR) (31), a state of the art field-normalized citation metric that calculates the citation impact of an article relative to the average NIH-funded paper in its co-citation network. RCR data is only available for publications that are at least one calendar year old and was available for 231 of the focal articles as of mid-2020.

Next, to retrieve author and altmetrics information, the list of publications was searched in Digital Science’s subscription-based Dimensions application (32) in 2020 (Dimensions also offers a free limited-access version). All 250 indexed publications were found in Dimensions, yielding a dataset that included the following for each article:

• Co-authors and their affiliated organizations.
• Altmetric Attention Score (AAS) (23), a rank-ordered index score, aggregated from a number of specific subcomponents, that reflects media and community attention paid to a published article as well as use of the article in subsequent public documents. Specific subcomponents of the AAS detailed in this study include references to publications in: news articles, blog posts, Twitter posts (33), Wikipedia articles (34), patent applications, policy documents, and Faculty Opinions [formerly F1000 Prime; (35)] peer faculty recommendations. We also included in our analysis reader downloads in the Mendeley Reference Management program (36), which are tracked by Dimensions but are not used in calculating the AAS.

Data analysis

First, in order to contextualize the content of this pediatric publication portfolio before the central analyses, we assessed descriptive characteristics of the portfolio, including the focus and diversity of the research via the frequency distributions of articles published in different journal outlets and representing different WoS Research Areas. To identify major clusters of research carried out in repeated collaborations, we employed network analysis for co-authorship data from Dimensions, using VOSviewer 1.6.15 [Visualization of Similarities (VOS)] (18). Major lines of work were then assessed for content based upon article content and author specialty. We then assessed the translational features of the portfolio, utilizing the metrics included in iCite’s Translational Features module.

In the main phase of analyses, to explore the splashes and ripples made by articles in this portfolio, we examined mean, maximum, and sum totals for journal- and article-level impact factors, Mendeley reader downloads, and Altmetric Attention Score components, describing the sources of altmetric attention. The short-term impact of the portfolio in the academic sphere is reflected by scholarly interest and readership that can be measured soon after publication and heighten potential for future citations. The journal impact factor, which is immediately available as it is based upon citation precedent, is associated with circulation, readership, and average rates of academic citation (27,37,38). Thus, articles published in high impact factor journals will, on average, receive more exposure within the academic community, and the impact factor is often used as a preliminary indicator of the expected influence of articles selected for publication in that journal. Official peer faculty recommendations in Faculty Opinions are reserved for articles thought to be of exceptional importance after qualitative review by a peer-nominated faculty of internationally-renowned researchers. Article downloads on the Mendeley reference management program is a particularly early measure of broad academic interest in a publication from around the globe. Identifying and downloading an article creates potential for that paper to be read, used, and cited in subsequent research. On the other hand, early interest in research in the public sphere is indicated by altmetrics that signify early community discourse on published research. Relevant components of the AAS for short-term impact include references to publications in news stories, blog posts, and Twitter posts shared to followers around the world, all of which tend to peak within days to weeks after publication.

Conversely, the long-term impact of the portfolio in the academic sphere is demonstrated by absolute and relative rates of academic citations accumulated over several years. Aggregate citation counts correspond to the amount of direct impact that the portfolio has had on the academic literature. The RCR corresponds to relative citation influence of articles within the context of their field of study and the time elapsed since publication. Long-term, downstream influence in the public sphere is indicated by relevant AAS components that reflect documented usage of the publications to inform public knowledge. References in broadly used and openly accessible Wikipedia articles reach and inform wide-ranging audiences. References in filed patent applications document usage of research in ways that may advance technology through new inventions or technological processes. References in policy documents from government agencies and NGOs document how research is used to influence public health policy.

Figure 1 summarizes the full metric toolkit of short-term splash versus long-term ripples analyzed in this article. Finally, we used the aforementioned metrics to identify several case example articles, with relatively high AAS values, that reflect high impact splashes and influential ripple effects, and present illustrations of the impact made by these articles.

Figure 1 Framework for bibliometric measures of short- and long-term impact within academic and public spheres.

Discussion

Portfolio summary: focus, diversity, and translational features

Descriptive analyses revealed that 250 Georgia CTSA-supported pediatrics articles were published at a steady rate from the inception of the program in 2007 through 2020. The articles were published in 112 different journals, most frequently Pediatrics (21 articles), The Journal of Pediatrics (20), and the Journal of Allergy and Clinical Immunology (18). The articles represented 44 WoS Research Areas, most often Pediatrics (104 articles), but with the most frequent field-specific areas being Immunology [42], Allergy [27], and Oncology [23]. The 93 Georgia CTSA authors collaborated with more than 1,550 co-authors from 236 different institutions across the US, as well as 16 other countries. The network of most frequent co-authorships (of at least 5 publications; 97 authors) indicated significant collaboration both within and across five major cohesive clusters centering on: premature infants, childhood asthma, liver and kidney disease, and autism.

The research in this portfolio covered a broad range of the translational spectrum from basic and foundational research intended to elucidate correlates, mechanisms, and molecular underpinnings of health problems to applied research intended to improve treatments, therapies, and protocols for particular health problems (39). The iCite Translational Features Module revealed that the majority (89%) of the research involved human subjects. The mean APT value for the portfolio was 0.67, meaning that on average, there is a 67% likelihood that articles in this portfolio will be translated to clinical use by being cited in a clinical article. Consistent with this, 59% of the articles have been cited by clinical articles thus far, and 25% are themselves clinical articles.

Big splashes & ripple effects

Table 2 summarizes findings for metrics that reflect splashes and ripples among the Georgia CTSA Pediatrics Program’s supported publications. Of the 250 articles, 93% had at least one academic citation and 70% had some form of altmetric attention.

Short-term impact in this portfolio is demonstrated by high average JIF, a number of rarely-granted recommendations from Faculty Opinions, thousands of Mendeley downloads and references in news stories (from sources such as Physician’s Briefing, MedicalXpress, and EurekAlert), blog posts (from blogs such as Neonatal Research, Latest BMJ Blogs, and JAMA Author Interviews), and Twitter posts (largely from science and medical professionals and organizations). These metrics have the advantage of typically manifesting at or soon after publication and providing indications of the potential importance and utility of recent, cutting-edge research without a long period of latency.

Long-term impact in this portfolio is demonstrated by thousands of academic citations, a mean RCR value indicating more than 3 times the expected number of citations based on field and publication year, references in Wikipedia articles (for Sepsis, Adrenarche, Sweetened beverage, and one for a study author), patent applications (such as for medications, formulas, and diagnostic tools), and policy documents (from agencies such as the World Health Organization, the Centers for Disease Control & Prevention, and international health agencies). These metrics have the advantage of portraying actual usage of articles in subsequent scholarly and public pursuits that move research along the translational pipeline.

Using this suite of metrics we identified specific publications that represented strong case examples of big splashes and broad ripple effects. Table 3 summarizes these case examples including a description of how Georgia CTSA Pediatrics Program supported the research and how the article represents a big splash or broad ripple effect, using the metrics in this toolkit. Examples a, b, and c (40-42) exemplify big splashes within both academic spheres (high impact factor journals, Faculty Opinions recommendation, Mendeley downloads) and public spheres (news, blog, and twitter references); examples d, e, and f (43-45) illustrate broad ripple effects across both academic (academic citations) and public spheres (Wikipedia, patent, and policy references).

Strengths and limitations

The limitations of using academic citations to measure impact have long been discussed (11,31,46), and newly updated metrics are periodically introduced to address the limitations of previous iterations (e.g., adding the g-index and m-quotient to the h-index to compare productivity among pediatric faculty) (47,48). In the context of translational goals, academic citations alone are especially limiting because they reflect influence within an insulated academic community that does not necessarily reach the relevant clinicians and patient populations at the bedside. Altmetrics, on the other hand, have the potential to capture some of the impact that research articles have beyond the boundaries of academia. One limitation of the relatively new AAS is that, while extensive, the metrics collected by Dimensions cannot be exhaustive of all community attention paid to research articles. Media communication is vast, ever-evolving, and sometimes ephemeral, and the AAS has been tracking some sources for less than ten years. We intentionally chose some of the most salient and well-tracked media platforms available (i.e., Twitter, Wikipedia), but the AAS includes additional, albeit, limited data from platforms such as Facebook, Google+ and Reddit, which we did not include in our analysis in the interest of more concise interpretability. Further, the content and quality of altmetric attention is not captured in the AAS and can be either positive or negative in nature, or may not have strong relevance for translational advancement. For example, Tweets can be susceptible to gaming by automated bots (49), which further reason to adopt multi-faceted methods for evaluating publications in the short term, which do not rely on singular metrics. Still, the AAS is a useful quantitative tool for identifying articles with significant attention to then examine their specific altmetrics on a more individual or qualitative basis. Indeed, Table 3 models the inclusion of twitter followers and users when showcasing media influence.

Our overall framework, summarized in Figure 1, provides a useful blueprint for using bibliometrics to compare/benchmark publication portfolios from different programs or centers, and to communicate findings to leadership and stakeholders as well as the general public. Indeed, demonstrating the impact of a funded program such as an NIH-sponsored CTSA center is critical when advocating for continued funding. The suites of metrics that reflect splashes and ripples leverage cutting-edge altmetrics together with established best practices for evaluating academic citation influence, forming a compelling methodology for illustrating impact, tailored to the specific characteristics of cross-cutting translational pediatric research. A strength of this framework is that it may be flexibly applied to other existing and emerging data sources, (e.g., PlumX metrics, clinical citations, Overton policy citations), as needed, to meet the needs and resources of a given program, such that the ‘academic sphere’ includes use by other researchers within the translational pipeline, the ‘public sphere’ includes use by those outside of the translational science pipeline, ‘short-term’ encompasses the first months and years after publication, and ‘long-term’ generally encompasses more than several years after publication. Although any one metric may yield an incomplete picture (e.g., journal-level impact factors represent averages across articles that may have with widely varying individual citation impact), we suggest that converging evidence of impact is key, whether it be academic and/or public, short and/or long term in nature.

Summary & conclusion

The short-term splashes and long-term ripples depicted by both academic measures and altmetrics tell the story of a pediatrics program’s research impact in ways that go further than traditional citation metrics and inform our understanding of how these research products are being used and translated in both the academic and the broader community. Our analyses characterize specific mechanisms by which this body of research is advancing findings from the bench to the bedside. Taken together, these findings establish the myriad ways that this research has engendered progress across diverse segments of pediatric medicine. Case examples further illustrate specific areas of notable influence.

This review provides a useful new methodological framework for illustrating the waves of impact made by pediatric research, that we advocate for as complementary to traditional publication evaluation metrics. Whether a publication makes a big splash of early interest that foreshadows eventual utility and impact, or shows ripple effects of cumulative influence over time, both reflect opportunities for a line of research to advance along the translational spectrum. In the future it may be useful to use bibliometrics to draw clearer connections between specific program inputs and relevant outputs. Future research could also further explore the connections between splashes and ripple effects. That is, does early attention paid to an article predict long-term influence, or do these forms of influence function separately? Past research has found a weak association between Altmetric score and citations among pediatrics publications (50), but others have identified ‘sleeping beauties’ in pediatric research that only become highly cited and influential after a long period with few citations (51). Additionally, assuming that all forms of altmetric attention are not equal, what are the implications of different types of near-term public attention? It is important to note that much of the altmetric attention observed in this review originated with academic or professional sources, such as a Tweet from a university account, or a blog written by a health professional, which may then be disseminated and re-shared broadly. These activities could bring findings to the attention of other researchers in adjacent fields, to relevant clinicians, or to potential patients or community stakeholders. This observation illustrates the dual responsibility of the scientific community in communicating findings through both scholarly and public channels, with a key distinction in the intended audience. Future research could benefit from qualitatively studying the pathways through which scientific knowledge flows through academic versus altmetric sources to both other researchers and to the public, and the implications of each in terms of potential inter-disciplinarity and translational advancement.

Acknowledgments

Funding: This work was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002378. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnote

Reporting Checklist: The authors have completed the Narrative Review reporting checklist. Available at https://tp.amegroups.com/article/view/10.21037/tp-21-506/rc

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://tp.amegroups.com/article/view/10.21037/tp-21-506/coif). AMF has received grants for the submitted work from NIH/NCATS under award number UL1TR002378. EJN reports that this work was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002378. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Committee to Review the C, Translational Science Awards Program at the National Center for Advancing Translational S, Board on Health Sciences P, Institute of M. The National Academies Collection: Reports funded by National Institutes of Health. In: Leshner AI, Terry SF, Schultz AM, editors. The CTSA Program at NIH: Opportunities for Advancing Clinical and Translational Research. Washington (DC): National Academies Press (US) Copyright © 2013, National Academy of Sciences.; 2013.
2. Califf RM, Berglund LPrincipal Investigators of National Institutes of Health Clinical and Translational Science Awards. Linking scientific discovery and better health for the nation: the first three years of the NIH's Clinical and Translational Science Awards. Acad Med 2010;85:457-62. [Crossref] [PubMed]
3. Comeau DL, Alvarez JA, Bhatti P, et al. Building Diverse Careers in Clinical and Translational Research: Evaluation of a Certificate Program in Translational Research. J Clin Transl Sci 2018;2:38-47. [Crossref] [PubMed]
4. Comeau DL, Escoffery C, Freedman A, et al. Improving clinical and translational research training: a qualitative evaluation of the Atlanta Clinical and Translational Science Institute KL2-mentored research scholars program. J Investig Med 2017;65:23-31. [Crossref] [PubMed]
5. Rollins L, Llewellyn N, Ngaiza M, et al. Using the payback framework to evaluate the outcomes of pilot projects supported by the Georgia Clinical and Translational Science Alliance. J Clin Transl Sci 2020;5:e48. [Crossref] [PubMed]
6. Morozumi K, Patel T, Kerr P, et al. Evaluating a CTSA-funded pilot grant program. J Clin Transl Sci 2020;5:e63. [Crossref] [PubMed]
7. Steketee M, Frechtling J, Cross D, et al. Final report on CTSA-supported publications: 2006-2011. Rockville, MD: Westat; 2021.
8. Yu F, Van AA, Patel T, et al. Bibliometrics approach to evaluating the research impact of CTSAs: A pilot study. J Clin Transl Sci 2020;4:336-44. [Crossref] [PubMed]
9. Llewellyn NM, Adachi J, Nehl EJ. Georgia Clinical & Translational Science Alliance (Georgia CTSA) 2017 Publication Content Evaluation. Atlanta, GA; 2018.
10. Llewellyn NM, Weber AA, Nehl EJ. Altmetric Impact of the Georgia CTSA Publication Portfolio from 2007-2020: Do Big Splashes lead to Ripple Effects on Translation? Atlanta, GA; 2021.
11. Carpenter CR, Cone DC, Sarli CC. Using publication metrics to highlight academic productivity and research impact. Acad Emerg Med 2014;21:1160-72. [Crossref] [PubMed]
12. Llewellyn N, Carter DR, Rollins L, et al. Charting the Publication and Citation Impact of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2016. Acad Med 2018;93:1162-70. [Crossref] [PubMed]
13. Garner RM, Hirsch JA, Albuquerque FC, et al. Bibliometric indices: defining academic productivity and citation rates of researchers, departments and journals. J Neurointerv Surg 2018;10:102-6. [Crossref] [PubMed]
14. Myers BA, Kahn KL. Practical publication metrics for academics. Clin Transl Sci 2021;14:1705-12. [Crossref] [PubMed]
15. Roldan-Valadez E, Salazar-Ruiz SY, Ibarra-Contreras R, et al. Current concepts on bibliometrics: a brief review about impact factor, Eigenfactor score, CiteScore, SCImago Journal Rank, Source-Normalised Impact per Paper, H-index, and alternative metrics. Ir J Med Sci 2019;188:939-51. [Crossref] [PubMed]
16. Ioannidis JP, Klavans R, Boyack KW. Multiple Citation Indicators and Their Composite across Scientific Disciplines. PLoS Biol 2016;14:e1002501. [Crossref] [PubMed]
17. Petersen AM, Wang F, Stanley HE. Methods for measuring the citations and productivity of scientists across time and discipline. Phys Rev E Stat Nonlin Soft Matter Phys 2010;81:036114. [Crossref] [PubMed]
18. Chhapola V, Tiwari S, Deepthi B, et al. Citation classics in pediatrics: a bibliometric analysis. World J Pediatr 2018;14:607-14. [Crossref] [PubMed]
19. Desai N, Veras LV, Gosain A. Using bibliometrics to analyze the state of academic productivity in US pediatric surgery training programs. J Pediatr Surg 2018;53:1098-104. [Crossref] [PubMed]
20. Jelicic Kadic A, Kovacevic T, Runjic E, et al. Research methodology used in the 50 most cited articles in the field of pediatrics: types of studies that become citation classics. BMC Med Res Methodol 2020;20:60. [Crossref] [PubMed]
21. Schwimmer MH, Sawh MC, Heskett KM, et al. A Bibliometric Analysis of Clinical and Translational Research in Pediatric Gastroenterology From 1970 to 2017. J Pediatr Gastroenterol Nutr 2018;67:564-9. [Crossref] [PubMed]
22. Qua K, Pelfrey CM. Using bibliometrics to evaluate translational science training: evidence for early career success of KL2 scholars. J Clin Transl Sci 2020;5:e24. [Crossref] [PubMed]
23. Elmore SA. The Altmetric Attention Score: What Does It Mean and Why Should I Care? Toxicol Pathol 2018;46:252-5. [Crossref] [PubMed]
24. PubMed. U.S. National Library of Medicine Pubmed 2017 [cited 2020 May]. Available online: https://www.ncbi.nlm.nih.gov/pubmed.
25. Clarivate Analytics InCites2021 [Available from: https://incites.clarivate.com/.
26. Clarivate Analytics Web of Science2016. Available online: https://webofknowledge.com/.
27. Thomson Reuters InCites Handbook II2014. Available online: https://services.anu.edu.au/files/system/indicators-handbook.pdf.
28. National Institutes of Health Office of Portfolio Analysis, iCite 2021. Available online: https://icite.od.nih.gov.
29. Weber GM. Identifying translational science within the triangle of biomedicine. J Transl Med 2013;11:126. [Crossref] [PubMed]
30. Hutchins BI, Davis MT, Meseroll RA, et al. Predicting translational progress in biomedical research. PLoS Biol 2019;17:e3000416. [Crossref] [PubMed]
31. Hutchins BI, Yuan X, Anderson JM, et al. Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level. PLoS Biol 2016;14:e1002541. [Crossref] [PubMed]
32. Dimensions: Digital Science 2021. Available online: https://www.dimensions.ai/.
33. Twitter. Available online: https://twitter.com/.
34. Wikipedia. Available online: https://www.wikipedia.org/.
35. Faculty Opinions. Available online: https://facultyopinions.com/.
36. Mendeley. Available online: https://www.mendeley.com/.
37. Carmine S. A study of journal publication attributes: Some considerations for academics in the information systems discipline. Webology 2009;6.
38. Moed H. Citation Analysis of scientific journals and journal impact measures. Current Science 2005;89:1990-6.
39. Khoury MJ, Gwinn M, Yoon PW, et al. The continuum of translation research in genomic medicine: how can we accelerate the appropriate integration of human genome discoveries into health care and disease prevention? Genet Med 2007;9:665-74. [Crossref] [PubMed]
40. Schwimmer JB, Ugalde-Nicalo P, Welsh JA, et al. Effect of a Low Free Sugar Diet vs Usual Diet on Nonalcoholic Fatty Liver Disease in Adolescent Boys: A Randomized Clinical Trial. JAMA 2019;321:256-65. [Crossref] [PubMed]
41. Maude SL, Laetsch TW, Buechner J, et al. Tisagenlecleucel in Children and Young Adults with B-Cell Lymphoblastic Leukemia. N Engl J Med 2018;378:439-48. [Crossref] [PubMed]
42. Sheehan WJ, Mauger DT, Paul IM, et al. Acetaminophen versus Ibuprofen in Young Children with Mild Persistent Asthma. N Engl J Med 2016;375:619-30. [Crossref] [PubMed]
43. Welsh JA, Sharma AJ, Grellinger L, et al. Consumption of added sugars is decreasing in the United States. Am J Clin Nutr 2011;94:726-34. [Crossref] [PubMed]
44. Langman CB, Greenbaum LA, Sarwal M, et al. A randomized controlled crossover trial with delayed-release cysteamine bitartrate in nephropathic cystinosis: effectiveness on white blood cell cystine levels and comparison of safety. Clin J Am Soc Nephrol 2012;7:1112-20. [Crossref] [PubMed]
45. Stoll BJ, Hansen NI, Bell EF, et al. Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network. Pediatrics 2010;126:443-56. [Crossref] [PubMed]
46. Coleman R. Impact factors: use and abuse in biomedical research. Anat Rec 1999;257:54-7. [Crossref] [PubMed]
47. Tschudy MM, Rowe TL, Dover GJ, et al. Pediatric Academic Productivity: Pediatric Benchmarks for the h- and g-Indices. J Pediatr 2016;169:272-6. [Crossref] [PubMed]
48. Klimo P Jr, Venable GT, Khan NR, et al. Bibliometric evaluation of pediatric neurosurgery in North America. J Neurosurg Pediatr 2014;14:695-703. [Crossref] [PubMed]
49. Haustein S, Bowman TD, Holmberg K, et al. Tweets as impact indicators: Examining the implications of automated bot accounts on Twitter. Journal of the Association for Information Science and Technology 2014;67.
50. Chang J, Desai N, Gosain A. Correlation Between Altmetric Score and Citations in Pediatric Surgery Core Journals. J Surg Res 2019;243:52-8. [Crossref] [PubMed]
51. Završnik J, Kokol P. Sleeping beauties in pediatrics. J Med Libr Assoc 2016;104:313-4. [Crossref] [PubMed]