Dear Editor

The rapid growth in medical research has popularized systematic reviews as comprehensive, accurate, reliable, and concise summaries of the best available information on a particular topic. ¹ The systematic review process involves a comprehensive search on a focused practical issue, followed by the inclusion of eligible studies based on clearly defined criteria, the quality assessment of each study, data extraction, and finally synthesizing the data from the included studies. In systematic reviews, the use of inappropriate tools for quality assessment or skipping this step entirely leads to inaccurate findings and increases research waste. In this editorial, we examine several quality assessment tools depending on their objective and provide tools that assess each component appropriately in light of the study designs.

When referring to quality in systematic reviews, quality of reporting and quality of conduct are the first two concepts that need to be distinguished clearly. While reporting guidelines are concerned with the quality of writing and whether all relevant components have been reported in adequate detail, quality appraisal tools concentrate on how well the study has complied with the methodological standards. Reporting guidelines, which are available on enhancing the quality and transparency of health research (EQUATOR) network website, can be beneficial for authors, as an outline for writing a good manuscript that is easy to read and likely to be cited, as well as editors and peer reviewers of scientific journals to make it easier to navigate the manuscript and ensure the quality of the paper. The last column of the table in the appendix provides the appropriate reporting guidelines for each study type/design (Appendix 1).

Misusing reporting guidelines instead of risk bias or critical appraisal tools can lead to a well-reported study with methodological bias be misinterpreted as “high quality” and a well-conducted study being judged as “poor quality” simply based on inadequate reporting. In other words, good-quality reporting does not guarantee the validity of the findings. Thus, using reporting guidelines for quality assessment in systematic reviews may lead to an inaccurate assessment of the findings’ validity.

Bias is defined as any systematic process at any stage of study design, conduct, and analysis that result in deviations of the study findings from reality. ^{2 , 3} Risk-of-bias assessment is a formal evaluation of the sources of bias, as well as the strength and direction of their effect on the findings of each particular study, to guide their synthesis. ^{2 , 4} Hence, when planning a meta-analysis, risk-of-bias assessment is the preferred method, which is carried out in of the following steps:

1- Selecting the appropriate tool according to the study design.

2- Using the checklist to identify the sources of bias in each study, and

3- Judging the risk of bias based on how each fault may have affected the findings.

These risks are frequently classified as low risk, some concerns (unclear risk), and a high risk of bias. Assessing the risk of bias is an essential component of a systematic review, because it completes the analysis and interpretation of the findings by demonstrating the reliability of the individual study findings. ⁵

Although the terms “risk-of-bias assessment” and “critical appraisal” are frequently used interchangeably in systematic reviews, ^{3 , 4} and both of them assess the validity of individual studies, they are not identical. Critical appraisal aims to assess the methodological quality, relevance, reliability, and applicability of the primary findings of the studies without considering how much these flaws may affect study results. ⁶ Notably, implementing all methodological precautions does not guarantee the absence of bias, and studies with methodological flaws may have a low risk of bias for some specific outcomes. In the absence of risk-of-bias assessment tools and especially when no quantitative synthesis is planned, critical appraisal tools can effectively replace the risk-of-bias assessment tools. ⁴

The Cochrane Review Group Editorial Board advises systematic review writers to assess the certainty of the evidence for their primary outcome measures. ⁷ GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) is a systematic approach designed for assessing the overall certainty and confidence in systematic review findings.

Although the GRADE approach was initially developed to address concerns regarding the effectiveness of interventions based on randomized and observational studies, it is now frequently used in other contexts, such as medical education research. Additionally, it can be used to assess either narrative or statistical syntheses. ⁸

Instead of focusing on individual studies, GRADE incorporates five domains (risk-of-bias assessment, inconsistency, indirectness, imprecision, and publication bias) to rate the overall level of confidence in the findings of the whole body of evidence (e.g., systematic review results, effect estimates from metaanalysis, etc.). ⁵ Importantly, risk-of-bias assessment is one domain of GRADE, making it impossible to implement GRADE without first determining the risk of bias in each study. These GRADE ratings can be used to summarize the findings of a systematic review as well as develop clinical practice guidelines.

In conclusion, risk-of-bias and critical appraisal tools assess the validity of each study. It is preferable to use risk-of-bias assessment tools for systematic reviews, unless such tools are unavailable. While reporting guidelines can be helpful for drafting and reviewing manuscripts to ensure that all required components of the study are well-reported, they do not assess the methodological quality and cannot be replaced by quality appraisal tools. Therefore, while risk-of-bias assessment, critical appraisal, reporting guidelines, and the GRADE approach share many components and domains, each tool has its special perspective and use. The appendix of this editorial is a powerful source for authors to choose the appropriate tools based on their requirements.

Conflict of Interest:

None declared.

References

1. Lasserson TJ, Thomas J, Higgins JP. Starting a review. Cochrane handbook for systematic reviews of interventions. 2019;1-12. DOI
2. Juni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001; 323:42-6. Publisher Full Text | DOI | PubMed
3. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009; 339:b4012. Publisher Full Text | DOI | PubMed
4. Furuya-Kanamori L, Xu C, Hasan SS, Doi SA. Quality versus Risk-of-Bias assessment in clinical research. J Clin Epidemiol. 2021; 129:172-5. DOI | PubMed
5. Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons: New Jersey; 2019. DOI
6. Aromataris E, Munn Z. JBI manual for evidence synthesis. Joanna Briggs Institute: Adelaide; 2020.
7. Furlan AD, Pennick V, Bombardier C, van Tulder M, Editorial Board CBRG. 2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group. Spine (Phila Pa 1976). 2009; 34:1929-41. DOI | PubMed
8. Murad MH, Mustafa RA, Schunemann HJ, Sultan S, Santesso N. Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med. 2017; 22:85-7. Publisher Full Text | DOI | PubMed