Chapter 4: Evidence Review Procedures and Methods

Following approval by the working group of the key questions (KQs) and their scope (e.g., population(s), intervention(s), comparator(s), outcome(s), timing, setting, [PICOTS]), and by the Working Group Chair, Science Team Lead and Science Team Manager of the final feasibility report (see Chapter 3), the Evidence Review and Synthesis Centre (ERSC) drafts the protocol. Once approved by the Task Force and reviewed by stakeholders and journal peer reviewers, the ERSC conducts the reviews.

The protocol (and term “evidence review”) covers all of the KQs of interest to the Task Force, each of which may be answered in a slightly different manner; further, a KQ may at times be fully or partly answered by one or more existing systematic reviews with minor revisions by the ERSC such as changing certainty ratings for the Canadian context. Unless otherwise stated, all of the KQs are answered using current guidance for systematic review methods; this chapter focuses on current methods for conduct and use of systematic reviews. Throughout this chapter the term ‘review’ will refer to a systematic review unless otherwise specified.

For each of the protocol and evidence review manuscripts, anyone who meets the criteria set out by the International Committee of Medical Journal Editors (ICMJE) for authorship will be considered an author. For the protocol, this will include members of the ERSC, Science Team, working group members, as well as any clinical or content experts. For the evidence review, members of the ERSC are the only authors because most of the review conduct is undertaken independently by them, decisions on eligibility and other aspects are their sole responsibility, and they are accountable for all aspects of the work including its accuracy and integrity.

Although the ERSC are the authors of the evidence reviews, Task Force members have several roles during this stage: (i) the working group members and Task Force Chair are required to approve any major methods deviations (see section 4.7.3), (ii) any member of the Task Force may provide input on decisions by the ERSC for synthesizing findings (e.g., similarity of interventions when considering meta-analysis; blinded to study results) or interpreting the evidence (e.g., making judgements on the magnitude of effects including thresholds used for decision making), or for discussing the limitations of the evidence, and (iii) the Task Force reviews and approves the manuscript for submission to a journal. The clinical/content experts supporting the working group may also provide insight for synthesis and interpretation of the evidence. These roles are made transparent in the reporting of the evidence review and those contributing are appropriately acknowledged.

4.2 Types of Evidence Syntheses

For each guideline topic, the evidence synthesis usually consists of multiple reviews and there may be different review methods applied for each of the KQs depending on the availability of existing, recent systematic reviews and their ability to cover the scope of the KQs comprehensively. While all KQs usually require the use of findings from systematic reviews (including overviews of reviews), the ERSC may not need to conduct a de novo systematic review for each KQ. Four main options for each KQ include:

Use of one or more existing systematic reviews, with the addition by the ERSC of certainty assessment using GRADE (if not conducted) or review of GRADE assessments with revisions as needed, based on reporting in the review, and required to contextualize the evidence for the Task Force (e.g., apply different thresholds of minimal effect, additional indirectness to Canada or the target population). This option will not usually be conducted for the main KQ about effectiveness of the screening or prevention intervention, where the Task force prefers very recent searches. If multiple reviews exist for a KQ, an overview of reviews (i.e., option C) may be used.
An update to an existing review that is not deemed recent enough and/or otherwise suitable for use “as is”
De novo review of primary studies or systematic reviews (as the unit of analysis in an overview of reviews)
In some cases, there may be a mixture of these methods to fully answer a KQ, for example when an update to an existing review covers one intervention but not others.

Further, as described below in section 4.6.4., some de novo Task Force reviews of primary studies may rely to some extent on other existing reviews for certain aspects such as locating studies to a certain date, but are not considered to be updates because often more than one review will be used and the majority of the review steps (e.g., data extraction, analysis and GRADE) are undertaken de novo and not all aspects or studies in the existing review may be relevant.

Systematic reviews of interventions conducted for the Task Force follow methods that are considered mandatory in the current version of the Cochrane Methodological Expectations of Cochrane Intervention Reviews (MECIR) (1) or acceptable by the current version of A MeaSurement Tool to Assess systematic Reviews (e.g., AMSTAR-2) (2). If the methods do not adhere to these minimum standards they are explicitly stated as such. Decisions not to adhere to methodological expectations may occur when the resources and time of the method involved are thought to outweigh the expected benefits. The latter will be considered mainly for KQs targeted towards indirect evidence (e.g., treatment effectiveness for recommendations on screening). Any methods not described or deviating from this manual will be transparently reported in the protocol and evidence review manuscript. When KQs are not related to the effects of interventions (e.g., accuracy, prognosis, patient preferences), other current guidance is followed, as applicable.

4.3 Protocol Development

The protocol incorporates some of the information from the scoping document, to help support the importance of the topic and the rationale and scope for each KQ. While there should be few or minor changes to the scope of the KQs at this stage, the protocol describes in detail the choice for the review approach for each KQ (e.g., de novo systematic review, overview of existing reviews) and all the steps for answering each KQ. As much as possible, this document ensures the objectivity of the review methods; decisions on various steps such as study inclusion, data extraction elements, and data synthesis, including which features of the study interventions/exposure and populations are proposed as possible confounders or effect mediators/moderators, are set out before reviewing the evidence. In many but not all cases, an a priori threshold for an important effect for each critical and important outcome will also be developed and described. Each protocol is reported in manuscript format and using the suitable and current Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) guidance. All protocols are registered with either PROSPERO or another online registry (e.g., Open Science Framework).

4.4 Methods for De Novo Reviews of Intervention Effects

4.4.1 Literature Searches

The search strategies are developed by an information specialist/research librarian and peer-reviewed by another experienced librarian using the Peer Review of Electronic Search Strategies (PRESS) checklist (3). At minimum, three databases (MEDLINE^®, Embase and

Cochrane Central Register of Controlled Trials (CENTRAL)) are searched using sensitive search strategies combining controlled vocabulary (MESH and Emtree) terms with free-text words. The searches are designed to capture all potentially eligible study designs and any limits on date or setting are justified. Additional subject-specific databases may be searched if warranted and feasible.

For KQs on intervention effects, searches for grey or unpublished literature are also conducted by either the librarian or a reviewer experienced in these searches and knowledgeable on the topic. Clinical trial registries (e.g., WHO’s International Clinical Trials Registry Platform (ICTRP), ClinicalTrials.gov) and several major topic-relevant professional organization websites are searched. If additional sources are considered, CADTH’s Grey Matters list is consulted for topic relevant sites (4).

In addition to searching in electronic databases and grey literature, the reference lists of included studies and recent (past 2-3 years) relevant national guidelines and systematic reviews are scanned by one reviewer. For studies not previously screened, full text screening is done by two reviewers. The ERSC will scan reviews and guidelines identified during the database searches as well as during the guideline scoping phase. Also, during their review of the protocol, stakeholders identified for the topic by the Task Force Office are asked to indicate any published or ongoing studies that might be relevant to the evidence review.

4.4.1.1 Search updates

The database searches are updated prior to the expected date of publication of the evidence review and guideline. If the ERSC is aware of completed trials whose results may be posted on trial registries before publication, they will monitor the trial registries. The timing of performing the updates will be decided together by the ERSC, Science Team, and working group in order to keep the search dates recent for the guideline development and final guideline publication (e.g., <1 year old). The timing may depend on knowledge of ongoing studies relevant to the KQs. In some cases, search updates may only be done for the KQ most directly relevant to the guideline recommendation(s) (typically the first KQ). If study designs other than randomized controlled trials (RCTs) were originally considered, the updated search may be limited to RCTs. In these situations, the evidence review manuscript will document the decision and provide the rationale. If findings from any new studies are not incorporated into the review analysis (e.g., one small study that is highly unlikely to change conclusions), the studies will be cited with a brief discussion of their results and/or implications for the review’s conclusions.

4.4.2 Screening of Articles

The ERSC applies the inclusion and exclusion criteria specified in the protocol to the literature search results to identify articles relevant for the systematic review using standardized forms piloted by the ERSC. There are two stages of screening. Initial screening involves title and abstract review of citations identified from the literature search. Titles and abstracts are reviewed independently, with all excluded records being screened by at least two reviewers (e.g., liberal accelerated method). There is no resolving of conflicts at this stage; either reviewer can pass a study through to full text.

Following completion of the initial screening, a more in-depth screening takes place involving the review of the full text articles identified as potentially relevant in the initial screening. Articles are screened for inclusion independently in duplicate with conflicts resolved by an assigned arbitrator. If questions arise about the eligibility criteria that were not anticipated at the protocol stage; the working group or clinical/content experts may be consulted, with study results blinded in these cases, to provide input about changes/refinements to the eligibility of interventions, comparators, or populations. The final decision on study eligibility is made by the ERSC only. Consensus is not required on the exact reason for exclusion as long as two reviewers exclude the study. The ERSC strives to include all studies where the outcomes of interest are measured, regardless of whether results for the outcome(s) are reported. Apart from the primary report of each study (as defined in the protocol), any associated/companion publications that are used for results data are cited in the manuscript; other reports used to inform about the intervention or study design and/or assessments of risk of bias are not cited.

A flow diagram of the study selection process is created following the most recent PRISMA reporting guidance for relevant review types.

If the ERSC is informed of any additional papers that others (e.g., working group, Science Team, clinical/content experts, stakeholders) believe might be relevant for inclusion, these are screened accordingly.

4.4.3 Data extraction

Using the pilot-tested data extraction forms, all data, except for results (i.e., outcomes data), are extracted by a single reviewer and verified by a second reviewer for accuracy and completeness. Outcome result data are extracted independently and in duplicate with conflicts arbitrated by a third reviewer, if needed. Study authors are contacted if necessary to try to obtain more complete study (e.g., intervention details) or outcome data (e.g., in cases of suspected missing outcomes, to obtain study-arm specific data if only p values are reported). This contact occurs via email and includes two reminders 1-2 weeks apart.

4.4.4 Study-level Risk of Bias Assessments

The risk of bias of each outcome is assessed independently, in duplicate using study design-specific tools with conflicts resolved in consultation with a third reviewer, where appropriate. The tools chosen must allow for assessment of the major potential biases relevant to each study design. For randomized controlled trials the Cochrane risk of bias tool (ROB 2.0) is used for all de novo reviews. Any questions in other tools that relate to anything other than risk of bias (e.g., applicability or study reporting) will not be considered for assessment of risk of bias but may be used for considering the directness of the evidence during assessments of the certainty of evidence. Each tool used in the review is pilot tested by all team members involved in this step. For non-randomized studies, the ERSC will determine a priori which confounders are the most important to assess for studies to assist with risk of bias assessment. Results for each outcome-comparison and for each domain/potential bias are reported in tables or figures. Although not directly considered in the risk of bias assessments, the ERSC documents possible financial conflicts of interest and assesses whether they judge a study to be of “notable concern about conflicts of interest.”

4.4.5 Analysis: Preparation and Conduct

To help deciding on what form(s) of analysis to undertake, the studies are usually charted by their PICOTS variables, with details on the variations within one population (e.g., by age), intervention (e.g., by screening interval or thresholds) or outcome (e.g., timing of assessment, tool used for patient-reported outcomes; continuous or dichotomized). Other specifics such as the effect measure used by the authors will also be charted. Thereafter, any classifications/coding (e.g., of interventions to a framework) or data conversions (e.g., odds ratios to relative risks) are undertaken. From this step it is possible to consider which further analysis is most appropriate.

Meta-analysis will be considered if there is more than one study reporting on an outcome for a comparison. Several considerations will be used when deciding about meta-analysis; cases where it may not be conducted include large variations in clinical or methodological aspects, large heterogeneity of the effects between studies (e.g., opposing direction), incomplete reporting (e.g., only p value reported) by multiple studies, and high suspicion of missing studies or outcome data (from studies not reporting on the outcome). The ERSC does not rely solely on statistical heterogeneity (e.g., I²). If meta-analysis is not performed, other approaches with relevant guidance will be used such as narrative synthesis (5) or other methods of synthesis without meta-analysis (6,7).

Regardless of whether meta-analysis is undertaken, the data within studies may need to be manipulated before synthesis, as able, to use common effect measures as much as possible for better comparisons across studies. Any calculations (e.g., transforming data to similar measures of effect, combining the results from two arms in a study) or imputations (e.g., missing data on variance) undertaken during analysis follow Cochrane or other recognized methodological guidance, with statistical consultation when necessary. Findings from cluster designs where the authors did not account for this effect are re-calculated by the ERSC and documented as such. If meta-analysis is used, a method appropriate to the data is applied (e.g., considering and allowing for rare or zero event studies), and findings from studies not used in the meta-analysis are considered when making overall conclusions.

Although results of meta-analysis of binary data will typically use relative effect measures (relative effects or odds ratios), for interpretation of the findings these will be converted to absolute effects. Absolute effects are typically calculated by applying the relative effects to the median or pooled estimate of the control event rate across the studies, but specific methods will be reported. Additional assumed risks may be considered, for example incorporating Canadian data external to the studies included in the systematic review, to investigate possible impact on the estimated absolute effects.

Assessing heterogeneity. As specified in the protocol, sensitivity analysis (i.e., removing one or more studies based on methodological decisions by the reviewers or concern(s) in the studies) and sub-group analysis (i.e., comparisons of effects between two groups of studies based on a priori specified potential moderators in terms of population, intervention, comparison, setting or timing) will be considered in cases of heterogeneity in direction or magnitude of effects. Meta-regressions may also be considered if there are sufficient studies. The variable(s) chosen for the subgroup analyses may be determined based on key populations on which the Task Force desires to focus their recommendation. It may also include variables that, based on clinical input or other rationale from the literature, are thought to potentially impact the effects on the intervention. The ERSC uses several criteria for determining the credibility of subgroup findings. If highly credible, they may report findings from each subgroup separately and use these for further assessments and conclusions. If the subgroup findings are not highly credible, the ERSC may re-consider pooling the results.

Dealing with missing data. If there are ten or more trials reporting on an outcome-comparison, small study bias will be assessed graphically and statistically. At the level of analysis for each outcome-comparison, the ERSC will consider the extent of possible missing outcome data (e.g., one or more studies measuring, as per protocol, but not reporting on the outcome) and use this assessment when assessing the certainty of evidence. The ERSC will attempt to limit missing data by employing comprehensive searches (including grey literature) and by contacting authors for reports of unpublished studies (e.g., reported only in trial registries) and for potentially missing outcome data.

4.4.6 Assessing the Certainty of the Evidence

The ERSC relies on current guidance from GRADE for assessing the certainty of the evidence (8). They keep up to date on changes to guidance and report which guidance is used in their reports, as suitable. If GRADE guidance does not yet exist for a specific review type, the ERSC applies the principles and still rates the certainty of the evidence while reporting on their methods. For reviews of interventions the assessments always include the five domains of risk of bias, indirectness, inconsistency, imprecision, and publication/reporting bias. When non-randomized studies are assessed, there is consideration (when other limitations are not serious) of reasons to upgrade the certainty (large magnitude of effect, dose-response gradient, and the effects of plausible residual confounding). Consultation with GRADE Members of relevant methods groups may be sought.

Every outcome rated as important or critical by the working group is assessed, and the certainty reported, even if no evidence on the outcome is found. The ERSC usually applies a minimally or partially contextualized approach to rate the absolute effects in the analysis, whereby each outcome is rated separately without regard to findings for other outcomes or other considerations such as cost/resources. The certainty is assessed with respect to whether the effects meet or surpass the threshold(s) chosen for decision making, whether this is in terms of direction (i.e., any magnitude but classified as benefit or harm based on the null [i.e., OR or RR of 0]) or a particular magnitude of benefit or harm. Each protocol and evidence review manuscript will detail the approach taken and the methods for determining any thresholds of effect used for these assessments.

If the intervention reviewed for one of the KQs is considered indirect in relation to the main intervention of interest reviewed for another KQ (e.g., treatment effects and/or accuracy of a screening test for a guideline on screening), the ERSC does not rate down the evidence for indirectness; this indirectness is taken into account by the Task Force when developing their recommendations.

Results are presented in Summary of Findings Tables with data on the studies contributing to each assessment (number of studies, total sample size, study designs), relative and absolute effects (in natural frequencies, e.g., X fewer per 1000), event rates in the intervention and control groups, number-needed to screen or treat (if statistically significant), narrative statements for each assessment (i.e., “may” and “probably” used for low and moderate certainty; “very uncertain” used for very low certainty), and the certainty of evidence rating. In addition, footnotes to the table provide detailed rationale for each rating, by outcome.

4.4.7 Reporting and Availability of Data

The reviews are reported using the most current reporting guidelines (typically PRISMA or a PRISMA extension) (9) found on the EQUATOR library (https://www.equator-network.org). Highly comprehensive reporting of all stages of the review is provided in detailed appendices, including a list of excluded studies at full-text screening, database search strategies, and analyses regardless of whether findings are reported in the main text. Generally, all data extracted from the studies are included, and if not, made available via a publicly accessible database. The completed evidence review will be made openly available through the Task Force website regardless of whether the evidence review is published in a peer-reviewed journal.

4.5 Non-intervention Reviews

4.5.1 Reviews of Diagnostic Accuracy or Risk Prediction

Many of the methods described above apply to reviews on diagnostic test accuracy or risk prediction (using single or multiple risk factors/models), but there are some differences. For example, the search for literature may use different databases, may not include trial registries or may involve less extensive searching of grey literature. Guidance and checklists are available from Cochrane and elsewhere (10) for conducting and reporting of systematic reviews, risk of bias assessments, and analysis (e.g., Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis TRIPOD)(11–16). For accuracy outcomes, meta-analysis will typically require more than three studies per outcome. Another difference between these questions and those on intervention effects is that all studies (not just RCTs) start at high rather than low certainty using GRADE (17–21).

4.5.2 Reviews on Patient Preferences or Other Perspectives

Reviews on patient preferences for outcomes (i.e., the relative importance for patients of the potential benefits and harms from the interventions) are often undertaken to help the working group weigh the balance of the magnitude of effects between the benefits and harms of the intervention. These questions follow GRADE guidance (22,23), with any deviations noted. The findings may also help determine whether, and the extent to which the balance in effects may differ between different people. The ERSC generally considers quantitative, rather than qualitative, studies as most informative for this question because information on the magnitude of relative preferences of outcomes is sought.

Apart from reviews on outcome preferences, there may be interest in evidence on other patient perspectives or interventions to inform the Task Force when considering the domains of feasibility, acceptability, and equity during development of their recommendations. Preferences for and/or adherence to different interventions deemed effective may help determine what interventions are most acceptable and feasible to recommend. Further, evidence about the acceptability of the intervention(s) or barriers to undertaking the intervention(s) may also be reviewed. Findings from these reviews may inform the direction and strength of recommendations, although in many instances would be more relevant to statements or discussions about considerations when implementing the recommendations. In some cases, rather than systematic reviews by the ERSC, the Task Force will use existing studies or reviews or primary data collection from focus groups or other methods used by Task Force Knowledge Translation activities to help inform these domains in their decision-making framework. In these cases, the question of interest is not considered a KQ, and is not included in the evidence review by the ERSC.

4.6 Other Approaches to Using Systematic Reviews

4.6.1 Conducting Overviews of Reviews to Address a Key Question

Overview of reviews rely on systematic reviews as the unit of analysis. An overview of reviews may be considered for some KQs. In most cases, this will be when there are multiple interventions of interest in the KQ and recent reviews are available on all interventions. In some cases, some of the interventions relevant to a KQ will be addressed using an overview of reviews, while other interventions (e.g., newly available) require a de novo review. The ERSC relies on the Cochrane handbook for guidance when planning for overviews (Chapter V) (24). The protocol for the evidence review will report any specific approaches, for example when selecting reviews to include, or synthesis methods used. The ERSC will assess the certainty of the evidence using the principles of GRADE. However, there may be some limitations based on the reporting or methodology within the included reviews (e.g., lack of reporting on control event rates to calculate absolute effects). Reporting of overviews follows current guidance (e.g., PRIOR) (25).

4.6.2 Updating Task Force Systematic Reviews or Other Existing Systematic Reviews to Address a Key Question

When the guideline is an update of a previous Task Force guideline, any KQs for the updated guideline with very similar eligibility as in the previous review conducted for the Task Force will be considered for an update. Slight modifications in study eligibility (e.g., fewer interventions or outcomes, only RCTs) or analytic plan may be considered as long as the original review would have included all studies meeting the criteria for the update (i.e., the scope of the update is the same or narrower than the original review). The new guideline may add one or more KQs, and the methods used for these reviews would therefore not be considered updates.

For guideline updates and for new guidelines, there may be another existing review where an update by the ERSC could be undertaken. A close examination of the searches, eligibility criteria, methods for risk of bias assessments and analysis will be undertaken by the ERSC to determine suitability of an update. A quality check of the data extraction with a random sample of the studies will be undertaken, to ensure accuracy, before planning to rely on the previously extracted data. The scope of the original review may be broader but not narrower (e.g., excluding populations of interest) than that of interest to the Task Force. If the Task Force scope and methods requires major modifications to the existing review, for example limiting the population, study designs, requiring new data extraction due to differing outcome definitions, and using a different analytic approach, the existing review will be considered a de novo review with integration of the existing review (see 4.7.4) but not as an update.

For both types of updates, the ERSC may rely on the original review authors’ data extraction and risk of bias assessments. They will perform a search update but may modify the search strategy, for example when new indexing terms are available or there is a narrower scope. Any search to update the evidence will overlap by at least 6 months from the date of the search in the original review to ensure location of studies not yet indexed by the databases at the time of the last search. In general, the ERSC will perform new GRADE assessments even if no new evidence was found in the update. If the existing review is only used to help identify studies and the ERSC does not rely substantially on work of the other review team, the approach will be considered integration of another systematic review rather than an update (see 4.6.4).

4.6.3 Reliance on Other Existing Systematic Reviews or Overview of Reviews to Address a Key Question

For some KQs, the Task Force may be satisfied with using an entire review (systematic review or overview of reviews) conducted by another review group. Often this will be for KQs that are considered to provide indirect evidence (e.g., accuracy of screening tests or effects of treatment, for a recommendation on screening effectiveness), but may also include KQs related to the main intervention (e.g., screening vs. no screening) if, for example, the focus of the guideline is not on whether or not to recommend the intervention but on other issues such as for whom and how to deliver the intervention. In general, the review would need to be quite recent (search within 2 years of guideline publication), unless the certainty of evidence for most of the outcomes was deemed moderate or high based on older studies (and therefore newer studies would be unlikely to change conclusions). The scope of the review needs to be considered well matched to that of interest to the Task Force, without any major deviations, and the review is well conducted and reported (e.g., zero to one critical flaws using AMSTAR-2). In these cases, the ERSC reports on the review methodology and findings, and critically examines the certainty assessments in view of the Task Force’s perspective. In some circumstances, the ERSC may rate the certainty differently, due to, for example, use of different thresholds for important effects or when considering key areas of indirectness to the Canadian context. Any changes to the original review author’s conclusions will be transparently reported, with rationale.

4.6.4 Integrating Existing Systematic Reviews into De Novo reviews

The ERSC may integrate existing systematic reviews into de novo Task Force reviews of primary studies. These methods were developed by reviewing guidance by others and making some modifications to maximize efficiency in the review processes while maintaining rigour (26,27). The major focus is to rely, when possible, on studies identified from comprehensive searches in other reviews, so the ERSC does not duplicate efforts to locate studies; thus, searches are only updated to ensure that more recent studies are found. In some cases, there may be some reliance on data extraction or risk of bias assessments done in other reviews, if suitable (e.g., choice and definition of outcomes are exactly the same). However, the analysis of results and assessments of certainty in the evidence will typically be undertaken de novo by the ERSC. The findings of individual studies are always considered rather than basing conclusions on a combination of other reviews’ syntheses and a separate synthesis of individual studies. These are not considered updates or overviews of reviews, but rather studies from existing reviews are included in a de novo review following the latter’s methods. Reviews that are integrated are cited in the evidence review.

4.7 Review and Approval Procedures

4.7.1 Protocol

All protocols are reviewed and approved by the working group (full approval) and subsequently by the full Task Force (approval by non-objection). After this approval, the Task Force Office circulates the protocol to external Canadian stakeholders (who have signed confidentiality agreements and completed conflict of interest statements) for review using structured questionnaires. Refer to section 1.7.2 for approval procedures. Simultaneously, the ERSC submits the protocol for publication in a peer-reviewed open access journal.

The protocol must be reviewed by a minimum of three peer reviewers (coordinated by the journal and the Task Force Office), including one with topic expertise, one Canadian, and one methods expert, and by external stakeholder organizations (coordinated by the Science Team and Task Force Office). Once comments have been received from the stakeholders and journal peer reviewers, all input is carefully considered by the Task Force and ERSC, and revisions are made where suitable. The revised protocol manuscript and responses to the journal are reviewed and approved by the working group (full approval) and the full Task Force (by non-objection); if very minor changes (e.g., to background information, clarifying terminology) have been made, the Working Group Chair may be the only one to review the changes. Stakeholder and peer reviewer comments, names and affiliations are added in the ERSC response to the journal and are made publicly available by the journal. In addition, a copy of the written consent of stakeholders is available for review by the Editors-in-Chief of the journal.

In order for the ERSC to start the review(s), the protocol (i) will have been reviewed by stakeholders and/or at least one round of peer review, (ii) needs to be publicly available at a protocol registry site (e.g., PROSPERO record), or as an “in process” manuscript in an accessible form either by the journal or in a pre-print server, and (iii) is at a state that the ERSC and working group are satisfied that no major changes will be made to the eligibility criteria for the evidence review. The final version of the protocol may not have yet been published at this stage. A link to the published protocol manuscript is added to the Task Force website.

4.7.2 Evidence Review

Once the evidence review is completed, the ERSC prepares a draft manuscript for review, sequentially with revisions at each stage, by the Science Team and Working Group Chair(s), followed by the working group and clinical/content experts, and then the entire Task Force. For at least the first round of reviews by the working group and Task Force, a disposition table with responses to comments is prepared by the ERSC. After full approval by the working group followed by Task Force approval by non-objection (as per section 1.7.2), the manuscript is simultaneously circulated to stakeholders by the Task Force Office and submitted by the ERSC to a peer-reviewed open access journal for peer review. The stakeholders and journal peer reviewers are often but not always the same individuals who reviewed the protocol. Revisions, as suitable, are made by the ERSC based on these reviews, and the working group approves (full approval) and Task Force approves (approval by non-objection) re-submission to the journal. Unblinded responses to the peer reviewers are posted online by the journal with the final publication and the stakeholder comments and responses are made available in additional files with the published manuscript. A link to the published manuscript is added to the Task Force website.

4.7.3 Deviations from Protocol and Methods

Deviations from the protocol are described in the manuscript of the evidence review. Any proposed deviations from the approval procedures and evidence review methods described herein must be approved by the Task Force Chair and/or Vice-Chair, Working Group Chair, Methods Working Group Chair and Science Team Manager. This may include the use of methods guidance that has become available in between updates of this chapter.

References

Higgins JP, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, et al. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates [Internet]. 2022 [cited 2023 Mar 2]. Available from: https://community.cochrane.org/sites/default/files/uploads/MECIR Version February 2022.pdf
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017 Sep;358:j4008.
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016 Jul 1;75:40–6.
Grey Matters: a practical tool for searching health-related grey literature | CADTH [Internet]. [cited 2021 Oct 13]. Available from: https://www.cadth.ca/resources/finding-evidence/grey-matters
Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews A Product from the ESRC Methods Programme Peninsula Medical School, Universities of Exeter and Plymouth. 2006;
McKenzie J, Brennan S. Chapter 12: Synthesizing and presenting findings using other methods. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021) [Internet]. Cochrane. 2021 [cited 2021 Oct 13]. Available from: www.training.cochrane.org/handbook
Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ [Internet]. 2020 Jan 16 [cited 2021 Oct 13];368. Available from: https://www.bmj.com/content/368/bmj.l6890
Journal of Clinical Epidemiology. GRADE Series [Internet]. 2021 [cited 2021 Sep 21]. Available from: https://www.jclinepi.com/content/jce-GRADE-Series
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ [Internet]. 2021 Mar 29 [cited 2021 Oct 13];372. Available from: https://www.bmj.com/content/372/bmj.n71
Roqué M, García LM, Solà I, Alonso-Coello P, Bonfill X, Zamora J, et al. Toolkit of methodological resources to conduct systematic reviews [version 3; peer review: 2 approved] . F1000Research [Internet]. 2020 [cited 2021 Dec 10];9:82. Available from: https://f1000research.com/articles/9-82/v3
Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: A revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med [Internet]. 2011 [cited 2021 Oct 13];155(8):529–36. Available from: https://pubmed.ncbi.nlm.nih.gov/22007046/
Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: A tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med [Internet]. 2019 Jan 1 [cited 2021 Oct 13];170(1):51–8. Available from: https://pubmed.ncbi.nlm.nih.gov/30596875/
Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med [Internet]. 2013 Feb 19 [cited 2021 Oct 13];158(4):280–6. Available from: https://pubmed.ncbi.nlm.nih.gov/23420236/
TP D, JA D, KI S, J E, L H, JB R, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ [Internet]. 2017 [cited 2021 Oct 13];356. Available from: https://pubmed.ncbi.nlm.nih.gov/28057641/
Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. BMJ [Internet]. 2015 Jan 7 [cited 2021 Oct 13];350. Available from: https://pubmed.ncbi.nlm.nih.gov/25569120/
Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist. PLoS Med [Internet]. 2014 [cited 2021 Oct 13];11(10). Available from: https://pubmed.ncbi.nlm.nih.gov/25314315/
Schünemann HJ, Mustafa RA, Brozek J, Steingart KR, Leeflang M, Murad MH, et al. GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy. J Clin Epidemiol. 2020 Jun;122:129–41.
Schünemann HJ, Mustafa RA, Brozek J, Steingart KR, Leeflang M, Murad MH, et al. GRADE guidelines: 21 part 2. Test accuracy: inconsistency, imprecision, publication bias, and other domains for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables. J Clin Epidemiol. 2020 Jun;122:142–52.
Foroutan F, Guyatt G, Zuk V, Vandvik PO, Alba AC, Mustafa R, et al. GRADE Guidelines 28: Use of GRADE for the assessment of evidence about prognostic factors: rating certainty in identification of groups of patients with different absolute risks. J Clin Epidemiol. 2020 May;121:62–70.
Yang B, Mustafa RA, Bossuyt PM, Brozek J, Hultcrantz M, Leeflang MMG, et al. GRADE Guidance: 31. Assessing the certainty across a body of evidence for comparative test accuracy. J Clin Epidemiol. 2021 Aug;136:146–56.
Iorio A, Spencer FA, Falavigna M, Alba C, Lang E, Burnand B, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ. 2015 Mar;350:h870.
Zhang Y, Alonso-Coello P, Guyatt GH, Yepes-Nuñez JJ, Akl EA, Hazlewood G, et al. GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences—Risk of bias and indirectness. J Clin Epidemiol [Internet]. 2019 Jul 1 [cited 2021 Oct 13];111:94–104. Available from: https://pubmed.ncbi.nlm.nih.gov/29452223/
Zhang Y, Coello PA, Guyatt GH, Yepes-Nuñez JJ, Akl EA, Hazlewood G, et al. GRADE guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences—inconsistency, imprecision, and other domains. J Clin Epidemiol [Internet]. 2019 Jul 1 [cited 2021 Oct 13];111:83–93. Available from: https://pubmed.ncbi.nlm.nih.gov/29800687/
Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter 5: Overviews of Reviews. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021) [Internet]. Cochrane. 2021 [cited 2021 Oct 13]. Available from: https://training.cochrane.org/handbook/current/chapter-v
Gates M, Gates A, Pieper D, Fernandes RM, Tricco AC, Moher D, et al. Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. BMJ. 2022 Aug;378:e070849.
Robinson KA, Chou R, Berkman ND, Newberry SJ, Fu R, Hartling L, et al. Integrating Bodies of Evidence. Agency Healthc Res Qual [Internet]. 2015 Feb 26 [cited 2021 Oct 13];15(xx):1–25. Available from: https://www.ncbi.nlm.nih.gov/books/NBK279904/
Robinson KA, Chou R, Berkman ND, Newberry SJ, Fu R, Hartling L, et al. Twelve recommendations for integrating existing systematic reviews into new reviews: EPC guidance. J Clin Epidemiol [Internet]. 2016 Feb 1 [cited 2021 Oct 13];70:38–44. Available from: https://pubmed.ncbi.nlm.nih.gov/26261004/