Procedure Manual

Archived Content

This archived information is provided for reference, research or record keeping purposes. It has not been altered or updated since it was archived. An updated version of the Procedure Manuel will be added in spring 2017.


This document replaces the first edition of the Procedure Manual, published in October 2011, and is intended for use by members of the Canadian Task Force on Preventive Health Care, the Evidence Review and Synthesis Centre, and the Prevention Guidelines Division of the Public Health Agency of Canada in the development of recommendations. This manual may also be useful for other organizations that are developing evidence-based guidelines.

1. Overview of the Canadian Task Force on Preventive Health Care

1.1 Mandate

The mandate of the Canadian Task Force on Preventive Health Care (CTFPHC) is to develop clinical practice guidelines (CPGs) that support primary care providers in delivering preventive health care.

The CTFPHC uses the same definition of primary care as the U.S. Institute of Medicine:

Primary care is the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and community (p. 31).1

1.2 Structure and function

The CTFPHC is an independent panel, composed primarily of clinicians and methodologists, that makes recommendations for clinical preventive services based on rigorous, systematic review and synthesis of evidence conducted by the Evidence Review and Synthesis Centre (ERSC). The CTFPHC makes recommendations directly to its key constituency of primary care providers, but its work is also directly relevant to other health care professionals, developers of preventive care programs, policy-makers, and Canadian citizens. It uses standardized methodology and transparent processes to review and synthesize evidence, to weigh the balance of benefits and harms, and to make recommendations. The CTFPHC also develops and fosters linkages between primary care and community or public health programs that support clinical preventive services, as well as linkages to enhance the dissemination and uptake of its recommendations. Finally, the CTFPHC works with researchers to advance the evidence base supporting preventive care.

The recommendations of the CTFPHC are aimed at improving clinical practice and promoting public health. The CTFPHC provides recommendations about primary and secondary preventive services targeting clinically relevant conditions. The services must be provided in primary care settings or available through primary care referral. Primary prevention is the prevention of a target condition in healthy patients and takes the form of activities such as counselling and chemoprevention. Secondary prevention is directed to asymptomatic individuals who have risk factors for a condition or preclinical disease but who do not have clinically evident disease.

1.3 Governance

The CTFPHC has independent decision-making authority in all aspects of its scientific mission, including the following activities:

  • final decisions about topics to be covered
  • setting of standards and expectations for review and synthesis of the evidence
  • development, public declaration, and dissemination of its recommendations

The CTFPHC contributes to improvements in the quality of health care by:

  • systematically reviewing the evidence for the effectiveness of preventive interventions in personal health care
  • producing high-quality evidence-based guidance for primary care practitioners
  • producing evidence-based recommendations for clinical preventive interventions
  • identifying gaps in current knowledge in clinical preventive care
  • focusing on health issues and interventions that will have the largest impact in improving the health of Canadians
  • being a source of expert knowledge independent of government policy

1.4 Overview of roles

1.4.1 Role of the CTFPHC

The CTFPHC is responsible for prioritizing the topics that will be reviewed and works with the Office of the CTFPHC (TFO) at the Public Health Agency of Canada (PHAC) to define the analytic framework and scope of each topic. The CTFPHC works closely with the ERSC and the TFO in preparing systematic reviews and developing recommendations for each topic. The CTFPHC is also primarily responsible for leading knowledge translation (KT) and dissemination activities and assists key stakeholders in designing and implementing an evaluation strategy to assess the impact of the CTFPHC’s products.

1.4.2 Role of the TFO

The TFO at the PHAC provides scientific support to the CTFPHC. A scientific research manager is assigned for each topic, and this person coordinates and provides scientific support during development of the systematic reviews and recommendations for the topic. A scientific research manager also serves as the co-chair of the various working groups of the CTFPHC.

1.4.3 Role of the ERSC

The ERSC conducts the systematic reviews that are used as the basis for the CTFPHC’s recommendations. The ERSC follows documented methods for its reviews of the topics specified by the CTFPHC and follows the methods of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group2 in assessing the evidence.

1.4.4 Role of the University of Alberta

PHAC and the University of Alberta signed a Contribution Agreement in 2012, with the aim of enhancing the CTFPHC’s ability to disseminate its recommendations to primary care practitioners. Under the agreement, the university provides KT, stakeholder engagement, evaluation, and administrative support to the CTFPHC.

1.5 Membership

The TFO will periodically solicit nominations for new members of the CTFPHC, including the chair and vice-chair, by contacting stakeholder groups and through other appropriate channels. Both new and previous nominations will be considered. Nominations may also be submitted by current members of the CTFPHC. Once one or more individuals have been nominated, the CTFPHC appoints a selection committee composed of the current chair and vice-chair of the CTFPHC, two other current members of the CTFPHC, one representative of PHAC, and one external representative appointed on a rotating basis by the chair and vice-chair of the CTFPHC. The selection committee reviews the qualifications of each nominee in relation to the required qualifications for selection (Table 1) and makes its recommendations for appointments directly to the CTFPHC, which votes on the recommendations. The Joint Appointment Committee (consisting of the Chief Public Health Officer and a representative from the College of Family Physicians of Canada [CFPC]) then reviews and approves the recommendations. After approval, new members are appointed to the CTFPHC.

Table 1. Qualifications for appointment to the CTFPHC.
  • National and international recognition in respective field of expertise
  • Knowledge and experience in critical appraisal of peer-reviewed publications
  • Knowledge and experience in systematic review methods
  • Knowledge and experience in applying evidence to decision-making or policy-making
  • Expertise in disease prevention and health promotion
  • Demonstrated ability to collaborate with peers
  • No conflict of interest (including financial or intellectual conflicts) that would impair the integrity of the CTFPHC
  • Expertise in methodology (e.g., medical decision-making, clinical epidemiology, health economics)

Adapted from the procedure manual of the US Preventive Services Task Force.3

1.5.1 Selection of chair and vice-chair

The selection committee makes a recommendation for the chair of the CTFPHC to the Joint Appointment Committee, who appoints the chair. In turn, the chair makes a recommendation for the vice-chair, on the basis of discussions with the Joint Appointment Committee, which then appoints the vice-chair.

1.5.2 Selection of members

For the inaugural membership of the CTFPHC, the selection committee made recommendations for individual members to the Joint Appointment Committee, which subsequently appointed the new members. When additional members are needed, the selection committee will make recommendations according to the criteria listed in Table 1. Individuals recommended for membership must be ratified by a majority vote of current members of the CTFPHC and are then appointed by the Joint Appointment Committee.

1.5.3 Terms of service

The CTFPHC consists of 12 to 16 voting members, consisting of 10 to 14 members, one vice-chair, and one chair. The initial term for each member, including the chair and the vice-chair, is three years, with a possible one-year extension. The extension of terms will be staggered, to ensure continuity of membership over time and overlap of terms. When members leave the CTFPHC, they are expected to continue keeping all CTFPHC business confidential. As members rotate off the CTFPHC, they are offered the opportunity to continue working on guidelines that are already under way. Working group chairs are asked to remain available for consultation about the need for updates on their topics for one year after their terms are complete.

1.5.4 Non-voting members of the CTFPHC

The chair may assign ex officio members to support the work of the CTFPHC.

The Director of the TFO will sit as a non-voting member of the CTFPHC. Scientific staff at the TFO are not official CTFPHC members but work with the CTFPHC through all stages of the guideline development process

1.6 Quorum and voting

A quorum for official votes is two-thirds of the members, including the chair. Major decisions about procedures and methods, recommendations, clinical practice statements, and the selection of new members all require a non-secret vote. Votes are taken by hand, by voice, or by proxy, and voting can be done electronically if necessary (e.g., if a quorum is not available during an in-person meeting). Members with a potential conflict of interest related to the topic of a particular vote must recuse themselves and are not eligible to vote. Votes are recorded as yes, no, abstain, or absent. In cases where a decision is not unanimous, no minority reports are permitted.

1.7 Conflict of interest

1.7.1 Declaration of affiliations and interests form and checklist

Before a potential participant (such as a peer reviewer, clinical expert, or review team member) becomes engaged in a CTFPHC-led initiative, he/she must disclose any information that might prevent him/her from discussing a specific topic in an impartial manner, by completing the Declaration of Affiliations and Interests Form and Checklist (Appendix I). This form is used to report any potential conflicts of interest (e.g., financial, business or professional, intellectual).

Disclosure is required from each participant for each new topic. Furthermore, each participant is responsible for informing PHAC of any changes that have occurred since a person’s initial disclosure. CTFPHC members must also sign the Declaration of Affiliations and Interests Form upon joining the CTFPHC and before each in-person meeting.

The form will be administered and collected by the TFO. A preliminary review will be completed by the scientific research manager. Approval may be sought from the chair of the CTFPHC topic working group, the CTFPHC chair, and the TFO director in cases where a potential participant is deemed ineligible to contribute or in the event that recusal from participation is recommended.

Completed forms are filed at the TFO. Declarations for CTFPHC members are made public on the website. The intention of the PHAC is to keep personal information for reviewers and others involved in the guidelines confidential, and it will comply with all applicable laws pertaining to privacy and confidentiality in dealing with the information of those who participate in CTFPHC reviews.

1.7.2 Confidentiality agreement

Upon joining a new project, each participant must sign a Confidentiality Agreement (Appendix II), whereby he/she acknowledges that all documents and information that he/she may receive from PHAC or the CTFPHC or that he/she may develop while working on a CTFPHC review or guideline are strictly confidential and shall not be disclosed to any third party without the prior written consent of the CTFPHC.

By way of the Confidentiality Agreement, each participant agrees not to use any confidential information for any purpose other than those indicated by the PHAC or the CTFPHC.

The Confidentiality Agreement must be signed upon joining any PHAC-led initiative and has no expiry. Completed forms are filed at the TFO.

1.7.3 Process for determining appropriate actions

The TFO reviews the disclosure forms in consultation with the chair or vice-chair of the CTFPHC to recommend the appropriate course of action, if any (see Table 2 for possible actions).

In making recommendations for action, the TFO and the chair and vice-chair of the CTFPHC will consider the transparency, integrity, and acceptability of CTFPHC recommendations and products. The recommended action will be reported to the affected member and kept on file. Even if the recommended action is to allow the member to participate in discussions and vote on the topic, the member may withdraw from discussions or voting on a topic at any time if he/she feels it is appropriate to do so.

Table 2. Possible actions following disclosure of potential conflict of interest
Action Description
No action No recusal necessary.
Disclosure of information to CTFPHC only Member may discuss and vote on the topic and may serve as discussion leader.
Recusal from topic lead, disclosure of information to CTFPHC Member may discuss and vote on the topic, but may not lead the discussion.
Recusal from all participation, disclosure of information to CTFPHC

Member may not lead or participate in discussion or vote on the topic and will not be present for discussion and voting.

Recusal will be noted with published recommendations.

Recusal from all participation in CTFPHC Member will no longer participate in any CTFPHC activities.

Adapted from the procedure manual of the US Preventive Services Task Force.3

1.8 Public activities

CTFPHC members are encouraged to discuss, disseminate, and defend the recommendations of the CTFPHC in public forums.

1.8.1 Dealing with the media

Under the Contribution Agreement, the University of Alberta coordinates responses to media inquiries. The chair, vice-chair, or other members of the CTFPHC may make comments or statements to the media at the discretion of the chair and vice-chair.

1.8.2 Expert testimony

Members of the CTFPHC may provide expert testimony on topics that have been considered by the CTFPHC. Any member of the CTFPHC who, within the previous five years, has provided expert testimony or has reviewed a case related to a topic to be considered by the CTFPHC must disclose these activities through the disclosure process described in section 1.7.1. To avoid potential financial conflicts of interest, members of the CTFPHC should refrain from accepting more than $10,000 per year for testimony or review.

1.9 Authorship

Authorship for journal articles or other documents for public dissemination is assigned in accordance with the recommendations of the International Committee of Medical Journal Editors. Other contributors who do not meet the criteria for authorship may be acknowledged.

2. Overview of the guideline development process

Figure 1: Steps in guideline development, associated meetings, responsible parties, and products.

Overview of the CTFPHC guideline development process, described below.

Further details are provided in the text under the corresponding section numbers.

CFPC = College of Family Physicians of Canada; CTFPHC = Canadian Task Force on Preventive Health Care; ERSC = Evidence Review and Synthesis Centre; GRADE = Grading of Recommendations Assessment, Development and Evaluation; KT = knowledge translation; PICO = population, intervention, comparator, and outcomes; PR = peer review; SR = systematic review; TFO = Office of the CTFPHC; WG = working group

The guideline development process starts at step 3.1.

  • Step 3.1 is topic selection; this section is lead by the topic working group and receives input from the CTFPHC members and and the CFPC.
  • Step 4.2 involves a scoping exercise that is lead by the OCTFPHC and ERSC. A scoping review document is produced from this step.
  • Step 4.2 is topic refinement; this involves discussing the scope and PICO, drafting outcomes and harms, and discussing questions. This step usually occurs during the first meeting and is lead by the ERSC. Input for this step is provided by the working group, OCTFPHC, and KT Program.
  • Step 5 is protocol development; there are six stages to protocol development.
  • Step 5.1 involves ranking outcomes and harms while drafting questions. This step is led by the ERSC predominantly over email. Input for this step is provided by the working group and OCTFPHC.
  • Step 5.2 occurs during the second meeting to finalize outcomes and harms, and research questions. This step is lead by the ERSC with input from the working group and KT program.
  • Step 5.3 involves drafting the protocol and having the working group review the protocol. The first draft protocol is produced in this step. This step is lead by the ERSC with input from the working group and OCTFPHC.
  • Step 5.4 occurs during the third meeting and involves discussing the protocol and addressing any issues. This step is lead by the ERSC with input from the working group and OCTFPHC.
  • Step 5.5 involves having the protocol peer-reviewed. This step also addresses any public relations comments. This step is led by the ERSC and occurs predominantly over email. Input is provided by the working group and OCTFPHC.
  • Step 5.6 involves having the CTFPHC review and approve the final protocol. The final protocol is registered with PROSPERO and posted on the CTFPHC website. This step is lead by the ERSC with input from the working group, CTFPHC, and OCTFPHC.
  • Step 6 is the systematic review; there are four steps for the systematic review.
  • Step 6.1 involves drafting the systematic review. This is led by the ERSC and all correspondence occurs over email. In this step a preliminary draft of the systematic review is produced.
  • Step 6.2 involved submitting a draft of the systematic review and reviewing the draft. This is led by the ERSC and all correspondence occurs over email. Input is provided by the working group and OCTFPHC.
  • Step 6.3 occurs during the fifth meeting. A draft of the systematic review (including GRADE ratings) is presented, and comments are addressed. The ratings are also discussed at this meeting. This is led by the ERSC and input is provided by the working group and OCTFPHC.
  • Step 6.4 involves revising the draft of the systematic review based on the working group feedback, and sending the systematic review to be peer-reviewed. Then all public relations comments are address and the systematic review is finalized. The final systematic review is produced in this stage.
  • Step 7 is guideline development; there are four stages to guideline development.
  • Step 7.1 involves drafting the recommendations. This is led by the working group chair, OCTFPHC, KT WG. Put is provided from the working group and ERSC. In this step the draft recommendations and draft KT tools are produced. This step occurs through email communication.
  • Step 7.2 involves discussing the draft recommendations at the sixth meeting. This step is led by the working group chair and OCTFPHC. Input is provided from the working group and ERSC.
  • Step 7.3 occurs over email and involves having the recommendations peer-reviewed. Any feedback is also addressed while finalizing the recommendations. This step is led by the OCTFPHC, the working group chair, and the working group. Input is provided by the CTFPHC and the ERSC.
  • Step 7.4 is the final step in guideline development. It involves presenting the final documents and voting at the seventh meeting. The final documents also also submitted for publication. In this step the final recommendation and KT tools are produced. This step is led by the working group chair with input from the OCTFPHC, CTFPHC, and the KT Program.

Figure 2. Process and timelines for review of a guideline, highlighting points in the process at which external review occurs

Outline of the process and timelines for review of a guidelines

There are six stages of the recommendation development process and external review occurs alongside each step.

  1. Step 1: Initiation

    External Review Activities—In this step an external review plan is developed, including identifying content experts/peer reviewers and stakeholder organizations.

  2. Step 2: Key Questions and Analytic Framework

    External Review Activities—This step involves engaging reviewers and confirming agreement to participate via email. There is also a formal request for completion of confidentiality forms and conflict of interest forms.

  3. Step 3: Protocol

    External Review Activities—External review of draft protocol.

  4. Step 4: Systematic Review

    External Review Activities—External review of draft systematic review.

  5. Step 5: Recommendation

    External Review Activities—External review of draft recommendations.

  6. Step 6: Knowledge Translation Tools

    External Review Activities—External review of draft knowledge translation tools.

The steps in the guideline development process are shown in Figure 1. Once a topic is identified and prioritized for a review, a working group is formed and the ERSC is notified. A protocol is prepared, the systematic review is prepared, and the guideline is developed. Details on these steps are described in sections 3 to 7. Working groups with responsibility for topic prioritization, methods, and KT assist in this process (see Appendix III).

2.1 External linkages and peer review

The CTFPHC identifies appropriate stakeholder groups and related organizations to ensure effective and meaningful external linkages. These external linkages take several forms and serve a variety of functions, including the following:

  • expert and peer review of protocols, drafts of reviews, final systematic reviews, and final reports and recommendations
  • feedback on reviews and recommendations for specified topics
  • enhancement of KT tools
  • suggestions of topics that might be reviewed

External linkages are developed with formal organizations (e.g., professional societies), government-related organizations, and, in certain cases, selected groups of practitioners. These organizational linkages are intended primarily to represent end-users who use the CTFPHC recommendations in their day-to-day practice. External linkages may also be set up with organizations that are in a position to enhance KT and uptake of CTFPHC recommendations.

Documents prepared by the CTFPHC are reviewed externally at several points during guideline development. Figure 2 shows the key steps in the process at which such input is solicited.

A list of peer reviewers and stakeholder organizations suitable for each document is identified by the TFO on the basis of scientific, clinical, or topic specific expertise. This list is approved by the working group and the CTFPHC chair.

Each potential reviewer (whether an individual or an organization) is asked to declare potential conflicts of interest using the standard declaration form (as described in section 1.7.1 and must sign the Confidentiality Agreement (Appendix II). Actions related to declared conflicts, including disqualification of potential reviewers, are managed by the chair of the working group and/or the chair of the CTFPHC in accordance with the conflict of interest guidelines of the CTFPHC, as detailed in section 1.7). Potential reviewers are advised at the time of invitation that their participation will be acknowledged on the CTFPHC website and in technical documents published by the CTFPHC.

Once peer review for a particular guideline is complete, the working group reviews all of the comments and prepares a written response, in the form of a summary document. The review comments and the working group’s responses are shared with the entire CTFPHC for consideration during review of the final version of the guideline and its products.

Review occurs at four steps in the development of a guideline: the protocol, the draft systematic review, the draft guideline, and the KT tools.

The protocol for a review, once developed, is approved by the full CTFPHC and undergoes formal peer review by stakeholders and experts. The ERSC also coordinates peer review of the analytic plan by a methodologist and review of the search strategy by a library scientist.
Draft systematic review
Once complete, the draft systematic review is reviewed, revised, and approved by the working group. It is then sent to three to six content experts and/or one review methodologist and to a variety of stakeholders for additional review. Some or all of these reviewers may have participated in reviewing the protocol. The draft systematic review and all of the reviewers’ comments are then presented and discussed at a meeting of the CTFPHC. If no meeting is scheduled, dissemination and discussion take place by email.
Draft recommendation
The draft recommendations are approved by all members of the CTFPHC and are then sent to external peer reviewers and stakeholders for additional review.
Knowledge Translation
Usability and heuristic testing is conducted on all KT tools.

When the final guideline is submitted to a journal for publication, is also subject to the journal’s peer review process.

3. Topic prioritization

3.1 Topic selection process

Topics are identified by members of the CTFPHC, stakeholders, practitioners, the PHAC, the ERSC, other organizations, and individuals. The CTFPHC may also solicit topic nominations (see Appendix IV).

The TFO periodically updates the list of topics to be considered for new reviews and review updates in the coming year. This list includes all nominated topics and any additional topics identified by CTFPHC members or stakeholders.

Topics are included on the list if they meet appropriateness criteria, specifically those meeting the mandate of the CTFPHC and having effective treatments available. Topics that meet the appropriateness criteria are then assessed against the following selection criteria:

  • Timing of most recent review: Priority is given to topics that have not been examined by the CTFPHC within the past five years.
  • Availability of new evidence: Priority is given to topics for which new or controversial evidence, which might lead to a change in existing recommendations, has emerged since the last time the topic was reviewed by the CTFPHC.
  • Input from primary care practitioners: Priority is given to topics that will address the needs of primary care practitioners. Input on such topics is obtained through communication with the CFPC.

The topic prioritization working group independently assesses the list and selects the topics they think best reflect these criteria. This process should reduce the number of topics to a short list, with a maximum of 30 topics. If necessary, further discussion and consensus are used to reduce the list to 30 topics.

The members of the topic prioritization working group examine and subjectively rank the short list according to the following criteria:

  • disease burden (prevalence, mortality, comorbidity, quality of life) and expected effectiveness of the preventive service in decreasing that burden
  • potential impact of recommendations in clinical practice
  • interest of the public or care providers
  • variation in care and potential for preventive service to decrease that variation
  • sufficiency of evidence
  • availability of new evidence

In the ranking process, all criteria are considered equally (i.e., the criteria are not weighted). The prioritization process takes into account the requirement that the topics for each year should cover various disease types, populations, and types of services (screening, prevention). Topics are then classified according to whether they will be the subject of new reviews, updates, partnerships, or critical appraisals.

Mean rankings for each of the topics are calculated, and this information is fed back to the working group members, who are then asked to re-rank the topics. If consensus on the top 10 topics is not achieved in the ranking process, the group is asked to discuss their results and reach a consensus. This list of potential topics and their respective priorities is then presented to the CTFPHC as a whole for discussion and approval.

Although this process is used as a guide, the CTFPHC maintains the flexibility to modify the process as required to take advantage of scientific developments or timely opportunities for partnerships.

3.2 Literature surveillance

A scan of the literature occurs every 12 months to capture information published that may affect current and previous CTFPHC guidance. This process is specific to guidance published by the revitalized CTFPHC since 2010. Other guidance may be considered for opportunistic updates based on available systematic reviews by other organizations, and will be reviewed on a case-by-case basis. Details on the literature surveillance process are in Appendix V.

3.3 Topic categories

CTFPHC topics are classified as either active or inactive. Inactive topics are those that the CTFPHC has considered but decided not to address, because they are no longer relevant to clinical practice, are not relevant to the primary care setting or for primary care providers, have a low burden in terms of public health, or are otherwise determined to be beyond the scope of the CTFPHC. Active topics (as described below) are posted at the CTFPHC website.

The CTFPHC considers five types of active topics:

  • new (de novo) topics (i.e., topics that the CTFPHC has not previously addressed)
  • topics for updates (i.e., topics that the CTFPHC or other national guideline organizations have addressed in the past)
  • topics for critical appraisal (i.e., topics for which guidelines by other major guideline organizations will be critically appraised)
  • topics for which existing CTFPHC recommendations will be reaffirmed
  • topics that will be referred to another organization

Each type of active topic has a defined process and timeline, along with specific resource requirements for associated activities and document formats arising from those activities (Table 3). Additional detail about each type of active topic is provided below.

3.3.1 New (de novo) topics

Subsequent sections of this manual describe the processes for developing protocols (section 5), preparing systematic reviews (section 6), and making recommendations (section 7) related to new topics.

3.3.2 Topics for updates

The processes for preparing updates for active topics are similar to those for new topics, as described in sections 5 to 7, with some exceptions. The process for updates applies to both CTFPHC guidance and guidance produced by other groups. Section 6.8 describes the process for updating reviews conducted by other organizations.

3.3.3 Topics for critical appraisal

The CTFPHC appraises guidance produced by other organizations as a complement to the production of its own CPGs. The appraisal process does not include a new systematic review of the source evidence, and thus it does not include a detailed content review of the recommendations or the appropriateness of the levels of evidence. Instead, the focus of a critical appraisal is on the quality of the methods used, with a commentary section that outlines some points for primary care practitioners to consider should they choose to implement the recommendations.

The primary objective of the critical appraisal process is to identify existing high-quality guidance that Canadian primary care practitioners can use to facilitate health care.

Details of the critical appraisal process are presented in Appendix VI.

3.3.4 Topics for reaffirmation

To keep its recommendations current, the CTFPHC has committed to re-examine each guideline within five years from the original date of publication or sooner if new research implies that there should be a change to an existing recommendation. As part of the reaffirmation process the CTFPHC searches the literature published since the previous review was completed. If no evidence is identified that would change the direction of the recommendation statement, the CTFPHC may reaffirm the previous guideline on the basis of the updated literature search.

3.3.5 Topics referred to other groups performing evidence-based reviews

If the CTFPHC deems that another organization is better suited to developing recommendations for a specific topic, it may refer the topic to that organization.

Table 3. Overview of the processes and timelines for active topics
Category of topic Definition Method of identifying topic Method of evidence review Time from protocol finalization to vote Frequency of consideration
New Topic never previously reviewed by CTFPHC Topic prioritization working group Full systematic review 12 months 5 years after publication of previous recommendations or when new evidence is available
Update Topic reviewed previously by CTFPHC or another guideline-producing group Topic prioritization working group Updated systematic review 9 months Previous CTFPHC topics, 5 years; topics reviewed by other groups, as they are identified or as new evidence becomes available
Critical appraisal Topic addressed by another major guideline organization Critical appraisal working group; topic prioritization working group Organization-dependent process 3 months Annual
Reaffirmation Topic with a well-established, evidence-based standard of practice; decision to reaffirm because topic is a priority and is within the scope of the CTFPHC or because there is a compelling reason to make a statement; recommendations changed only if there is a very high level of new evidence Topic prioritization working group Brief literature search and consultation with experts and partners to identify high-level evidence 3 months 5 years after publication of previous recommendations
Referral to another organization Topic of importance for which the CTFPHC decides that another organization is better positioned to make accurate and timely recommendations Topic prioritization working group No systematic review; topic is sent to external organization 3 months As identified

3.4 Topic working groups

For every topic selected by the CTFPHC, a working group is formed, consisting of hree CTFPHC members who volunteer to join the working group (one of whom is selected as chair), a scientific research manager and officer from the TFO (whose role is to provide scientific support), and members from the ERSC. At least one family physician is needed on each working group. The working group may also include members from partner organizations, if any such organizations are involved for the particular topic. The responsibilities of members of working groups are described in Appendix VII. The working group reports to the CTFPHC.

In the selection of a working group chair, an adequate and balanced level of participation from all CTFPHC members is considered. The scientific research manager identifies members with the least number of assignments, those who are interested in the topic, and those without conflicts of interest for the topic in question and then makes a recommendation for the working group chair to the CTFPHC chair and co-chair on the basis of these factors. If more than one CTFPHC member is interested in the position of chair of the working group, then the selection will be based on availability, level of participation, compatibility, and previous working group assignments. The CTFPHC chair and co-chair make the final decision in the selection of working group chairs.

4. Topic refinement

4.1 Search for recent guidelines and reviews

The scientific research manager and/or science officer and members of the ERSC search for recent guidelines and systematic reviews on the topic. This scoping exercise is not a thorough search but is limited to identifying the most relevant information needed to inform the systematic review. This information is summarized and distributed to members of the topic-specific working group before the first meeting of the working group. Working group members are expected to read the summary of the scoping exercise and any associated papers of particular relevance before the first call. The results of the scoping exercise and the summary are then used to inform development of the key questions and analytic framework. The CTFPHC process for scoping exercises is detailed in Appendix VIII.

4.2 Inclusion and exclusion criteria

Refining the topic includes identification of the PICO elements relevant to the guideline: the target population (P), the intervention (I), its comparator (C), and the outcomes (O). Other criteria for inclusion or exclusion are also discussed, as well as study designs to be considered. In the case of a topic update, the inclusion and exclusion criteria will be the same as reported in the previous review protocol or systematic review that is used as the foundational document.

5. Development of protocol

Figure 3. Template for an analytic framework.

Template for an analytic framework

Developing a protocol involves specification of the key and contextual questions, the associated analytic framework, and the inclusion and exclusion criteria. The ERSC uses the template in Appendix IX to prepare the protocol.

5.1 Select and rank outcomes and harms

At its initial meeting, the working group develops a draft list of outcomes and harms that will be considered for each key question. Members of the working group then independently rank the outcomes and harms according to the GRADE methodology, taking into consideration the whole population, rather than specific subgroups.

A working group meeting is organized to discuss the rankings and come to consensus about overall assessments of outcomes and harms as critical, important, or not important. The initial GRADE rankings range from 1 to 9.4 Outcomes and harms ranked from 7 to 9 are critical for decision-making, those ranked from 4 to 6 are important but not critical, and those ranked from 1 to 3 are not important. Only outcomes and harms considered important (rating 4–6) or critical (rating 7–9) are included in the evidence profile. Only outcomes and harms considered critical (rating 7–9) are primary factors influencing a recommendation; these are used to determine the overall quality of evidence supporting the recommendation. Whenever possible, the number of critical and important outcomes and harms should be limited to five outcomes and five harms.

Critical outcomes and harms are decisive or of essential importance, indispensable, and likely to determine a decision about care. They should be included in the summary-of-findings tables and are considered when determining the overall quality of evidence. Important outcomes and harms are meaningful, consequential, and significant and may influence a decision. They will usually be included in the summary-of-findings tables, but their inclusion may depend on the total number of important outcomes and harms. The overall quality of the evidence is not influenced by important outcomes and harms. Outcomes and harms of limited or no importance are those of little consequence or significance and are unlikely to influence a decision. They are not included in the summary-of-finding tables and not considered when determining the overall quality of the evidence.

Factors to consider when ranking outcomes and harms:

  • Rankings are judgements based on values.
  • Judgement of patients’ values and preferences should be considered.
  • The CTFPHC attempts to focus on outcomes that it believes will be important for clinicians to discuss or highlight with a patient when presenting the potential benefits and harms of a preventive service.
  • The judgements are relative, not absolute (i.e., weighing the importance of each outcome in relation to other relevant outcomes for the specific decision that is being considered).
  • The relative importance of an outcome depends on how likely it is to be affected by the intervention.
  • Surrogate outcomes are important only to the extent that they reliably indicate directly important outcomes, where directly important outcomes have not been measured and reported in studies.
  • Although the CTFPHC strives to select outcomes at the outset of protocol development, there may be rare cases when the outcomes specified in the original protocol may have to be changed as a result of findings from the literature review.

5.2 Surrogate outcomes

In general, the CTFPHC prefers to use clinically relevant outcomes but will consider the use of surrogate outcomes if trial evidence about clinically relevant outcomes (e.g., mortality) is lacking. In most cases, use of a surrogate outcome would result in downgrading the quality of evidence. If there is insufficient trial evidence for clinically relevant outcomes, surrogate outcomes that meet the following criteria may be considered:

  • There must be a relationship between the surrogate outcome and the clinically relevant outcome, defined as the following:
    • A high proportion of people with the surrogate outcome are expected to experience the condition or the outcome
    • Intervention directed toward the surrogate outcome leads to improvements in the clinically relevant outcome
    • Intervention directed toward the surrogate outcome results in a net benefit (e.g., benefits should outweigh harms)
  • The validation is based on multiple studies

5.3 Create analytic framework

The working group uses an analytic framework to illustrate the key and contextual questions that the literature review must answer to determine whether the proposed preventive service will safely prevent clinically relevant adverse outcomes. The analytic framework links interventions and outcomes to help structure the systematic review.

Analytic frameworks do not incorporate all factors associated with the clinical preventive service. Furthermore, they are not decision algorithms and do not incorporate all possible outcomes.

In an analytic framework, actions (such as the performance of a screening test) are depicted by arrows, and outcomes (such as decreased morbidity) are depicted by rectangles (see template in Figure 3). An analytic framework distinguishes between clinically relevant outcomes (those that are perceived by the patient) and intermediate outcomes, including surrogate outcomes and clinical correlates (which cannot be perceived by the patient). All critical and important clinically relevant outcomes must be specified. The CTFPHC considers intermediate outcomes only when evidence about clinically relevant outcomes is lacking. In this situation, the intermediate outcomes must be specified a priori. Use of intermediate outcomes may lead to a downgrading of the quality of the evidence in the final systematic review. The association of intermediate outcomes to the final outcome is depicted with a dashed line.

Whenever cause-specific and all-cause mortality are available, they should be used as outcomes in the analytic framework.

The analytic framework specifies populations, actions, and outcomes (Figure 3).

  • The population consists of the patients for whom the proposed preventive service is intended.
  • The actions link the population to the outcomes (or they may link outcomes directly) and may include screening and treatment. The name of each action appears in a label above its respective arrow. Adverse events, which are considered to be “actions” and which are denoted by curved arrows, can also appear in the framework.
  • Clinically relevant or intermediate outcomes result from actions or from previous outcomes. Clinically relevant outcomes are depicted as rectangles with square corners, whereas intermediate outcomes are depicted as rectangles with rounded corners.

Each arrow is associated with a key question that must be addressed by the systematic review, and all of the key questions are listed within the analytic framework.

Figure 3 shows the template for the analytic frameworks used in CTFPHC guidance. The framework shows (from left to right) the population identified for study (i.e., persons at risk), the activities to be studied (i.e., screening and early detection), the intermediate and ultimate health outcomes being sought, and the desired association between them. Adverse effects of screening are shown as ovals below the main flow. Each element in the flow chart is related to one of the key questions.

5.4 Identify key questions

The key questions are directly linked with the analytic framework and serve to focus the systematic review. They specify the population, interventions, and outcomes for the topic under consideration and are critical to conducting the literature search and the systematic review and to developing the recommendations. Key questions for an updated review may focus on a limited aspect of the topic and may be used to examine gaps in the evidence for the previous review or to examine new evidence published since the previous review.

The following are examples of well-constructed key questions relating to the template in Figure 3:

  • What is the evidence that screening for X in patient population Y reduces morbidity and mortality?
    • What is the optimal screening interval for screening for X?
  • What are the sensitivity, specificity, and positive and negative predictive values for screening test X for condition X?
  • What are the harms of screening test X for patient population Y?

Questions on the appropriate interval for screening and special considerations for high-risk groups should also be included as key questions. Sub-questions may be included if they are directly related to the main key question. Only evidence identified for the key question may be used to address the sub-question.

5.5 Select high-risk groups

The following process is used for selecting high-risk groups for inclusion in the review protocol:

  1. During the topic refinement phase, the ERSC conducts a search, with input from the working group, for high-risk groups affected by the condition of interest, using three to five reputable sources (such as national organizations).
  2. A list of high-risk groups is included in the executive summary of the refinement results and is shared with the working group.
  3. The working group discusses the high-risk groups identified:
    1. Populations may be excluded from the list by consensus of the working group if they would not be encountered in primary care.
    2. Additional high-risk populations may be added by consensus of the working group if supported by evidence and a suitable rationale.
    3. The working group will identify populations that will be:
      1. included and may have different recommendations from those of the average-risk population
      2. included and may have the same recommendations as the average-risk population
  4. Decisions to exclude or add other high-risk groups are documented in the Record of Decisions before the high-risk groups are included in the protocol.

5.6 Identify contextual questions

Contextual questions (which are identified by the working group) are not associated with the analytic framework, but the CTFPHC requires responses to such questions as context for the recommendations. The contextual questions may relate to risk factors, prevalence, cost-effectiveness, equity, patient values and preferences, comorbidities, and performance measures. Although questions about treatment effectiveness are not usually the focus of a CTFPHC guideline, such questions may be included to inform a screening guideline.

The following are examples of contextual questions often included in systematic reviews:

  • What is the cost-effectiveness of intervention for disease/condition in population?
  • What are the patient values and preferences for intervention for disease/condition?
  • What risk assessment tools have been identified in the literature to assess the risk of disease/condition?
  • What is the evidence for a higher burden of disease, a differential treatment response, differential performance of intervention, or barriers to implementation of for disease/condition in particular subgroups, such as the Aboriginal population, rural or remote populations, or other ethnic populations?

Contextual questions are addressed not through a formal systematic review, but instead through literature review. Data from all study designs may be considered in answering these questions, but studies are limited to those published in the past five years.

5.7 Select study design

The approach for determining the study designs to be included to address key questions should be documented a priori in the protocol and should be explained in the final review, to ensure that the process is transparent, defensible, and reproducible. Any changes made to the search parameters after the review is under way should be documented.

The levels of evidence used for a review, which are determined in part by study design, vary by the types of questions addressed. The following is a general hierarchy for evidence based on study design:

  1. systematic reviews of randomized controlled trials (RCTs)
  2. RCTs with a minimum sample size of 30 in each arm
  3. systematic reviews of non-randomized controlled trials
  4. non-randomized controlled trials
  5. observational studies with controls (prospective and retrospective cohorts, case–control studies, studies with before-and-after designs)
  6. observational studies without controls (cross-sectional, case series)
  7. ecologic studies and surveys

In some cases, the CTFPHC will consider the use of modelling studies to answer key and contextual questions. The process for incorporating modelling studies for key questions is reported in Appendix X. The decision to include modelling studies as a supplemental source of data is made by the working group, and these studies would not generally be used as the sole source of data to answer a key question.

Once the key and contextual questions have been developed, the working group determines, on the basis of their own knowledge and input from the ERSC’s technical experts, which study designs would be most appropriate to answer each of the research questions. At this point, members of the working group may determine whether they would like to focus exclusively on systematic reviews and RCTs or whether they will expand the search to include observational and modelling studies. The group may also decide on a staged approach to the search, whereby they first collect data from RCTs and decide whether supplemental observational or modelling data are needed once the RCT data have been reviewed. In such cases, the process and criteria for supplementing the RCT data should be documented in advance.

The working group should come to a consensus, based on a clear rationale, about the study designs that will be admissible for the review and should document these decisions. For example, in an examination of the impact of harms, the working group may decide to include large cohort studies, as these are more likely than RCTs to detect effects. Decisions about inclusion criteria that are based on study design should be sent to the methods working group for discussion, and input from technical experts should be sought as required.

If a decision is made to include observational studies, the working group may decide to limit the amount of observational data collected on the basis of sample size, study characteristics, or other relevant criteria. A minimum sample size of 1000 is suggested for inclusion of observational data.

In addition, the working group may consider the following questions when deciding whether to include observational data:

  • Are there sufficient high-quality RCTs to answer the key questions?
  • In cases where a staged approach to the search is being employed, would the inclusion of new data from observational studies change the conclusions of the review or the guideline recommendation?
  • Are the findings from RCTs homogeneous, or are there inconsistencies in the results that observational data might help to address?
  • What are the costs and benefits of including observational data (i.e., will doing so substantially increase the workload with little additional benefit to the review)?
  • For the topic under consideration, is it important to be as comprehensive as possible by collecting all of the available evidence? Such a comprehensive approach may be necessary if study results vary widely or the topic is particularly controversial.
  • Is the key question better answered with observational studies?
  • Are certain types of observational studies (e.g., cohort or large-sample studies) better suited to answer the research question than other types of observational or RCT designs?
  • Are the observational data current?
  • Would the inclusion of observational data change the strength of the evidence that will form the basis of the recommendation? For example, the GRADE Working Group recommends that if the quality of evidence differs across critical outcomes and the outcomes point in different directions — toward benefit and toward harm — the lowest-quality evidence for any of the critical outcomes determines the overall quality of the evidence. This means that if both RCT and observational data are used for an outcome, the overall data quality would be low.2

5.8 Consult content experts

The ERSC may contact content experts for advice on the protocol and methodology and must consult at least one content expert on each systematic review. Topic experts must complete the conflict of interest form (Appendix I) and must sign the Confidentiality Agreement (Appendix II); the signed forms are then sent to TFO.

5.9 Develop search strategy

An ERSC librarian prepares the search strategy according to protocol parameters, including timeframe, databases, study designs, and key words. Separate searches may be conducted for systematic reviews, RCTs, and observational studies. The search strategy for the key questions is peer reviewed by another librarian; once finalized, it is included in the protocol.

5.10 Timelines

The timelines for each deliverable is included in the protocol. The deliverables include the final protocol, the draft systematic review, the final systematic review, the draft guideline and the final guideline. The ERSC, the TFO, and the chair of the working group review the proposed timelines and sign-off to indicate agreement.

5.11 Send protocol to peer reviewers

The protocol must be reviewed by three to six content and/or methodology experts before it is finalized. The ERSC develops a list of potential peer reviewers, which is reviewed and approved by the working group. The ERSC then coordinates the peer review. Once comments have been received and incorporated into the protocol, a summary of the comments and action taken are presented to the working group for approval.

5.12 Seek approval of protocol from CTFPHC

The final peer-reviewed protocol is first approved by the working group and then by the full CTFPHC.

5.13 Register protocol and modifications

After the protocol has been approved by the working group, the review coordinator at the ERSC registers the project with PROSPERO (International Prospective Register of Systematic Reviews). The working group is notified when the PROSPERO file is complete. The review coordinator logs decisions that affect the protocol as the project evolves. Changes to the protocol are made in the existing document, which is thus kept current and correct. Revised versions are dated and labelled with a modification number. Appropriate revisions are also made to the PROSPERO registration file. The review coordinator notifies members of the review team of any protocol revisions that have implications for their work. The ERSC sends the final protocol to the CTFPHC for posting on its website.

6. Development of systematic review

Once the protocol has been finalized and approved, the ERSC conducts a systematic review of the available evidence following standard methods. The standard operating procedures of the ERSC in the conduct of systematic reviews are reported in Appendix XI and the template is reported in Appendix XII. The systematic review is reviewed by the working group, the CTFPHC, and external peer reviewers.

6.1 Types of reviews

To address the key questions associated with a topic, the ERSC undertakes a series of systematic reviews. Several approaches are used to ensure overall efficiency:

  • a full systematic review (the most common approach) to address each key question in the analytic framework, with existing high-quality systematic reviews being used if they are relevant to the research questions
  • targeted systematic reviews for a limited number of key questions in the analytic framework that address critical gaps in knowledge, for which established or current evidence may not be available (a common approach for updates)
  • staged reviews to address key questions in the analytic framework that must be answered before a full review can proceed (used as a means of informing the CTFPHC that there is sufficient evidence to proceed with other questions in the analytic framework)

6.2 Search procedure

6.2.1 Main search

The search strategy is implemented, and the date on which the search is conducted is recorded, along with the list of search terms for each database. The reference lists of included studies are checked and working group members and other experts are consulted for missing or up-and-coming studies that might be relevant to the review. If any studies are found by this means, they are assessed in relation to the inclusion and exclusion criteria and are incorporated into the review as appropriate.

The search results are downloaded into a reference database (EndNote).Removal of duplicate citations is accomplished, first by using the EndNote duplicate tool, and then by visually scanning the citations, as organized by EndNote according to author name and document title. The de-duplicated citations are loaded into DistillerSR. If, at any time, key questions are added, removed, or changed, further searches may be done and added to the search string.

The databases of, the ISRCTN trial register, and the World Health Organization’s International Clinical Trials Registry Platform are also searched to identify trials in progress that may be relevant to the topic.

A search of the grey literature should also be conducted to identify relevant high-quality Canadian data that have been disseminated from governmental and non-governmental organizations such the PHAC, the Canadian Institutes of Health Research, Statistics Canada, and the Canadian Agency for Drugs and Technologies in Health. This type of information is incorporated into the review as contextual information and is not assessed using the GRADE methodology.

Expedited searches are conducted to answer contextual questions. In these expedited searches, the ERSC searches selected databases to identify systematic reviews published in the past five years that present evidence relating to identified subgroups. This search is supplemented by a search of key journals and websites for additional primary studies disseminated in the past two years (i.e., potentially too recent to have been included in published reviews). For these expedited reviews, the ERSC uses Canadian data sources wherever possible. The list of journals and databases to be searched is determined by the working group, with input from the ERSC and clinical and content experts. Input on this list is usually solicited when the protocol is sent for external review (see section 5.11).

6.2.2 Updated search

If the CTFPHC is updating a previous review or a review from another organization, the search strategy used in the original review that is now being updated is reviewed by the ERSC librarian. Before commencing the update, the librarian assesses the search strategy and provides a recommendation to the working group as to whether the strategy is appropriate to answer the key questions. This search strategy is peer reviewed by another librarian. Section 6.8 described steps required for updates of reviews conducted by other organizations.

6.3 Screening of articles

The ERSC applies the a priori inclusion and exclusion criteria to the results of the literature search to identify articles suitable for the systematic review. There are two rounds of screening. Level 1 screening involves review of the title and abstract of each article. Two reviewers screen all titles and abstracts independently. On the basis of this review, all citations are coded as “included” or “excluded.” Any studies that are screened in by only one of the two reviewers at this stage are automatically selected for level 2 screening.

The articles selected during level 1 screening then undergo level 2 screening by at least two reviewers. This screening involves review of the full text of each article, with each article again being coded as “included” or “excluded.” Studies that are screened in at this stage must be coded to specify the key question addressed, and excluded studies are coded with the reason for exclusion.

6.4 Abstraction of data

Data extracted during the review include the characteristics of studies used to answer key question 1 (concerning benefits). In addition, all quantitative data for all key questions are extracted using standardized forms. These data are presented in a meta-analysis when appropriate.

6.5 Quality assessment

The ERSC uses the GRADE process to assess the internal and external validity of each included study.

As defined by the GRADE Working Group, the quality of evidence is the “extent to which our confidence in an estimate of the effect is adequate to support a particular recommendation.”2 The CTFPHC considers the quality of evidence related to all critical and important outcomes when developing its guidance.

The GRADE handbook2 provides information about assessing the quality of evidence, which is the basis for the system used by the CTFPHC. The ERSC first grades the quality of evidence for each outcome of importance to patients and then determines the overall quality of evidence across all outcomes. The quality of evidence for an individual outcome can be affected by a number of factors (some of which are described in the subsections below), and judgement is used to determine the overall quality of a study. In both cases, the quality of evidence is classified according to the GRADE system2 into one of four grades: high, moderate, low, or very low. If the ERSC confidence in an effect is unlikely to be influenced by additional research, the grade is high, whereas a low grade indicates that the ERSC is uncertain of the effect. The moderate and low grades are used for effects that are likely (moderate) or very likely (low) to be influenced by further research. The GRADE handbook2 provides additional information about factors affecting the quality of the evidence, as summarized here:

Study design
In general, the GRADE approach considers RCTs as representing stronger evidence than observational studies. However, the limitations of specific RCTs or the strengths of specific observational studies may affect the quality of evidence from these studies. The GRADE handbook2 and the Cochrane handbook5 provide further information about grading evidence on the basis of study design.
Risk of bias

The GRADE approach includes an assessment of risk of bias.2 Limitations of RCTs that may result in bias include, but are not limited to, lack of allocation concealment or blinding, lack of reporting of loss to follow-up, lack of adherence to intention-to-treat analysis, and incomplete reporting of outcomes. The limitations of observational studies include, but are not limited to, inappropriate eligibility criteria, inaccurate measurement of outcomes, lack of control of confounders, and incomplete follow-up. The ERSC considers a study’s limitations and potential bias when rating the quality of evidence and reporting the risk of bias for outcomes (Appendix XIII). When case–control and cohort studies are included in the review, the ERSC will complete the Newcastle-Ottawa Scale6 to assess the risk of bias. This information is used to determine if the “limitations” component of the GRADE quality assessment should be downgraded. Ecologic studies and surveys, modeling studies, and uncontrolled observational studies are given a very low quality rating, as currently no tools exist to assess the quality of studies with these designs.

Once the risk-of-bias assessment is complete, the ERSC assesses whether the limitations can be considered negligible (no downgrading), serious (downgrade quality assessment by one level), or very serious (downgrade by two levels). The remainder of the GRADE quality assessment categories (inconsistency, indirectness, imprecision, and publication bias) can be completed as for RCTs.

When studies of diagnostic test accuracy are included in the review, the ERSC uses the QUADAS-2 tool to assess study quality.7 In this situation, risk of bias is assessed through four domains: patient selection; index test; reference standard; and flow of patients through the test and timing of the test. Applicability of the test is assessed via the first three of these domains.

Inconsistency across studies
Differences in results across studies (heterogeneity) may occur because of differences in the populations studied, the interventions applied, or the outcomes evaluated. Therefore, any assessment of the quality of the evidence should consider heterogeneity.
Indirectness of evidence
Indirect comparisons or the use of indirect populations, interventions, comparators, or outcomes will affect the quality of the evidence. Indirect comparisons are used when the two interventions of interest are not compared directly, but rather are both compared to another intervention. The two interventions can then be compared indirectly. Indirectness can also occur when the population, intervention, comparator, or outcome being investigated varies from the evidence available in the literature (e.g., use of evidence from a population different from the population of interest).
Imprecision of evidence
Results with wide confidence intervals are imprecise and carry less weight than more precise results. The precision of results should therefore be considered in the assessment of the quality of the evidence. The GRADE handbook2 provides additional information about imprecision in dichotomous and continuous outcomes and its role in the assessment of quality of evidence.
Publication bias
When assessing the quality of evidence, the ERSC considers the potential that there has been selective publication of studies resulting in publication bias.
Large effect sizes
The quality of evidence associated with observational studies can be upgraded if the studies are of high quality and have no limitations, and the effect size is large (relative risk [RR] < 0.5 or RR > 2) or very large (RR < 0.2 or RR > 5).
Plausible confounding
Confounding that may cause an increase or decrease in the effect is considered when the quality of evidence is assessed. The assessment of quality may be upgraded for studies with confounding, on the basis that if only very ill patients receive an intervention and recover, then it is likely that the actual effect of the intervention is greater than the data suggest.
Dose–response effect
The presence of a dose–response effect is considered in assessments of the quality of evidence. Such an effect may support the conclusion that the intervention has an effect. The assessment of quality may be upgraded for studies with a dose–response effect.

6.6 Summary of the evidence

6.6.1 Characteristics of included studies

The characteristics of all studies that meet the inclusion criteria are summarized by key question. Characteristics commonly recorded include, but are not limited to, the name of the study; the objective of the study; the methods (design, study selection); participants (study sample, key demographic and clinical characteristics); details on the intervention; outcomes (reported by the study authors and reported in the review). The template for the study characteristics table can be found in Appendix XIV.

6.6.2 Preparation of GRADE tables (summary of findings and evidence profile)

The main results of the systematic review are reported in a summary-of-findings table (for an example, see Appendix XV). This table, described in detail in the GRADE handbook,2 presents outcomes, assumed risk, corresponding risk, relative magnitude of effect, number of participants, number of studies, overall quality rating, and additional information as appropriate. Where possible, the ERSC provides both relative and absolute measures of effect in the summary-of-findings tables. These tables are completed regardless of whether a meta-analysis is appropriate and even if only one study is included in the review. A GRADE evidence profile is also prepared for each key question, according to the process described in the GRADE handbook.2 These tables summarize the body of evidence for each outcome (both benefits and harms) and contain four key pieces of information: a quality assessment, with details about limitations, inconsistency, indirectness, imprecision, and other considerations; a summary of findings, with selected summary statistics; the quality rating (very low, low, moderate, or high); and the level of importance of that particular body of evidence. For completeness, a GRADE evidence profile is generated even if the systematic review reports that no studies meeting the inclusion criteria for a particular outcome were found. For an example, see Appendix XVI.

If there is only one study for a particular outcome or if the data are not pooled, there are additional considerations in conducting the quality assessment for the domains of precision and consistency. In these cases, precision can be assessed on the basis of size of the confidence intervals (with a narrow interval suggesting precision), sample size, and effect size and through calculation of the optimal information size.

The consistency domain is downgraded when evidence is found to suggest inconsistency in results. Therefore, if only one study is included for a particular outcome, this domain will not be downgraded, because there would be no evidence to suggest inconsistency. In the event that data are summarized qualitatively, consistency can be assessed on the basis of overlap in confidence intervals and/or variation in effect size (RR, odds ratio, hazard rate) among the different estimates.

The numbers needed to screen, to treat, or to harm (NNS, NNT, or NNH, respectively) are also calculated and added to the evidence table. The NNS, NNT, or NNH are calculated using the RR method: a weighted RR value is calculated, the number of lives saved per million (i.e., [1 – RR] multiplied by event rate per million in the control group) is calculated, and finally the NNS, NNT, or NNH is calculated. In general, meta-analyses using relative measures (such as the relative risk) are associated with less heterogeneity than meta-analyses of absolute measures (such as the risk difference). When there is variation in control event rates, using the relative risk method is preferred.

6.6.3 Additional evidence syntheses

Additional methods for evidence synthesis, including forest plots, funnel plots, and assessments of risk of bias, may be applied.

As many calculations are involved in producing final numbers for the evidence profile table, all calculations are conducted to the fourth decimal place, then rounded to two decimal places for final presentation.

6.6.4 Applicability of the evidence in relation to key questions

The CTFPHC considers the applicability of the findings to the key questions. The evidence is assessed to determine if any clinically important differences in the results are relevant to those expected in the Canadian primary care setting. This assessment should consider the following possibilities:

  • whether the evidence suggests that the intervention will be effective in the Canadian primary care setting
  • whether the benefit that was achieved in the reported studies is similar to the benefit that would be achieved in the Canadian primary care setting
  • whether the harms that occurred in the reported studies are similar to the harms that would occur in the Canadian primary care setting
  • whether the relation between benefits and harms in the reported studies is similar to the relation between benefits and harms in the Canadian primary care setting
  • whether the effort needed to provide the interventions would be possible in the Canadian primary care setting
  • whether the intervention is feasible for Canadian patient populations and primary care providers in terms of time, effort, and cost
  • whether it is feasible to extrapolate from the data in the reported studies to the larger asymptomatic Canadian population

6.7 Other considerations for evidence assessments

6.7.1 Subgroup analyses

The CTFPHC may develop certain recommendations for specific populations. As such, systematic reviews may incorporate appropriate subgroup analyses. The CTFPHC analyzes the evidence for the subgroups to determine the quality of the information and the feasibility of including it. The working group will limit subgroup analyses to one or two critical outcomes. The choice of sub-analyses should be factors that may change efficacy of proposed intervention, which may or may not be risk factors for the condition. The key consideration for choosing subgroups for analysis will be whether that analysis will impact the guideline. For example, subgroup analyses for which the working group may make different recommendations may include age, intervention type, or gender. For subgroup analyses, use of preplanned analyses from trials (not post hoc analyses) is preferred.

6.7.2 Ecologic evidence

Ecologic evidence consists of data at the level of a population, rather than an individual. Such evidence is often reported as population averages. Comparisons of outcomes in ecologic studies may be in the form of comparisons between different populations at a single time point or comparisons over time within the same population. Ecologic studies are often used to estimate the effect of geographic differences. An “ecologic fallacy” occurs when conclusions from an ecologic study are drawn at the level of individuals, rather than at the aggregate ecologic level.

Because of potential biases, the CTFPHC generally does not use ecologic evidence to determine the effectiveness of an intervention. However, the CTFPHC may consider such evidence for use as background information, for example, if other guideline groups have used well-known ecologic evidence in their recommendations or if an ecologic study has yielded substantive results.

If the CTFPHC decides to include ecologic evidence in a review, the study must be appraised. The criteria for the appraisal might include how the outcomes, exposure, or confounders were measured; whether adjustments were made for confounders; and whether the populations and interventions are comparable to and relevant for those in the primary care setting.

6.7.3 Mortality as an outcome

The CTFPHC considers both all-cause mortality and cause-specific mortality (when such data are available) in developing its recommendations.

In situations where the condition of interest commonly causes death, the CTFPHC may consider all-cause mortality, rather than cause-specific mortality, as a final health outcome. Any difference in the effect of the intervention between all-cause mortality and cause-specific mortality should also be considered. Such differences may be attributed to there being a benefit of the intervention for the condition of interest but an increase in mortality related to other conditions. Alternatively, the difference may occur because there is a decrease in cause-specific mortality but no change in all-cause mortality, which would indicate a potential harm of the intervention for patients with other conditions. Differences between all-cause and cause-specific mortality may also occur if the condition of interest is rare or if the population is subject to other causes of mortality, in which case the intervention has little or no effect on all-cause mortality.

Methodologic issues may contribute to differences between all-cause and cause-specific mortality. Accurately ascertaining the cause of death for participants in clinical studies is potentially difficult, and deaths may be attributed to a chronic condition even in cases where the condition did not contribute to the death. Conversely, when physicians know that their patients are involved in a study (as is the case for some interventions where blinding is impossible) and are uncertain of the actual cause of death, they may be reluctant to attribute the death to the condition of interest. This could lead to a false reduction in cause-specific mortality and a false increase in all-cause mortality. Moreover, participants enrolled in the active-intervention arm of a trial may be followed more closely before death than those in the passive, no-intervention arm, which may mean that selected information about those in the intervention arm is available even to the external adjudicators of cause of death. This may in turn lead to biased estimates of cause-specific mortality.

6.7.4 Relative versus absolute risk reduction

The CTFPHC is interested in reducing the risk for both populations and individuals, although its focus is on individuals seen in primary care practices. Therefore, the CTFPHC considers both relative and absolute risk reduction, with an emphasis on the latter.

6.7.5 Screening and case-finding

The difference between screening and case-finding should be considered when recommendations for screening are being developed. Some groups use the terms “screening” and “case-finding” interchangeably. However, the CTFPHC maintains the following distinctions. Screening is the examination of an asymptomatic population, using a specific tool, to identify a condition of interest, whereas case-finding is the examination of an individual or group suspected of having the condition or at risk for the condition. Case-finding is a targeted approach to identify conditions in selected patients, who may already have symptoms. It usually does not involve the use of a specific tool. The CTFPHC may decide to investigate screening or case-finding, depending on the topic.

6.8 Updates of reviews from other organizations

The following steps are conducted when the working group is considering updating a review completed by another organization:

  1. The quality of the systematic review will be assessed using AMSTAR. The AMSTAR rating will be done by ERSC, for all reviews under consideration for update. Results will be presented to working group, highlighting areas where the rating is low. The working group will make a decision about which review to use for an update.
  2. The ERSC can revise the search, which will be peer reviewed, and update it from when the search ended in original review.
  3. The data extraction of the review being updated will be will be checked by one person from ERSC.
  4. If risk-of-bias and GRADE tables are part of the original review, one person from the ERSC will review those results to ensure they are consistent with how ERSC does ratings/ extraction/ decision rules. If the results are not consistent, they will be redone by the ERSC.
  5. The inclusion/exclusion criteria will not be expanded and may be narrowed.
  6. A new search will be run for contextual questions.

6.9 Incorporating other systematic reviews in CTFPHC reviews

The CTFPHC may incorporate existing high-quality systematic reviews or meta-analyses into its own reviews to address all or some of the key questions or to serve as the evidence base for a specific time period. An existing review may also be used as a reference, to confirm the findings of the current CTFPHC review.

To assess the methodological quality of systematic reviews, the CTFPHC uses the AMSTAR measurement tool.8 The existing review must be relevant to one or more of the key questions being addressed in the CTFPHC review. It must also report the relevant study designs, populations, settings, interventions, comparators, and outcomes. The CTFPHC considers the publication date of the previous review to determine its relevance and to determine if updated searches are required.

6.10 Incorporating evidence for contextual questions

Evidence used to address contextual questions does not require quality assessment and may be examined by only one reviewer

6.10.1 Population subgroup

The CTFPHC attempts to assess whether its guidance has particular implications for the equitable delivery of preventive services to specific subgroups. To inform this issue, the CTFPHC considers the following questions:

  • How does the burden of disease (especially mortality) for the subgroup differ from the burden of disease for the population as a whole?
  • Is there reason to believe that the screening tool may not perform as well for the subgroup as for the population as a whole (e.g., because of language or cultural barriers, education level, genetic variation, providers’ adherence in the delivery of screening)?
  • How do the effectiveness and harms of the preventive intervention or treatment differ for the subgroups (e.g., because of language or cultural barriers, socioeconomic barriers, genetic differences, patient preferences, or physicians’ adherence to treatment recommendations)?
  • Are there unique implementation issues for the subgroup?

If the working group reaches consensus that the answer to one or more of these questions is “Yes” for any particular subgroup, this triggers targeted searches for evidence to address the issues. These searches aim to identify any subgroups for which there is literature to support differential burden, effectiveness, harms, or implementation issues, so specific subgroups need not be identified in advance. The decision to include recommendations for specific subgroups in the final guideline is based on evidence from these searches.

Subgroups that are routinely considered for examination include Aboriginal peoples, remote or rural dwellers, women, children and adolescents, elderly people, immigrant populations, and ethnic subgroups in Canada. The working group may consider other subgroups at its discretion.

6.10.2 Consideration of resource use

The cost-effectiveness of interventions is addressed by adding a contextual question on costs to all searches. Modelling studies may be appropriate to answer these questions. Section 7.3 discusses how the CTFPHC incorporates information on expected costs and resource use related to specific recommendations.

6.10.3 Consideration of values and preferences in the target population

A search of the literature is performed to determine the values and preferences of the target population in relation to the intervention in question. The CTFPHC uses this information to consider patients’ preferences and to incorporate preferences into the formulation of recommendations, as appropriate.

6.10.4 Consideration of concomitant medical conditions

During the literature review, concomitant medical conditions are considered by means of the following questions:

  • What is the population being studied?
    • general population
    • primary care population (from practices)
    • secondary or tertiary care population (from specialized care settings)
  • Did the study report patients’ characteristics?
  • Did the study report specifically on comorbidities, either those associated with the condition of interest or unrelated comorbidities?
  • Were specific subgroup analyses performed for patients with particular comorbidities?
  • Are the benefits of the study of importance for all subgroups?
  • Are there any elements that could compromise the generalizability of the results to the primary care population that will be the target of the guideline?

6.11 Updating systematic reviews before publication

6.11.1 Review less than one year old at time of guideline publication

If the systematic review is expected to be less than one year old at the time of publication of the guideline, the only additional update required is a targeted update for RCTs addressing the key questions. This update is performed 6 weeks before publication.

6.11.2 Review more than one year old at time of guideline publication

If the systematic review is expected to be more than one year old at the time of publication of the guideline, an additional update is performed. In this update, the ERSC searches for evidence (from any study design) related only to the key questions; no updates are done for contextual questions. Any evidence that is found is included in the review and is incorporated into the GRADE tables and, if relevant, the recommendations. This update is timed so that it begins about one to two months before submission of the guideline to CMAJ or another appropriate peer-reviewed journal agreed upon by the CTFPHC. As such, the systematic review and the recommendations that are submitted for publication will not include data from the update at the time of initial submission. Instead, these documents are updated when peer review comments are received from the journal.

7. Development of recommendations

During peer review of the systematic review, the chair of the working group and the scientific research manager prepare an initial set of draft recommendations, based on the findings of the draft review. The draft recommendations are shared with and approved by the working group, and are then considered and approved by the broader CTFPHC. The process for reviewing evidence and developing recommendations is based on guidance in the GRADE handbook.2 The recommendations should specify the target population and the intervention, and their phrasing should be consistent throughout related documents and with other CTFPHC guidance, when possible. If appropriate, performance indicators for guideline implementation are included in the final guideline.

7.1 Application of the GRADE approach in formulating recommendations

When developing recommendations, the CTFPHC must first, in accordance with the GRADE approach, agree on the critical and important outcomes to be reviewed (see section 5.1). In addition, the CTFPHC must agree on the evidence to be included and the assessment of its quality; as such, the CTFPHC should review and discuss the systematic review (see section 6). The factors used in determining the strength of eventual recommendations (as discussed in section 7.2) should be considered in the development of those recommendations. Voting may be needed to reach agreement on a particular recommendation and its strength. The results of such voting may be reported in the final guideline documents.

To facilitate formulation of recommendations, the GRADE Working Group has developed the Evidence to Recommendation Framework. The framework outlines six criteria important for formulating health system recommendations: the problem, the benefits and harms of the intervention, resource use, equity, acceptability, and feasibility. As the working group works through the framework, its members must make judgements about these criteria (e.g. “Are the anticipated desirable effects large?”). Use of the completed framework facilitates the working group’s discussion about the direction of the guideline, by ensuring that all key criteria are considered. To ensure transparency when formulating recommendations, the working group considers and describes the balance between desirable and undesirable effects, as well as the basis for any judgements.

7.2 Strength of recommendations

In accordance with GRADE guidance, recommendations are classified as strong or weak.

A recommendation is rated as strong if the CTFPHC determines that the benefits of the intervention outweigh its harms or vice versa. A recommendation is rated as weak if the CTFPHC determines that the benefits of the intervention probably outweigh its harms or vice versa.2

In determining the strength of a recommendation, the CTFPHC considers the baseline risk of the outcome, the effect size of the intervention, and the precision of the effect. The quality of the evidence, patients’ values and preferences, and the balance between benefits and harms are also considered. Further information about the strength of recommendations can be found in the GRADE handbook.2

7.3 Incorporating cost and resource use into recommendations

The cost of an intervention and associated resource use may also be considered in the development of recommendations. The decision to include costs is made on a topic-by-topic basis, depending on whether the working group thinks this factor will be important in making a decision about a recommendation. When costs are considered, the CTFPHC usually takes the perspective of the health care payer or the societal perspective. The quality of evidence about costs and resource use should also be considered.

The GRADE Working Group does not recommend inclusion of cost-effectiveness or cost-utility modelling, 2 but these approaches may be used to help inform decisions of the CTFPHC. Further information about incorporating costs and resources into recommendations is provided in the GRADE handbook.2

7.4 Relative importance of efficacy and effectiveness

Because recommendations of the CTFPHC are intended for widespread implementation throughout Canada, the CTFPHC considers both efficacy (benefit in the ideal setting) and effectiveness (benefit in the usual setting) when determining the potential overall benefit of an intervention. Some jurisdictions have more resources than others, so the CTFPHC attempts to provide recommendations suitable for a variety of settings.

7.5 Overall quality of evidence across outcomes

When assessing the overall quality of the evidence, the CTFPHC considers only critical outcomes. In cases where studies have different results (benefit or harm) for critical outcomes, the lowest-quality evidence determines overall quality. In cases where all studies have the same result for a particular critical outcome, the highest-quality evidence determines overall quality. The quality of evidence for all critical outcomes is combined to determine the overall quality of evidence for each recommendation.

7.6 CTFPHC vote on draft recommendations

Once the recommendations have been drafted and approved by the working group, a vote of the CTFPHC is required. During a meeting of the CTFPHC, the ERSC presents the overall findings of the final systematic review, and the working group presents the draft recommendations. Members of the CTFPHC discuss the systematic review and draft recommendations and may propose changes to the wording of the recommendations. The CTFPHC then votes on the draft recommendations. The timeline from approval of the protocol to presentation of the draft recommendations to the CTFPHC is usually 9 to 15 months.

7.7 External review of draft recommendations

Following discussion and voting during a CTFPHC meeting, the chair of the working group or the scientific research manager revises the recommendations and shares the revised version with all members for the CTFPHC for approval. The approved statement of recommendations is then sent to external peer reviewers and stakeholders for comment.

7.8 Approval of final recommendations

Comments from peer reviewers are shared with the working group and the scientific research manager, who decide whether any changes are required. If substantial revisions are required or if the recommendations are controversial, the entire CTFPHC may be asked to review and discuss the comments, at the discretion of the working group chair and/or the CTFPHC chair. The CTFPHC approves the final recommendations at its next meeting or by email, if no meeting is scheduled.

7.9 Release of recommendations and systematic review

CTFPHC recommendation statements are published in the peer-reviewed literature. An agreement has been reached with the CMAJ giving the journal right of first refusal to publish recommendation statements. The recommendations should be released within six months from the time of the CTFPHC vote described in section 7.6. The systematic review and recommendations are published on the CTFPHC website in accordance with arrangements with the journal publishing the recommendations for a particular topic. Six weeks before publication, one final search for RCTs addressing the key questions is conducted by the ERSC, to ensure that no new information that could change the nature of the recommendations has been published. This information is provided to the working group, which then decides if and how such information should be incorporated. All materials intended for publication or release are submitted to the Chief Public Health Officer for information at least six weeks before any public announcement or release.

8. Knowledge translation

The KT working group develops KT strategies and partnerships aimed at advancing the uptake of CTFPHC guidance into clinical practice.

The KT working group has the following key objectives:

  • To disseminate CPGs to stakeholders through the development of decision tools, publications, presentations, and media.
  • To develop and maintain relationships with a range of stakeholders, including primary care practitioners, members of the general public, disease-specific and general organizations, and policy-makers.
  • To evaluate the activities of the CTFPHC and the impact and uptake of its CPGs to ensure that KT activities are effective, appropriate, and consistent.

8.1 Dissemination of information

The KT working group uses a multi-pronged approach to disseminate information to stakeholders. Dissemination strategies include the following:

  • Development of KT tools for primary care practitioners, the general public, and policy-makers.
  • Publication of guidance, methods, and KT tools in peer-reviewed journals.
  • Presentation of guidance and methods at major scientific meetings such as Family Medicine Forum and the annual meeting of the North American Primary Care Research Group.
  • Engagement of a wide audience through a comprehensive public communications campaign with the release of each guideline.
  • Ongoing communication about CTFPHC activities through a dedicated website, newsletters, and bulletins.

End-of-guideline KT incorporates diffusion, dissemination, and application. Diffusion focuses on passive strategies, such as peer-reviewed publications and newsletters, with targeting of open access journals. The CTFPHC website also serves as a passive diffusion tool. Dissemination involves activities that tailor the message and medium to a particular audience. Application moves research into decision-making when the strength of the evidence is sufficient.

For each of these approaches, the CTFPHC considers the following questions:

  • Who are the end-users of the guideline and who will be interested in its results?
  • What are the key messages for each of the end-users?
  • Who are the principal target audiences, organizations, and groups for each of these messages?
  • What are the barriers and facilitators to uptake of the guideline for each of these end-user groups?
  • What KT strategy will be used to facilitate uptake of the guideline?
  • What is the impact of uptake of the guideline?
  • Is uptake of the guideline having a sustained effect? How can the CTFPHC optimize sustainability?

8.2 Development of KT Tools

A wide array of KT tools (e.g., decision tools, mobile and electronic medical record applications) are developed to assist primary care practitioners in their understanding of the CPGs and the corresponding methodology and to facilitate integration of CPGs into clinical practice. The development process is based on the knowledge-to-action framework, and the resulting tools incorporate emerging best practices for KT.

Protocol for Development of KT Tools
  1. A member of the Knowledge Translation Program (KTP) sits in on meetings of the guideline working group once a direction for the recommendations has been selected, even if the final draft of the guideline has not been completed. The KTP member communicates the guideline direction and anticipated timeline for the KT tools to the KT working group.
  2. The KTP conducts a needs assessment to determine what tools will be needed to disseminate the guideline to primary care clinicians and their patients:
    1. Internally reviews the direction for the recommendations and discusses potential issues or knowledge gaps and corresponding strategies for the KT tools.
    2. Researches potential tool designs to be discussed during a meeting of the KT working group.
    3. Identifies the resources that will be needed to complete the KT tools within the anticipated timeline.
    4. When required, obtains feedback from guideline knowledge users (through interviews or focus groups) to identify the KT tools most appropriate for the guideline.
  3. The chair of the guideline working group and/or the scientific research manager officer share the first version of the guideline (as prepared for submission to a peer-reviewed journal) with the KTP and the KT working group.
  4. The KTP, the chairs of the KT working group and the guideline working group chair, and the scientific research manager discuss and decide on the KT tools to be developed for the guideline and discuss anticipated issues and the content to be included in the suggested tools.

    (Note: When multiple tools are developed, the following steps will be repeated for each tool and may occur on a staggered timeline.)

  5. The KTP develops an initial, unformatted version of the KT tool and presents it to the chairs of the KT and guideline working groups and the science officer for discussion.
  6. The KTP revises the unformatted KT tool to incorporate feedback. This process may involve several iterations, depending on the complexity of the tool.
  7. Once the overall content of the KT tool has been established, the KTP takes the tool to a graphic designer who creates two or three design concepts.
  8. The KTP brings formatted samples of the tools to the KT and guideline working groups for further feedback on layout, content accuracy, and messaging. This process also may involve several iterations until the working groups agree upon a formatted version of the tools. The working groups may select more than one tool design to be sent for usability testing.
  9. The KTP and graphic designer revise the formatted version of the tools according to feedback from the working groups.
  10. The KTP conducts usability testing with the formatted tool.
    1. Facilitators and interviewers are engaged for focus groups and interviews.
    2. The tool or tools are presented to the applicable population (clinicians or patients).
    3. Usability sessions are transcribed, and the transcripts are delivered to a coder.
    4. The coder identifies themes based on discussions during usability testing and shares a summary with the KTP.
  11. The KTP updates and format the tool on the basis of the usability testing results. If appropriate, the tool will be sent to heuristics experts to identify and remedy problems in the user interface design and to ensure user interaction is as simple and efficient as possible
  12. The KTP consults with the science officer and the guideline working group to ensure that any changes to the tool are accurate and in line with planned messages.
  13. The KTP brings the updated tool to the KT working group for review and incorporates final changes.
  14. Once the KT working group has made a final decision on the tool, the KTP presents the final version of the tool to the guideline working group for final approval.
  15. If the tool is to be posted on the website without publication in a journal, the KTP posts the tool when the guideline is released.
  16. If the tool is to be published in the journal, the KTP works with the journal’s editorial office to finalize tool formatting. The journal is usually responsible for French translation of any KT tools being published.
  17. The journal supplies the final English and French versions of the tool for approval.
  18. The KTP and the science officer review and approve the journal’s final English and French versions of the tool.
  19. The KTP arranges for translation into additional languages when required.
  20. When appropriate, the KTP evaluates the KT tools to inform the possible development of additional tools.

8.3 Stakeholder engagement

The KT working group develops and maintains relationships with stakeholders (primary care practitioners, the general public, general and disease-specific organizations, and policy-makers) and identifies opportunities for engagement throughout all stages of the guideline development and dissemination process.

Effective engagement of stakeholders is central to the successful management of both uptake and impact of CPGs. Therefore, CTFPHC strives to engender positive relationships with stakeholders by maintaining open and transparent communication.

The CTFPHC’s stakeholder engagement is an iterative process involving the following activities:

  1. Identifies and describes key project stakeholders who are invested in CTFPHC activities.
  2. Engages stakeholders and sustains relationships.
  3. Develops stakeholder awareness of CTFPHC activities through outreach and education at various increments throughout the CPG development.
  4. Monitors and evaluates ongoing stakeholder relationships on a guideline by guideline basis while continuing to seek out new engagement opportunities.
8.3.1 Identification of stakeholders

The CTFPHC aims to identify any group that can affect or that may be affected by its guidance. Stakeholders include but are not limited to the following organizations:

Primary care and public health organizations
These organizations are national in scope and are generally included on the list of stakeholders for every CPG released by the CTFPHC. The CTFPHC currently engages the following organizations, among others:
  • Canadian Medical Association
  • Canadian Nurses Association
  • Canadian Partnership Against Cancer
  • Canadian Public Health Association
  • Chronic Disease Prevention Alliance of Canada
  • College of Family Physicians of Canada
  • Royal College of Physicians and Surgeons of Canada
  • United States Preventive Services Task Force
  • Agency for Healthcare Research and Quality
Disease-specific organizations
These organizations, identified by the working group, include national non-governmental organizations and professional associations having a specific affiliation with the guideline topic.
Federal, provincial, and territorial government organizations
These organizations are engaged by the CTFPHC because of their key role in developing policy and/or delivering health care to Canadians. They require advance notice of the contents of each guideline to prepare their program delivery groups and to prepare for media interactions. The CTFPHC currently engages the following organizations of this type, among others:
  • Canadian Institutes of Health Research
  • Conference of Deputy Ministers of Health
  • Council of Chief Medical Officers of Health
  • Health Canada
  • Public Health Agency of Canada
  • Public Health Network Council
  • Provincial ministries of health
Stakeholders likely opposed to the CPG
In some instances, the CTFPHC knows in advance that certain organizations are likely to oppose a particular CPG. Regardless of the stakeholder’s position, the CTFPHC will provide advance notification of the contents of the guideline, as a courtesy.
Researchers and research funding agencies
These stakeholders have interests in research on topics specific to each guideline. The CTFPHC will notify these stakeholders of gaps in evidence that were identified during the guideline development process.
General public
As users of the health care system, members of the general public bring a unique and important perspective to CTFPHC activities and can be engaged at various stages of guideline development and dissemination. The CTFPHC uses both print and social media to make direct contact with the public.

Not all stakeholders will be engaged either because of their own internal capacity issues or because the CTFPHC has not regarded the involvement of certain stakeholders relevant to a particular guideline. Each guideline topic will have its own unique set of stakeholders and a corresponding plan for KT.

8.3.2 Engagement and maintenance of relationships

The CTFPHC may engage stakeholders at various points during the guideline development process. Areas of interaction may include, but are not limited to, guideline development, endorsement, partnership in the creation of tools for dissemination, and provision of CPG materials including KT tools.

The steps for engagement include:

Guideline development
Stakeholders may be engaged at various stages of the guideline development process. General and disease-specific organizations and individual health care practitioners may be asked to provide feedback on early drafts of recommendations, and feedback is solicited from all categories of stakeholders during development of KT tools. Members of the general public may also be asked to provide feedback on the selection of guideline topics, outcomes, or patient preferences. Methods used to garner stakeholder feedback at this stage include questionnaires, interviews, focus groups, and online surveys.
Pre-launch engagement

Stakeholders may be engaged before the launch of a new guideline to given them lead time to prepare for the release. For example, they may need time to prepare for media interactions and/or to make changes to policies or programs. In addition, organizations likely to oppose a particular CPG should be engaged so they have a full understanding of the contents of the guideline.

Pre-launch engagement may take the form of in-person meetings, teleconferences, letters announcing upcoming CPGs, technical briefings, webinars, and early release of CPGs and KT tools. For each CPG launch, a critical path is created, with a pre-determined list of activities and dates. Each organization that receives advance materials is required to sign and return the Confidentiality Agreement (Appendix II).

Early release of CPG-specific summary methods, with explanation of GRADE methodology
To assist stakeholders in understanding the rigour with which each CPG is developed, all stakeholders are given a synopsis of the CTFPHC methods as applied to the particular topic and the one-page KT tool about the GRADE methodology. They are also referred to the full Procedure Manual for more detailed information.
Early release of CPGs
Depending upon their particular requirements and their respective relationships with the CTFPHC, some organizations are given embargoed copies of the CPG and frequently asked questions (FAQs) before the release.
Early release of KT tools
Organizations that require a fuller briefing and those responsible for delivering front-line services are given KT tools in advance.
Supplementary engagement, such as debriefings and presentations
For each CPG, certain stakeholders are invited to attend a briefing before the release. This session gives stakeholders the opportunity to ask questions and raise concerns regarding the development and roll-out of the guideline. These briefings can also generate early feedback for the CTFPHC, allowing it to manage risk and adapt the media message to respond to concerns.
Pre-launch endorsement

For each CPG, the CTFPHC will seek formal endorsement from the CFPC. The CFPC was selected as the primary endorsement organization because of its unique historical relationship with the CTFPHC and the importance of primary care practitioners. The endorsement process involves completing a Confidentiality Agreement and review of an advance copy of the final approved version of the CPG by CFPC representatives for endorsement. The CFPC reviewers may submit questions or concerns regarding the guideline to the CTFPHC at this time. Next, the CFPC executive committee reviews the CTFPHC materials and recommendations from the CFPC reviewers and, if the guideline is deemed acceptable, grants endorsement. Once it has decided to endorse the guideline, the CFPC sends a formal letter of endorsement to the CTFPHC. If the endorsement is received before publication of the guideline, the CTFPHC may notify the journal and may mention the endorsement in media materials. The CTFPHC may also post the endorsement on its website and use this information in its KT and public relations materials. The CTFPHC will provide the CFPC with a media package (including news release, key messages of the guideline, and media narrative) and can arrange media training for CFPC spokespersons.

Depending upon the topic of the guideline, pre-launch endorsement may be sought from other organizations, according to a similar process.

8.3.3 Stakeholder outreach and education following guideline launch

Ongoing education through various KT initiatives is an integral component of the CTFPHC stakeholder engagement plan. These initiatives may be simple (distributing a pamphlet) or more involved (establishing partnerships to develop KT tools for a specific audience).

Other educational initiatives include publications in major peer-reviewed journals, presentations and distribution of KT materials at major conferences, and development of modules for continuing medical education.

Stakeholders may also be enlisted to assist with dissemination of CTFPHC materials following the launch (e.g., by posting a link to the CTFPHC guideline on their respective websites).

8.3.4 Monitoring and evaluation of engagement

Qualitative and quantitative data on all stakeholder engagement activities is collected on an ongoing basis for the CTFPHC’s annual evaluation of its KT strategy and stakeholder engagement activities. The following are some indicators of optimal engagement:

  • number of organizations reached
  • number of endorsements received
  • number of organizations responding to information
  • number of partnerships developed
  • nature of comments from stakeholders (e.g., supportive or negative)
  • achievement of desired outcomes

8.4 Evaluation of KT Activities

The CTFPHC evaluates all of its activities annually, as outlined in Table 4, to assess the impact of dissemination activities and the uptake of CPGs by stakeholders, and to ensure that all KT activities are consistently aligned with CTFPHC’s overall objectives.

The results of the annual evaluation will be reviewed by the CTFPHC members, and if necessary, adjustments will be made to the KT strategy. In addition, an annual report of KT activities is submitted to the PHAC.

Table 4. Summary of activities to evaluate knowledge translation (KT)
Objective Outcome Measurement method Timeline
To determine if statements of the CTFPHC’s objectives are aligned with its implementation efforts Alignment Document review Completed once, before beginning the evaluation
To determine the impact of KT activities aimed at disseminating CPGs to the primary knowledge users (primary care practitioners) Reach Publications, Presentations, Stakeholder dissemination, Website usage measures, Media hits Annually, for all CPG topics
To determine primary care practitioners’ level of awareness and knowledge of key recommendations, the current source(s) of information they use to guide their practice (in the context of the guideline topic), and their general perceptions of the CPG, the CTFPHC, and KT processes used Awareness of CTFPHC, CPGs, Tools; Knowledge of CPG recommendations; Agreement with CPGs Surveys/interviews Annually, for all CPG topics
To determine the impact of more active KT activities aimed at increasing use and uptake of CPG knowledge by primary care practitioners and their patients, where these activities include dissemination of KT tools for primary care practitioners (e.g., algorithms, summaries, and clinical decision support tools) and patients (e.g., summaries, FAQ sheets, and decision aids) Planned practice change Surveys/interviews Annually, for selected CPG topics for which evaluation of KT tools is feasible and appropriate or where KT activities represent major changes to previous practice

CPG = clinical practice guideline, CTFPHC = Canadian Task Force on Preventive Health Care, FAQ = frequently asked questions, KT = knowledge translation

Appendix I: Declaration of affiliations and interests form and checklist

See PDF for the original form. A summary of the content follows.

Public Health Agency of Canada Office of the Canadian Task Force on Preventive Health Care Declaration of Affiliations and Interests Form { #declaration-of-affiliations-and-interests-form }

I have reviewed my current activities and those of recent years, particularly as they relate to the attached Affiliations and Interests Checklist. I have also considered the activities of my spouse and immediate family members in so far as they could be viewed to affect my impartiality.

I hereby certify that I am not in a position of real, potential or apparent conflict of interest except as disclosed above.

I undertake to inform the Public Health Agency of Canada’s Office of the Canadian Task Force on Preventive Health Care of any changes in circumstances that may place me in a position of real, potential or apparent conflict of interest.

Public Health Agency of Canada Office of the Canadian Task Force on Preventive Health Care Affiliations and Interests Checklist { #affiliations-and-interests-checklist }

In reviewing your activities (and those of your spouse and immediate family members) to determine whether they affect your impartiality or create a real, potential or apparent conflict of interest, consider the following, among other things:

  • Investments in a business enterprise (other than mutual funds or Registered Retirement Savings Plans that are not self-directed)
  • Participation as investigator in clinical trials of relevance to the Committee’s mandate o Previous, present and potential contracts, grants and/or contributions
  • Pending negotiations regarding potential contracts
  • Honoraria and other sources of personal income
  • Advice to or close association with international organizations
  • Gifts and hospitality of significant value
  • Travel sponsorship
  • Promotion of a product(s) of relevance to the Committee’s mandate o Publications
  • Public statements
  • Lobbying activities
  • Membership in special interest groups
  • Expert testimony in court
  • Access to confidential information
  • Any interest or activity which may create a reasonable apprehension of bias

If for any reason you feel you cannot sign this statement as worded, or if you have further questions, please contact the Prevention Guidelines Division at 613‒957‒9429.

Appendix II: Confidentiality agreement

See PDF for the original agreement. A summary of the content follows.

The above individual acknowledges that information which is confidential and/or commercially sensitive (“Confidential Information”) may be disclosed.

  1. The above individual acknowledges that he/she shall procure that all persons associated with them, whether as directors, employees or otherwise):
    1. keep all the documents and information that the above individual may receive from the Public Health Agency of Canada (PHAC) in the course of carrying out his/her responsibilities, or that the Canadian Task Force on Preventive Health Care (CTFPHC) may develop while performing its mandate, strictly confidential;
    2. not use any Confidential Information for any purpose other than those indicated by PHAC or the CTFPHC;
    3. not disclose any Confidential Information to any third party without the prior written consent of PHAC or the CTFPHC, and in the event that such disclosure is permitted, the above individual shall procure that said third party is fully aware of and agrees to be bound by these undertakings.
  2. No Waiver of Privilege—The above individual acknowledges that the Confidential Information is the property of PHAC and the CTFPHC (and as some cases may allow, a third party), and that none of the latter intend to and do not waive, any rights, title or privilege they may have in respect of any of the Confidential Information.
  3. Specific Exclusions—The above individual’s obligation to protect Confidential Information hereunder does not apply to Confidential Information which, even if it may be marked “confidential”, in the following circumstances:
    1. In public domain—the information was legally and legitimately published, or otherwise part of the public domain (unless due to the disclosure or other violation of this Confidentiality Agreement by the above individual);
    2. Already known to the above individual—the information was already in the possession of the above individual at the time of its disclosure to the above individual and was not acquired by the above individual, directly or indirectly, from PHAC;
    3. Third party discloses—the information becomes available from an outside source who has a lawful and legitimate right to disclose the information to others;
    4. Independently developed—the information was independently developed by the above individual without any of the Confidential Information being reviewed or accessed by the above individual.
  4. The above individual acknowledges that there are no conflicts of interest or if there are, that they are indicated on the attached Conflict Disclosure form.

Appendix III: Functional working groups

Topic prioritization working group

The topic prioritization working group assists the Canadian Task Force on Preventive Health Care (CTFPHC) in selecting topics to consider for guideline development, according to the process outlined in section 3 of the Procedure Manual. Criteria for topic selection have been developed and are applied to ensure transparency, reproducibility, and objectivity in the topic-selection process. On the basis of these criteria, the topic prioritization working group solicits and considers input from the CTFPHC and its partners concerning the topics that should be addressed. Topic priorities are re-examined every 6 months.

The topic prioritization working group has also developed criteria to determine which of the following guideline types should be used for a given topic:

De novo
new topics that neither the CTFPHC nor anyone else has tackled lately and for which no clinical practice guidelines are pending from major organizations
topics for which the CTFPHC has produced guidance in the past, for which the literature search and recommendations need updating (currently few in number but anticipated to grow); e.g., screening for breast cancer
Critical appraisal
topics of interest to the CTFPHC or on its short list for prioritization for which other groups have developed guidance that the CTFPC can appraise
topics for which the CTFPHC has previously developed guidance and for which no evidence exists that would change the direction of the guidance

The topic prioritization working group is led by a member of the CTFPHC or the Office of the CTFPHC (TFO) and is composed of other interested individuals from the CTFPHC and a representative of the Evidence Review and Synthesis Centre (ERSC).

Methods working group

The methods working group assists the CTFPHC in maintaining the highest methodologic standards in guideline development. Output from this working group ensures that CTFPHC guidance and the methods used to produce such guidance are methodologically sound, scientifically defensible, reproducible, and well documented.

The methods working group is responsible for the ongoing review and updating of the CTFPHC Procedure Manual, which documents the methods used by the ERSC, the CTFPHC, and the TFO to develop reviews and recommendations for clinical preventive services. According to a regular schedule, the methods working group identifies areas where modifications, expansions, or updates are required.

In addition, the methods working group addresses important scientific and methodologic issues as they arise, including but not limited to reviewing the existing tools for appraisal (e.g., Appraisal of Guidelines for Research and Evaluation instrument [AGREE II]), adapting guidelines developed by other organizations (e.g., using ADAPTE), integrating performance measurement into the guideline development process, and making recommendations about the types of studies to be included in systematic reviews. All decisions related to methods issues are documented in the Procedure Manual.

The methods working group is led by a member of the TFO and is composed of other interested individuals from the CTFPHC, a representative of the ERSC, and other representatives from the TFO.

Appendix IV: Solicitation of nominations for topics for the Canadian Task Force on Preventive Health Care (CTFPHC)

See PDF for the original solicitation. A summary of the content follows.

Public Health Agency of Canada
Solicit new topic nominations

The CTFPHC invites nominations for topics to review and develop recommendations for primary care. Topics should be for primary or secondary prevention. Recent or current topics reviewed by the CTFPHC are attached.

The CTFPHC is an independent panel of experts that develops evidence-based recommendations on interventions for primary or secondary prevention in asymptomatic individuals, including screening, counselling and preventive treatment.

Individuals, organizations, evidence-based practice centres or the CTFPHC can nominate topics, which will then be reviewed and prioritized by the CTFPHC. The following criteria will be used to consider topics:

  • Disease burden (prevalence, mortality, comorbidity, quality of life) and expected effectiveness of the preventive service in decreasing that burden
  • Potential impact of recommendations in clinical practice
  • Interest of the public or care providers
  • Variation in care and potential for preventive service to decrease that variation
  • Sufficiency of evidence
  • New evidence, especially high-quality evidence in a stable field (i.e., an area where the evidence and state of knowledge are not changing rapidly)

Topics will be prioritized that have the potential to impact clinical practice. Topics previously reviewed by the CTFPHC will also be considered. To nominate a topic, please describe in no more than 500 words the topic and the rationale for conducting a review. Rationale will include the relevance of the topic to the primary care setting, whether the intervention is for primary or secondary prevention, the public health importance, summary of new evidence, and the potential impact of the review. Citations and supporting information can be included which does not count toward the 500-word limit.

Nominations for topics can be submitted to:

Prevention Guidelines Division, Public Health Agency of Canada 785 Carling Avenue, Address Locator 6807B Room 516B2 Ottawa, ON K1A 0K9 or to

The CTFPHC solicits nominations to create a balanced portfolio of topics. Topics will be selected based on the criteria described here, the CTFPHC prioritization process and the current expertise of the CTFPHC.

Appendix V: Literature surveillance

The literature is scanned by the Office of the Canadian Task Force on Preventive Health Care (TFO) every 12 months to capture published information that may affect current and previous guidance from the Canadian Task Force on Preventive Health Care (CTFPHC). This process is specific to guidance published since 2010 by the revitalized CTFPHC. Other guidance may be considered for opportunistic updates based on systematic reviews performed by other organizations and will be reviewed on a case-by-case basis.

The annual literature surveillance process consists of two steps:

  1. Targeted MEDLINE searches for guidelines, systematic reviews, and individual trials relating to previously published CTFPHC guidance.
  2. Grey literature search for guidelines using the checklist for clinical practice guidelines in [Grey Matters]((, a publication of the Canadian Agency for Drugs and Technology in Health.

Database searching is conducted automatically through pre-programmed searches of Ovid MEDLINE, Ovid MEDLINE, In-Process & Other Non-Indexed Citations, Ovid MEDLINE Daily, and Ovid OLDMEDLINE. The search strategies for this step are replications of those published in the systematic reviews for each topic performed by the Evidence Review and Synthesis Centre. Search results are sent to the TFO Refworks account on a quarterly basis, which is then reviewed annually.

Grey literature searching is conducted independently by a science officer, at similar time intervals (i.e., annually). Refer to the “Clinical Practice Guidelines” section of Grey Matters.

For each guideline, a literature surveillance summary report is generated by a science officer within the TFO and circulated to the chair of the topic prioritization working group. The summary report contains a summary of new evidence by type of publication and summary statements on the implications for the guideline under consideration. If evidence has been published that may necessitate an update to a guideline, the report is shared with the topic prioritization working group and the chair of the working group. It may be necessary to consider an update if new information has been published that changes the evidence of benefit or harm associated with the intervention, if relevant outcomes or available interventions have changed, or if changes in current practice have occurred.

Appendix VI: Critical appraisal process for existing CPGs

Step Components of the process Most responsible person(s) Detailed activities and considerations
1. Identification of a topic or guideline to consider for the CTFPHC appraisal process TFO science officer

CPGs or topics that are suitable for critical appraisal are identified through the following mechanisms:

  • Direction from the chair of the topic prioritization working group to the scientific officer for topics on the guideline short list that are not planned for production.
  • Semi-annual literature searches for new guidance.
  • Suggestions from CTFPHC members, external CPG developers, and/or the ERSC.
  • Input from the Canadian Medical Association (e.g., sharing existing list of topics with CMA and seeking input on prioritization, or receiving suggestions from CMA based on Infobase web analytics).
2. Review of guideline or topic for relevance to the CTFPHC mandate TFO science officer in consultation with the critical appraisal working group
  • The critical appraisal working group chair and the scientific officer review and discuss all guidance and topics that are brought to their attention.
  • Each CPG/topic is reviewed for relevance according to the following criteria:
    • Disease burden (affected population, incidence, prevalence)
    • Public or provider interest in the topic
    • Expected effectiveness of preventive service in decreasing the burden based on available evidence
    • Variations in care and potential for preventive services to decrease that variation
    • Potential impact of recommendation on clinical practice and opportunities for practice improvement
    • New evidence published since guideline was released or updated that has not been considered in the current CPG and that would affect the recommendations
    • Degree of alignment with CTFPHC topic priorities
3. Scan of the literature for recently published guidance on that topic, or, if a guideline has already been suggested, search for other recently published guidance on the same topic TFO science officer
  • Once a CPG has been selected for this process through opportunistic methods, it is important to determine whether other guideline groups have also recently published recommendations on the same topic.
  • No date limit has been specified. However, should the volume of guidance be high (i.e., more than five documents), limit to those published within the past five years.
  • A scan of the literature (via both PubMed database search and grey literature search) is used to identify other guidance on the same topic.
  • All guidance identified will move through step 4 of the process.
4. Completion of the guideline selection template and selection of a guideline (or guidelines) for appraisal TFO science officer

Each guideline identified through the literature scan (step 3) is evaluated against the following six criteria:

  1. Whether the guideline was produced by a national group
  2. Whether the guideline was produced by a generalist organization
  3. Who the target audience is
  4. Whether the recommendations are based on a systematic review of the literature (which is available)
  5. Whether the guideline developers applied the GRADE system
  6. Whether a family doctor was in the author list

The contents of the completed template are discussed at a meeting of the critical appraisal working group. Members select the guideline (or guidelines) to be critically appraised through a qualitative assessment of the criteria.

5. Evaluation of the CPG development process using the AGREE II review tool and scoring system, along with additional criteria identified by the CTFPHC TFO science officer
  • The scientific officer sends the CPG and supporting documents to a minimum of four CTFPHC members and two TFO employees for completion of the AGREE II assessment. Assessors are given two weeks to complete and submit reviews.
  • Reviews are submitted electronically to the scientific officer according to the standardized AGREE II evaluation form provided.
  • All scores are tabulated and summary scores calculated (according to the AGREE II scoring methodology) by the scientific officer.
  • Anonymized comments from CTFPHC members are collated.
6. Generation of summary report for dissemination TFO science officer, critical appraisal working group members, CTFPHC members
  • The final report is generated for each CPG that undergoes appraisal and meets the CTFPHC’s criteria for high quality (i.e., scores of at least 60% on the AGREE II domains of Scope and Purpose, Rigour of Development, and Editorial Independence).
  • A critically appraised guideline that does not score at least 60% on these three domains does not meet the CTFPHC criteria for high quality, and a summary report is not written.
  • For a high-quality guideline, the summary report is completed by the PGD scientific officer and is circulated to the AGREE II appraisers for comments.
  • Once comments have been incorporated, the final version is circulated to all members of the CTFPHC for approval.
7. Review and vote by CTFPHC members CTFPHC members CTFPHC members review the materials provided and vote on whether to support a positive appraisal.
8. Peer review TFO science officer
  • The completed report is sent to at least two external peer reviewers.
  • Comments are incorporated as required. A formal response to comments is not provided to the peer reviewers.
  • If changes are considered substantial, a second vote by members of the CTFPHC is required. Otherwise, proceed to step 9.
9. Feedback to the CPG developer group TFO science officer The appraisal is sent to the lead of the originating development group for information purposes and to correct any factual inaccuracies.
10. Review of KT opportunities and collaboration through KT working group Critical appraisal working group chair, TFO science officer, and members of the KT working group The KT working group reviews all KT tools available from the guideline producer and assesses whether they can be included in the disseminated package.
11. Dissemination by the KT working group KT working group The KT working group reviews the final appraisals and prepares them for dissemination through posting in a designated section of the CTFPHC website.
12. Maintenance of currency of existing critical appraisals TFO, critical appraisal working group
  • Most guideline documents are updated either on an ad hoc basis or according to regular update schedules, as determined by the originating CPG development organization.
  • Guidance that has been critically appraised and posted on the CTFPHC site is reviewed annually to determine if any updates have been published. Guideline developers are also asked to notify the CTFPHC of updates.
  • Depending on the extent of an update, a new appraisal process may be required.

AGREE = Appraisal of Guidelines for Research and Evaluation instrument, CTFPHC = Canadian Task Force on Preventive Health Care, CPG = clinical practice guideline, ERSC = evidence review and synthesis centre, GRADE = grading of recommendations assessment, development, and evaluation, KT = knowledge translation, TFO = Office of the CTFPHC)

Appendix VII: Roles and responsibilities of Canadian Task Force on Preventive Health Care topic working groups and their chairs

Each topic working group consists of two to five members of the Canadian Task Force for Preventive Health Care (CTFPHC), a scientific research manager from the Public Health Agency of Canada, and members from the Evidence Review and Synthesis Centre. In the case of a partnership with an external organization, one or two members of that organization will also be appointed as members of the topic working group.

Members of each topic working group are expected to actively participate in all aspects of the guideline development process, including developing the questions and analytic framework, reviewing the evidence, and drafting the recommendations. This work involves attending regular conference calls and providing prompt feedback as required.

The chair and co-chair of the CTFPHC select the working group chairs. The chair of the topic working group has the following responsibilities:

  • Work with the scientific research manager (who co-chairs the working group) to set the agenda and chair meetings of the working group. The chair is responsible for ensuring that the work proceeds according to pre-determined timelines.
  • Liaise with the scientific research manager to provide updates about the work and to coordinate meetings.
  • Ensure that the scope of the review is clear for all working group members (e.g., the analytic model, benefits and harms).
  • Ensure that working group members are comfortable with the process in which they are engaged and attempt to identify and deal with concerns and issues as they arise.
  • Lead the assessment of evidence for each key question according to the criteria of the Grading of Recommendations Assessment, Development and Evaluation Working Group.2

The scientific research manager leads the drafting of the recommendations, in close collaboration with the chair of the topic working group. The chair of the topic working group presents the recommendations, along with a proposal regarding the certainty and grading of the evidence and recommendations, at a meeting of the CTFPHC.

Teleconference calls

  • The TFO schedules calls for the topic working group, taking into consideration the chair’s schedule. The chair should respond promptly to requests about these calls from the TFO or members of the working group.
  • For a call to proceed, the chair and at least one other member of the working group must be available.
  • Working group members who cannot attend a call may provide comments, before or after the call, to the chair, the scientific research manager, or all working group members.

Appendix VIII: Scoping exercise process

The purpose of the scoping exercise is to provide an overview of the evidence (including areas where evidence is lacking) and to identify key literature that the working group is expected to read and understand in order to formulate the key and contextual questions. It provides a general framework from which to establish what the guideline will include and what will not be covered. The output from the exercise is a summary of the evidence, a list of key studies and guidance that the working group is expected to read and understand, and a comparative analysis of relevant guidance. The working group conducts the topic refinement exercise with input from the ERSC.


  1. After a guideline topic has been selected, the TFO, with the ERSC, scans the literature and provides the chair of the working group with a listing of relevant studies (i.e. list of references).

    The aim of this step is to identify previous evidence, but the search should not be exhaustive and should not address potential review questions in detail; rather, the scan is used to reasonably inform the content of the review and to identify any potential challenges with the evidence. The literature scan is staged. First, a search for existing guidance and systematic reviews is conducted. If no relevant evidence is identified, a search for primary studies is conducted. If appropriate, clinical trial registries are searched for studies that have ended recently or that will likely have results during the course of the proposed systematic review. The expectation is that the literature search will be focused, not extensive, and the scope will be limited to what can be completed in a week. The search is not limited to randomized controlled trials.

  2. The TFO summarizes the evidence found in the scan, identifies the articles that working group members should read (with previous guidance being key), and conducts a comparison of analytic frameworks and key questions from key guidance identified in the literature.

    The summary includes an explicit comparison of any guideline using the PICO framework as well as key differences in search strategies, perspectives, recommendations, and implications for research and policy. The ERSC clinical expert identifies what is known and not known and then maps the extent of evidence relating to key morbidity and mortality outcomes.

  3. Working group members reviews the material that has been identified through the literature scan.

    The aim of this step is to give working group members the opportunity to read material that will allow them to understand the background of the disease in question, the prevalence and burden of the disease, its etiology, its natural history and the consequences if left untreated, risk factors, the rationale for screening and screening strategies, preventive interventions, current clinical practice, existing systematic reviews, previous recommendations, related clinical practice guidelines, and other relevant material. In-depth study of every page may not be required, but it is important that all working group members be familiar with the identified material to ensure that the group’s decisions are well informed.

    After the working group has read the material, the TFO presents the evidence to the working group at a meeting, so that the working group can ask questions related to the evidence and discuss the direction of the guideline. The working group is expected to read the results of the literature scan and related material before the presentation.

  4. TFO documents the working group’s decisions and key points raised during the presentation and later meets with the working group to start drafting the wording of the key questions and contextual questions, the PICO terms, and the inclusion and exclusion criteria that will be inserted into the protocol.

Appendix IX: Protocol template

The following template is used to develop the protocol for a systematic review for the Canadian Task Force on Preventive Health Care (CTFPHC). The protocol is completed on the basis of information gathered during calls of the topic working group, including the chair, the co-chair (a scientific research manager from the Public Health Agency of Canada), and a representative of the Evidence Review and Synthesis Centre (ERSC). The protocol should focus on the key questions, contextual questions, inclusion and exclusion criteria, literature search, and analytic framework. Information intended as context to the topic should be included in the background section, whereas information needed as context to the recommendation may be included with the contextual questions (section 5.6). The protocol should be no more than 10 pages long, excluding any appendices and references.

  • Project Title:
  • ERSC Project Lead Investigator:
  • ERSC Project Staff:
  • CTFPHC Working Group Chair:
  • CTFPHC Working Group Members:
  • TFO Scientific Research Manager:
  • TFO Science Officer:
  • Suggested citation:

Section I. Purpose and background

Describe the purpose of the report (used by CTFPHC to develop recommendations), and address whether the project is new or an update.

The background section of the protocol (and systematic review) should provide the context for the topic. Briefly define the condition and discuss the prevalence and burden. The rationale for screening as well as current recommended or practiced strategies should be discussed.

Section II. Previous CTFPHC recommendations and recommendations from other guideline developers

For updated topics, describe the previous CTFPHC review, the results and recommendations. The questions used in a previous CTFPHC review can be included, and describe any limitations to the previous review.

Describe recommendations from other guideline development groups, such as the USPSTF, SIGN, NICE, and other relevant organizations. Recommendations currently being followed by the provinces and territories and any other contextual information to describe why the guideline is being updated should be included.

Section III. Scan of new evidence since previous recommendation

For updated topics, report the new evidence identified from the scoping exercise. If ongoing studies were identified in the previous review, these should be discussed.

Section IV. Review approach

If the topic is an update, provide information about how the review will be updated (new systematic review, updated, focused, or staged) and whether all key questions from the previous review will be updated (key questions for which there is no new evidence may not be updated).

Analytic framework and key questions

The analytic framework, key questions, and contextual questions are reported in this section.

Standard contextual questions may include:

  1. What is the cost-effectiveness of for in ?
  2. What are the patient values and preferences for for ?
  3. What process and outcome performance measures or indicators have been identified in the literature to measure and monitor the impact of for ?
  4. What is the optimal screening intervalf or for ?
  5. What risk assessment tools are identified in the literature to assess the risk of ?
  6. What is the evidence for a higher burden of disease, a differential treatment response, differential performance of , or barriers to implementation of for in subgroups, such as the Aboriginal population, rural or remote populations, or other ethnic populations?
Analytic plan and subgroup analysis

Discuss the analytic plan and identification of high-risk groups and how these will be considered in the recommendations.

Literature search

Describe databases, time periods, and any other relevant information about the search strategy.

Inclusion and exclusion criteria

Report all information about the population, intervention, comparator, and outcomes included and excluded. Study designs and settings included and excluded, language and date limits, and any other information on inclusion and exclusion criteria should also be described. The data abstraction and article screening forms should be supplied.

Section V. Planned schedule and timeline

The project schedule with deliverables and milestones is listed. Include the plan for updating the search before publication.

Section VI. References cited

Generate the list of references used in developing the protocol.

Appendix X: Process to incorporate and assess the quality of modelling studies that address key questions


The following process is designed for use when modeling studies are being used to answer key questions for a Canadian Task Force on Preventive Health Care (CTFPHC) systematic review. If the working group decides that modeling data are to be considered, the Evidence Review and Synthesis Centre (ERSC) conducts a search of the literature to identify modeling studies that can be used to answer the questions.


The current GRADE approach emphasizes the need to determine the quality of evidence supporting the clinically important benefits and harms attributable to use of an intervention. Randomized controlled trials (RCTs) or meta-analyses of RCTs remain the gold standard in terms of evidence for benefits. RCTs may provide high quality of evidence for harms, but given the rarity of harms, it is now recognized that prospective observational studies may be the best source of evidence for uncommon or rare harms. Identifying evidence relevant for patient-important outcomes remains the central goal.

In the field of clinical prevention, an intervention may include several components applied in sequence (such as screening followed by treatment in identified cases) and, due to slow progress of a disease, may include intermediate outcomes rather than clinically important final outcomes. While in an ideal world an RCT for benefits and a prospective cohort study for harms would provide the highest quality of evidence for screening interventions, these may not be available for the general population or subgroups within a population. In addition, guideline developers may also have questions concerning the frequency of screening and the cost effectiveness of screening. Developing de novo models or micro-simulations, or using evidence from published models, may provide an important source of new evidence.

A CTFPHC working group may choose to incorporate evidence from modeling and cost effectiveness analysis (CEA) studies to inform the estimate of benefits and harms of a preventive intervention or to inform the resource use related to an intervention. When modeling and CEA studies are sought to help inform the benefits and harms of an intervention for a general population or subgroups of a population, then the ERSC will use the six-step process described herein to systematically search, appraise and judge the quality of the CEA/modeling study.

The results of a CEA/modeling study will be incorporated into the systematic review only if it is considered to be methodologically rigorous (“well done” or “very well done”). At this stage, the ERSC assigns a GRADE quality rating related to the evidence for benefits and harms related to patient-important outcomes.

Step 1

The ERSC screens the papers identified in the search for “applicability”. The applicability criteria are based on the National Institute for Health and Care Excellence methodology checklist for economic evaluations.9 The guideline is assessed to determine if the study population and the intervention is appropriate for the guideline, and an overall judgement of applicability is given:

Directly applicable : the study meets all applicability criteria, or fails to meet one or more applicability criteria but this is unlikely to change the conclusions about cost effectiveness.

Partially applicable : the study fails to meet one or more applicability criteria, and this could change the conclusions about cost effectiveness.

Not applicable : the study fails to meet one or more applicability criteria, and this is likely to change the conclusions about cost effectiveness. Such studies would be excluded from further consideration and there is no need to continue with the Drummond Plus table.

Studies that are directly or partially applicable move on to step 2.

Step 2

The ERSC completes an assessment of the quality of the modelling studies, using questions from the Drummond checklist10 and those developed by the CTFPHC (Plus).

Step 2: Quality appraisal of economic studies adapted from Drummond10
Source Question Study 1 Study 2
Drummond 1. Was a well-defined question posed in answerable form?
Drummond 2. Was a comprehensive description of the competing alternatives given (i.e., can you tell who did what to whom, where, and how often)?
Drummond 3. Was the effectiveness of the program or services established?
Drummond 4. Were all the important and relevant costs and consequences for each alternative identified?
Drummond 5. Were costs and consequences measured accurately in appropriate physical units (e.g., hours of nursing time, number of physician visits, lost work-days, and gained life-years)?
Drummond 6. Were the cost and consequences valued credibly?
Drummond 7. Were costs and consequences adjusted for differential timing?
Drummond 8. Was an incremental analysis of costs and consequences of alternatives performed?
Drummond 9. Was allowance made for uncertainty in the estimates of costs and consequences?
Drummond 10. Did the presentation and discussion of study results include all issues of concern to users?
Plus 11. Were all the relevant comparators considered?
Plus 12. Were all the relevant outcomes considered?
Plus 13. Does the study population consider appropriate subgroups that require special attention for the guideline (e.g., high-risk population)?
Plus 14. Were the ethical/distributional implications discussed?
Plus 15. Is there no potential conflict of interest (includes funding considerations)?
Plus 16. Was the generalizability of outcomes discussed?
Plus 17. Are the outcomes and input parameters applicable to the Canadian context?
Plus 18. Are the conclusions of the evaluation justified by the evidence presented?

####Step 3

A modeling consultant completes the “characteristics of included modeling studies table” (below) and use these data and the results presented in step 2 to evaluate the level of methodological quality for each study. Studies identified with an overall quality assessment of “very well done” and “well done” move on to step 4 of the process.

The overall methodological study quality of the economic evaluation is assessed in two ways:

  • level of limitations (minor, potentially serious, very serious)[^1]
  • overall quality of the model (very well done, well done, fair, poor)

Limitations definitions:

Minor limitations : the study meets all quality criteria, or the study fails to meet one or more quality criteria but this is unlikely to change the conclusions about cost effectiveness.

Potentially serious limitations : the study fails to meet one or more quality criteria and this could change the conclusions about cost effectiveness.

Very serious limitations : the study fails to meet one or more quality criteria and this is highly likely to change the conclusions about cost effectiveness. Such studies should usually be excluded from further consideration.

Step 3: Characteristics of included modeling studies
Characteristic Study 1 Study 2
Screening for
Screening mechanism
Screening programs
Model format
Time horizon
How risk of condition modeled
Quality of model
How time to detection modeled
Other risk factors changing prior to detection
Treatment at point of diagnosis
Funding source
Conclusion Step 3
Overall assessment of the quality of the model (very well done, well done, fair, poor)
Limitations assessment (minor, potentially serious, very serious)
Provide details of limitations

Step 4

The modeling consultant completes the following table for those studies that perform well in steps 1–3 and chooses the ones to include into the systematic review by applying the criteria below.

Step 4: Selecting the studies that will be incorporated into the evidence
Criteria Study 1 Study 2
Criteria 1. Relevance of the model’s focus to the key questions and contextual questions addressed by the guideline: High/Medium/Low
Criteria 2. Relevance of the model’s sensitivity and scenario analyses to the key questions and contextual questions addressed by the guideline: High/Medium/Low
Criteria 3. Capacity to use the model for de novo analyses relevant to key questions and contextual questions
Criteria 4. Is the model up to date? Yes/No
Criteria 5. Other considerations? List other reasons why the selected model is the best to move forward with for inclusion.

Step 5

The modeling consultant applies GRADE criteria to the final included study/studies. The approach to rating quality of evidence for modeling studies is as follows:


“Modeling”, including mathematical models, decision analyses, and economic analyses. These studies always start at “very low” quality, to account for the inherent and often hidden risk of bias that accompanies the modeling process. In addition, the CTFPHC considers admitting evidence only from modeling studies that are determined to be “well done” or “very well done.”


Used to highlight appraisal issues related to the modeling studies, the rule being that identification of “very serious limitations” would negate any later increase in the evidence rating. This category should also be used to highlight limitations that emerge due to individual studies that are used in the model, where well-done RCTs would be considered to have “minor limitations” (meaning there could be a rating up to maximum of two points, depending on results for subsequent categories) and studies with other designs (i.e., observational studies) or poorly done RCTs would have very serious limitations (which would limit the rating to a maximum of one point). Since it is not possible to go below “very low” for the quality of the evidence, no further downgrading is necessary. Furthermore, given the inherent assumptions of models, it is suggested that it is impossible to have a model with “no study limitations,” so this designation would not apply.


In the case of “well done” modeling studies that show inconsistent results (for example, substantial variability between clinically plausible scenarios), the quality should be downgraded. Note: Modeling studies cannot be pooled, given the nature of these studies.


Modeling studies are usually designed to provide a more direct estimate of benefits and harms for specific groups. However, if the groups in the study are not the groups of interest for the guideline, the CTFPHC will downgrade the rate for indirectness.


In the case of modeling studies, this characteristic is not applicable, as there will be no confidence intervals for relative risks. (“NA” is inserted in the corresponding box on the evidence profile.)

Other considerations

This criterion can be used to increase the rating by one or two points, based on the limitations and three additional variables:

  1. Range of comparators: at least three comparators
  2. Outcomes: at least one outcome from the list of important and critical outcomes
  3. Validated: the standard definition used in relation to modeling studies applies and evidence of demonstration of sensitivity analysis is also considered.
Number of patients and effect

Does not apply, given the nature of these studies. (“NA” is inserted in the corresponding box on the evidence profile.)

A sample of the GRADE Evidence Profile and Summary-of-Findings Table for modeling studies using diabetes data is shown at the end of this appendix.

Step 6

ERSC or staff or the modeling consultant prepares a summary of the evidence that addresses the key questions, which is included in the systematic review.

Example of a GRADE Evidence Profile and Summary-of-Findings Table for modeling studies using diabetes data
Quality Assessment Summary of Findings Importance
No. of patients Effect Quality
No. of Studies Design Limitations Inconsistency Indirectness Imprecision Other Considerationsx Screening Control Relative (95% CI) Absolute
2i modeling studiesii serious limitationsiii no serious inconsistencyiv no serious indirectnessv no serious imprecisionvi Use of appropriate range of comparators.vii Evidence of sensitivity analysis.viii Validation of model.ix NA NA NA NA ⊕⊖⊖⊖ Low Critical

Appendix XI: Standard operating procedures

McMaster Evidence Review and Synthesis Centre (MERSC) Systematic Reviews for the CTFPHC

☑ = procedure for input (I) and/or quality control (QC)

A. Protocol registration and modifications

  1. After the protocol is approved by the full Canadian Task Force on Preventive Health Care (CTFPHC) the review coordinator registers the project with PROSPERO (International Prospective Register of Systematic Reviews). The working group is notified when the PROSPERO file is complete. ☑ (QC)

  2. The review coordinator logs decisions that affect the protocol as the project evolves. Changes, additions or modifications to the protocol are made in the existing document to keep it current and correct. Revised versions are dated and labelled with a modification number. Appropriate revisions are made to the PROSPERO registered file. ☑ (QC)

  3. The review coordinator notifies review team members of protocol revisions that have implications for their work.

B. Search for citations

Original search
  1. The librarian prepares the search strategy based on protocol parameters including timeframe, databases, study designs and key words. Separate searches may be conducted for systematic reviews and randomized controlled trials, observational studies and contextual questions.

  2. The search strategy is peer reviewed and any suggested changes to the strategy can be incorporated prior to the implementation of the search.

  3. The search is conducted by the librarian; the date the search is conducted is recorded with the list of search terms for each database.

  4. The search results are downloaded into a reference manager database such as EndNote.

  5. Deduping of citations takes place, first using the EndNote duplicate tool,then using visual scans of the citations by author then by title.

  6. A Research Assistant checks for missing or incorrect information and fixes errors. ☑ (QC)

The deduped and corrected citations are loaded into DistillerSR. Any remaining duplicates that are found during the screening process will be quarantined in DistillerSR.

Updated search
  1. The literature search for all key questions is updated prior to the completion of the draft of the systematic review to ensure that the working group members have access to the most current literature.

  2. If the update was conducted more than one year prior to thes ubmission of the guideline and/or the short paper for publication, a second full updated search of the key questions is undertaken. These citations will follow processes in Sections C–G and results are incorporated in the review.
Pre-guideline targeted search
  1. Approximately 6 to 8 weeks prior to the public release of the CTFPHC guideline, the ERSC conducts a targeted search of MEDLINE for relevant RCTs regarding key questions, released since the updated search was conducted. These studies do not become part of this review, but are identified in order that the CTFPHC spokesperson is aware of any recent research findings.

C. Screening on DistillerSR

Title and abstract screening (key questions)
  1. Based on basic inclusion/exclusion criteria specified in the protocol the first level title and abstract screening questions are developed by the Review Coordinator.

  2. The questions are set up as a screening form on DistillerSR by the review coordinator and checked by the methods manager. ☑ (QC) A companion guide (with criteria, examples, definitions, etc.) is developed and linked to the screening form.

  3. MERSC review team members test the screening form/guide on at least 50 citations randomly selected from the database. All staff working on the review topic screen the same titles and abstracts to establish consistency and understanding of the questions. ☑ (QC) Refinements to the screening questions and/or the guide are made accordingly.

  4. First level screening proceeds with two screeners reviewing each title/abstract. For a citation to move to full text review, one or both screeners must indicate it meets the inclusion criteria or they cannot tell. ☑ (QC)

  5. The review coordinator performs random checks to monitor the progress and consistency of the screeners. ☑ (QC) Individual and/or group conversations will take place to address questions or problems that arise during screening.

  6. After all citations have been screened to exclude obviously irrelevant items(e.g., commentaries, not the disease or population of interest), the review coordinator may develop a second level title and abstract screening form with several more refined questions. A second round of relevance testing by two experienced screeners maximizes efficiency by using the topic and research knowledge gained through the first round of title and abstract screening to reduce the number of irrelevant articles put through for full text screening. As in the initial round, the questions are circulated to the MERSC review team for input and necessary modifications are made to the tool before screening begins. ☑ (I)

  7. Questions, concerns, results and/or progress reports from title and abstract screening may be brought to working group discussions by the review coordinator. ☑ (I, QC)
PDF retrieval
  1. Once the second round of title and abstract screening is underway, the full papers of citations moving to full text review are retrieved.

  2. Research Assistants retrieve, store (electronically at MERSC) and upload full text PDFs into DistillerSR.

  3. If a full text article is not found through the McMaster Health Scienceslibrary, it is ordered through Racer Inter-Library Loan at the McMaster Library.

  4. If a full text article cannot be found at the McMaster library or through ILL, other methods are used to access the paper including:

    1. contacting the author;

    2. purchasing through a journal/organization;

    3. consulting working group members to see if they have the article or have any way of retrieving the paper.
  5. If the full text paper cannot be accessed by the above methods, the citation is marked as “unable to be retrieved” in the reference manager database and is excluded from the review.
Full text screening (key questions)
  1. Based on the key questions and the inclusion/exclusion criteria specified in the protocol the full text screening questions are developed by the review coordinator.

  2. Steps 2 to 4 as outlined above in the Title and Abstract Screening section are used to refine and finalize the full text screening form and guide. ☑ (I, QC)

  3. MERSC review team members test the full text screening tool on a group of papers randomly selected from the database. All the people involved in full text screening will review the same titles and abstracts to establish consistency and understanding of the questions. ☑ (QC) Refinements to the screening questions and/or the guide are made accordingly.

  4. Where possible, the same people involved in title and abstract screening complete full text screening, limiting the need for extensive new training. Clarification regarding study designs for inclusion and particulars of outcomes or key questions form the basis of the training for this stage of the review. Training is done by the review coordinator. ☑ (QC)

  5. Two screeners are assigned to each paper. For a paper to advance to data extraction, both screeners must agree the study meets all the inclusion criteria. When screeners’ answers are not consistent, they must resolve their conflicts. ☑ (QC)

  6. When full text screening is complete, reference lists for each key question are circulated by the Review Coordinator to the working group for review and comment. The working group informs the review coordinator of any additional papers they believe might be relevant for inclusion. ☑ (I) These papers are retrieved and reviewed in relation to the inclusion/exclusion criteria.

  7. Reasons for exclusion are prepared for all papers excluded at this stage. These reasons are entered into the reference manager database and are reported in the Excluded Studies List attached to the final review.

  8. Questions, concerns, results and/or progress reports from full text screening may be brought to working group discussions by the review coordinator. ☑ (I, QC)
Screening for contextual questions
  1. Screening for contextual questions will be done by one person, usually the author for that section of the review.

  2. Included studies will not be quality appraised and there will be no GRADE tables for contextual questions.

D. Quality assessment of included studies for key questions

  1. Two review team members independently assess the methodologic quality of each of the included studies. Reviewers must agree on their ratings, consulting another team member if consensus cannot be reached. ☑ (QC)

  2. The AMSTAR tool is used to assess systematic reviews. The Cochrane risk-of-bias tool is used to assess RCTs. The Newcastle–Ottawa Scale is used to assess case-control and cohort analytic studies. The QUADAS-2 scale is used for screening/diagnostic tests. Drummond criteria are used for economic analyses and Drummond Plus for modeling studies.

  3. The results of the quality assessment are summarized in one or more tables that are included as an appendix in the systematic review.

  4. Questions, concerns, results and/or progress reports from the quality assessment process may be brought to working group discussions by the review coordinator. ☑ (I, QC)

E. Data extraction

Key questions
  1. Once studies for the key questions have been identified, a review team meeting (possibly one or two days in duration depending on the number of included studies) takes place, involving the review coordinator, methods manager, senior methodologist, content/clinical expert, statistician, and any other staff involved in data extraction. The purpose of this meeting is to confirm the relevance of the included studies, decide if meta-analyses can be performed, determine the most appropriate statistical tests, identify which pieces of data need to be extracted from the studies and decide if any data need to be transformed (e.g., inverted) or if additional data need to be computed (e.g., confidence intervals, logs). ☑ (I, QC)

  2. Based on decisions reached at the review team meeting and if necessary in consultation with the working group ☑ (I) a data extraction form is constructed in Microsoft Excel or in DistillerSR and a companion guide is developed.

  3. Two members of the MERSC review team test the data extraction tools on a small number of papers (up to five) to ensure consistency and understanding of questions. ☑ (QC) Refinements to the form and/or guide are made accordingly.

  4. Data extraction for each included study is done independently by two members of the review team who then compare their answers. Conflicts that cannot be resolved by the two reviewers are presented to another member of the team (e.g., statistician, senior methodologist) for resolution.

  5. Once data are entered into DistillerSR or Excel, they will be checked by the project coordinator or the statistician. Regular reports will be run by the project coordinator looking for inconsistencies in the data and corrections will be made accordingly. Any issues arising will be discussed internally at the project meetings.

  6. Data will also be checked by the person responsible for writing the section.

  7. Once there is agreement on the extracted data, the completed form is used for analysis (meta-analyses and/or narrative syntheses). ☑ (QC)

  8. If any data need to be transformed or computed to ensure consistency in reporting across studies and/or to permit meta-analyses, these new numbers (and any formulas used to derive them) are entered into the data extraction table and a statistician or the senior methodologist reviews them to verify their accuracy. ☑ (QC)

  9. A review team member not involved in the data extraction task performs an additional verification step to ensure the numbers in the data table match the numbers reported in the original papers. To document this check the person will highlight the verified numbers in hard copies of each included study. ☑ (QC)

  10. Questions, concerns, results and/or progress reports from data extraction may be brought to working group discussions by the review coordinator. ☑ (I, QC)

F. Analyses

Meta-analyses (key questions)
  1. To avoid transcription errors, whenever possible numbers used for analyses are transferred directly from the Excel or DistillerSR data extraction form into the analytic program. ☑ (QC)

  2. The senior methodologist is involved in deciding which analyses to do and which software programs to use. Meta-analyses are conducted by the statistician and/or the review coordinator. The senior methodologist checks all statistical results to verify analyses are done correctly. ☑ (QC)

  3. Whenever possible the data used for analyses and presented in the evidence sets (e.g., forest plots, GRADE tables) are reported to four decimal places.

  4. Forest plot graphs should use a consistent scale (X axis) and the scale should be set at the smallest possible range.

  5. Analytic figures (e.g., forest and funnel plots) included in the evidence sets are copied directly from the statistical software program used for analysis. ☑ (QC)

  6. A review team member not involved in th eanalysis checks the output to verify that the numbers are consistent with the data extraction tables, the correct tests have been used, and the plots are formatted appropriately. ☑ (QC)

  7. All analysis will be reviewed by the senior methodologist assigned to the project. ☑ (QC)

  8. Questions, concerns, results and/or progress reports about the meta-analyses are brought to working group discussions by the review coordinator. ☑ (I, QC)
Narrative syntheses (key questions and contextual questions)
  1. If the data cannot be pooled, the range of individual study results may be presented in the GRADE tables.

  2. Statistical findings reported in the results section are taken directly from the Excel or DistillerSR data extraction file. ☑ (QC) These numbers are rounded to two decimal places.

  3. A review team member other than the author checks these sections against the data extraction file to ensure numbers have been transferred and rounded accurately. ☑ (QC)

  4. Questions, concerns, results and/or progress reports related to writing the narrative syntheses may be brought to working group discussions by the review coordinator. ☑ (I, QC)


Preparation of GRADE tables (evidence profile and summary of findings) for key questions
  1. GRADE tables are produced only for outcomes that the working group rates as critical or important.

  2. Two review team members, with knowledge of the studies included for a key question, independently rate the body of evidence according to the five GRADE categories. Reviewers must agree on their ratings, consulting a third member of the team if consensus cannot be reached. The basis for category ratings (reasons for downgrading or not downgrading evidence) are included as footnotes in the GRADE tables. ☑ (QC)

  3. Whenever possible data are imported directly into the GRADE table from the statistical software program/file used to conduct the meta-analysis. ☑ (QC)

  4. Data used and reported in the GRADE tables are reported to four decimal places.

  5. A review team member who did not create the GRADE tables checks them to ensure numbers have been transferred accurately and footnotes are appropriate and complete. ☑ (QC)

  6. Questions, concerns, results and/or progress reports about GRADE table preparation may be brought to working group discussions by the Review Coordinator. ☑ (I, QC)

  7. Completed GRADE tables are circulated to the working group with the draft report. ☑ (I)

  8. Project staff will attend the working group meetings. Notes from these meetings are circulated by the PHAC staff and decisions recorded.

  9. Local project team meetings will be held as needed (usually weekly). Notes will be taken at these meetings and circulated to the group. A decision log will be created for each project. The review coordinator will be responsible for ensuring this is completed. The logs will be stored in the shared drive and updated on a weekly basis.

  10. Progress reports will be created by the review coordinator and can be circulated at the working group meetings. These can be done in point form, providing updates to the progress of screening and data extraction. Providing examples of problems, as well as examples of clear decisions, helps the working group to understand the complexity of the task.

H. Systematic review documents

  1. All non-finalized products (e.g., review drafts, preliminary evidence sets, summary tables) sent to the working group or the larger CTFPHC require an appropriate watermark (e.g., Draft, Draft for Comment, For Consideration, Preliminary Data) ☑ (QC)
Version control
  1. All documents prepared and circulated for review are labeled (filename and/or footer) with the version number, date and/or name of the author or the last person to make changes to the document. ☑ (QC)

  2. For documents circulated outside of the MERSC review team (e.g., to the working group, the CTFPHC, or external reviewers) a record is kept (either in a separate log or in the filename) of which document version is sent for review. ☑ (QC)
Writing the systematic review
  1. The author(s) are provided with a full text copy (electronic and/or hard copy) of each study included in their section(s) as well as a copy of the data extraction table. Additional requests from review authors for summary statistics are submitted to the Review Coordinator and prepared by MERSC staff.

  2. Authors submit their completed section(s) to the Review Coordinator. Any questions or comments are returned to the author. Review and editing of the section(s) is a collaborative process between the author and the Review Coordinator as questions or comments arise.
Formatting the systematic review
  1. The systematic review is formatted according to the template developed by MERSC (CTFPHC Procedure Manual Appendix XII). ☑ (QC)
Editing the systematic review
  1. A trained editor (internal staff or contract hire), who is aware of the project but is not necessarily part of the review team, reviews the full document and suggests edits to improve readability, consistency and basic coherence. Comments, questions and suggestions are referred to the review coordinator and/or author of the section. Any inconsistencies in the text are checked with the data provided in the original paper(s). ☑ (QC)
Responding to Working Group Comments
  1. After receiving the working group’s comments on the systematic review draft ☑ (I) the review coordinator, in consultation with the methods manager, determines which comments can/should be addressed. These decisions are documented and shared with the PHAC scientific research manager and the working group chair. ☑ (I)

  2. The review coordinator or another member of the review team makes the appropriate changes to the document and records the specific actions taken in the systematic review comment table. ☑ (QC)

  3. The working group then has the opportunity to review and approve the document to be sent for peer review.

I. Project meetings

  1. Project staff will attend the working group meetings. Notes from these meetings are circulated by the PHAC staff and decisions recorded.

  2. Local project team meetings will be held as needed (usually weekly). Notes will be taken at these meetings and circulated to the group. A decision log will be created for each project. The review coordinator will be responsible for ensuring this is completed. The logs will be stored in the shared drive and updated on a weekly basis.

  3. Progress reports will be created by the review coordinator and can be circulated at the working group meetings. These can be done in point form, providing updates to the progress of screening and data extraction. Providing examples of problems, as well as examples of clear decisions, helps the team to understand the complexity of the task.

Appendix XII: Systematic review template

Project Title [Level 1 Title: Times New Roman, Bold, Size 24]

Date [Level 6 Title: Times New Roman, Size 12]

MERSC Group Authors
McMaster University
Hamilton Ontario Canada


PGD Scientific Research Manager:

CTFPHC Working Group Members:

Suggested citation:

  • Abstract [Level 2 Title: Times New Roman, Bold, Size 18]
    • Background: [Level 5 Title: Times New Roman, Bold, Size 12]
    • Purpose:
    • Data Sources:
    • Study Selection:
    • Data Abstraction:
    • Results:
    • Data Synthesis:
    • Limitations:
    • Conclusions:
  • Table of Contents
  • Abbreviations/ Glossary
  • Chapter 1: Introduction [Tables included throughout body of text]
    • Purpose [Level 3 Title: Times New Roman, Bold, Size 16]
    • Condition Background
    • Definition [Level 4 Title: Times New Roman, Bold, Size 14]
    • Prevalence and Burden of Disease
    • Etiology and Natural History
    • Consequences if Left Untreated
    • Risk Factors
    • Rationale for Screening
    • Screening Strategies
    • Interventions/Treatments
    • Current Clinical Practice
    • Previous Review and CTFPHC Recommendations
  • Chapter 2: Methods
    • Analytic Framework and Key Questions
    • Search Strategies
    • Study Selection
    • External Review
    • Quality Assessment, Data Abstraction and Analysis
  • Chapter 3: Results
    • Summary of the Literature Search
    • Results for Key Questions
    • Key Question 1a:
    • Key Question 1b:
    • Key Question 1c:
    • Key Question 2a:
    • Key Question 2b:
    • Key Question 2c:
    • Results for Contextual Questions
    • Contextual Question 1:
    • Contextual Question 2:
    • Contextual Question 3:
    • Contextual Question 4:
  • Discussion
    • Limitations
    • Future Research
  • Conclusion
  • Reference List
  • Figure 1: Analytic Framework and Key Questions
  • Figure 2: Search Results
  • Table: Characteristics of Included Studies
  • Table: Risk of Bias Table
  • Evidence Sets
    • Evidence Set 1:
    • Evidence Set 2:
    • Evidence Set 3:
    • Evidence Set 4:
  • Appendix 1: Search Strategies
  • Appendix 2: Grey Literature Search
  • Appendix 3: List of Reviewers (Protocol)
  • Appendix 4: List of Reviewers (Review)
  • Appendix 5: Tools
  • Appendix 6: List of Excluded Studies
  • Acknowledgements:

Appendix XIII: Headings for a GRADE risk of bias table

Author/Year Sequence generation Allocation concealment Blinding of outcome assessors Incomplete reporting Selective reporting Other Overall risk of bias

Appendix XIV: Characteristics of included studies

Study Objective Methods Participants Intervention Measurement (screening) tool Outcomes Comments
Author(s), Date, Country
  • Design:
  • Selection: Recruitment, inclusion/exclusion
  • Blinding:
  • Confounders:
  • Sample: total N
  • Intervention: study group n = and control group(s) n =
  • Characteristics:
  • Loss to follow-up:
  • Other relevant information such as years of recruitment:
Description of intervention and control, duration of intervention, length of follow-up Related to the key questions Study limitations identified by the study or review authors

Appendix XV: Example of a GRADE summary-of-findings table

Does screening with mammography (film and digital) reduce all cause mortality?
Outcomes Illustrative Comparative Risksxi Relative Effect (95% CI) No. of Participants (Studies) Quality of the Evidence (GRADE)
Assumed Risk


Corresponding Risk

Screening with Mammography (film and digital)

All Cause Mortality for Ages 39–49

Follow-up: 10–16 years

18,070 per 1,000,000 17,528 per 1,000,000 (16,443 to 18,793) RR 0.97 (0.91 to 1.04) 211,270 (2 studiesxiv) ⊕⊕⊕⊕ highxiixiii
All Cause Mortality for Ages 50–59 35,040 per 1,000,000 37,142 per 1,000,000 (33,638 to 41,347) RR 1.06 (0.96 to 1.18) 39,465 (1 study) ⊕⊕⊕⊕ highxiiixiv

CI: Confidence interval; RR: Risk ratio

GRADE Working Group grades of evidence

High quality:
Further research is very unlikely to change our confidence in the estimate of effect.
Moderate quality:
Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.
Low quality:
Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
Very low quality:
We are very uncertain about the estimate.

Note: GRADE = Grading of Recommendations Assessment, Development and Evaluation.

Appendix XVI: Example of a GRADE evidence table

Summary of findings KQ 1—Effect of community-based suicide prevention program (including screening for depression)—incidence of suicide (overall)
Quality assessment No. of patients Effect Quality Importance
No. of studies Design Risk of bias Inconsistency Indirectness Imprecision Other considerations CDS screening—Elderly Control Relative (95% CI) Absolute
Overall (follow-up 5–10 years; assessed with: CDS (community depression screening))
5xv observational studies no serious risk of biasxvi no serious inconsistencyxvii very seriousxviiixix no serious imprecisionxx nonexxi 65/70,053 (0.09%) 145/113,324 (0.13%) RR 0.5006 (0.3213 to 0.7802) 1 fewer per 1,000 (from 0 fewer to 1 fewer) ⊕⊝⊝⊝ very low critical