Overview—Appraisal Process

Home Overview—Appraisal Process
PDF Version
Download PDF

Introduction

The Canadian Task Force on Preventive Health Care (CTFPHC) appraises guidelines produced by other organizations as a complement to the production of CTFPHC clinical practice guidelines. The focus of the critical appraisal is on the quality of the methods used, with a commentary section that outlines some points for primary care practitioners to consider should they choose to implement the recommendations.

 

Guideline Appraisal Process

Background

  1. Topic selection: assemble appraisal topic list with input from various sources.

    1. Decision:

      • Topic falls within CTFPHC mandate.

      • Topic does not fall within CTFPHC mandate.

  2. Guideline selection: conduct literature search to identify other recently published guidelines on same topic and apply guideline selection template to systematically select a guideline for appraisal.

    1. Selection of CPG(s)

  3. AGREE II assessment: guideline is assessed by 4–6 reviewers.

    1. Assessment:

      • Meets CTFPHC criteria for high quality.

      • Does not meet CTFPHC criteria for high quality.

  4. Generation of summary report.

  5. CTFPHC review and approval; review by guideline developer.

  6. Publication.

The Canadian Task Force on Preventive Health Care (CTFPHC) develops and disseminates clinical practice guidelines (CPG) for primary preventive care, based on systematic analysis of scientific evidence.

The CTFPHC, in cooperation with the Prevention Guidelines Division at the Public Health Agency of Canada and the Evidence Review and Synthesis Centre (ERSC), meets this mandate in several ways.

The CTFPHC develops new guidelines or updates existing guidelines on preventive health services, and also identifies additional high quality guidance produced by other organizations on topics that are relevant to the CTFPHC’s mandate.

What is a critical appraisal?

The CTFPHC has implemented a process to critically appraise eligible guidelines produced by other groups. The CTFPHC critical appraisal process does not include a new systematic review of the source evidence referred to in the guidelines. As a result, the CTFPHC does not undertake a detailed content review of the recommendations or the appropriateness of levels of evidence. The focus of the critical appraisal is on the quality of the methods used to produce the guideline, with a Commentary section that outlines some points for primary care to consider if they choose to implement the recommendations in practice.

Objectives

The primary objective of the critical appraisal process is to identify high quality guidelines that Canadian practitioners can use to facilitate preventive health care. A secondary objective is for the CTFPHC website to serve as a repository of guidelines identifying best practices for primary and secondary prevention.

 

Process

Step 1

Identification of a topic or guideline to consider for the CTFPHC appraisal process.

Most Responsible Person(s)
PGD Scientific Officer

CPGs or topics that are suitable for critical appraisal can be identified through the following mechanisms:

  • The Scientific Officer will obtain direction from the Chair of the Topic Prioritization WG for topics on the Guideline short list that are not planned for production.
  • Semi-annual literature searches for new guidelines.
  • Suggestions from CTFPHC members, external CPG developers, and/or the ERSC.
  • Input from the Canadian Medical Association (e.g. sharing the list of topics with CMA and input on prioritization, or suggestions from CMA based on Infobase web analytics).

 

Step 2

Review of guideline or topic for relevance to the CTFPHC mandate.

Most Responsible Person(s)
PGD Scientific Officer in consultation with the Critical Appraisal WG

  • The Critical Appraisal WG chair and and Scientific Officer will review and discuss all guidelines and topics that are brought to their attention.
  • CPG/topic will be reviewed for relevance according to the following criteria:
    • Disease burden (affected population, incidence, prevalence),
    • Public or provider interest in the topic
    • The expected effectiveness of preventive service in decreasing the burden based on available evidence
    • Variations in care and the potential for preventive services to decrease that variation
    • Potential impact of recommendation on clinical practice and opportunities for practice improvement
    • New evidence published since guideline last released or updated that has not been considered in the current CPG and would impact the recommendations
    • Degree of alignment with CTFPHC topic priorities Last modified:

 

Step 3

Conduct a literature scan for recently published guidelines on that topic. Or, if a guideline has already been suggested, search for other recently published guidelines on that same topic.

Most Responsible Person(s)
PGD Scientific Officer

  • Once a CPG has been selected for this process through opportunistic methods, it is important to determine whether other guideline groups have also recently published recommendations on the same topic.
  • No date limit has been specified. However, should the volume of guidelines be high (i.e. n > 5), limit to those published within the last 5 years.
  • A scan of the literature (via both grey literature search and through Pubmed) will identify other guidelines on the same topic.
  • All guidelines identified will move through Step 4 of the process.

 

Step 4

Complete the Guideline Selection template and choose a guideline (or guidelines) for appraisal.

Most Responsible Person(s)
PGD Scientific Officer

Each guideline identified through the literature scan (Step 3) will be evaluated based on 6 criteria:

  1. Whether the guideline was produced by a national group.
  2. Whether the guideline was produced by a generalist organization.
  3. Who the target audience is.
  4. Whether the recommendations are based on a systematic review of the literature (which is available).
  5. Whether the guideline applied the GRADE system.
  6. Whether a family doctor was in the author list.

The contents of the completed template will be discussed at a Critical Appraisal WG meeting. Members will select the guideline (or guidelines) that will be critically appraised through a qualitative assessment of the criteria.

 

Step 5

Evaluation of the CPG development process using the AGREE II review tool and scoring system. This includes additional criteria identified by the CTFPHC.

Most Responsible Person(s)
PGD Scientific Officer

  • The Scientific Officer will send the CPG and supporting documents to a minimum of 4 CTFPHC members and 2 OCTFPHC employees for completion of AGREE II assessment. Assessors will be given 2 weeks to complete and submit reviews.
  • Reviews will be submitted electronically to the Scientific Officer on the standardized AGREE II evaluation form provided.
  • All scores will be tabulated and summary scores calculated according to the AGREE II scoring methodology by the Scientific Officer.
  • Anonymized comments from CTFPHC members will be collated.

 

Step 6

Generation of summary report for dissemination.

Most Responsible Person(s)
PGD Scientific Officer, Critical Appraisal WG members, and CTFPHC members

  • The final report will be generated for each CPG that undergoes appraisal and meets the CTFPHC’s criteria for high quality.
    • Scores of at least 60% on the AGREE II domains of Scope and Purpose, Rigor of Development and Editorial Independence.
    • At least 60% consensus on the question that asks whether the CPG should be supported or not.
  • Critically appraised guidelines that do score at least 60% on the above domains do not meet the CTFPHC criteria for high quality. Summary reports will not be written.
  • For high quality guidelines, the report will be completed by the PGD Scientific Officer and circulated to the AGREE II appraisers for comments.
  • Once incorporated, the final version will be circulated to the full Task Force for approval.

 

Step 7

Review and vote by CTFPHC members.

Most Responsible Person(s)
CTFPHC members

CTFPHC members will review and vote on the final decision to support a positive appraisal.

 

Step 8

Peer review.

Most Responsible Person(s)
PGC Scientific Officer

  • The completed report is sent to at least 2 external peer reviewers.
  • Comments are incorporated as required. A formal response to comments is not provided to the peer reviewer.
  • If changes are considered substantial, report must be re-voted on by the CTFPHC members. Otherwise, proceed to Step 9.

 

Step 9

Feedback to the CPG developer group.

Most Responsible Person(s)
PGD Scientific Officer

The appraisal will be sent to the lead of the originating development group for information purposes and to correct any factual inaccuracies.

 

Step 10

Review of KT opportunities and collaboration through Knowledge Translation Working Group (KT WG).

Most Responsible Person(s)
Critical Appraisal WG Chair, PGD Scientific Officer, and members from the KT WG.

Review all KT tools available by guideline producer and assess whether they can be included in the disseminated package.

 

Step 11

Dissemination by the Knowledge Translation Working Group.

The KT WG will review the final appraisals and prepare them for dissemination through posting in a designated section of the CTFPHC website.

 

Step 12

Maintaining currency of the critical appraisals.

Most Responsible Person(s)
PGD, Critical Appraisal WG

  • Most guidelines are updated either ad hoc or on regular update schedules. This is determined by the originating CPG development organization.
  • Guidelines that have been critically appraised and posted on the CTFPHC site will be reviewed annually to determine if any updates have been published. Guideline developers will also be asked to notify the CTFPHC of updates.
  • Depending on the extent of the update, a guideline may need to undergo a new appraisal process to reflect the most recent CPG.