Chapter 5: Development of Recommendations

After reviewing the GRADE tables and draft evidence review prepared by the Evidence Review and Synthesis Centre (ERSC), the Working Group Chair and the Science Team prepare an initial set of draft recommendations using the Evidence-to-Decision framework (1). The working group will discuss and revise the draft recommendations, then approve them by consensus. Following working group approval, the Task Force will meet to deliberate recommendations and propose revisions if necessary. The Task Force will then approve the final version of the recommendations by consensus.

As discussed in Chapter 3, the evidence review questions are designed to inform the guideline recommendations. The process for reviewing evidence and developing recommendations is based on GRADE guidance (2,3). The recommendations should specify the target population and the intervention, and, when possible, phrasing should be consistent throughout related documents and with other Task Force guidance.

The Task Force will always make recommendations on the intervention(s), even when evidence is uncertain. This follows best practices from experts in guideline development, as clinicians prefer having recommendations in the face of uncertainty and guideline panels are well positioned to provide such guidance (4).

5.1 Application of the GRADE Approach to Formulate Recommendations

When developing recommendations, the working group must first, in accordance with the GRADE approach, formulate the key question(s) and agree on the PICOTS elements including the critical and important outcomes that are required for decision-making (see Chapter 3). In addition, the working group must agree on the evidence to be included and the assessment of its certainty. They will also review and discuss the evidence review findings, including GRADE tables, at an in-person or virtual meeting with the full Task Force. This evidence base will support development of recommendations on the intervention(s).

To facilitate a systematic and transparent approach to moving from evidence to recommendations, the GRADE Working Group Evidence-to-Decision (EtD) Framework will be used(1,5). This framework asks questions addressing the important determinants for formulating health recommendations, including:

Is the problem a priority?
How substantial are the desirable and undesirable anticipated effects?
What is the overall certainty of the evidence of effects?
Is there important uncertainty about or variability in how much people value the main outcomes?
Does the balance between the desirable and undesirable effects favour the intervention or the comparison?
How large are the resource requirements?
What is the certainty of the evidence of resource requirements?
Does the cost effectiveness of the intervention favour the intervention or the comparator?
What would be the impact on health equity?
Is the intervention acceptable to key stakeholders?
Is the intervention feasible to implement?

The evidence review which addresses each of these questions will be summarized in an EtD table and should clearly describe the evidence that was assessed for all key questions. Note that questions included in the EtD framework may vary depending on the type of key question (e.g., EtD sections could differ from the standard template for a guideline including a question on diagnostic test accuracy). Specific populations which may require unique recommendations based on the available evidence should also be described. The table can provide linkages to additional information such as the summary of findings table or evidence reviews.

As the members of the working group advance through the framework, they make judgements about the answers to these questions. Use of the completed EtD framework facilitates the working group’s discussion about the direction and strength of the recommendation(s) by ensuring that all key determinants are considered. To ensure transparency when formulating recommendations, the working group considers and describes the balance between desirable and undesirable effects of the intervention, as well as the basis for any judgements. Rationale for decisions about the direction and strength of the recommendation should be reported clearly and transparently (2,6).

5.2 Strength of Recommendations

The strength of a recommendation in a guideline may be either strong or conditional (2). The strength of a recommendation may be defined as “the extent to which one can be confident that the desirable consequences of an intervention outweigh its undesirable consequences” (6).

5.2.1 Strong Recommendation

The working group may propose a strong recommendation in favour of an intervention if they determine that its desirable effects clearly outweigh undesirable effects, while a strong recommendation against an intervention is made when the undesirable effects outweigh the desirable effects. The overall certainty of the evidence available on outcomes will also inform the strength of the recommendation; higher certainty evidence provides greater confidence in the balance of effects. Finally, the strength of a recommendation is also influenced by considerations such as availability, feasibility, acceptability, equity, cost and resource use. For example, a less costly, effective intervention is more likely to warrant a strong recommendation for an intervention. Similarly, situations with confidence in greater harms or resource use, but uncertainty of benefits, are more likely to warrant a strong recommendation against an intervention (6).

In effect, a strong recommendation is often made when “all or almost all informed people would make the recommended choice for or against an intervention” (6). From a policy-maker’s perspective, a strong recommendation can be adopted as policy in most settings or situations.

5.2.2 Conditional Recommendation*

The working group may propose a conditional recommendation when the desirable effects of the intervention probably outweigh the undesirable effects or vice versa. When the magnitude of benefits versus harms are similar, or there is uncertainty in those effects, or uncertainty or variability about their value, a conditional recommendation may be warranted (2,6). The working group also takes considerations such as availability, feasibility, acceptability, equity, cost and resource use into account.

A conditional recommendation implies that not all people will be best served by the recommended course of action. With a conditional recommendation, shared decision-making can be utilized to ensure an individual’s values and preferences are considered in healthcare decisions as well as evidence (7). This may mean that primary care providers should allocate more time to shared decision-making in the context of a conditional recommendation. From a policy-maker’s perspective, a conditional recommendation may require further discussion before being implemented in a particular setting or situation.

*Because of concern that the term ‘weak recommendation’ may be confused with weak evidence, the Task Force decided to replace it with the term ‘conditional recommendation’ (9).

5.3 Determinants of the Direction and Strength of a Recommendation

The working group uses the EtD table to make judgements about the GRADE determinants of the strength and direction of a recommendation.

5.3.1 Balance between Desirable and Undesirable Effects and Consideration of Patient Values and Preferences

For this determinant, the working group considers the balance between the desirable and undesirable outcomes) of the intervention using best available estimates and considers the importance of the outcomes to decision-making. Also considered are the values and preferences that patients apply to these outcomes; this information may be obtained from surveys or focus groups of patients, or evidence reviews (6).

The magnitude of the effects is judged as trivial, small, moderate or large. The larger the difference between the desirable and undesirable effects of the intervention in comparison to the comparator, the more likely the working group will suggest making a strong recommendation. On the other hand, if the difference in effects is very small, a conditional recommendation may be warranted. Information on patient values and preferences helps inform understanding of the magnitude of effect.

5.3.2 Uncertainty and Variability in Values and Outcomes

Little or no information may be available in the literature around patient values and preferences for a particular outcome leading to uncertainty in this area. To reduce this uncertainty, the working group may work with the KT team to conduct patient/public engagement activities such as focus groups to gather more information.

In situations where patient values and preferences are uncertain, or where considerable variability is demonstrated to exist in how patients value the benefits and harms of an intervention, a conditional recommendation, typically emphasizing shared decision-making, on the intervention is more likely.

5.3.3 Confidence in Estimates of Effect (Certainty of Evidence)

The working group assesses certainty of evidence for each critical or important outcome using GRADE methods. It is frequently the case that the certainty of the evidence differs across those outcomes, but an overall certainty must be determined across all outcomes. The working group applies the following rules to deal with such differences:

Considers only those outcomes that have been deemed critical or important for decision-making.
If the certainty of evidence is the same for all critical or important outcomes supporting a recommendation, then this becomes the overall certainty of the evidence.
If the certainty of evidence differs across all or some critical outcomes, then the lowest certainty of evidence for a critical outcome supporting that particular recommendation will determine the overall certainty of the evidence for that recommendation. There is an exception to this rule: if a critical outcome is of lower certainty than other critical outcomes, and the range of plausible effects of that outcome would not change the direction or strength of the recommendation based on the other critical outcomes, then the overall certainty of the recommendation can be based on the other outcomes (8).

Given that different outcomes are examined to support different recommendations, the certainty of evidence may be different for each recommendation included in a single guideline document, even when they relate to the same topic. Similarly, certainty of evidence may vary for the same outcomes being assessed for different populations in a guideline.

In most situations, a strong recommendation is associated with high, or at least moderate, confidence in estimates of the effect of an intervention (6).

5.3.3.1 Strong Recommendations when Evidence is Uncertain

In the GRADE approach, a strong recommendation may be warranted despite low or very low certainty in effect estimates in the following situations (6):

When low certainty evidence suggests benefit in a life-threatening situation (evidence of harms can be low or high)
When low certainty evidence suggests benefit and high certainty evidence suggests harm or a very high cost
When low certainty evidence suggests equivalence of two alternatives, but high certainty evidence of less harm for one of the competing alternatives
When high certainty evidence suggests equivalence of two alternatives and low- certainty evidence suggests harm in one alternative
When high certainty evidence suggests modest benefits and low/very low- certainty evidence suggests possibility of catastrophic harm

Being mindful of resource constraints of our health care system, the Task Force may make a strong recommendation against the implementation of a prevention intervention in primary care practice when there is no credible evidence of benefit (9). The Task Force places high value on utilizing scarce primary care resources to deliver services of known or likely benefit.

5.3.4 Incorporating Feasibility, Acceptability, Equity, Cost and Resource Use into Recommendations

As part of the EtD framework, the working group considers feasibility of implementing interventions, acceptability to key stakeholders, potential impact on health equity, as well as cost-effectiveness and resource requirements (e.g., primary care provider time) of interventions. The working group may incorporate these considerations into recommendations if they are deemed important or critical for decision-making.

In some situations, resource use might not be considered in depth where no reliable information on outcomes of the intervention is available, or the desirable effects so greatly outweigh undesirable effects (or vice versa) that resource considerations would not change the final judgement (6). In a situation where the working group decides not to explicitly consider costs and resource use when developing a recommendation, for example because the intervention is not effective, this decision and its rationale should be made clear in the guideline document.

5.4 Task Force Vote on Draft Recommendations

Once recommendations have been drafted and approved by the working group (by consensus), they are brought to the full Task Force for discussion and approval. During a meeting of the Task Force, the Working Group Chair presents the EtD Table and the judgements for the draft recommendations. Members of the Task Force discuss and may propose changes to the judgements and wording of the recommendations. The Task Force then approves the recommendations by consensus, as per methods described in section 1.7. Note that if the ERSC makes significant changes to the evidence review (e.g., based on new evidence identified by the pre-publication search update), the working group and Task Force will revisit draft recommendations.

5.5 Draft Guideline

Following discussion and voting during a Task Force meeting, the Science Team and Working Group Chair revise recommendations as indicated. The Science Team drafts the guideline manuscript, including key messages to patients and/or the public. The Working Group Chair reviews and approves the guideline, then the working group reviews and approves. After the working group provides full approval, the Science Team shares the guideline manuscript with all members for the Task Force for their approval to send for external stakeholder and peer review. See Chapter 6 for more detail on external review processes.

5.6 Approval of Final Guideline

The Science Team compiles comments from peer reviewers and stakeholders and shares them with the working group in a disposition table. Members of the working group determine whether any changes to the recommendations or guideline manuscript are required. Typically changes to the strength or direction of recommendations are not made; however, clarification of wording may be needed. If substantial revisions are required or if the recommendations are controversial, the entire Task Force may be asked to review and discuss the comments, at the discretion of the Working Group Chair and/or the Task Force Chair. The working group reviews and approves the revised draft guideline and disposition table. Following the working group’s full approval, the Task Force reviews and provides full approval of the guideline and disposition table for submission to a peer-reviewed journal.

5.7 Release of Guideline

Task Force guidelines are published in peer-reviewed journals. The EtD framework and analytical frameworks are provided as appendices. The final guideline is submitted to a peer reviewed journal within six months of approval by the Task Force. An agreement has been reached with the Canadian Medical Association Journal (CMAJ) giving the journal right of first refusal to publish recommendation statements.

The Task Force website publishes links to the guideline in accordance with arrangements with the publishing journal. The process of publishing and posting the guidelines is managed by the Task Force Office under the Contribution Agreement with the Public Health Agency of Canada. In addition to the guideline manuscript, the Task Force website also publishes links to the evidence review, excluded studies, reviewer comments and responses, CMAJ author podcasts, clinician summaries and KT tools (discussed further in Chapter 7).

References

Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016 Jun;353:i2016.
Schünemann H, Brożek J, Guyatt G, Oxman A. GRADE handbook [Internet]. 2013 [cited 2021 Oct 25]. Available from: https://gdt.gradepro.org/app/handbook/handbook.html
Journal of Clinical Epidemiology. GRADE Series [Internet]. 2021 [cited 2021 Sep 21]. Available from: https://www.jclinepi.com/content/jce-GRADE-Series
Neumann I, Schünemann HJ. Guideline groups should make recommendations even if the evidence is considered insufficient. C Can Med Assoc J = J l’Association medicale Can. 2020 Jan;192(2):E23–4.
Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016 Jun;353:i2089.
Andrews JC, Schünemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol. 2013 Jul;66(7):726–35.
Grad R, Légaré F, Bell NR, Dickinson JA, Singh H, Moore AE, et al. Shared decision making in preventive health care. Can Fam Physician [Internet]. 2017 Sep 1;63(9):682 LP – 684. Available from: https://www.cfp.ca/content/cfp/63/9/682.full.pdf
Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, et al. GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2013 Feb;66(2):151–7.
Thombs BD, Straus SE, Moore AE. Update on task force terminology and outreach activities: Advancing guideline usability for the Canadian primary care context. Can Fam Physician. 2019 Jan;65(1):12–3.

Appendix

Appendix 1: Key Elements of Task Force Guidelines