Should the same evidence lead to the same decision outcomes in different decision-making contexts? In order to improve comprehension of this issue, this study considers how context influences evidence utilization in the development of health policy recommendations. We used an embedded multiple case study design to study how four expert groups formulated policy recommendations for breast, cervical, colorectal, and prostate cancer screening in Ontario, Canada. We interviewed expert group members and analysed meeting agendas/minutes, interim/final reports and other case-related documents. Our analyses revealed varying policy objectives; the use, neglect, or overextended consideration of three key decision support tools; the varying skills/abilities of expert group members in using different decision support tools; the varying impact of effect modifiers, resource constraints and political interests; and the differing development/consideration of context-specific evidence to address uncertainty in the external decision-making context. While more work is needed to determine if these findings are generalizable beyond cancer screening policy, we believe the central challenge for evidence-based policy is not to develop international evidence, but rather to develop more systematic, rigorous, and global methods for identifying, interpreting, and applying evidence in different decision-making contexts. Our analyses suggest that identification of evidence must distinguish between different policy objectives in order to link a broad conceptualization of evidence to appropriate policy questions. Interpretation of evidence must acknowledge the varying nature of evidence for different policy objectives, balancing existing emphasis on evidentiary quality with more sophisticated methods for assessing the generalizability of evidence. The application of evidence must also acknowledge different policy objectives, appropriately employing rule-based grading schemes and agreement-based consensus methods that are sensitive to the nature of the evidence and contexts involved.