BACKGROUND: Clinical guidelines for improving the quality of care are a familiar part of clinical practice. Formal consensus methods such as the nominal group technique are often used as part of guideline development, but little is known about factors that affect the statements produced by nominal groups, and on their consistency with the research evidence. METHODS: Cognitive behavioural therapy, behavioural therapy, brief psychodynamic interpersonal therapy, and antidepressants for irritable bowel syndrome, chronic fatigue syndrome, and chronic back pain were selected for study. 16 nominal groups in a factorial design allowed comparison of GP-only with mixed groups of GPs and specialists, provision of a literature review with no provision, and ratings made in the context of realistic or ideal levels of health-care resources. Participants rated appropriateness independently, and again after a facilitated meeting. Audiotapes of four group discussions were analysed. FINDINGS: There was agreement with the research evidence for 51% of 192 scenarios. Agreement was more likely if the group was GP-only, if a literature review was provided, or if the evidence was in accordance with clinicians' beliefs. Assumptions about the level of resources available had no impact. Clinical and social cues had mixed effects, irrespective of the research evidence. Qualitative analysis showed the modifying effect of clinical experience and beliefs about research evidence. INTERPRETATION: Guidelines cannot be based on data alone; judgment is unavoidable. The nominal group technique is a method of eliciting and aggregating judgments in a transparent and structured way. It can provide important information on levels of agreement between experts. However, conclusions can be at odds with the published literature. If they are, reasons need to be explicit.