There is current interest in the development of new or improved outcome measures for rheumatological diseases. In the early stages of development, attention is usually directed to how well the measure distinguishes between patients and whether different observers attach similar values of the measure to the same patient. An approach, based on variance components, to the assessment of outcome measures is presented. The need to assess different aspects of variation associated with a measure is stressed. The terms 'observer reliability' and 'agreement' are frequently used in the evaluation of measurement instruments, and are often used interchangeably. In this paper, we use the terms to refer to different concepts assessing different aspects of variation. They are likely to correspond well in heterogeneous populations, but not in homogeneous populations where reliability will generally be low but agreement may well be high. Results from a real patient exercise, designed to study a set of tools for assessing myositis outcomes, are used to illustrate the approach that examines both reliability and agreement, and the need to evaluate both is demonstrated. A new measure of agreement, based on the ratio of standard deviations, is presented and inference procedures are discussed. To facilitate the interpretation of the combination of measures of reliability and agreement, a classification system is proposed that provides a summary of the performance of the tools. The approach is demonstrated for discrete ordinal and continuous outcomes. © 2011 Taylor & Francis.