BACKGROUND: The research community relies heavily on measures of healthcare utilization from household surveys to understand health seeking choices and to evaluate interventions in developing countries. Such measures are known to suffer from recall problems but there is limited evidence of whether the method of data collection affects evaluation findings. We compared the results of a randomized trial of free healthcare using utilization data from two sources. METHODS: Data are from a study in Ghana, in which 2,194 households containing 2,592 children under 5 y old were randomized into a prepayment scheme providing free primary and some referral care, or to a control group whose families paid user fees for healthcare. Data on morbidity and health seeking behaviour were collected using a standard household survey administered at endline and a pictorial diary given to households over a six month period, collected at monthly intervals. RESULTS: Self-reported measures of morbidity and healthcare utilization were substantially lower in the household survey than the pictorial diary when the recall period was over a month. Introducing free healthcare had a positive effect on primary care visits based on the pictorial diary and a non-significant negative effect according to the household survey. Using any clinic visit in the past month as the outcome, the difference in the effect of free care between the two data collection methods was 3.6 percentage points (p = 0.078). CONCLUSIONS: The findings raise methodological concerns about measures of healthcare utilization from household surveys, particularly in the evaluation of health financing interventions.