Study design and randomisation

Potential participants were randomized to receive either “knowledge of” or “no knowledge of” a financial incentive—in this instance gift vouchers (from the online retailer Amazon) to the value of £10. Amazon gift vouchers (distributed via the Amazon email gift certificate facility) were sent to all participants who completed the questionnaire regardless of the study group to which they were randomized.

Random allocation of participants using computer-generated numbers was undertaken independently by a statistician at the Medical Research Council (MRC) General Practice Research Framework.

Administration

On October 13, 2008, both groups were contacted by email (Textbox 1). Participants were told the purpose of the study and invited to complete an online questionnaire via an embedded URL. The online questionnaire was hosted on the SurveyMonkey website [16] and was based on an instrument previously used to assess the practices of intramural MRC Research Units in an earlier phase of the project. The questionnaire comprised a combination of 36 open and closed questions that could be completed in 20 to 30 minutes. The questionnaire was piloted prior to use.

The email sent to the participants in the “knowledge” group stated that those who completed the online questionnaire would receive a £10 Amazon gift voucher. The study design specified that four reminder emails (sent from the same email address as the initial invitation) would be sent out to nonrespondents at one, two, three, and four weeks following the initial invitation; a fifth postal reminder would be sent to nonrespondents if the response rate was considered to be low. Participants who completed the online questionnaire were deemed to have given their consent. Questionnaires not returned by December 31, 2008, were deemed to be nonresponses.

As this RCT was nested within a larger survey, the primary concern was to maximize response rates. Given this, it was determined thatif the difference in the response rate between the two groups was such that it was likely to adversely affect the main aims of the survey, then knowledge of the incentive would be provided to the “no knowledge” group, but not before the third reminder. At the third reminder, we provided “knowledge of” the incentive to the “no knowledge” group to limit any adverse effects on total response to the survey.

A combination of IP address and questionnaire responses were used to identify multiple responses from a single participant [17]. Where multiple responses from a single participant occurred, the most recently completed questionnaire was retained for analysis. Noninvited responses from individuals not part of the study sample were excluded from the analysis.

Comparison with other studies

In terms of overall response, our rate of 50% compares favourably with those reported for other Web surveys. For example, in a review of comparisons of Web survey versus other survey modes, only 6 of 45 Web surveys managed to obtain a response rate higher than 50% [14]. In a second meta-analysis, which reported an 11% difference in response rates in favor of postal over Web modes, only 10 of 39 comparisons obtained a Web survey response rate higher than 50% [15]. Previous randomized evaluations of our choice of incentive (an Amazon gift voucher) [1] have shown mixed effects in different populations and settings. However, of three previous studies similar to ours included in the Cochrane review, researchers compared the effects of: a $5 cash incentive versus a $5 gift voucher [19]; no incentive versus entry into a lottery for $50, $100, $150, or $200 gift vouchers [20]; and unconditional $15 or $25 gift vouchers versus $15 or $25 gift vouchers conditional on completion of the survey [21].

Strengths and limitations of study

In developing our survey, we adhered to recommendations for the design of email questionnaires [18]. These included deriving an appropriate sample, using an embedded URL, using incentives, and sending the request for information from a recognized academic source. One recommendation beyond our control was that the research be perceived to be relevant to the population surveyed. As stated above, there is renewed emphasis on increasing the uptake and transfer of publicly funded research into policy and practice, and those responding indicated that dissemination of the results of research was highly relevant to their work. However, we had no way of knowing beforehand whether the topic or goal would be deemed relevant or of interest by those we surveyed.

In our study, we utilized a 36-item questionnaire and stated that it would take participants up to 30 minutes to complete. Shorter postal questionnaires are associated with increased response rates [1]. It may be that the perceived return (£10) for time invested in completing the 36 items was deemed inadequate compensation by some participants, especially if considered in relation to their incomes as professional researchers. We do not know whether an increase in the financial incentive relative to participant income would have made any difference in this instance. Another consideration relates to the nature of the incentive offered. Receipt of the gift voucher was dependent on the participant completing the questionnaire. There is evidence that response rates can be higher when an incentive is given up front unconditionally rather than given conditional on completion [1]. The use of unconditional versus conditional incentives merits further investigation.

In this study, members of the population of interest have high levels of Internet and email access. Yet, around a fifth of all returned completed questionnaires were paper copies that had been mailed out as part of the postal reminder. This decision to adopt a mixed mode approach in the event of a low response rate appears sensible in light of feedback from two of the respondents. They indicated that they found it hard to find the time to respond to Web surveys, and as they were often out of the office, it was easier to complete a survey that used a paper-and-pen format. Although we recognize that our experience relates to a very specific population and suggest some caution in generalizing these findings to other populations, designers of future Web surveys may wish to consider using this mixed mode approach.

This randomized study was undertaken as part of a wider survey to assess what steps public health and health services researchers working across the United Kingdom are taking to disseminate the findings of their research. This nesting approach offered a cheap and efficient method of adding to our knowledge of the utility of different survey modes. However, undertaking such an approach was not without potential challenges. Normally in randomized studies, one would compare an intervention against standard practice when the outcome is unknown. But in this instance our primary concern was to maximize response rates to the wider survey. In doing so it was possible we limited the duration of the intervention making it difficult to determine what the true effect of the incentive would have been over a longer time period. Future web surveys should consider nesting a randomized element to further test the utility of incentives but should also consider whether the time frame for response is adequate to determine the true effect.

This research was funded by MRC Population Health Sciences Research Network.