The methodological studies reported here were embedded in DYD-RCT, a large trial of an online intervention to help hazardous drinkers reduce their alcohol consumption [9]. The parent study included a pilot phase followed by the main trial. The pilot phase involved an unusually large sample, greater than that required for the main trial. We undertook the present methodological experiment in both phases of the parent trial with one alteration made to the design of the second study (see below). We also explored the effects of incentives on follow-up rates in randomized studies, which have been separately reported and do not influence the findings of the present study.

Potential study participants originally accessed a webpage inviting them to “find out if you are drinking too much,” and were then asked to complete a brief 3-item screening test. If eligible, they were invited to take part and given access to a consent page after an information page. Eligible participants were people drinking potentially unhealthy levels of alcohol who were also willing to consider changing their behaviour. After a password had been created and email details validated, participants completed the EQ-5D, a well known brief health-related quality of life measure, and calculated their past week alcohol consumption based on specific alcohol brands and volumes. This is a complex task requiring time and effort varying with amount and patterns of drinking. Participants subsequently answered two questions on confidence and intentions before arriving at a final questionnaire prior to being told their parent trial group allocation. Without their knowledge, participants had been randomly allocated to one of four different questionnaires (described below) to be completed as this final questionnaire. Participants were thus blinded to the conduct of this study.

All participants thus completed common trial entry and baseline research assessments with the sole difference between the study groups being the secondary outcome measure (ie, the final questionnaire) to which they had been randomly allocated. In both phases, randomization was performed by a computer-generated randomization procedure. Randomization could not be subverted, therefore, by the study team, and allocation was fully concealed. Randomization to a particular secondary outcome measure applied to baseline and both subsequent follow-ups. Participants were thus offered the same secondary outcome questionnaire at all three time points. Randomization was performed separately and independently from randomization to intervention and control conditions in the parent trial [9]. The numbers of participants in the present study slightly exceed those in the parent trial as some participants completed the first randomization to secondary outcome questionnaire and did not complete the subsequent randomization to parent trial study condition.

Participants were sent email requests for follow-up data in the pilot phase after 1 and 3 months (study 1) and in the main trial phase after 3 and 12 months (study 2). Data collected at follow-up consisted of past week alcohol consumption, the EQ-5D, single-item measures of confidence and intention, and the same secondary outcome measure completed at baseline. Up to three reminders were sent at 7-day intervals to nonresponders, with the final reminder containing a request for participants to tell us their past week alcohol consumption only. Ethical approval was obtained from University College London ethics committee.

In both studies the sole outcome evaluated here was the proportion of participants who responded, that is, completed the primary outcome questionnaires within 40 days of the email request. The three alcohol problems measures used in both studies were the Alcohol Use Disorders Identification Test (AUDIT), which is the screening test for hazardous and harmful drinking recommended by the World Health Organization [10]; the Leeds Dependence Questionnaire (LDQ), which assesses severity of alcohol dependence [11]; and the Alcohol Problems Questionnaire (APQ), which assesses problems other than dependence [12]. We used the core 23 items of the APQ. The AUDIT and LDQ both comprise 10 items. These instruments all require the respondent to report whether drinking is responsible for a range of difficulties they may experience. Mental health was assessed with different versions of the same instrument: the full 23 or 34 item CORE-OM (Clinical Outcomes in Routine Evaluation-Outcome Measure) in study 1 and the newer, briefer 10-item CORE-10 in study 2 [13-14]. This instrument makes no reference to alcohol.

The analyses followed an analysis plan that was written before the relevant data were analyzed. The main analyses compared the proportion responding at each time point between those randomized to longer (APQ and CORE-OM) and shorter (AUDIT, LDQ, and CORE-10) questionnaires and between those randomized to questionnaires relevant to alcohol problems (AUDIT, LDQ, and APQ) and questionnaires less relevant to alcohol problems, being concerned with mental health (CORE-OM and CORE-10). Comparisons were expressed as differences in proportions (risk differences) for interpretability and odds ratios for comparability with other literature.

As a sensitivity analysis, we used logistic regression to adjust for the following baseline variables that were previously found to be predictive of attrition: parent trial group allocation (DYD or comparator), age, gender, educational attainment (degree level or not), ethnicity (white British or other), whether an address was given at study entry, health state, baseline weekly alcohol consumption, and intention (scored 1 to 5). Pooled analyses (that were not prespecified) combined all four follow-up assessments and allowed for the correlation between the two follow-up assessments for the same person using generalized estimating equations adjusting for study and occasion [15]. Prespecified subgroup analyses explored, using interaction tests on both scales, whether any effects differed by gender, parent trial group allocation, educational attainment (university or college degree obtained or not), and baseline weekly alcohol consumption. Baseline weekly alcohol consumption was dichotomized so that women drinking under 35 United Kingdom (UK) units of alcohol and men drinking under 50 UK units of alcohol in the past week were classified as lighter drinkers and those drinking above these levels were deemed heavier drinkers with 1 UK unit being equivalent to 8 grams of ethanol.

The unusual decision to randomize to secondary outcome measures was made to minimize attrition, both because we were persuaded by existing high quality evidence of the effects of questionnaire length on response rate and also because attrition was correctly anticipated as a formidable challenge in a trial undertaken completely online. We did not, however, investigate overall assessment burden, which could have been done by making a randomized comparison between the total burden, that is, completion of all secondary outcome measures, which is standard practice, versus one only. This would have required a comparison that assigned a large proportion of participants to a condition expected to be unfavorable to retention in the trial, and, therefore, we chose not to do this. This original study design decision is reemphasised here because of the implications for the interpretation of study findings.

We found that asking participants to answer an additional 23 or 34 questions rather than an additional 10 questions did not influence the likelihood of retention in the study. The unit of analysis in previous postal studies has been the number of pages per questionnaire [2] rather than the number of items per questionnaire, as was used in the present study. In both the previously cited review and in a related systematic review and meta-regression study, Edwards and colleagues identified significant unexplained heterogeneity in the effects of questionnaire length [16]. Effects were greatest when postcards were compared with conventional questionnaires. In six trials comparing one page against either two or three pages, no differences in response rate were observed. Only in the five trials in which a one-page questionnaire was compared against four or more pages did effects on response rate emerge [16].

We are aware of only one previous experimental study in a similar population of drinkers thinking about quitting or reducing their consumption that was not included in previous reviews [17]. It found that a brief alcohol consumption measure yielded a much higher response rate (51%) than did a more detailed and relatively time-consuming measure (22%). Both are commonly used approaches, though the time commitments involved have not been studied.

There are two previous online studies of the effects of questionnaire length on response rate. Both studies found shorter questionnaires to increase response rates by approximately 50% to 100%, which is in line with the mean size of effects observed in postal surveys. Deutskens and colleagues [18] compared a questionnaire taking approximately 15 to 30 minutes to complete with one taking 30 to 45 minutes to complete and found response rates of 24.5% and 17.1% respectively. Marcus and colleagues [19] compared a questionnaire with 91 items taking approximately 10 to 20 minutes to complete with one comprised of 359 items and described as taking 30 to 60 minutes to complete and obtained response rates of 30.8% and 18.6% respectively with the odds of response calculated as 0.51 (95% CI 0.42-0.62). Although the online follow-up context of the present study complicates a direct comparison with the wider literature, it seems very likely that the comparison we made is thus consistent with previous findings in that the difference between the two questionnaire lengths was simply too small to impact upon attrition.

Those participants asked more relevant questions in the form of items addressing alcohol problems rather than mental health were on average 3% less likely to be lost to follow-up. These additional questions followed detailed questions about recent alcohol consumption. These findings suggest that the perceived relevance of research assessments could indeed influence attrition.

Our emphasis here is on perceived relevance in the context of an alcohol rather than a mental health trial, even though the perception itself has not been directly assessed. Some participants undoubtedly did have mental health difficulties and may have seen the mental health instrument as being just as relevant to their situation as an alcohol problems measure had they been offered one. Study findings indicate that it is some unspecified property of this instrument that leads to lower follow-up rates in comparison with the others. We assumed at the outset, however, that across the study population as a whole, the mental health content of the additional questionnaire would be viewed as less relevant than an alcohol problems one, and this assumption formed the basis of the hypothesis and the operationalization of the relevance construct. This remains our interpretation of the characteristic most likely to be responsible for the observed difference, though the possibility must be recognized that other features may also be at work.

The existing literature on relevance is rather less extensive than that available for questionnaire length, though again observed effects are much larger than were observed here (unadjusted OR = 1.14, adjusted OR = 1.11). Relevance has also been operationalized heterogeneously in these previous studies. There were three postal studies included in the review by Edwards and colleagues [2] for which the combined odds of response were approximately doubled when more and less relevant questionnaires were compared (OR = 2.00, 95% CI 1.32-3.04). These studies compared the effects on response rates of questionnaires on (1) skipping classes among undergraduates versus PhD students [20], (2) providing bowling versus restaurant feedback among participants in an amateur bowling competition [21], and (3) “a variety of interesting topics” versus “ a boring topic in-depth” in market research [22] with the latter condition in each study being deemed to be less relevant. The only online relevance experiment of which we are aware found a similar effect size to ours (OR = 1.85, 95% CI 1.52-2.26) when comparing a highly salient questionnaire on the motives and personality of website owners against one on psychological aspects of Internet usage, which was deemed to be of relatively low salience among website owners [19]. As with the questionnaire length study, although comparisons are necessarily indirect, our relevance experiment involves a much less pronounced contrast than any previously studied, including this online experiment. We compared two questionnaires both judged relevant to the needs of the study population at the outset though differing in likely degree of relevance as perceived by study participants, whereas in the postal studies, the relevance experiments have been designed to compare relevant with not relevant.

Our findings are strengthened by the large sample sizes employed, the randomized design, and the absence of any missing data given the nature of the study. The online context of the present study is important, as the Internet is likely to be the vehicle for an increasing number of studies of delivering health care and health promotion in the future, as well as many other types of research. The generalizability of data from this study population of hazardous and harmful drinkers to other populations is unknown.

The original decision to randomize to secondary outcome measures was influenced by the emerging literatures on “assessment reactivity” in the alcohol field [23-24] and on “mere measurement effects” elsewhere [25]. Work in this area suggests that participating in research studies and completing questionnaires can itself influence the target behavior under investigation, which, though relevant to all research designs, could be a particular problem in trials. The present findings on relevance, set in the context of the literature in this area, underscore how little is known about the unintended impacts of our research decisions on participant experience and behavior. The findings also indicate that attrition itself may be a useful proxy measure for unintended adverse effects of research design decisions.

We isolated two aspects of methodological decision making for experimental study here. Qualitative differences in questionnaire content were related to attrition, which suggests the possibilities that the aggregate effects of our methodological decisions may have a large influence not only on attrition but probably also on participant engagement with research in other ways. The absence of an effect of additional questionnaire length on attrition suggests that not all our decisions will do so. This suggestion is coherent with existing online findings of interactions between characteristics affecting response rates in surveys [19].

Further methodological studies of this type are important to pursue specifically in the context of both online and conventional trials and also more broadly, as the lack of prior study of the dynamics of response and attrition in different study designs should be rectified. Surely whether prospective research participants decide to enter studies or not, or stay in them if they do, depends upon what it is that is being asked of them.