Is aid helping, hindering, or having no effect on development and health? The answer to this question is highly contested, with proponents on all sides adhering strongly to their competing interpretations. We ask how it is possible for those who are often using the same data to hold such divergent views. Here, we employ an epidemiological perspective and find that, in many cases, the arguments are characterised by methodological weaknesses. There may be selective citation of results and failure to account for bias and confounding, such as where an extraneous factor influencing the outcome is correlated with increased aid or, in confounding by indication, where increased aid is a consequence of a country being in an especially adverse situation. Studies may also lack external validity, whereby lack of data (a widespread problem) or similar considerations mean that analyses are undertaken on an unrepresentative subset of countries. Multiple outcome measures can also be problematic, where the main outcome of interest is not specified in advance. Many studies fail to account for differential time lags between changes in aid and the outcomes being studied. Some studies may also be underpowered to detect an association where one exists. Although, ideally, this debate should be informed by large scale randomised controlled trials, this will often be unfeasible. Given this limitation, it is essential that those engaged in it are cognisant of the many methodological issues that face any observational study.