Aim: To identify a potential efficacy–effectiveness gap and possible explanations (drivers of effectiveness) for differences between results of randomized controlled trials (RCTs) and observational studies investigating glucose-lowering drugs. Methods: A systematic literature review was conducted in English language articles published between 1 January, 2000 and 31 January, 2015 describing either RCTs or observational studies comparing glucagon-like peptide-1 analogs (GLP-1) with insulin or comparing dipeptidyl peptidase- 4 inhibitors (DPP-4i) with sulfonylurea, all with change in glycated hemoglobin (HbA1c) as outcome. Medline, Embase, Current Content, and Biosis were searched. Information on effect estimates, baseline characteristics of the study population, publication year, study duration, and number of patients, and for observational studies, characteristics related to confounding adjustment and selection- and information bias were extracted. Results: From 312 hits, 11 RCTs and 7 observational studies comparing GLP-1 with insulin, and from 474 hits, 16 RCTs and 4 observational studies comparing DPP-4i with sulfonylurea were finally included. No differences were observed in baseline characteristics of the study populations (age, sex, body mass index, time since diagnosis of type 2 diabetes mellitus, and HbA1c) or effect sizes across study designs. Mean effect sizes ranged from −0.43 to 0.91 and from −0.80 to 1.13 in RCTs and observational studies, respectively, comparing GLP-1 with insulin, and from −0.13 to 2.70 and −0.20 to 0.30 in RCTs and observational studies, respectively, comparing DPP-4i and sulfonylurea. Generally, the identified observational studies held potential flaws with regard to confounding adjustment and selection- and information bias. Conclusions: Neither potential drivers of effectiveness nor an efficacy–effectiveness gap were identified. However, the limited number of studies and potential problems with confounding adjustment, selection- and information bias in the observational studies, may have hidden a true efficacy-effectiveness gap.