While there is growing evidence of associations between infections and dementia risk, associations with cognitive impairment and potential structural correlates of cognitive decline remain underexplored. Here we aimed to investigate the presence and nature of any associations between common infections, cognitive decline and neuroimaging parameters. The UK Biobank is a large volunteer cohort (over 500,000 participants recruited aged 40-69) with linkage to primary and secondary care records. Using linear mixed effects models, we compared participants with and without a history of infections for changes in cognitive function during follow-up. Linear regression models were used to investigate the association of infections with hippocampal and white matter hyperintensity (WMH) volume. 16,728 participants (median age 56.0 years [IQR 50.0-61.0]; 51.3% women) had baseline and follow-up cognitive measures. We found no evidence of an association between the presence of infection diagnoses and cognitive decline for mean correct response time (slope difference [infections versus no infections] = 0.40 ms, 95% CI: -0.17-0.96 per year), visual memory (slope difference 0.0004 log errors per year, 95% CI: -0.003-0.004, fluid intelligence (slope difference 0.007, 95% CI: -0.010-0.023) and prospective memory (OR 0.88, 95% CI: 0.68-1.14). No evidence of an association was found between infection site, setting or frequency and cognitive decline except for small associations on the visual memory test. We found no association between infections and hippocampal or WMH volume. Limitations of our study include selection bias, potential practice effects and the relatively young age of our cohort. Our findings do not support a major role for common midlife infections in contributing to cognitive decline for this cohort. Further research is warranted in individuals with more severe infections, for infections occurring later in life.