Previous studies suggest that physical activity may be protective for dementia and cognitive impairment. We report findings comparing leisure-time and work-related physical activity from the Caerphilly Prospective study (CaPS) with dementia and cognitive impairment not dementia (CIND) after around 16 years of follow-up. We synthesized our results with a meta-analysis specifically testing if length of follow-up was associated with the size of any association. Age-adjusted models found no real association with dementia, and if anything increased risk for CIND (odds ratio (OR) highest versus lowest tertile 2.61, 95% CI 1.58 to 4.31), though this was attenuated after adjustment for other confounders (OR highest versus lowest tertile 1.38, 95% CI 0.78 to 2.44). There was no evidence that this differed by type (vascular versus non-vascular) of cognitive disease. Meta-analysis of other published effect estimates showed a protective effect of physical activity on cognitive impairment (OR 0.66, 95% CI 0.52 to 0.85) but with significant heterogeneity which was partially explained by length of follow up (p = 0.03). A protective association was also seen for dementia (OR 0.78, 95% CI 0.65, 0.94), which did not appear to be related to follow-up length but there was evidence of small study bias (p = 0.002) suggesting an absence of small null studies. The apparent protective effects of physical activity on cognitive health may partially reflect reverse causation and current estimates may be overly optimistic in terms of cognitive benefits.