Increased rates of brain atrophy measured from serial magnetic resonance imaging precede symptom onset in Alzheimer's disease and may be useful outcome measures for prodromal clinical trials. Appropriate trial design requires a detailed understanding of the relationships between β-amyloid load and accumulation, and rate of brain change at this stage of the disease. Fifty-two healthy individuals (72.3 ± 6.9 years) from Australian Imaging, Biomarkers and Lifestyle Study of Aging had serial (0, 18 m, 36 m) magnetic resonance imaging, (0, 18 m) Pittsburgh compound B positron emission tomography, and clinical assessments. We calculated rates of whole brain and hippocampal atrophy, ventricular enlargement, amyloid accumulation, and cognitive decline. Over 3 years, rates of whole brain atrophy (p < 0.001), left and right hippocampal atrophy (p = 0.001, p = 0.023), and ventricular expansion (p < 0.001) were associated with baseline β-amyloid load. Whole brain atrophy rates were also independently associated with β-amyloid accumulation over the first 18 months (p = 0.003). Acceleration of left hippocampal atrophy rate was associated with baseline β-amyloid load across the cohort (p < 0.02). We provide evidence that rates of atrophy are associated with both baseline β-amyloid load and accumulation, and that there is presymptomatic, amyloid-mediated acceleration of hippocampal atrophy. Clinical trials using rate of hippocampal atrophy as an outcome measure should not assume linear decline in the presymptomatic phase.