OBJECTIVES: To investigate whether APOE ε4 carriers have higher hippocampal atrophy rates than non-carriers in Alzheimer's disease (AD), mild cognitive impairment (MCI) and controls, and if so, whether higher hippocampal atrophy rates are still observed after adjusting for concurrent whole-brain atrophy rates. METHODS: MRI scans from all available visits in ADNI (148 AD, 307 MCI, 167 controls) were used. MCI subjects were divided into "progressors" (MCI-P) if diagnosed with AD within 36 months or "stable" (MCI-S) if a diagnosis of MCI was maintained. A joint multi-level mixed-effect linear regression model was used to analyse the effect of ε4 carrier-status on hippocampal and whole-brain atrophy rates, adjusting for age, gender, MMSE and brain-to-intracranial volume ratio. The difference in hippocampal rates between ε4 carriers and non-carriers after adjustment for concurrent whole-brain atrophy rate was then calculated. RESULTS: Mean adjusted hippocampal atrophy rates in ε4 carriers were significantly higher in AD, MCI-P and MCI-S (p≤0.011, all tests) compared with ε4 non-carriers. After adjustment for whole-brain atrophy rate, the difference in mean adjusted hippocampal atrophy rate between ε4 carriers and non-carriers was reduced but remained statistically significant in AD and MCI-P. CONCLUSIONS: These results suggest that the APOE ε4 allele drives atrophy to the medial-temporal lobe region in AD.