To assess the relationship between carriage of APOE ε4 allele and evolution of white matter hyperintensities (WMHs) volume, we longitudinally studied 339 subjects from the Alzheimer's Disease Neuroimaging Initiative cohort with diagnoses ranging from normal controls to probable Alzheimer's disease (AD). A purpose-built longitudinal automatic method was used to segment WMH using constraints derived from an atlas-based model selection applied to a time-averaged image. Linear mixed models were used to evaluate the differences in rate of change across diagnosis and genetic groups. After adjustment for covariates (age, sex, and total intracranial volume), homozygous APOE ε4ε4 subjects had a significantly higher rate of WMH accumulation (22.5% per year 95% CI [14.4, 31.2] for a standardized population having typical values of covariates) compared with the heterozygous (ε4ε3) subjects (10.0% per year [6.7, 13.4]) and homozygous ε3ε3 (6.6% per year [4.1, 9.3]) subjects. Rates of accumulation increased with diagnostic severity; controls accumulated 5.8% per year 95% CI: [2.2, 9.6] for the standardized population, early mild cognitive impairment 6.6% per year [3.9, 9.4], late mild cognitive impairment 12.5% per year [8.2, 17.0] and AD subjects 14.7% per year [6.0, 24.0]. Following adjustment for APOE status, these differences became nonstatistically significant suggesting that APOE ε4 genotype is the major driver of accumulation of WMH volume rather than diagnosis of AD.