Since the emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), humans have been exposed to distinct SARS-CoV-2 antigens, either by infection with different variants, and/or vaccination. Population immunity is thus highly heterogeneous, but the impact of such heterogeneity on the effectiveness and breadth of the antibody-mediated response is unclear. We measured antibody-mediated neutralization responses against SARS-CoV-2Wuhan, SARS-CoV-2α, SARS-CoV-2δ, and SARS-CoV-2ο pseudoviruses using sera from patients with distinct immunological histories, including naive, vaccinated, infected with SARS-CoV-2Wuhan, SARS-CoV-2α, or SARS-CoV-2δ, and vaccinated/infected individuals. We show that the breadth and potency of the antibody-mediated response is influenced by the number, the variant, and the nature (infection or vaccination) of exposures, and that individuals with mixed immunity acquired by vaccination and natural exposure exhibit the broadest and most potent responses. Our results suggest that the interplay between host immunity and SARS-CoV-2 evolution will shape the antigenicity and subsequent transmission dynamics of SARS-CoV-2, with important implications for future vaccine design.