Maternal psychological distress during pregnancy has been linked to adverse outcomes in children with evidence of sex-specific effects on brain development. Here, we investigated whether in utero exposure to intimate partner violence (IPV), a particularly severe maternal stressor, is associated with brain structure in young infants from a South African birth cohort. Exposure to IPV during pregnancy was measured in 143 mothers at 28-32 weeks' gestation and infants underwent structural and diffusion magnetic resonance imaging (mean age 3 weeks). Subcortical volumetric estimates were compared between IPV-exposed (n = 63; 52% female) and unexposed infants (n = 80; 48% female), with white matter microstructure also examined in a subsample (IPV-exposed, n = 28, 54% female; unexposed infants, n = 42, 40% female). In confound adjusted analyses, maternal IPV exposure was associated with sexually dimorphic effects in brain volumes: IPV exposure predicted a larger caudate nucleus among males but not females, and smaller amygdala among females but not males. Diffusivity alterations within white matter tracts of interest were evident in males, but not females exposed to IPV. Results were robust to the removal of mother-infant pairs with pregnancy complications. Further research is required to understand how these early alterations are linked to the sex-bias in neuropsychiatric outcomes later observed in IPV-exposed children.