We aimed to model the incidence of infection with Mycobacterium tuberculosis among adults using data on infection incidence in children, disease prevalence in adults, and social contact patterns. We conducted a cross-sectional face-to-face survey of adults in 2011, enumerating "close" (shared conversation) and "casual" (shared indoor space) social contacts in 16 Zambian communities and 8 South African communities. We modeled the incidence of M. tuberculosis infection in all age groups using these contact patterns, as well as the observed incidence of M. tuberculosis infection in children and the prevalence of tuberculosis disease in adults. A total of 3,528 adults participated in the study. The reported rates of close and casual contact were 4.9 per adult per day (95% confidence interval: 4.6, 5.2) and 10.4 per adult per day (95% confidence interval: 9.3, 11.6), respectively. Rates of close contact were higher for adults in larger households and rural areas. There was preferential mixing of close contacts within age groups and within sexes. The estimated incidence of M. tuberculosis infection in adults was 1.5-6 times higher (2.5%-10% per year) than that in children. More than 50% of infections in men, women, and children were estimated to be due to contact with adult men. We conclude that estimates of infection incidence based on surveys in children might underestimate incidence in adults. Most infections may be due to contact with adult men. Treatment and control of tuberculosis in men is critical to protecting men, women, and children from tuberculosis.