Abstract Background Social distancing is an important public health intervention to reduce or interrupt the sustained community transmission of emerging infectious pathogens, such as SARS-CoV-2 during the coronavirus disease 2019 (COVID-19) pandemic. We aimed to explore the impact on the epidemic curve of fewer contacts when individuals reduce the time they spend on selected daily activities. Methods We combined the large-scale empirical data of a social contact survey and a time-use survey to estimate contact matrices by age group (0-15, 16-24, 25-44, 45-64, 65+) and daily activity (work, schooling, transportation, and four leisure activities: social visits, bar/cafe/restaurant visits, park visits, and non-essential shopping). We assumed that reductions in time are proportional to reductions in contacts. The derived matrices were then applied in an age-structured dynamic-transmission model of COVID-19 to explore the effects. Findings The relative reductions in the derived contact matrices were highest when closing schools (in ages 0-14 years), workplaces (15-64 years), and stopping social visits (65+ years). For COVID-19, the closure of workplaces, schools, and stopping social visits had the largest impact on reducing the epidemic curve and delaying its peak, while the predicted impact of fewer contacts in parks, bars/cafes/restaurants, and non-essential shopping were minimal. Interpretation We successfully augmented contact matrices with time-use data to predict the highest impact of social distancing measures from reduced contacts when spending less time at work, school, and on social visits. Although the predicted impact from other leisure activities with potential for close physical contact were minimal, changes in mixing patterns and time-use immediately after re-allowing social activities may pose increased short-term transmission risks, especially in potentially crowded environments indoors. Research in context Evidence before this study We searched PubMed for mathematical models using social contact matrices and time-use data to explore the impact of reduced social contacts as seen from social distancing measures adopted during the coronavirus disease 2019 (COVID-19) pandemic with the search string ((social OR physical) AND distancing) OR (contact* OR (contact matri*)) AND (time-use) AND (model OR models OR modeling OR modelling) from inception to May 06, 2020, with no language restrictions. We found several studies that used time-use data to re-create contact matrices based on time spent in similar locations or to calculate the length of exposure. We identified no study that augmented social contact matrices with time-use data to estimate the impact on transmission dynamics of reducing selected social activities and lifting these restrictions again, as seen during the COVID-19 pandemic. Added value of this study Our study combines the empirical data of two large-scale, representative surveys to derive social contact matrices that enrich the frequency of contacts with the duration of exposure for selected social activities, which allows for more fine-grained mixing patterns and infectious disease modelling. We successfully applied the resulting matrices to estimate reductions in contacts from social distancing measures such as adopted during the COVID-19 pandemic, as well as the effect on the epidemic curve from increased social contacts when lifting such restrictions again. Implications of all the available evidence Social distancing measures are an important public health intervention to limit the close-contact transmission of emerging infectious pathogens by reducing the social mixing of individuals. Our model findings suggest a higher fraction of close-contact transmission occurs at work, schools, and social visits than from visits to parks, bars/cafes/restaurants, and non-essential shopping. The minimal predicted impact is suggestive of lifting the restrictions on certain activities and excluding them from the list of social distancing measures, unless required to maintain sufficient healthcare capacity. However, potential replacement effects of activities and in mixing patterns remain unclear, particularly immediately after re-allowing social activities again.