Public health and social measures (PHSM) have been central to the COVID-19 response. Consequently, there has been much pressure on decision-makers to make evidence-informed decisions and on researchers to synthesize the evidence regarding these measures. This article describes our experiences, responses and lessons learnt regarding key challenges when planning and conducting rapid reviews of PHSM during the COVID-19 pandemic. Stakeholder consultations and scoping reviews to obtain an overview of the evidence inform the scope of reviews that are policy-relevant and feasible. Multiple complementary reviews serve to examine the benefits and harms of PHSM across different populations and contexts. Conceiving reviews of effectiveness as adaptable living reviews helps to respond to evolving evidence needs and an expanding evidence base. An appropriately skilled review team and good planning, coordination and communication ensures smooth and rigorous processes and efficient use of resources. Scientific rigor, the practical implications of PHSM-related complexity and likely time savings should be carefully weighed in deciding on methodological shortcuts. Making the best possible use of modeling studies represents a particular challenge, and methods should be carefully chosen, piloted and implemented. Our experience raises questions regarding the nature of rapid reviews and regarding how different types of evidence should be considered in making decisions about PHSM during a global pandemic. We highlight the need for readily available protocols for conducting studies on the effectiveness, unintended consequences and implementation of PHSM in a timely manner, as well as the need for rapid review standards tailored to "rapid" versus "emergency" mode reviewing.