The onset of the COVID-19 pandemic drove a widespread, often uncoordinated effort by research groups to develop mathematical models of SARS-CoV-2 to study its spread and inform control efforts. The urgent demand for insight at the outset of the pandemic meant early models were typically either simple or repurposed from existing research agendas. Our group predominantly uses agent-based models (ABMs) to study fine-scale intervention scenarios. These high-resolution models are large, complex, require extensive empirical data, and are often more detailed than strictly necessary for answering qualitative questions like “Should we lockdown?” During the early stages of an extraordinary infectious disease crisis, particularly before clear empirical evidence is available, simpler models are more appropriate. As more detailed empirical evidence becomes available, however, and policy decisions become more nuanced and complex, fine-scale approaches like ours become more useful. In this manuscript, we discuss how our group navigated this transition as we modeled the pandemic. The role of modelers often included nearly real-time analysis, and the massive undertaking of adapting our tools quickly. We were often playing catch up with a firehose of evidence, while simultaneously struggling to do both academic research and real-time decision support, under conditions conducive to neither. By reflecting on our experiences of responding to the pandemic and what we learned from these challenges, we can better prepare for future demands.