Abstract Research has demonstrated that pilots contain multiple shifting purposes, not all of which relate to simple policy testing or refinement. Judging the success of policy pilots is therefore complex, requiring more than a simple judgment against declared goals. Marsh and McConnell provide a framework against which policy success can be judged, distinguishing program success from process and political success. We adapt Boven's modification of this framework and apply it to policy pilots, arguing that pilot process, outcomes and longer‐term effects can all be judged in both program and political terms. We test this new framework in a pilot program in the English National Health Service, the Vanguard program, showing how consideration of these different aspects of success sheds light on the program and its aftermath. We consider the implications of the framework for the comprehensive and multifaceted evaluation of policy pilots.