OBJECTIVE: To evaluate the pilot implementation of the World Health Organization Performance Assessment Tool for Quality Improvement in hospitals (PATH). DESIGN: Semi-structured interviews with regional/country coordinators and Internet-based survey distributed to hospital coordinators. SETTING: A total of 37 hospitals in six regions/countries (Belgium, Ontario (Canada), Denmark, France, Slovakia, KwaZulu Natal (South Africa)). PARTICIPANTS: Six PATH regional/country coordinators and 37 PATH hospital coordinators. INTERVENTION: Implementation of a hospital performance assessment pilot project. OUTCOME MEASURE: Experience of regional/country coordinators (structured interviews) and experience of hospital coordinators (survey) with the pilot implementation. RESULTS: The main achievement has been the collection and analysis of data on a set of indicators for comprehensive performance assessment in hospitals in regions and countries with different cultures and resource availability. Both regional/country coordinators and hospital coordinators required seed funding and technical support during data collection for implementation. Based on the user evaluation, we identified the following research and development tasks: further standardization and improved validity of indicators, increased use of routine data, more timely feedback with a stronger focus on international benchmarking and further support on interpretation of results. CONCLUSIONS: Key to successful implementation was the embedding of PATH in existing performance measurement initiatives while acknowledging the core objective of the project as a self-improvement tool. The pilot test raised a number of organizational and methodological challenges in the design and implementation of international research on hospital performance assessment. Moreover, the process of evaluating PATH resulted in interesting learning points for other existing and newly emerging quality indicator projects.