BACKGROUND: The aims of the current study were three-fold: i) to estimate the reliability and predictive validity of a therapy quality measure for use by peers; ii) to assess the extent to which peer delivery agents could be trained to evaluate their peers' counsellors as reliably as experts; and iii) to identify barriers and facilitators of several implemented models of peer supervision. METHODS: 26 peers (called 'Sakhis' in the study context), with no previous experience or formal training in mental health care delivery, were trained by experts to deliver the Thinking Healthy Program Peer-delivered (THPP) and conduct peer-led supervision. Using the Therapy Quality Scale (TQS)-an 18 item Likert scale (0-2) measuring both general and treatment-specific skills-both peers and experts independently rated 167 individual sessions to estimate: a) the psychometric properties of TQS; and b) the mean difference between peer and expert TQS ratings; these data were analyzed using SAS 9.3. This was complemented with qualitative data (two rounds of in-depth interviews with four experts and focus group discussions with all Sakhis) which were analyzed using framework analysis. RESULTS: We observed good internal consistency on TQS ratings among expert (α = 0.814) and Sakhis (α = 0.843) and good to excellent scores of inter-rater reliability among experts (ICC = 0.779) and Sakhis (ICC = 0.714). TQS ratings were not significantly related to patient depressive symptoms at 6-months post-child birth but were significantly related to patient activation scores (r = 0.375, p < 0.01 for treatment-specific skills and 0.313, p < 0.01 for general skills) at 3-months post-child birth, which in turn were significantly related to depressive symptom scores at 6-months post-child birth (r = -0.455, p < 0.001), highlighting a potential temporal pathway between therapy quality, patient behaviours and patient outcomes. Following additional training and with growing expertise, Sakhis eventually evaluated their peers' counselling sessions as reliably as experts-demonstrating no significant mean differences on general (t = -0.42, p > 0.05) or treatment-specific (t = -1.44, p > 0.05) subscale scores. Qualitative findings were also consistent between experts and Sakhis: barriers included peers' initial difficulties in rating the TQS and leading supervision which declined over time. Most Sakhis and experts reported the benefits of using a structured scale to rate therapy quality which in turn facilitated consistent and relevant feedback and motivation to ultimately improve Sakhis' counselling skills. In addition, most Sakhis and experts found that peer supervision methods were acceptable and feasible, particularly when linked to financial incentives and expert supervisor. CONCLUSION: With time, non-specialist or lay providers can be trained to implement peer supervision and assess therapy quality as reliably as experts using a psychometrically-sound measure. However, peer supervision with experts was more preferred than peer supervision without experts to facilitate structured, reliable feedback. Additional studies are required to address this challenge and test solutions to facilitate the dissemination of non-specialist delivered psychosocial interventions at a global level.