INTRODUCTION: Improved pneumonia diagnostics are needed in low-resource settings (LRS); lung ultrasound (LUS) is a promising diagnostic technology for pneumonia. The objective was to compare LUS versus chest radiograph (CXR), and among LUS interpreters, to compare expert versus limited training with respect to interrater reliability. METHODS: We conducted a prospective, observational study among children with World Health Organization (WHO) Integrated Management of Childhood Illness (IMCI) chest-indrawing pneumonia at two district hospitals in Mozambique and Pakistan, and assessed LUS and CXR examinations. The primary endpoint was interrater reliability between LUS and CXR interpreters for pneumonia diagnosis among children with WHO IMCI chest-indrawing pneumonia. RESULTS: Interrater reliability was excellent for expert LUS interpreters, but poor to moderate for expert CXR interpreters and onsite LUS interpreters with limited training. CONCLUSIONS: Among children with WHO IMCI chest-indrawing pneumonia, expert interpreters may achieve substantially higher interrater reliability for LUS compared to CXR, and LUS showed potential as a preferred reference standard. For point-of-care LUS to be successfully implemented for the diagnosis and management of pneumonia in LRS, the clinical environment and amount of appropriate user training will need to be understood and addressed.