OBJECTIVES: To quantify reader agreement for the British Society of Thoracic Imaging (BSTI) diagnostic and severity classification for COVID-19 on chest radiographs (CXR), in particular agreement for an indeterminate CXR that could instigate CT imaging, from single and paired images. METHODS: Twenty readers (four groups of five individuals)-consultant chest (CCR), general consultant (GCR), and specialist registrar (RSR) radiologists, and infectious diseases clinicians (IDR)-assigned BSTI categories and severity in addition to modified Covid-Radiographic Assessment of Lung Edema Score (Covid-RALES), to 305 CXRs (129 paired; 2 time points) from 176 guideline-defined COVID-19 patients. Percentage agreement with a consensus of two chest radiologists was calculated for (1) categorisation to those needing CT (indeterminate) versus those that did not (classic/probable, non-COVID-19); (2) severity; and (3) severity change on paired CXRs using the two scoring systems. RESULTS: Agreement with consensus for the indeterminate category was low across all groups (28-37%). Agreement for other BSTI categories was highest for classic/probable for the other three reader groups (66-76%) compared to GCR (49%). Agreement for normal was similar across all radiologists (54-61%) but lower for IDR (31%). Agreement for a severe CXR was lower for GCR (65%), compared to the other three reader groups (84-95%). For all groups, agreement for changes across paired CXRs was modest. CONCLUSION: Agreement for the indeterminate BSTI COVID-19 CXR category is low, and generally moderate for the other BSTI categories and for severity change, suggesting that the test, rather than readers, is limited in utility for both deciding disposition and serial monitoring. KEY POINTS: • Across different reader groups, agreement for COVID-19 diagnostic categorisation on CXR varies widely. • Agreement varies to a degree that may render CXR alone ineffective for triage, especially for indeterminate cases. • Agreement for serial CXR change is moderate, limiting utility in guiding management.