PURPOSE: To assess the reproducibility of subjective interpretation of computed x-ray tomography for 8 features associated with thyroid eye disease. METHODS: Patients with confirmed thyroid eye disease had 3 distinct orbital computed x-ray tomography sections presented as anonymized montages to 3 masked observers (#1 orbital radiologist, #2 general radiologist, and #3 orbital surgeon). Eight features were graded: superior orbital fissure clarity, degree of orbital fat prolapse through the superior orbital fissure, loss of fat space at the apex, muscle enlargement, increase in orbital fat volume, vascular congestion, superior ophthalmic vein size, and lamina papyracea bowing. Thirty montages were randomly triplicated within the completed image-testing-file. RESULTS: Each observer provided 3296 assessments of montages from 146 patients (68% female). Observer #2 had the highest rate of "indeterminate" gradings (13.3%), while #1 had the lowest (6.7%). For intraobserver agreement, the kappa statistics were "substantial" to "almost perfect" for apical crowding, muscular enlargement, and medial bowing, whereas orbital fat expansion and vascular congestion showed only "slight" to "moderate" agreement. Excluding superior ophthalmic vein size (where indeterminacy was too great for statistical analysis), there was a wide and statistically significant interobserver variation for the other 7 features, with no consistent ranking of observer scores. CONCLUSIONS: Subjective interpretation of computed x-ray tomography images for patients with thyroid eye disease has high variability, particularly for interobserver comparisons. Only the assessment of apical crowding, muscular enlargement, and bowing of the lamina papyracea showed fairly consistent intraobserver gradings. The results suggest that variability in the interpretation of such images might only be improved with the use of objective measures applied to the computed x-ray tomography images.