Despite the rising popularity of indicators of women's empowerment in global development programmes, little work has been done on the validity of existing measures of such a complex concept. We present a mixed methods validation of the use of the Relative Autonomy Index for measuring Amartya Sen's notion of agency freedom in rural Nepal. Analysis of think-aloud interviews (n = 7) indicated adequate respondent understanding of questionnaire items, but multiple problems of interpretation including difficulties with the four-point Likert scale, questionnaire item ambiguity and difficulties with translation. Exploratory Factor Analysis of a calibration sample (n = 511) suggested two positively correlated factors (r = 0.64) loading on internally and externally motivated behaviour. Both factors increased with decreasing education and decision-making power on large expenditures and food preparation. Confirmatory Factor Analysis on a validation sample (n = 509) revealed good fit (Root Mean Square Error of Approximation 0.05-0.08, Comparative Fit Index 0.91-0.99). In conclusion, we caution against uncritical use of agency-based quantification of women's empowerment. While qualitative and quantitative analysis revealed overall satisfactory construct and content validity, the positive correlation between external and internal motivations suggests the existence of adaptive preferences. High scores on internally motivated behaviour may reflect internalized oppression rather than agency freedom.