Global gridded crop models (GGCMs) are the workhorse of assessments of the agricultural impacts of climate change. Yet the changes in crop yields projected by different models in response to the same meteorological forcing can differ substantially. Through an inter-method comparison, we provide a first glimpse into the origins and implications of this divergence - both among GGCMs and between GGCMs and historical observations. We examine yields of rainfed maize, wheat, and soybeans simulated by six GGCMs as part of the Inter-Sectoral Impact Model Intercomparison Project-Fast Track (ISIMIP-FT) exercise, comparing 1981-2004 hindcast yields over the coterminous United States (US) against US Department of Agriculture (USDA) time series for about 1000 counties. Leveraging the empirical climate change impacts literature, we estimate reduced-form econometric models of crop yield responses to temperature and precipitation exposures for both GGCMs and observations. We find that up to 60% of the variance in both simulated and observed yields is attributable to weather variation. A majority of the GGCMs have difficulty reproducing the observed distribution of percentage yield anomalies, and exhibit aggregate responses that show yields to be more weather-sensitive than in the observational record over the predominant range of temperature and precipitation conditions. This disparity is largely attributable to heterogeneity in GGCMs' responses, as opposed to uncertainty in historical weather forcings, and is responsible for widely divergent impacts of climate on future crop yields.