Market oriented factors predict Hugging Face likes and downloads more reliably than solely capability benchmarks

Abstract

Large language models (LLMS) are primarily evaluated using capability benchmarks such as Massive Multitask Language Understanding (MMLU), which measure performance on academic and reasoning tasks. While these benchmarks have become the standard for comparing models, the largely ignore non-capability metrics. Consequently, strong benchmark performance does not necessarily translate into real world adoption, leaving it unclear whether capability benchmarks predict public usage. To address this gap, we construct a Market Index using Principal Component Analysis (PCA) over sever features: price, context length, documentation depth, licensing openness, model age, tag count, and commercial hosting status. These datapoints were collected from Hugging Face Hub API, the Open LLM Leaderboard, and OpenRouter’s pricing API. We compare the predictive power of the Market Index with MMLU-PRO using regression models on Hugging Face downloads and likes. The Market Index outperformed MMLU-Pro in out-of-samples prediction for both downloads and likes, improving R² by 0.06 and 0.098, respectively. A combined model including both the Market Index and MMLU-Pro further increased out-of-sample R² by approximately 0.07 for downloads and 0.11 for likes relative to the MMLU-only model. With the combined models, the Market Index had a positive and statistically significant coefficient (beta = 0.239 for downloads; beta = 0.288 for likes, p<0.001), while MMLU-Pro had a negative and significant coefficient (beta = 0.107 for downloads; beta = -0.087 for likes, p<0.001). These results suggest that, overall, market-oriented factors are more reliable predictors of Hugging Face model likes and downloads than capability benchmarks alone.