电脑科学家 Moritz Hardt 认为，作为人工智慧评估工具的基准测试（Benchmarks）虽然在过去六十年间推动了机器学习的重大进展——甚至决定了 1980 年代的 AI 寒冬与 2012 年 ImageNet 引发的深度学习革命——但其生态系统本质上已经崩坏。传统机器学习理论要求将测试集严格封存以确保客观性，但实务上研究人员却将测试集公开于网路并反复迭代。这导致模型在单一基准测试上的绝对准确率毫无实质意义，只要微调测试集，分数就会剧烈波动，因此科技公司将高分直接等同于专家级推理能力的宣传其实是误导性的。

尽管绝对分数不可信，机器学习领域过去却能侥幸逃过「可重复性危机」，这主要归功于模型之间的「相对排名」在不同资料集上表现出惊人的稳定性。换言之，在基准测试竞赛中胜出的模型，在真实世界或其他资料集上通常也确实表现较佳。然而，这种偶然的可靠性有一个关键前提：所有受测模型必须使用相同的训练集。进入大型语言模型（LLM）时代后，这个前提不复存在，因为开发者各自使用不公开的资料组合，甚至针对特定基准进行策略性的「应试训练（benchmaxxing）」，导致模型在发布前的旧基准上表现优异，但在发布后的新基准上却大幅落后，彻底瓦解了排名的有效性。

为了化解当前的基准测试危机，Hardt 借镜 1968 年墨西哥城奥运允许所有选手进行「高原训练」的共识，提出在测试前对所有 AI 模型进行统一的基准相关数据微调，借此拉平起跑线，让相对排名重新具备参考价值。他坦言，基准测试的胜利在某种程度上令人遗憾，因为它绕过了对内部机制的深入科学理解；但在缺乏宏大数学理论的情况下，正如 CASP 蛋白质折叠竞赛一样，完善的基准测试长期而言仍能以矛盾却强大的方式推动全新科学知识的诞生。

Computer scientist Moritz Hardt argues that while AI benchmarks have driven machine learning progress for six decades—governing both the 1980s AI winter and the 2012 ImageNet deep learning revolution—the benchmarking ecosystem is fundamentally broken. Although textbook theory dictates locking away test sets using the holdout method, the discipline publicly shared them online and iterated on them continuously. Consequently, absolute accuracy scores are meaningless and fluctuate wildly with minor dataset changes, making tech companies' claims that high benchmark scores equate to expert-level reasoning highly misleading.

Despite invalid absolute numbers, machine learning historically escaped a replication crisis mostly by sheer luck because relative rankings between models remained remarkably stable across diverse datasets. A model that won a benchmark competition reliably performed better in the real world. However, this accidental validity relied on a crucial caveat: all models had to train on the exact same training set. In the era of large language models (LLMs), training data is opaque, and developers strategically assemble data mixes to "benchmax" or study for the test, causing new models to ace existing benchmarks while failing miserably on newly released ones, thus destroying ranking reliability.

To resolve this benchmarking crisis, Hardt draws an analogy to the 1968 Mexico City Olympics, suggesting that instead of banning optimization, the field should standardize "altitude training" for AI. By subjecting all models to a uniform preparation protocol on benchmark-relevant data prior to testing, researchers can level the playing field and restore the validity of relative rankings. While Hardt notes it is somewhat sad that benchmarking bypasses a mechanistic scientific understanding of how models work, he concludes that, much like the CASP protein-folding competition, rigorous benchmarks paradoxically remain a powerful engine for generating new scientific knowledge in the long run.