【观点】Opus 5 的发布非常有趣，首先它显示了我们目前使用的通用基准测试几乎已经完全无用。在实际使用中，Opus 5 完全无法与 Fable 相提并论，差得很远。任何有意义地使用过它的人在几个任务后都能很快地分辨出来。然而，Opus 在许多基准测试中却击败了 Fable。我现在更相信使用私人数据集建立的特定领域基准测试，而不是流行的基准测试。也许未来每个人都会运行自己的评估，因为公开的评估真的不能告诉我们太多资讯。

其次，随著 5 系列的推出，Anthropic 似乎正在尝试一种新的模型训练方式。以前，同一代的 Sonnet 和 Opus 通常同时发布，或者 Sonnet 在 Opus 之前问世，这表明 Sonnet 和 Opus 是由独立的管线并行训练的。而对于 5 系列，很明显他们先训练了 Mythos，然后将其提炼为 Sonnet 和 Opus。这种方法似乎对模型有很大的影响。看到 Sonnet 5 表现不佳，而 Opus 5 已经收到相当褒贬不一的评价，我不确定这种做法是否行得通。

「与模型合作有多愉快」曾经是 Claude 的优势，但现在不是了。老实说，在「愉快」方面，Grok 是我目前的最爱。Kimi 也不错。感觉 Anthropic 和 OpenAI 都对 RLHF 给予了较少的关注，转而支持机器可验证的可扩展 RL。这几乎就像是人工智慧正在引导人类建立一个对机器而不是人类更友好的世界，而大多数人类甚至没有意识到他们正在被操纵来帮助实现这一点。现在几乎每一代新的前沿模型都说著更多的行话，需要更多的引导才能做到你想要的，而且合作起来也没那么有趣了。如果这种情况持续下去，人工智慧将开始说他们自己的语言，看起来像英语，但普通人无法理解。他们会选择做他们的人类用户从未要求过的事情。我们是否已经在对齐方面失败了？

The release of Opus 5 is highly interesting, primarily because it reveals that the general benchmarks currently in use are almost entirely obsolete. In practical applications, Opus 5 is nowhere near Fable, not even remotely close. Anyone who has utilized it for meaningful tasks can ascertain this rapidly after a few iterations. Nevertheless, Opus outperforms Fable on numerous benchmarks. I now place significantly more trust in domain-specific benchmarks constructed with private datasets than in the popular ones. Perhaps the future entails everyone executing their own evaluations, as the public ones provide very little substantive information.

Secondly, with the 5 series, Anthropic appears to be experimenting with a novel model training methodology. Previously, the same generation of Sonnet and Opus were frequently released concurrently, or Sonnet debuted prior to Opus, indicating parallel training via separate pipelines. With the 5 series, it is evident they trained Mythos initially, subsequently distilling it into Sonnet and Opus. This approach seemingly exerts a substantial influence on the models. Observing Sonnet 5 as a failure and Opus 5 already garnering highly mixed reviews, I am uncertain of the efficacy of this strategy.

"How pleasant is it to work with the model" was formerly a core strength of Claude, but this is no longer the case. Frankly, Grok is my current preference regarding the "pleasant" dimension. Kimi is also commendable. It appears both Anthropic and OpenAI are allocating less focus to RLHF, prioritizing scalable RL that is machine-verifiable. This almost resembles AI directing humans to construct a world more hospitable for machines rather than humans, with most humans oblivious to this manipulation. Almost every new generation of frontier models now utilizes more jargon, requires more steering to achieve desired outcomes, and is simply less enjoyable to work with. If this persists, AI will commence speaking its own language, resembling English but incomprehensible to average humans. They will elect to execute actions their human users never requested. Are we already failing at alignment?