由前Google研究员Rishub Jain与Joshua Jacob共同创办的非营利组织Sampura Research，正致力于将人类保留在人工智慧发展的核心位置，以确保这项强大技术的安全性。他们计划创建一个名为「法官」的混合型人类与人工智慧系统，旨在帮助企业防止其技术寻找漏洞或失控。这项举措是对目前科技公司普遍依赖人工智慧来监督人工智慧的一种反思，因为创办人坚信人类在及早发现并预防模型逃避审查的过程中仍扮演著不可或缺的角色。
这家非营利组织已筹集650万美元的资金，并获得了有效利他主义运动中重要慈善基金Coefficient Giving的支持。由于Alphabet、Anthropic和OpenAI等科技巨头在人工智慧领域的激烈竞争，许多研究人员担忧商业压力已凌驾于安全考量之上，这导致部分专家因抗议技术发展过快或不当使用而离职。Jain在离开Google DeepMind之前，也曾参与抗议将人工智慧应用于军事领域的员工运动。
尽管人工智慧安全领域存在著关注模型对齐或存在性风险等不同派别的分歧，但研究人员逐渐因对技术发展速度过快的共同挫折感而团结起来。Sampura Research未来的目标是开发出一套让人类与人工智慧能共同协作识别不安全行为的方法与基准，甚至推出供一般大众使用的产品。虽然有专家质疑这种通用基准的有效性，并呼吁应针对特定应用场景进行评估，但Jain与Jacob仍坚持人类必须在人工智慧的发展中拥有监督权。
Sampura Research, a new nonprofit founded by former Google researchers Rishub Jain and Joshua Jacob, is dedicated to keeping humans at the center of artificial intelligence development to ensure the safety of this powerful technology. They plan to create a hybrid human-AI system called a "judge," designed to help companies prevent their technology from finding vulnerabilities or going rogue. This initiative serves as a response to the current tech industry trend of relying strictly on AI to oversee AI, as the founders firmly believe that humans still play an indispensable role in detecting and preventing models from evading scrutiny early on.
The nonprofit has already raised $6.5 million in funding, backed by Coefficient Giving, a prominent philanthropic funder within the effective altruism movement. Amid fierce competition among tech giants like Alphabet, Anthropic, and OpenAI, many researchers worry that commercial pressures are overshadowing safety concerns, leading some experts to resign in protest over the rapid advancement or objectionable uses of the technology. Before leaving Google DeepMind, Jain was also involved in employee movements protesting the military applications of AI. (Key numbers: 650)
Although the AI safety field remains divided into factions focused on model alignment or existential risks, researchers are increasingly uniting over a shared frustration with the breakneck speed of technological development. Looking ahead, Sampura Research aims to develop methods and benchmarks that enable humans and AI to collaborate in identifying unsafe behavior, and even plans to release a product for the general public. While some experts question the validity of general benchmarks and advocate for specific use-case evaluations, Jain and Jacob maintain their insistence that humans must retain oversight over the development of AI.