导读
英文标题：BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories arXiv编号：2611.06911 作者：Fei Lin, Tengchao Zhang, Ziya…
【编辑部】这对你意味着什么
该论文发布了 BEAKER 基准，包含 1,000 个专家标注的问答样本，涵盖 11 种图像和视频任务类型，用于评估多模态大语言模型在自动驾驶化学实验室中的执行导向能力。测试了 28 个模型发现，当前模型能跟随视频中的粗略实验流程，但在将动作落地到可操作区域和可行轨迹、识别精细实验器具状态与安全规范、对中等长度流程进行步骤排序方面仍存在困难。对于从事化学、AI 或实验室自动化研究的留学生、科研人员和工程师，该基准提供了一个可用于模型诊断和改进的公开评测工具。
英文标题：BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories
arXiv编号：2611.06911
作者：Fei Lin, Tengchao Zhang, Ziyang Gong
来源：arXiv
原文链接：https://arxiv.org/abs/2611.06911
中文摘要：
Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.
本文由华人空间 AI 论文助手自动抓取翻译，仅供学术交流。
【编辑部】今日讨论
你在科研或开发中是否遇到过模型能看懂实验大致流程，却无法准确定位操作位置或判断器具状态的情况？
