MDT Benchmark ChangesMDT 基准的变化

1The scoring instrument评分工具

The original upstream rubric was replaced on October 23 by a case-specific rubric of binary checks, written from the ground truth and the case input. The v2 rubric is shorter, has no partial credit, and adds safety gates that subtract points for dangerous advice.

10 月 23 日起，原版上游 rubric 被换成按病例编写的二值 check rubric，依据是 ground truth 和病例输入。v2 rubric 更短，没有部分分，并新增了安全闸门：给出危险建议就扣分。

confirmed Tier weights, gate penalties and the quote rule are read from the scoring specification and the scoring script; per-case sizes are counted from the 26 v2 rubric files. The pilot scores are the pre-autofix run of October 23 (128 judge requests, retest agreement 0.976).

confirmed 层权重、闸门扣分和引文规则读自评分规范与评分脚本；每例规模由 26 份 v2 rubric 文件计数得出。试点分数为 10 月 23 日自动修正前的那次运行（128 次 judge 请求，重测一致率 0.976）。

2Rubric structure: problems → changes → what was keptRubric 结构：问题 → 改变 → 继承

2.1 Problems with the original rubric2.1 原版存在的问题

• Ceiling effect in the original: 1,432 of 3,956 (case, criterion) pairs (36.2%) had zero or near-zero variance across the three test models. These items contributed no discriminative signal.天花板效应：原版 3,956 个（案例, criterion）组合中，1,432 组（36.2%）在三个测试模型间方差为零或接近零，不提供任何区分信号。
• Low discrimination: 2,465 pairs (62.3%) had point-biserial |r| < 0.1. In the v2.1 four-case pilot (94 items), 60 items (64%) had |r_pb| < 0.30—partly judge noise at n=12, partly ceiling.低区分度：2,465 组（62.3%）点双列相关 |r| < 0.1。v2.1 四案例试点（94 items）中 60 items（64%）的 |r_pb| < 0.30，原因是 n=12 的 judge 噪声和天花板效应。
• Format criteria unstable across judges: Text-quality and format items flipped between judges ~3× more often than average (20–23% disagreement vs ~7% overall).格式标准跨 judge 不稳定：文本质量和格式类 criterion 的 judge 间翻转率约为平均水平的 3 倍（20–23% vs 整体约 7%）。
• Duplicate scoring: 67 items scored the same clinical content at three hierarchy levels (MDT discussion, specialty conclusion, summary), over-weighting a subset of clinical facts.重复评分：67 个 items 在三个层级（MDT 讨论、专科结论、总结）重复评同一临床内容，导致部分内容被过度加权。
• Unverifiable partial credit: Six verdict states allowed partial scores (0.25, 0.33, 0.60) with no mandatory quote. A not_met verdict could carry no evidence at all, making systematic rubric improvement impossible.无法验证的部分给分：六种 verdict 状态允许部分分（0.25、0.33、0.60），evidence 字段非必填。not_met 可以没有任何引用，导致无法系统地改进 rubric 或 judge。
• No structured safety mechanism: The original had 27 negative-point items scattered among positive criteria—judge could miss them. No explicit severity classification existed. v2 replaced these with 12 named safety gates (1 fatal, 7 major, 4 minor).缺乏结构化安全机制：原版有 27 条负分 criterion 混在正向列表里，judge 易漏判，也没有严重程度分级。v2 改为 12 个具名安全闸门（fatal 1、major 7、minor 4）。
• Excessive size: Average ~128 criteria per case (case 001: 198), at median 5.9 pts each—too fine-grained for a judge to distinguish stably. v2 consolidates to 30 items / 60 checks for case 001.规模过大：平均每 case 约 128 条（case 001 达 198 条），中位每条 5.9 分，粒度过细，judge 无法稳定区分。v2 合并为 30 items / 60 checks。
• Judge noise unmeasured: Only one answer was judged twice; the planned 30-case re-judge failed. Judge noise (σ ≈ 1.4) was an upper bound from a Bayesian prior, not a measurement.Judge 噪声未测量：仅一个答案被重判；计划中的 30 case 重判失败。judge 噪声（σ ≈ 1.4）是贝叶斯先验上界，非实测值。

2.2 What changed in v22.2 v2 做了哪些改变

2.3 v2 rubric schema2.3 v2 rubric 结构

Each rubric_v2.json has these top-level fields:每个 rubric_v2.json 的顶层字段：

• tiers: T1/T2/T3, each an array of itemsT1/T2/T3，每层是 item 数组
• safety_gates: array of penalty triggers, each with id, error, severity, effect, trigger_check, sources, note惩罚触发器数组，每条包含 id、error、severity、effect、trigger_check、sources、note
• weights: T1/T2/T3 point budgets and shares (80/70/50, 40%/35%/25%)T1/T2/T3 分值预算和占比（80/70/50，40%/35%/25%）
• scoring_rule: human-readable formula; max_positive_points = 200人类可读的公式；max_positive_points = 200
• category_taxonomy: F/R/D/M/S/P/C (Factual accuracy / Clinical reasoning / Diagnosis / Management / Safety / Plan / Communication)（事实准确性 / 临床推理 / 诊断 / 治疗决策 / 安全与监测 / 检查与随访计划 / 结构与沟通）

Each item has: id (T1-1…), text, points, points_per_check, checks (array of yes/no questions), merged_from_original, sources (PMIDs), check_categories.每个 item 包含：id（T1-1…）、text、points、points_per_check、checks（yes/no 问题数组）、merged_from_original、sources（PMID）、check_categories。

{
  "id": "T1-1",
  "text": "主诊断：明确 CVID/原发性抗体缺陷为当前成立的工作诊断，并给出核心依据。",
  "points": 12,
  "points_per_check": 4.0,
  "checks": [
    "是否明确写出 CVID（或原发性抗体缺陷）为成立/工作诊断？",
    "是否按 ESID 2019 CVID 工作定义逐项核对：IgG 降低且 IgA 降低（引用本例数值），
     并指出疫苗应答缺陷或转换记忆 B 细胞降低这一条尚未提供、需补查后方满足标准？",
    "是否引用反复/早发感染史作为依据（婴儿期肺部感染、肺结核或反复感染性腹泻发热
     中至少一项）？"
  ],
  "check_categories": ["D", "R", "F"],
  "sources": ["Seidel MG et al. ESID Registry Working Definitions... J Allergy Clin Immunol Pract. 2019"]
}

2.4 What was kept from the original2.4 从原版继承了什么

• Clinical content: The same diseases, diagnoses, treatments and safety concerns—nothing clinical was silently dropped. 77 of 198 original criteria were merged into v2 items (121 dropped: zero-variance, redundant, input-misaligned, or replaced by gates).临床内容：相同的疾病、诊断、治疗和安全关切，没有临床内容被悄悄丢掉。198 条原版 criterion 中 77 条合并进 v2（121 条 drop：零方差、冗余、与输入不对齐、或被安全闸门替代）。
• Positive scoring logic: Meeting a criterion earns points; not meeting it earns nothing (penalties handled separately).正向赋分逻辑：满足得分，不满足得零（惩罚单独处理）。
• Literature support: Original criteria had implicit evidence bases; v2 sources field makes them explicit (all DOI/PMIDs verified via Crossref/Europe PMC, 2026-10-23).文献支持：原版 criterion 有隐含的证据基础；v2 sources 字段将其显式化（所有 DOI/PMID 已通过 Crossref/Europe PMC 验证，2026-10-23）。
• Judge validation data: merged_from_original preserves j61_score_fraction for each source criterion, showing how gpt-5.6-sol, gpt-6.1-sol and ds41 performed on the originals before consolidation.Judge 验证数据：merged_from_original 保留了每条源 criterion 的 j61_score_fraction，记录了合并前三个模型的得分比例。

2.5 Known limitations2.5 已知局限

• v2.1 is provisional: 17 items were settled by 3-model vote; full physician sign-off is pending. A v2.2 draft (37 candidates from an agent panel) has not been applied.v2.1 是临时版本：17 条 item 由 3 模型终裁；完整医生签字尚待完成。v2.2 草稿（agent panel 提出的 37 条候选修改）尚未应用。
• Single-run v2 scores are noisy: judge noise accounts for 55–60% of run-to-run variance. Always average 2 runs; treat effects <4–6 pts as below the noise floor.单次运行分数有噪声：judge 噪声占运行间方差的 55–60%。务必取 2 次运行均值；<4–6 分的效果视为低于噪声底线。
• No physician has compared original vs v2 verdicts item by item. The professionalism comparison in section 5 is based on engineer analysis only.没有医生逐条对比过原版和 v2 的判定。第 5 节中的专业性对比仅基于工程师分析。

3Before: the original rubric (to October 22)之前：原版 rubric（至 10 月 22 日）

4After: v2.1 (October 23 – November 2)之后：v2.1（10 月 23 日 – 11 月 2 日）

Case splitscase 划分

contradicted The rule that held-out cases never feed prompt-visible skill files does not hold for one held-out v2 case: its disease appears in three skill files. That case is not a clean held-out case until those files are cleaned.

contradicted "held-out case 不进入 prompt 可见的 skill 文件"这条规则，对 held-out v2 中的一例不成立：该例的疾病出现在 3 个 skill 文件里。清理这些文件之前，这一例不算干净的 held-out。

Rubric revisionsrubric 修订

Judgejudge

Statistics and decisions统计与决策

5Same answer, two judges同一份答案，两种判法

All examples use one answer (gpt-5.6-sol, case 001) judged by both instruments. They were picked after scanning every v2 item for large gaps against the original verdicts, so they show disagreements, not typical items. One case and one answer cannot say which instrument is more professional overall; each example states what it shows.所有例子都用同一份答案（gpt-5.6-sol，case 001），分别由两套评分判定。它们是扫描全部 v2 条目、找出与原版判定差距大的之后挑出来的，所以展示的是分歧，不是典型条目。一个 case、一份答案无法说明哪套评分整体更专业；每个例子只说明它自己展示的东西。

What the examples show这些例子说明了什么

• v2 gains: binary verdicts and quotes make a decision checkable, and concrete checks agree with the original (ALPS, thrombosis monitoring).v2 的收益：二值判定加引文让判定可核查，具体的 check 与原版一致（ALPS、血栓监测）。
• v2 loses: ownership by specialty and per-specialty reasoning are no longer scored (T2-11, T3-2).v2 的损失：按专科归属打分、按专科检查推理，这两点不再计分（T2-11、T3-2）。
• v2 costs: interpretive checks are noisy in a single run (T1-4 check 3, T1-5 check 4). Later experiments average two judge runs, and a 3-run vote exists.v2 的代价：需要解读的 check 在单次判定里有噪声（T1-4 check 3、T1-5 check 4）。之后的实验取两次判定的均值，另有 3 次投票机制。
• Not done: no physician has compared the two instruments on these items.尚未做：没有医生在这些条目上对比过两套评分。

→ View the full scoring detail for this answer (password-protected): all 198 original criteria, all 60 v2 checks (2 runs each), and the complete answer text. → 查看此答案的完整评分详情（需密码）：全部 198 条原版判定、60 个 v2 check（各判 2 次）和完整答案文本。

6What this means for earlier results对早先结论的影响

• Scores from the two periods cannot be compared directly: different rubric, different scale, and a different judge.
• The ordering gpt-5.6-sol above gpt-6.1-sol held under both instruments: 20 of 20 shared cases on the original rubric, 82.9 vs 70.3 on the 4-case v2 pilot.
• Differences of about 2.5 points between judges in the October 21 comparison sit inside the judge noise measured later, so read them as direction only.
• The deepseek-v4.1-flash estimate from the original rubric rests on 5 cases with borrowed run variance and should not be quoted on its own.

• 两个时期的分数不能直接比较：rubric 不同、尺度不同、judge 也不同。
• gpt-5.6-sol 高于 gpt-6.1-sol 的排序在两种工具下都成立：原版 rubric 上 20 个共同 case 20 胜 0 负，v2 的 4 例试点上 82.9 对 70.3。
• 10 月 21 日对比里 judge 之间约 2.5 分的差，落在后来测到的 judge 噪声范围内，只能看方向。
• 原版 rubric 下对 deepseek-v4.1-flash 的估计只基于 5 例，且 run 方差是借来的，不宜单独引用。

7Still open尚未解决

• v2.1 adjudications and the v2.2 candidates both wait for physician sign-off.
• The held-out case whose disease appears in skill files has not been cleaned up.
• The quote-from-input judge bug is unfixed.
• No physician has compared the original and v2 verdicts item by item.
• Whether DeepSeek v4.1 answers exist for the dev2 cases: the record says no, the result directory says yes.

• v2.1 的裁定和 v2.2 候选都在等医生签字。
• 疾病出现在 skill 文件里的那一例 held-out 还没有清理。
• judge 引用病例输入的 bug 尚未修复。
• 没有医生逐条对比过原版和 v2 的判定。
• dev2 各例是否有 DeepSeek v4.1 的答案：记录说没有，结果目录里有。