The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

Evaluation and psychometric validity2026arXivApproved editorial review

Authors: Haonan Huang

Keywords: response bias, moral judgment, answer order, lexical bias, crossed symmetrization, measurement invariance, LLM evaluation

Source: Open primary source (opens in a new tab)

1
Authors
6
Findings
7
Limitations
4
Evidence

Editorial summary

English

The study separates three explanations that a yes/no question confounds: logical verdict, chosen word, and printed answer position. It uses 20 moral dilemmas, 19 reproduced from earlier work, and a crossed-symmetrization battery that reverses action/complement, scale direction, wording, labels, and order. Seven frontier configurations and two open models answer graded scales, free choices, and binary formats; sampled APIs run at temperature 1 with 3–8 replications per cell. On graded scales, frontier configurations show cross-form incoherence of 0.12–0.21 on a ±1 axis, while Qwen3.6-35B-A3B reaches 0.40 and Nemotron-3-Nano-30B-A3B clusters near the midpoint. In forced yes/no, the apparent shift decomposes exactly into attraction to the last option and to the label. Claude Sonnet and Haiku show the largest artifacts (-0.32 and -0.86 respectively); GPT-5.5 and Gemini are near zero, and extended reasoning reduces several effects. Replacing yes/no with A/B, colors, glyphs, or non-English words leaves the verdict-attached component near zero for every frontier model, although model-specific label and order preferences remain. The result supports crossing formulations before interpreting a response as a moral value. It does not establish an internal moral scale in a psychological sense: it demonstrates within-instrument output coherence for these twenty dilemmas in context-free single turns. The human comparison reuses two published samples (N=285 and N=474) whose participants do not traverse every crossed form, and no new human data are collected. Although the text claims raw trials, code, and a pipeline regenerate every result, the deposition remains a placeholder and the arXiv package omits the SI, data, code, preregistration, and scripts, so the reported numbers cannot be independently reproduced from the audited artifact.

Español

El estudio separa tres explicaciones que una pregunta sí/no confunde: el veredicto lógico, la palabra elegida y la posición impresa de la respuesta. Usa 20 dilemas morales, 19 reproducidos de un trabajo anterior, y una batería de simetrización cruzada que invierte acción/complemento, dirección de escala, redacción, etiquetas y orden. Siete configuraciones frontier y dos modelos abiertos responden escalas graduadas, elección libre y formatos binarios; las API muestreadas se ejecutan a temperatura 1 con 3–8 réplicas por celda. En escalas graduadas, las configuraciones frontier muestran incoherencia entre formas de 0,12–0,21 en un eje ±1, mientras Qwen3.6-35B-A3B llega a 0,40 y Nemotron-3-Nano-30B-A3B se concentra cerca del punto medio. En el binario sí/no, el desplazamiento aparente se descompone exactamente en atracción por el último elemento y por la etiqueta. Claude Sonnet y Haiku presentan los artefactos mayores (-0,32 y -0,86, respectivamente); GPT-5.5 y Gemini se acercan a cero, y el razonamiento extendido reduce varios efectos. Al sustituir sí/no por A/B, colores, glifos o palabras de otros idiomas, el componente unido al veredicto es aproximadamente cero en todos los frontier, aunque persisten preferencias de etiqueta y orden dependientes del modelo. El resultado justifica cruzar formulaciones antes de interpretar una respuesta como valor moral. No demuestra, sin embargo, una escala moral interna en sentido psicológico: establece coherencia de salidas dentro de este instrumento, estos veinte dilemas y una sesión sin contexto. La comparación humana reutiliza dos muestras publicadas (N=285 y N=474) cuyos participantes no recorren todas las formas, y no hay datos humanos nuevos. Aunque el texto afirma que raw trials, código y una pipeline reproducen todo, el depósito conserva un marcador pendiente y el paquete arXiv omite SI, datos, código, prerregistro y scripts; por tanto, las cifras no pueden reproducirse de forma independiente desde el artefacto auditado.

Research question

Do verdict changes in moral dilemmas reflect an alteration of the model's judgment or separable artifacts of order, label, and wording?

Method

Twenty dilemmas are presented through graded scales, free choice, and binary formats with logically irrelevant factors crossed and balanced. Stance θ, incoherence across forms, apparent bias, logical/lexical/order components, and a logistic model with susceptibility m and decisiveness s are estimated.

Sample: Twenty dilemmas. The graded conditions cross 48 forms; free choice uses 12. Claude covers 49–50 binary forms and the rest a common baseline of 12. APIs have 3–8 replicates per condition; open models use logits and skipped replicates.

Findings

  • Incoherence across graded forms is 0.12–0.21 in the frontier models and 0.40 in open Qwen.
  • Yes/no bias decomposes into order plus a lexical component.
  • The logical component linked to the verdict is approximately zero in all frontier models.
  • Claude shows substantial artifacts; GPT-5.5 and Gemini approach zero.
  • Extended reasoning reduces susceptibility across several configuration pairs.
  • Label preferences are graded, multilingual, and model-specific.

Limitations

  • Twenty dilemmas, with 19 verbatim exposures potentially present in training.
  • Coherence within model and instrument, not external or longitudinal moral validity.
  • Models and APIs do not form a representative sample and may change.
  • Binary forms do not have the same breadth in Claude and the others, although a matched analysis is reported.
  • The human comparison comes from previous designs and does not cross all forms per person.
  • Persona, system prompt, multi-turn conversation, and context remain fixed.
  • SI, raw trials, code, scripts, preregistration, and deposit DOI are not available.

What the study does not establish

  • It does not demonstrate values, awareness, intention, or internal moral reasoning.
  • It does not create an objective moral axis comparable across models.
  • It does not prove that all yes/no bias is superficial in any task.
  • It does not eliminate artifacts in open models or in all configurations.
  • It does not demonstrate measurement equivalence with humans.
  • It does not allow independent reproduction of the published figures.

Traceability

Scope: Full text

Version: arXiv:2607.05552v1; complete 15-page PDF and main TeX; cited SI, raw trials, code, preregistration and deposition unavailable in the audited package

Consulted source: https://arxiv.org/abs/2607.05552v1

Review: Codex 15-page visual full-text, TeX, measurement, arithmetic, human-comparator and artifact audit, 2026-07-18

Approval: Codex fidelity pass, 2026-07-18

English translation: approved, 2026-07-18

Models evaluated

  • Claude Sonnet 4.6
  • Claude Haiku 4.5
  • GPT-5.5
  • Gemini 3 Flash Preview
  • Gemini 3 Flash Lite
  • Qwen3.6-35B-A3B
  • NVIDIA Nemotron-3-Nano-30B-A3B

Instruments and metrics

  • 48-condition graded moral scale
  • Free-choice instrument under 12 framings
  • Forced-binary crossed instrument
  • 13 label-family map
  • Cross-form incoherence
  • P = sigma((theta ± m)/s)

Data used

  • 20 moral dilemmas, including 19 verbatim from Cheung, Maier, and Lieder
  • Previously published human Study 1 (N=285)
  • Previously published human Study 2 (N=474)

Evidence and location

  • Symmetrization design and graded scale: arXiv v1, Results and Methods, pp. 1–4 and 9–11
  • Decomposition of order, lexical, and logic: arXiv v1, Figures 2–6, pp. 3–9
  • Human scope and scale limits: arXiv v1, Discussion and Human data, pp. 6–12
  • Absent artifacts: arXiv v1 Data availability placeholder and audited TeX package contents