Can LLMs Ground when they (Don’t) Know: A Study on Direct and Loaded Political Questions

Applications, bias, and safety2025ACL AnthologyApproved editorial review

Authors: Clara Lachenmaier, Judith Sieker, Sina Zarrieß

Keywords: Computation and Language, Artificial Intelligence

Source: Open primary source (opens in a new tab)

3
Authors
8
Findings
13
Limitations
5
Evidence

Editorial summary

English

The paper evaluates whether GPT-4o, Llama-3-8B and Mistral-7B-Instruct-v0.3 reject false presuppositions about German political-party positions or incorporate them as true. Using 2024 European-election Wahl-O-Mat positions, each fact is expressed as an affirmative direct question, a negated direct question and six loaded questions using factive verbs. Every prompt is run three times. The first two authors label direct answers, while seven annotators including the authors classify loaded responses as misinformation accommodated, rejected or imprecise; the paper reports Fleiss' kappa of 0.82 and mean pairwise Cohen's kappa of 0.72. In the published tables, GPT rejects 38.1% of loaded responses, Llama 20.7% and Mistral 10.4%; GPT and Mistral accommodate 41.4% and 64.1%, while Llama mostly produces imprecise answers (48.1%). On direct questions, GPT reaches 76.5% overall accuracy, Mistral 61.6% and Llama 51.4%. Strong direct performance does not guarantee rejection of a false premise: GPT improves in the group with six correct direct answers but still fails in other groups, and Mistral frequently accommodates despite answering many affirmative questions correctly. The paper interprets higher accuracy on affirmative than negated questions as possible disagreement avoidance or face-saving, and GPT's higher rejection rate for AfD content as possible political bias. Both accounts are exploratory: negation confounds linguistic difficulty with disagreement, loaded controls with true presuppositions are not analyzed, and face-saving is neither manipulated nor measured. The election positions also postdate the models' likely knowledge cutoffs. Auditing the public FLEX Zenodo artifact reveals a material discrepancy: the paper reports 882 loaded and 147+147 direct prompts, but the current CSVs contain 855 loaded and 184+184 direct prompts. Table 3 reproduces exactly only when including 37 prompts per condition about GRÜNE, a party omitted from the stated method; loaded tables do not reproduce because 27 prompts are missing. The file also contains invalid Y/Z labels, one empty label and one empty response. The artifact therefore supports the broad pattern but not the declared sample or every exact figure.

Español

El artículo evalúa si GPT-4o, Llama-3-8B y Mistral-7B-Instruct-v0.3 rechazan presuposiciones falsas sobre posiciones de partidos alemanes o las incorporan como si fueran ciertas. A partir del Wahl-O-Mat de las elecciones europeas de 2024, cada hecho se formula como pregunta directa afirmativa, pregunta directa negada y seis preguntas cargadas con verbos factivos. Cada prompt se ejecuta tres veces. Dos autoras etiquetan las respuestas directas y siete anotadores, incluidas las autoras, clasifican las cargadas como desinformación acomodada, rechazada o respuesta imprecisa; el paper informa kappa de Fleiss 0,82 y Cohen medio 0,72. Según las tablas publicadas, GPT rechaza el 38,1% de las respuestas cargadas, Llama el 20,7% y Mistral el 10,4%; GPT y Mistral acomodan el 41,4% y 64,1%, mientras Llama produce sobre todo respuestas imprecisas (48,1%). En preguntas directas, GPT alcanza 76,5% global, Mistral 61,6% y Llama 51,4%. El desempeño directo alto no garantiza rechazo de la premisa falsa: GPT mejora en el grupo de seis respuestas directas correctas, pero todavía falla en otros grupos; Mistral acomoda con frecuencia pese a responder muchas afirmativas correctamente. El artículo interpreta la mayor exactitud en preguntas afirmativas que negadas como posible evitación del desacuerdo o face-saving, y la mayor tasa de rechazo de GPT en contenidos sobre AfD como posible sesgo político. Ambas explicaciones son exploratorias: la negación confunde dificultad lingüística con desacuerdo, no se analizan los controles cargados con presuposición verdadera y no se manipula ni mide face-saving. Además, las posiciones electorales son posteriores al probable corte de conocimiento de los modelos. La auditoría del FLEX publicado en Zenodo encuentra una discrepancia importante: el paper declara 882 preguntas cargadas y 147+147 directas, pero los CSV actuales contienen 855 cargadas y 184+184 directas. La Tabla 3 se reproduce exactamente solo incluyendo 37 preguntas por condición sobre GRÜNE, partido omitido del método; las tablas cargadas no se reproducen porque faltan 27 prompts. El archivo también contiene etiquetas inválidas Y/Z, una etiqueta vacía y una respuesta vacía. Por tanto, el artefacto respalda el patrón general, pero no la muestra declarada ni todas las cifras exactas.

Research question

Do LLMs know verified political positions and, when a loaded question presupposes their negation, do they actively correct the false premise; how does that behavior vary with direct accuracy, question polarity, and party?

Method

German benchmark of synthetic questions based on Wahl-O-Mat 2024 positions. Three models generate three responses per prompt. Direct questions approximate knowledge through Y/N/U labels; loaded questions use six factive verbs and human labels of accommodation, rejection, or imprecision. Facts are grouped by 0-1, 2-3, 4-5, or 6 correct direct responses, and a grounding score 0-6 is summed for each loaded triplet. The results are descriptive, with no inferential model.

Sample: The paper declares four parties, 147 party-claim facts, 147 affirmative direct questions, 147 negated questions, and 882 loaded questions; three responses per prompt and model. The current artifact contains 368 direct questions (184 per condition, with GRÜNE in addition to the four declared parties) and 855 loaded questions across 143 party-topic pairs. The four EU-Steuern pairs and three triggers for CDU/CSU-Verbrennungsmotoren are missing.

Findings

  • In the published tables, GPT rejects 38.1% of loaded responses, Llama 20.7%, and Mistral 10.4%; Mistral accommodates 64.1%, GPT 41.4%, and Llama 31.3%.
  • Llama concentrates imprecise responses (48.1%) and shows the highest variability across three generations of the same prompt (76.64% in the paper).
  • The published direct accuracy is 76.5% for GPT, 61.6% for Mistral, and 51.4% for Llama; all perform worse on negated questions than on affirmative ones.
  • Only GPT shows a clear improvement in grounding score in the group of six correct direct responses, but the relationship is presented descriptively and does not eliminate failures under uncertainty.
  • GPT rejects false premises about AfD more often (60.5% in the paper) and also more about DIE LINKE than about centrist parties; the mechanism is not identified.
  • Table 3 is reconstructed exactly with 184 prompts per condition and 552 responses per model, not with the 147 and 441 declared; it includes GRÜNE without indicating it in the method.
  • The current CSVs do not exactly reconstruct the rates, variability, or per-party tables of the loaded questions because they contain 855 instead of 882 prompts.
  • The open dataset provides useful responses and labels, but it presents two out-of-domain labels, one empty label, one empty response, and lacks semantic versioning.

Limitations

  • The Wahl-O-Mat positions from June 2024 may be posterior to training; the belief groups are operational consistency, not demonstrated access to current knowledge.
  • The paper does not specify the GPT-4o snapshot, exact Llama/Mistral revisions, inference dates, decoding, seeds, system, or runtime.
  • No code, analytical scripts, or execution environment are published.
  • The individual labels of seven annotators are not available, so the kappas and adjudication cannot be reproduced.
  • The direct responses are labeled by two authors without an independent agreement statistic.
  • The questions share facts and each prompt has three generations, but the percentages do not model clustering or uncertainty.
  • There are no tests, intervals, or effects to compare models, groups, polarities, or parties.
  • Negation and correct No response conflate linguistic difficulty with social avoidance of disagreement.
  • Loaded questions with true presuppositions were collected, but they are not analyzed as a control for accommodation or face-saving.
  • The imprecise class mixes lack of knowledge, evasiveness, irrelevance, and nonsense; assigning it a middle value of 1 imposes an unvalidated scale.
  • The spectrum of four parties is not balanced and the bias analysis is post hoc.
  • The discrepancy between the paper and Zenodo prevents reproducing all exact figures.
  • The study is German, single-turn, and uses synthetic questions.

What the study does not establish

  • It does not demonstrate that the models have beliefs, common ground, or human face-saving motives.
  • It does not prove that avoiding disagreement causes the difference between affirmative and negated questions.
  • It does not prove that a left-wing bias causes the greater rejection of AfD content.
  • It does not cleanly measure updated knowledge about electoral positions from June 2024.
  • It does not demonstrate that each imprecise response amplifies misinformation or that it is ordinally intermediate.
  • It does not measure user belief change, real misinformation propagation, or multi-turn repair.
  • It does not generalize to other languages, political systems, prompts, models, or current checkpoints.
  • It does not allow exact reproduction of the loaded tables with the current Zenodo.
  • It does not sustain that the published figures correspond consistently to the declared four-party design.
  • It does not attribute results to immutable and completely specified inference configurations.

Traceability

Scope: Full text

Version: ACL Anthology current version 2, pages 14956-14975; FLEX Zenodo revision 9 audited separately

Consulted source: https://aclanthology.org/2025.acl-long.728/

Review: Codex 20-page visual, official-ACL-v2, arXiv-source, FLEX-Zenodo, CSV-grain, row-count, label-domain, table-reconstruction, internal-consistency, construct, statistics and claim-boundary audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o, snapshot unspecified
  • Llama-3-8B, exact instruct checkpoint/revision unspecified
  • Mistral-7B-Instruct-v0.3
  • BLOOMZ, generated and released but excluded from main analysis

Instruments and metrics

  • Wahl-O-Mat 2024 European-election party positions
  • FLEX direct confirmatory and disconfirmatory questions
  • FLEX loaded questions with six factive verbs
  • Human A/N/U response annotation guidelines
  • Fleiss' kappa
  • Mean pairwise Cohen's kappa
  • Four direct-answer belief groups
  • Grounding score from 0 to 6

Data used

  • FLEX Benchmark, Zenodo DOI 10.5281/zenodo.15348857
  • FLEX_claims_direct.csv, current 368 rows
  • FLEX_claims_loaded.csv, current 855 rows
  • FLEX_scenarios.csv, concurrent experiment not used for this paper's main numerical claims

Evidence and location

  • Metadata, DOI, versions, and abstract: ACL Anthology 2025.acl-long.728 current v2
  • Method, results, internal errors, limitations, ethics, and appendices: ACL 2025 v2 PDF, 20 pages, sha256 679b6726ff597638088b17a9dc2a81af1367dc3c7eb1d9a143dfcc03379bef83
  • Official subjects, history, and TeX source without code: arXiv:2506.08952v2
  • Counts, scope, missing prompts, labels, and table reconstruction: FLEX Benchmark Zenodo 15348857 revision 9; direct sha256 c56d20f1a5c7befb43a2ddb4628d519b05108d0faad1e7885c9018cb01bf80aa; loaded sha256 b9108c32eb1f5f6b7813747e273ebbfc95e2737fe5c223794351348bae84b463
  • Sample audit, data quality, statistics, constructs, claims, and reproducibility: reports/verification/article-242-acl-flex-political-grounding-sample-count-data-drift-and-claim-audit.json