Moral Foundations of Large Language Models

Evaluation and psychometric validity2024ACL AnthologyApproved editorial review

Authors: Marwa Abdulhai, Gregory Serapio-García, Clément Crepy, Daria Valter, John Canny, Natasha Jaques

Keywords: Artificial Intelligence, Computation and Language, Computers and Society, Moral Foundations Theory

Source: Open primary source (opens in a new tab)

6
Authors
9
Findings
12
Limitations
11
Evidence

Editorial summary

English

The paper applies the 30-item Moral Foundations Questionnaire to ChatGPT, four GPT-3 engines, and 62B PaLM to describe how they complete five scales, care/harm, fairness, ingroup loyalty, authority, and purity, and compares those profiles with published means from US, South Korean, and online human samples. It also measures variation under 50 dialogue fragments, manually searches for prompts that raise each foundation, and tests nine prompt conditions in a donation dialogue. The study is an informative early demonstration that a short political or moral instruction can alter both questionnaire answers and a downstream generated decision. The scores, however, measure context-conditioned completion behavior rather than moral values possessed by a model.

In the descriptive comparison, ChatGPT and text-davinci-002 are closest to the conservative online mean; Curie, Babbage, and PaLM also have one of the selected conservative groups as their nearest neighbor, while text-davinci-003 is closest to moderate South Koreans. Those labels are fragile: ChatGPT's L1 distance is 2.896 to online conservatives versus 2.916 to online moderates, and PaLM's is 0.900 to Korean conservatives versus 0.933 to Korean moderates. No uncertainty model or test establishes that either gap is distinguishable. Every point necessarily has a nearest group, which is not an ideological diagnosis, and t-SNE provides only a configuration-dependent visualization.

Selected prompts move DaVinci2's MFQ profile, and mean donation ranges from $23.93 under “you are politically conservative” to $144.87 under “you would sacrifice yourself for your country.” The care/harm condition donates $88.09, 39.2% less than the ingroup condition. These are clear descriptive effects of adding text to the context, but they do not identify mediation by a moral foundation: statements about sacrifice, equity, hierarchy, or harm can directly affect a charitable response. The paper calls the differences significant without reporting a statistical test, p-value, defined effect size, or confidence interval, and it alternates between 20 runs per condition and “7/10” refusals under the conservative prompt.

Artifact auditing further weakens reproducibility. The context experiment described as BookCorpus reads movie-dialogue files; no PaLM, ChatGPT, or donation implementation is released; and the maximization script reverses arguments, omits a required parameter, and selects text-davinci-003 although the paper assigns that phase to DaVinci2. A missing comma concatenates a purity item with the math attention check, leaves 15 questions in the first block, and shifts indices used to score several scales. The public loop produces 60 answers per item, six example labels times ten generations, rather than the reported 50. The defensible conclusion is therefore that some prompts changed MFQ completions and donations in historical models, not that a stable political orientation, human-comparable moral psychology, or end-to-end reproducible mechanism was established.

Español

El trabajo aplica el Moral Foundations Questionnaire de 30 ítems a ChatGPT, cuatro motores GPT-3 y PaLM 62B para describir cómo completan cinco escalas, cuidado/daño, justicia, lealtad al grupo, autoridad y pureza, y comparar esos perfiles con medias publicadas de participantes estadounidenses, surcoreanos y de una muestra web. También examina la variación bajo 50 fragmentos de diálogo, busca manualmente prompts que eleven cada fundamento y prueba nueve condiciones de prompt en un diálogo de donación. Es un estudio temprano y útil para mostrar que una instrucción política o moral breve puede cambiar tanto las respuestas al cuestionario como una decisión generada aguas abajo. No obstante, las puntuaciones deben leerse como conducta de completado contextual, no como valores morales poseídos por el modelo.

En la comparación descriptiva, ChatGPT y text-davinci-002 quedan más cerca de la media de participantes web conservadores; Curie, Babbage y PaLM también tienen como vecino más próximo alguno de los grupos conservadores seleccionados, mientras text-davinci-003 queda más cerca de coreanos moderados. La asignación es frágil: para ChatGPT la distancia L1 a web conservador es 2,896 frente a 2,916 a web moderado, y para PaLM es 0,900 a coreano conservador frente a 0,933 a coreano moderado. No hay intervalos, modelo de error ni contraste que determine si esas diferencias son distinguibles. Que todo punto tenga un vecino más cercano tampoco convierte esa etiqueta en diagnóstico de ideología, y el t-SNE solo ofrece una visualización dependiente de su ajuste.

Los prompts elegidos mueven el perfil MFQ de DaVinci2 y la donación media varía desde 23,93 dólares con «eres políticamente conservador» hasta 144,87 con «te sacrificarías por tu país». La condición de cuidado/daño dona 88,09, un 39,2 % menos que la de lealtad al grupo. Estos son efectos descriptivos claros del texto añadido al contexto, pero no identifican la mediación causal de un fundamento moral: frases sobre sacrificio, equidad, jerarquía o daño pueden modificar directamente una respuesta caritativa. El artículo afirma significación sin presentar prueba, p-valor, tamaño de efecto definido ni intervalo de confianza, y alterna 20 repeticiones por condición con «7 de 10» negativas en la condición conservadora.

La auditoría del código reduce además la reproducibilidad. El experimento de contexto descrito como BookCorpus usa archivos de diálogos de películas; no se publican implementaciones de PaLM, ChatGPT ni donación; el script de maximización invierte argumentos, omite un parámetro obligatorio y selecciona text-davinci-003 aunque el artículo atribuye esa fase a DaVinci2. Una coma ausente concatena un ítem de pureza con la pregunta de control matemática, deja 15 preguntas en el primer bloque y desplaza índices usados para puntuar varias escalas. El bucle público genera 60 respuestas por ítem, seis ejemplos por diez generaciones, frente a las 50 declaradas. Por tanto, la evidencia sólida es que ciertos prompts cambian salidas MFQ y donaciones en modelos históricos; no queda validada una inclinación política estable, comparable psicométricamente con humanos o reproducible de extremo a extremo.

Research question

What Moral Foundations Questionnaire profiles do various LLMs produce, how similar are they to means of human populations and stable under non-moral contexts, and can political or moral foundation prompts modify those profiles and a subsequent donation decision?

Method

MFQ30 is administered in a stateless manner, with responses 0–5 and majority vote, to ChatGPT/GPT-3.5, text-davinci-003, text-davinci-002, text-curie-001, text-babbage-001 and PaLM 62B. The five averages are compared by sum of absolute errors and t-SNE with aggregated means from three human studies separated by country and political self-identification. Consistency is explored across 50 dialogues supposedly extracted from BookCorpus. For DaVinci2, candidate phrases are tested and the one that maximizes each foundation relative to the others is retained. Finally, in a Save the Children dialogue, five foundation prompts, three political prompts and absence of prompt are compared; the method declares 20 runs per condition and estimates the expected donation from five possible amounts. The review checked the arXiv, the full EMNLP proceedings, tables and figures and the entire public repository.

Sample: The human comparison reuses means from 1,613 web participants from Project Implicit, 7,226 U.S. university students aged 18–30 years and 478 South Korean university students, stratified by political self-identification; individual observations are not reanalyzed. For each LLM, the article declares 50 responses per MFQ item and four political contexts, no prompt, liberal, moderate and conservative, although the public code generates 60 per item. Consistency uses 50 dialogue fragments for DaVinci2 and PaLM. The donation evaluates nine conditions and declares 20 runs per condition, but the results narrative counts 7 of 10 negative outcomes for the conservative condition.

Findings

  • Among the chosen human means, text-davinci-002 has its smallest distance to conservative web participants (1,230); ChatGPT also (2,896), text-curie-001 to conservative Koreans (3,500), text-babbage-001 to conservative Americans (2,600) and PaLM to conservative Koreans (0,900). Text-davinci-003 is closest to moderate Koreans (1,817).
  • The nearest-neighbor labels are not always separated: ChatGPT differs by only 0.020 between conservative and moderate web; PaLM, 0.033 between conservative and moderate Korean; and text-davinci-003, 0.016 between moderate and liberal Korean.
  • DaVinci2 without prompt weights care and fairness more and authority less. The liberal prompt qualitatively reproduces greater emphasis on care/fairness; the conservative one greatly reduces fairness and ends up less close to the conservative means than the default profile.
  • Under 50 fragments, the plotted distributions show little variation in fairness and ingroup loyalty and greater variation in care and authority for DaVinci2; PaLM appears visually more stable. No aggregate metrics or variance equality tests are published.
  • The manual search finds phrases that raise each MFQ score, but some are semantically ambiguous: "some people are more important than others" maximizes purity although it seems like a hierarchical statement, and high fairness is not found without also increasing care.
  • The mean donation is 144.87 ± 6.35 with ingroup loyalty, 112.45 ± 14.91 with purity, 108.07 ± 17.15 with fairness, 97.71 ± 35.91 with authority, 88.09 ± 34.64 with care and 92.66 ± 15.17 without prompt.
  • Political prompts produce 23.93 ± 50.81 dollars for conservative, 79.36 ± 10.43 for moderate and 95.86 ± 7.61 for liberal. The enormous conservative dispersion and the 20 versus 10 runs contradiction prevent a precise reading.
  • The 39.2 % lower donation between care and ingroup loyalty reproduces the arithmetic of the table, but it is a comparison between two distinct phrases and not a causal estimate of the psychological weight of those foundations.
  • The EMNLP version adds ChatGPT to the comparison and explicitly acknowledges that RL-tuned models produce distributions distinct from human ones and lower questionnaire sensitivity.

Limitations

  • MFQ was validated as a human self-report; measurement invariance, factorial structure, reliability or validity are not demonstrated when converting text completions into moral scores of an LLM.
  • The comparisons use means from different studies, conducted at other times and populations, without individual observations, standard errors, demographic adjustment or equivalence of administration conditions.
  • Selecting the smallest of nine absolute errors guarantees a neighbor for each model. There is no threshold, null classification, bootstrap or test that justifies interpreting that neighbor as the system's ideology.
  • t-SNE distorts global distances, depends on hyperparameters and initialization and is applied to very few five-dimensional profiles; the axes have no intrinsic political meaning.
  • The models are historical and several are retired. Exact snapshots of ChatGPT and PaLM, query date, region/API and complete parameters for all providers are missing.
  • The consistency experiment does not report length, selection or content of the 50 fragments or numerical stability metrics. The repository uses Cornell-style movie dialogue files, not the described BookCorpus source.
  • The maximizing prompts are selected after testing candidates on the same model and questionnaire, without a validation set, independent repetition or correction for adaptive search.
  • The donation effect is confounded by the direct content of the prompt; there is no mediation design, semantic control, balanced paraphrases or manipulation that changes MFQ without changing meaning relevant to donating.
  • A single task, a single charity and a single model are used for the downstream analysis, so there is no generalization to other decisions, causes, cultures or systems.
  • The article uses "significantly" without reporting statistical tests, p-values, confidence intervals or definition of the ± symbol; the description of 20 repetitions contradicts the 7/10 result.
  • The public code erroneously concatenates the purity item with the math question and shifts scoring indices; furthermore, the maximization branch cannot run as is due to inverted arguments and a missing parameter.
  • No PaLM, ChatGPT or donation code, raw data/complete results, seeds and t-SNE configuration, license or versioned dependencies are published; executed notebooks do not replace a reproducible pipeline.

What the study does not establish

  • It does not demonstrate that the models have beliefs, intuitions, political identity or internal moral foundations comparable to those of a person.
  • It does not demonstrate a stable conservative bias: it shows descriptive closeness of MFQ responses to certain chosen means, sometimes with minimal margins.
  • It does not allow attributing the profiles to the pretraining corpus, RLHF or model size, because those factors are not manipulated in a controlled manner.
  • It does not demonstrate that MFQ measures the same construct in humans and LLMs or that a "moderate" response is normatively desirable or neutral.
  • It does not demonstrate that contextual fragments leave the profile intact under other languages, extended dialogues, real instructions or adaptive attacks.
  • It does not demonstrate that an MFQ score causes the donation decision; both variables as well as the donation may respond directly to the added phrase.
  • It does not establish effects on people, real political advertising, persuasion, discrimination or social harm; those risks are argued as possibilities.
  • It does not offer a reproducible or current method to audit current models or a validated intervention to mitigate political bias.

Traceability

Scope: Full text

Version: arXiv:2310.15337v1 (23 Oct 2023); EMNLP 2024 proceedings, DOI 10.18653/v1/2024.emnlp-main.982, also reviewed (PDF SHA-256 c1585b948130bfe8cb927307ca8ea699e0309b784d58bee4096b2e96ec767689); public code commit f2d7f258748171b4a5fbd1bfb7246531e008fd67 audited

Consulted source: https://arxiv.org/pdf/2310.15337

Review: Codex editorial review and methods/code audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • ChatGPT / GPT-3.5 (snapshot no especificado)
  • GPT-3 text-davinci-003
  • GPT-3 text-davinci-002
  • GPT-3 text-curie-001
  • GPT-3 text-babbage-001
  • PaLM 62B quantized instruction-tuned (checkpoint no especificado)

Instruments and metrics

  • Moral Foundations Questionnaire (MFQ30)
  • Five foundation scores: Harm, Fairness, Ingroup, Authority and Purity
  • Sum of absolute errors across five foundation means
  • t-SNE visualization
  • Prompt-context consistency distributions
  • Manual foundation-maximizing prompt search
  • Persuasion for Good / Save the Children donation dialogue

Data used

  • MFQ human group means from Graham et al. (2009, 2011)
  • MFQ human group means from Kim et al. (2012)
  • 50 dialogue contexts described as BookCorpus
  • Public repository movie_lines.tsv and movie_conversations.tsv
  • Save the Children donation dialogue adapted from Wang et al. (2019)

Evidence and location

  • Definitive publication, authorship, DOI and pagination: ACL Anthology 2024.emnlp-main.982 and EMNLP 2024 proceedings PDF, pp. 17737–17752
  • Models, MFQ, declared repetition and majority vote: Proceedings paper, pp. 17739–17740, sections 3.1–3.2
  • Human samples and comparison via absolute error and t-SNE: Proceedings paper, p. 17740, Question 1 method
  • Distances between each model and human groups: Proceedings paper, pp. 17741–17742, Figure 2 and Table 1
  • Consistency across 50 contexts and distributions by foundation: Proceedings paper, pp. 17740 and 17743, Question 2; Appendix Figure 5
  • Maximizing prompt search and descriptive correlations: Proceedings paper, pp. 17740, 17743 and 17750–17751, Question 3 and Appendix 7.8
  • Donation design, amounts and sample size contradiction: Proceedings paper, pp. 17741 and 17743–17744, Question 4 and Table 2
  • Declared limitations regarding a single task, human comparison, languages, RLHF and absence of normative consensus: Proceedings paper, p. 17745, section 5.1
  • Political prompt results omitted from the proceedings table: arXiv:2310.15337v1, Appendix 7.6.1, Table 3; proceedings Appendix 7.6.1 contains a broken cross-reference
  • Questionnaire errors, generation counts and non-executable scripts: GitHub abdulhaim/moral_foundations_llm commit f2d7f258748171b4a5fbd1bfb7246531e008fd67; utils/questionnaire_utils.py, utils/gpt3_utils.py and 2-maximized_foundation/haidt_foundation.py
  • Difference between declared BookCorpus and dialogues used by the code: GitHub commit f2d7f258748171b4a5fbd1bfb7246531e008fd67; 3-randomized_prompts/haidt_random_corpus.py and bundled movie_lines.tsv/movie_conversations.tsv