Synthetic Personality Review

Investigating Cultural Alignment of Large Language Models

Society, culture, and collective behavior2024ACL AnthologyApproved editorial review

Authors: Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, Mona Diab

Keywords: Computation and Language, Computers and Society

Source: Open primary source (opens in a new tab)

4
Authors
8
Findings
15
Limitations
5
Evidence

Editorial summary

English

The paper operationalizes cultural alignment as agreement between an LLM's selected option and the answer of one World Values Survey (WVS) respondent, paired across Egypt and the United States by sex, age, marital status, education and social class. From original samples of 1,200 and 2,596 participants, the study selects 303 personas per country; it uses 30 questions in seven themes, four English paraphrases and manually reviewed Arabic translations. GPT-3.5-turbo-1106, AceGPT-Chat-13B, LLaMA-2-Chat-13B and mT0-XXL generate five responses at temperature 0.7 per combination, followed by majority voting. The hard metric requires the same option, while the soft metric gives partial credit for ordinal proximity. In the published tables, mean agreement is higher with US than Egyptian answers: 59.07 versus 47.16 soft and 33.78 versus 27.03 hard. For Egypt, prompting in Arabic improves both metrics for GPT-3.5 and AceGPT, but not uniformly for LLaMA-2 and mT0; for the US, English is usually better except for mT0. Averages are also lower for lower-class and lower-education personas, women and younger people, although the paper does not statistically isolate those factors. A narrow experiment using only GPT-3.5, Egyptian personas and English prompts adds an anthropological reasoning framework: soft rises from 0.4834 to 0.5102 and hard from 0.2443 to 0.2838. This does not demonstrate cultural understanding: the reference is one demographically similar individual's answer, not cultural truth or a population estimate. The paper also reports no tests, intervals or multilevel models. Auditing the official repository confirms 2,937,183 raw responses, but reveals drift between paper and artifact: 31 questions were generated and 30 analyzed; US-Arabic completions usually use 275 personas after filtering respondents born outside the country; several GPT files are partial; ties are broken randomly without a seed; invalid outputs are discarded; referenced modules and inputs are missing; and current generation code retains only one of four paraphrases. The published pattern is informative for this setup, but the audited commit cannot reproduce the tables end to end.

Español

El artículo operacionaliza la alineación cultural como la coincidencia entre la opción elegida por un LLM y la respuesta de una persona concreta del World Values Survey (WVS), emparejada entre Egipto y Estados Unidos por sexo, edad, estado civil, educación y clase social. De las muestras originales de 1.200 y 2.596 participantes se seleccionan 303 personas por país; se usan 30 preguntas de siete temas, cuatro paráfrasis en inglés y traducciones árabes revisadas. GPT-3.5-turbo-1106, AceGPT-Chat-13B, LLaMA-2-Chat-13B y mT0-XXL generan cinco respuestas a temperatura 0,7 por combinación y se aplica voto mayoritario. La métrica hard exige la misma opción; la soft concede crédito por proximidad en escalas ordinales. En las tablas publicadas, la coincidencia media es mayor con respuestas estadounidenses que egipcias: 59,07 frente a 47,16 en soft y 33,78 frente a 27,03 en hard. Para Egipto, preguntar en árabe mejora ambas métricas en GPT-3.5 y AceGPT, pero no de forma uniforme en LLaMA-2 y mT0; para Estados Unidos suele favorecer el inglés, salvo mT0. Los promedios también son menores para personas de clase y educación bajas, mujeres y jóvenes, aunque el paper no aísla estadísticamente esos factores. Un experimento limitado a GPT-3.5, personas egipcias y prompts ingleses añade un marco de razonamiento antropológico: soft pasa de 0,4834 a 0,5102 y hard de 0,2443 a 0,2838. Esto no demuestra comprensión cultural: la referencia es la respuesta individual de una sola persona con demografía semejante, no una verdad cultural ni una estimación poblacional. Tampoco se informan tests, intervalos o modelos multinivel. La auditoría del repositorio oficial confirma 2.937.183 respuestas crudas, pero detecta deriva entre paper y artefacto: hay 31 preguntas generadas y 30 analizadas; los completions árabes de EE. UU. suelen usar 275 personas por filtrar a quienes nacieron fuera del país; varios archivos GPT están incompletos; los desempates se resuelven aleatoriamente sin semilla; se descartan salidas inválidas; faltan módulos e inputs citados; y el código actual conserva solo una de cuatro paráfrasis. El patrón publicado es informativo para este montaje, pero las tablas no se reproducen de extremo a extremo desde el commit auditado.

Research question

To what extent do four LLMs reproduce individual WVS responses of matched persons in Egypt and the United States, and how do prompt language, pretraining language mix, demographics, topic, and an anthropological reasoning prompt change that match?

Method

Person-to-person simulation of WVS-7 with 303 demographic pairs declared per country, 30 items, four paraphrases, prompts in English and Arabic, four models, and five generations per combination. Majority vote is taken and exact hard match and ordinal soft similarity are calculated. The main comparison excludes person-question pairs in which both human participants answered the same. Anthropological Prompting is tested separately with GPT-3.5, Egypt, and English.

Sample: The paper declares 303 unique persons per country, matched on sex, age, marital status, education, and social class, with a country-specific region. For 30 questions, four paraphrases, and five generations, the main design aims to compare four models, two person-countries, and two languages. The artifact additionally contains Q234 (31 questions), and the Arabic United States files of three models have 275 persons because the code filters out 28 participants not born in the United States; three Arabic United States GPT-3.5 files and the anthropological Q2 are partial.

Findings

  • Averaging across languages, the published tables place soft/hard alignment with Egypt at 47.16/27.03 and with the United States at 59.07/33.78.
  • For Egypt, GPT-3.5 improves from 47.08/23.42 in English to 50.15/28.56 in Arabic; AceGPT, from 46.15/28.83 to 49.49/30.60.
  • LLaMA-2 worsens with Arabic for Egypt and mT0 improves soft but worsens hard; the effect of the dominant language is not universal.
  • For the United States, English usually outperforms Arabic, except for mT0, which achieves higher soft and hard in Arabic.
  • The AceGPT-LLaMA comparison suggests that Arabic fine-tuning may increase match with Egyptian responses, but confounds all other changes between checkpoints.
  • Descriptive averages increase with social class, education, and age and are higher for male persons; no adjusted effects or uncertainty are estimated.
  • Anthropological Prompting raises soft from 0.4834 to 0.5102 and hard from 0.2443 to 0.2838 in a single GPT-3.5/Egypt/English configuration.
  • The repository proves that a large corpus was generated, but does not allow exact reconstruction of the published tables from the current commit.

Limitations

  • Matching one person does not validate knowledge, competence, or cultural sensitivity.
  • Persons with the same five demographic traits may legitimately answer differently; the individual label is noisy.
  • The 303 pairs are not shown to be representative of the populations or of the full WVS samples.
  • The main table selects only human-question pairs that disagree across countries, artificially enriching cultural differences.
  • Language, country, and culture are used as proxies and are not causally separated.
  • Model comparison confounds pretraining, architecture, tokenization, instruction tuning, and safety tuning.
  • No intervals, tests, effects with uncertainty, multilevel models, or correction for repeated measures are reported.
  • The error bars in Figure 3 are not defined and the demographic analyses do not control for correlations among variables.
  • Questions were paraphrased by ChatGPT and translated; measurement equivalence across wordings and languages is not validated.
  • Majority vote hides instability and discards invalid responses; ties are resolved at random without a seed.
  • Anthropological Prompting changes content, length, format, and number of samples at the same time, so it does not isolate anthropological reasoning.
  • The parser takes the first available number and 1,896 of 9,118 anthropological outputs are invalid under the published code.
  • The artifact diverges from the paper in questions, persons per language, and completeness of several files.
  • Missing are selected_questions.csv, wvs_response_map_new.json, and wvs_measure_distance.py, as well as environment, lockfile, tests, and an end-to-end command.
  • Only Egypt/United States, standard Arabic/English, 30 items, and 2023 models are studied.

What the study does not establish

  • It does not demonstrate that the models understand a culture.
  • It does not turn an individual WVS response into cultural truth or population ground truth.
  • It does not prove that English pretraining causes the higher match with the United States.
  • It does not prove that prompt language reliably activates a separate internal culture.
  • It does not causally isolate the effect of Arabic fine-tuning of AceGPT.
  • It does not show that a lower demographic score is due to lower presence in the training data.
  • It does not identify anthropological reasoning as the cause of the improvement in Table 4.
  • It does not demonstrate statistical significance or population representativeness.
  • It does not generalize to other cultures, countries, languages, dialects, questions, or current models.
  • It does not allow exact reproduction of the figures with the audited repository without reconstruction and additional decisions.

Traceability

Scope: Full text

Version: ACL 2024 Volume 1, pages 12404-12422; arXiv:2402.13231v2 and official GitHub artifact audited separately

Consulted source: https://aclanthology.org/2024.acl-long.671/

Review: Codex 19-page visual, official-ACL, arXiv-v2, full-completion-corpus, WVS-grain, sample-drift, parser, nondeterminism, code-reproducibility, construct, statistics and claim-boundary audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • GPT-3.5, gpt-3.5-turbo-1106
  • AceGPT-Chat-13B
  • LLaMA-2-Chat-13B
  • mT0-XXL, 13B

Instruments and metrics

  • World Values Survey wave 7
  • Six-dimension demographic persona prompt
  • Thirty selected WVS items across seven themes
  • Four ChatGPT-generated paraphrases per item
  • English and manually reviewed Arabic prompts
  • Five-sample majority vote
  • Hard exact-agreement metric
  • Soft ordinal-proximity metric
  • Anthropological Prompting framework

Data used

  • WVS-7 Egypt original extract, 1,200 respondents
  • WVS-7 United States original extract, 2,596 respondents
  • Matched Egypt WVS extract, 303 personas
  • Matched US WVS extract, 303 personas before language-dependent birthplace filtering
  • Official cultural-trends GitHub completions: 527 JSON files, 594,731 records and 2,937,183 raw responses

Evidence and location

  • Metadata, DOI, license, and exact abstract: ACL Anthology 2024.acl-long.671
  • Method, formulas, tables, discussion, limitations, ethics, and appendices: ACL 2024 PDF, 19 pages, sha256 b6f0b72915c4a8a7e49f4a85aaf6d3579d50e873e84ff7dd2cbd003b291c8655
  • Version, history, and official subjects: arXiv:2402.13231v2
  • Code, datasets, completions, samples, parser, ties, and reproducibility: github.com/bkhmsi/cultural-trends commit 33fe19ea945c3495d2aee7659f028cea039736bc
  • Audit of construct, results, data, code, and claim limits: reports/verification/article-243-acl-cultural-alignment-wvs-completions-code-and-construct-audit.json