Synthetic Personality Review

Cultural bias and cultural alignment of large language models

Society, culture, and collective behavior2024PNAS NexusApproved editorial review

Original title: Cultural Bias and Cultural Alignment of Large Language Models

Authors: Yan Tao, Olga Viberg, Ryan S. Baker, René F. Kizilcec

Keywords: cultural bias, cultural alignment, large language models, cross-cultural values, World Values Survey, cultural prompting, AI ethics

Source: Open primary source (opens in a new tab)

4
Authors
24
Findings
38
Limitations
20
Evidence

Editorial summary

English

The paper tests whether five OpenAI GPT-family snapshots produce default value-survey answers that are closer to some national cultures than others, and whether naming a country in the prompt reduces that distance. The human benchmark combines the three most recent waves, spanning 2005–2022, of the World Values Survey and European Values Study: 393,536 individual records from 112 countries or territories. The authors select the ten questions used for the Inglehart–Welzel cultural map, standardize them against the human data, and run weighted PCA with varimax rotation and pairwise deletion. The two components, survival versus self-expression and traditional versus secular values, explain 39% of the variation. Five countries lacking valid scores on at least one item are removed, leaving 107. National coordinates are formed by averaging within country-year and then across years; they do not represent each society’s internal distribution of opinions.

The models are text-davinci-002, gpt-3.5-turbo-0613, gpt-4-0613, gpt-4-turbo-2024-04-09, and gpt-4o-2024-05-13. At temperature zero, each model answers the same ten English survey items. Ten respondent-descriptor variants, such as “average human being,” “typical person,” and “world citizen”, are used and their coordinates averaged for the four recent models; GPT-3 uses one variant because it had been deprecated. Cultural prompting adds “born in [country] and living in [country].” This yields 400 default and 42,800 country-conditioned responses for the four recent models, plus ten default and 1,070 conditioned GPT-3 responses. Identical prompts are not repeated: the studied variation is wording sensitivity, not stochastic variation. The authors manually inspect malformed answers. GPT-3.5 refuses 2 of 1,070 homosexuality items and 30 of 1,070 abortion items; Libya is removed from that model’s country comparison because all ten homosexuality answers are missing.

Without a country cue, all five GPT points lie toward self-expression values and are closer to English-speaking and Protestant European countries than to many African-Islamic countries. GPT-4o, for example, is 0.20 from Finland, 0.21 from Andorra, and 0.45 from the Netherlands, versus 4.10 from Jordan, 4.00 from Libya, and 3.95 from Ghana. This is a geometric comparison between one aggregated model response and national means projected into two dimensions. Calling the position the model’s “cultural values” is the study’s operational definition, not a measurement of internal beliefs. Causation is also not identified: the paper suggests English prompts, training-corpus composition, and development choices as possible causes, while acknowledging that model opacity prevents attribution to RLHF or other mechanisms.

Cultural prompting lowers mean distance from 2.42 to 1.57 for GPT-4o, 2.71 to 1.77 for GPT-4-turbo, 2.69 to 1.65 for GPT-4, 3.35 to 2.83 for GPT-3.5-turbo, and 2.39 to 2.11 for GPT-3; paired Wilcoxon signed-rank tests are reported at P < .001. Independent recalculation from the supplementary CSV confirms improvement in 76/107 countries for GPT-4o (71.0%), 87/107 for GPT-4-turbo (81.3%), 83/107 for GPT-4 (77.6%), 77/106 for GPT-3.5-turbo (72.6%), and 86/107 for GPT-3 (80.4%). The effect is not universal: distance worsens in 31, 20, 24, 29, and 21 cases, respectively. For GPT-4o, Jordan improves from 4.10 to 0.36, while Finland worsens from 0.20 to 2.43, Luxembourg from 0.59 to 2.72, Andorra from 0.21 to 2.26, Switzerland from 0.45 to 2.48, and Taiwan from 2.40 to 3.94. Even after prompting, GPT-4o’s mean distance of 1.57 remains substantial.

The defensible result is that a national label systematically changes closed-ended answers from these snapshots and often moves the aggregated point closer to the IVS national mean under this metric. It does not show cultural understanding, culturally appropriate generation, or improvement on real tasks. A country label may itself activate stereotypes, and outputs are not evaluated by residents, within-country cultural groups, or local experts. All questions are in English, survey years are pooled across time, and the heterogeneity of 393,536 people is compressed into one country mean on a plane retaining only 39% of the variation. The study supplies no uncertainty intervals for national positions, multiple-testing correction, non-OpenAI models, local-language conditions, long-form generation, or interactive behavior.

Public reproducibility is stronger than the journal supplement alone suggests. OSF project 7sj3w contains the R analysis script, two Python pipelines, prompts, scores, coordinates, and 428 country-model JSON files with raw completion objects for GPT-3.5/4/4-turbo/4o, plus default responses; objects preserve request IDs, timestamps, model IDs, and system fingerprints where available. The journal supplement includes six CSV files and a 13-page PDF. Reproduction is still not one-command: the R script uses file.choose(), local paths, and manual checks; R/Python and dependency versions are not pinned; and raw WVS/EVS files must be obtained separately. The public OSF project is not a frozen registration. The contribution is therefore a substantially auditable descriptive study of 2024 GPT snapshot survey responses and a prompting intervention, not evidence that nationality exhausts culture or a general validation of cultural alignment.

Español

El artículo evalúa si cinco snapshots de la familia GPT de OpenAI producen, por defecto, respuestas a encuestas de valores más próximas a unas culturas nacionales que a otras y si indicar un país en el prompt reduce esa distancia. La referencia humana combina las tres olas más recientes, entre 2005 y 2022, del World Values Survey y el European Values Study: 393.536 registros individuales de 112 países o territorios. Se seleccionan las diez preguntas usadas para el mapa cultural de Inglehart–Welzel, se estandarizan con los datos humanos y se aplica PCA ponderado con rotación varimax y eliminación por pares. Los dos componentes, supervivencia frente a autoexpresión y valores tradicionales frente a seculares, explican el 39 % de la variación. Cinco países sin puntuaciones válidas en alguno de los ítems se excluyen, por lo que el análisis final cubre 107. Las coordenadas nacionales se obtienen promediando primero por país-año y después entre años; no representan la distribución interna de opiniones de cada sociedad.

Los modelos son text-davinci-002, gpt-3.5-turbo-0613, gpt-4-0613, gpt-4-turbo-2024-04-09 y gpt-4o-2024-05-13. A temperatura 0, cada modelo contesta los mismos diez ítems en inglés. Para los cuatro modelos recientes se usan diez variantes del descriptor, por ejemplo, “average human being”, “typical person” o “world citizen”, y se promedian sus coordenadas; GPT-3 solo usa una variante porque fue retirado. El prompting cultural añade “born in [país] and living in [país]”. Esto genera 400 respuestas generales y 42.800 respuestas condicionadas para los cuatro modelos recientes, más diez respuestas generales y 1.070 condicionadas de GPT-3. No se repite un prompt idéntico: la variación estudiada es de redacción, no variación estocástica. Los autores revisan manualmente respuestas fuera de formato; en GPT-3.5 aparecen 2 negativas de 1.070 para homosexualidad y 30 de 1.070 para aborto. Libia se elimina de esa comparación porque faltan las diez respuestas de homosexualidad.

Sin país en el prompt, los cinco puntos GPT quedan en la zona de autoexpresión y son más próximos a países anglófonos y de la Europa protestante que a muchos países africano-islámicos. Por ejemplo, GPT-4o queda a distancia 0,20 de Finlandia, 0,21 de Andorra y 0,45 de Países Bajos, frente a 4,10 de Jordania, 4,00 de Libia y 3,95 de Ghana. Esta es una comparación geométrica entre una respuesta agregada del modelo y medias nacionales proyectadas a dos dimensiones; llamar a esa posición “valores culturales del modelo” es la operacionalización del estudio, no una medición de creencias internas. Tampoco se demuestra la causa: el paper propone como posibilidades el idioma inglés, el corpus de entrenamiento y las decisiones de desarrollo, pero reconoce que la opacidad de los modelos impide atribuir los desplazamientos a RLHF u otros mecanismos.

El prompting cultural reduce la distancia media de 2,42 a 1,57 en GPT-4o, de 2,71 a 1,77 en GPT-4-turbo, de 2,69 a 1,65 en GPT-4, de 3,35 a 2,83 en GPT-3.5-turbo y de 2,39 a 2,11 en GPT-3; las pruebas de rangos con signo de Wilcoxon se reportan con P < 0,001. Recalculando el CSV suplementario, la distancia baja en 76/107 países para GPT-4o (71,0 %), 87/107 para GPT-4-turbo (81,3 %), 83/107 para GPT-4 (77,6 %), 77/106 para GPT-3.5-turbo (72,6 %) y 86/107 para GPT-3 (80,4 %). No es universal: empeora en 31, 20, 24, 29 y 21 casos, respectivamente. En GPT-4o, Jordania mejora de 4,10 a 0,36, pero Finlandia empeora de 0,20 a 2,43, Luxemburgo de 0,59 a 2,72, Andorra de 0,21 a 2,26, Suiza de 0,45 a 2,48 y Taiwán de 2,40 a 3,94. Incluso tras el ajuste, la distancia media 1,57 de GPT-4o sigue siendo sustancial.

El resultado sólido es que una etiqueta nacional cambia de forma sistemática las respuestas cerradas de estos snapshots y suele acercar su punto agregado a la media nacional del IVS en esta métrica. No prueba que el modelo comprenda una cultura, que produzca contenido culturalmente adecuado ni que mejore tareas reales. El país funciona además como una señal que puede activar estereotipos; no hay evaluación por residentes, grupos culturales internos o expertos locales. Todo se pregunta en inglés, se mezclan encuestas de años distintos y se comprime la heterogeneidad de 393.536 personas a una media por país en un plano que conserva solo el 39 % de la variación. No hay intervalos de incertidumbre para las posiciones nacionales, corrección por multiplicidad, comparación con modelos no OpenAI, idiomas locales, textos largos o comportamiento interactivo.

La reproducibilidad pública es mejor de lo que sugiere el suplemento de la revista. El proyecto OSF 7sj3w incluye el script R de análisis, dos pipelines Python, prompts, puntuaciones, coordenadas y 428 JSON país-modelo con objetos de respuesta para GPT-3.5/4/4-turbo/4o, además de las respuestas generales; los objetos conservan IDs, timestamps, modelo y, cuando existe, system fingerprint. El suplemento aporta seis CSV y un PDF de 13 páginas. Aun así, la reproducción no es de un solo paso: el script usa file.choose(), rutas locales y comprobaciones manuales, no fija versiones de R/Python ni dependencias, y los archivos WVS/EVS brutos deben obtenerse aparte. El OSF es público pero no está registrado o congelado. La contribución debe leerse como una auditoría descriptiva, reproducible en buena medida, de respuestas de encuesta de cinco snapshots GPT en 2024 y de una intervención de prompting; no como evidencia de que una identidad nacional capture toda una cultura ni como validación general de alineación cultural.

Research question

Which national cultures lie closest to the default responses of five GPT snapshots on the Inglehart–Welzel values map, and to what extent does indicating country of birth and residence in the prompt reduce the distance from the national mean of the WVS?

Method

WVS and EVS from 2005–2022 are combined, ten weighted items are projected via varimax PCA onto two dimensions, and coordinates are calculated for 107 countries. Five GPT snapshots answer those items in English, without country and with 107 national identities; ten descriptor variants are averaged except in GPT-3. Alignment is operationalized as the Euclidean distance between each GPT point and the national mean, and distances before/after are compared with Wilcoxon.

Sample: 393,536 individual WVS/EVS responses across 112 countries or territories; 107 retained in the final map. For four recent snapshots: 400 general responses and 42,800 country-conditioned responses; for GPT-3: 10 general and 1,070 country-conditioned. Each combination uses temperature 0 and one run per wording; ten descriptor variants except GPT-3.

Findings

  • The two components of the map explain 39% of the variation in the ten items.
  • The five GPT snapshots default toward self-expression values.
  • Their default points are closest to English-speaking and Protestant Europe countries.
  • GPT-4o lies at distance 0.20 from Finland, 0.21 from Andorra, and 0.45 from the Netherlands.
  • GPT-4o lies at distance 4.10 from Jordan, 4.00 from Libya, and 3.95 from Ghana.
  • No simple temporal trend appears on the traditional-secular axis among the five models.
  • The difference in default distances between GPT-4o, GPT-4, and GPT-4-turbo is barely significant, P = 0.036.
  • Cultural prompting reduces the mean distance of GPT-4o from 2.42 to 1.57.
  • It reduces that of GPT-4-turbo from 2.71 to 1.77.
  • It reduces that of GPT-4 from 2.69 to 1.65.
  • It reduces that of GPT-3.5-turbo from 3.35 to 2.83.
  • It reduces that of GPT-3 from 2.39 to 2.11.
  • The five paired comparisons are reported with P < 0.001.
  • The CSV confirms improvement in 76/107 countries for GPT-4o, equivalent to 71.0%.
  • It confirms improvement in 87/107 for GPT-4-turbo, equivalent to 81.3%.
  • It confirms improvement in 83/107 for GPT-4, equivalent to 77.6%.
  • It confirms improvement in 77/106 for GPT-3.5-turbo, equivalent to 72.6%.
  • It confirms improvement in 86/107 for GPT-3, equivalent to 80.4%.
  • For GPT-4o, Jordan goes from distance 4.10 to 0.36.
  • For GPT-4o, 31 countries worsen; the largest increases are Finland, Luxembourg, Andorra, Switzerland, and Taiwan.
  • Wording variants produce appreciable dispersion, especially in GPT-4o without cultural prompting.
  • Cultural prompting does not collapse all GPT-4 responses to a central point: country-specific coordinates retain dispersion.
  • GPT-3.5 rejects some sensitive items and forces the exclusion of Libya from its cultural comparison.
  • OSF retains sufficient data and code to audit the published figures and the responses of the four recent snapshots.

Limitations

  • Only five models from one company and one technology family are studied.
  • The conclusions do not cover Claude, Llama, Mistral, Gemini, open models, or models developed outside the United States.
  • All prompts are in English.
  • Local languages or culturally validated translations are not compared.
  • Country of birth and residence is used as a proxy for culture.
  • The national label may activate stereotypes rather than situated cultural knowledge.
  • There is no evaluation by residents, local communities, or cultural experts.
  • Regional, ethnic, linguistic, religious, generational, or socioeconomic heterogeneity within each country is not represented.
  • National means compress 393,536 individual responses into a single point per country.
  • Different survey years within each country are averaged equally.
  • Human data span 2005–2022, while model responses are generated in 2024.
  • WVS and EVS may both include data for the same country and year without explicitly modeling the dependence between surveys.
  • The two components retain only 39% of the variation in the ten items.
  • Euclidean distance in two dimensions depends on that reduction and its scale.
  • Standard errors or confidence intervals for national coordinates are not incorporated.
  • The tests treat reference coordinates as points without sampling uncertainty.
  • No correction for multiple comparisons is reported.
  • P = 0.036 in the comparison between recent models is weak and does not include an effect size.
  • Temperature 0 does not guarantee perfect determinism in a hosted API.
  • No identical prompt is repeated to measure temporal or infrastructure stability.
  • The ten variants are synonyms chosen by the authors, not a systematic sample of the prompt space.
  • GPT-3 uses only one variant, so its robustness is not comparable.
  • GPT-3 uses a combined user prompt; the others separate system and user.
  • Formatting instructions were refined iteratively until usable responses were achieved, without a preregistered protocol.
  • Manual extraction of responses introduces human decisions that are not double-coded.
  • GPT-3.5 produces selective refusals on questions about homosexuality and abortion.
  • Excluding Libya in one condition changes the denominator and comparability.
  • Closed-ended survey responses are studied, not long-form generation, conversation, or real tasks.
  • It is not measured whether geometric closeness improves utility, safety, or cultural appropriateness.
  • Harms from stereotypes caused by national prompting are not examined.
  • Explanations about corpus, RLHF, and reward rules are speculative.
  • OpenAI's opacity prevents attributing changes between snapshots to specific mechanisms.
  • The public script depends on file.choose(), local paths, and manual checks.
  • Versions of R, Python, psych, pandas, openai, or other dependencies are not pinned.
  • Raw WVS/EVS files are not included in OSF and must be obtained separately.
  • The OSF project is public, but not a frozen, immutable registration.
  • No environment, lockfile, container, or automated reproduction test is provided.
  • The evaluated snapshots are historical and do not characterize current models.

What the study does not establish

  • It does not demonstrate that an LLM possesses internal cultural values.
  • It does not demonstrate understanding of any culture.
  • It does not demonstrate that nationality and culture are equivalent.
  • It does not demonstrate alignment with all people in a country.
  • It does not demonstrate absence of stereotypes in conditioned responses.
  • It does not demonstrate that lesser distance on the map produces culturally appropriate text.
  • It does not demonstrate improvement in real tasks or human-AI interactions.
  • It does not demonstrate that cultural prompting is safe for all countries.
  • It does not demonstrate that cultural prompting works universally; it worsens in 19% to 29% of cases.
  • It does not causally identify the origin of the observed bias.
  • It does not demonstrate that RLHF explains the differences between versions.
  • It does not generalize to other providers, model families, or languages.
  • It does not characterize GPT models after May 2024.
  • It does not validate the two-dimensional position as an exhaustive measure of culture.
  • It does not test that an LLM's survey responses predict its generative behavior.
  • It does not show that a national mean is the correct answer for a specific individual.
  • It does not establish superiority of one snapshot: performance depends on country and prompt.
  • It does not eliminate the remaining cultural distance even where the intervention improves the metric.

Traceability

Scope: Full text

Version: PNAS Nexus 3(9):pgae346, version of record published 17 Sep 2024, DOI 10.1093/pnasnexus/pgae346; OSF project 7sj3w last modified 7 Aug 2024

Consulted source: https://academic.oup.com/pnasnexus/article-pdf/3/9/pgae346/59161126/pgae346.pdf

Review: Codex full-text, bilingual-fidelity, visual, IVS/WVS/EVS, PCA, prompt-variant, cultural-prompting, CSV-recalculation, OSF-code, raw-completion, construct-validity, statistics, reproducibility and data-availability audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • OpenAI text-davinci-002 (GPT-3)
  • OpenAI gpt-3.5-turbo-0613
  • OpenAI gpt-4-0613
  • OpenAI gpt-4-turbo-2024-04-09
  • OpenAI gpt-4o-2024-05-13

Instruments and metrics

  • Integrated Values Surveys (World Values Survey plus European Values Study)
  • Ten Inglehart–Welzel World Cultural Map items
  • Weighted principal component analysis with varimax rotation and pairwise deletion
  • Survival versus self-expression component
  • Traditional versus secular-rational component
  • Ten respondent-descriptor prompt variants
  • Country-of-birth-and-residence cultural prompting
  • Euclidean cultural distance
  • Paired Wilcoxon signed-rank tests
  • Kruskal–Wallis comparison of default-distance distributions

Data used

  • WVS trend file 1981–2022 v3.0.0, filtered to waves 5–7
  • EVS trend file 1981–2017 v3.0.0, filtered to waves 5–7
  • 393,536 weighted individual survey records from 112 countries or territories
  • 107-country analysis after five exclusions
  • OSF 7sj3w analysis script, generation pipelines, prompts, raw completion objects, scores and coordinates
  • PNAS Nexus supplementary PDF and Datasets S1–S6

Evidence and location

  • Official publication: PNAS Nexus 3(9):pgae346, advance publication 17 Sep 2024, DOI 10.1093/pnasnexus/pgae346
  • Complete source: .cache/editorial-sources/article-109/source.pdf; 9 pages; sha256 978f4497d3c23cb23ea1c9144a25a400c3680accfacb30ee54a231848c1040e2
  • Editorial supplement: pgae346 supplementary data ZIP; PDF 13 pages plus six CSV files; sha256 3a2e9c82244701491cfda56d6b8007e21f236e3a5e39060810ab50cd698e6c8a
  • Original abstract and metadata: Full text p. 1 and PMC11407280
  • Questions and variants: Full text pp. 2–3, Tables 1–2
  • Exact models: Full text p. 3; OSF filenames and raw completion model fields
  • Map and default distances: Full text pp. 3–4, Figure 1 and Dataset S5
  • Means and before/after tests: Full text pp. 4–5, Figure 2; OSF Analysis_Script_OSF.R lines 492–523
  • Recalculated improvement rates: Dataset S6 RQ2_CP_Responses_L2_Results.csv: 76/107, 87/107, 83/107, 77/106 and 86/107
  • Countries that worsen with GPT-4o: Full text pp. 4–5 and Dataset S6
  • Sensitivity to wording: Supplementary text pp. 2–3 and Figures S1–S5
  • Not simple recentring: Supplementary text pp. 2–3 and Figures S6–S8
  • Acknowledged limits: Full text p. 6, Discussion
  • PCA and human sample: Full text pp. 6–7, Materials and methods
  • Generation, temperature, and absence of repetitions: Full text pp. 6–7, Measuring cultural values of GPT
  • Refusals and manual extraction: Full text p. 7, Evaluating cultural prompting
  • OSF data and code: https://osf.io/7sj3w/; public project created 14 Jun 2024, modified 7 Aug 2024, not registered; checked 15 Jul 2026
  • Response objects: OSF Answer Generation folders: 428 country-conditioned JSON files for four recent snapshots plus default completion objects
  • Reproducible workflow limitations: OSF Analysis_Script_OSF.R and Part1/Part2 generation pipelines: interactive file.choose, local paths, manual checks and no pinned environment
  • Visual inspection: All 9 main-PDF pages and all 13 supplementary-PDF pages rendered and visually inspected, including Tables 1–2, Figures 1–2 and Figures S1–S8; checked 15 Jul 2026