Synthetic Personality Review

CultureLLM: Incorporating Cultural Differences into Large Language Models

Society, culture, and collective behavior2024arXivApproved editorial review

Authors: Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

Keywords: Computation and Language, Artificial Intelligence, Machine Learning

Source: Open primary source (opens in a new tab)

5
Authors
7
Findings
20
Limitations
11
Evidence

Editorial summary

English

CultureLLM neither builds a cultural knowledge base nor evaluates personality. It starts with 50 selected World Values Survey (WVS) questions, assigns each culture the average answer from one or more countries, and creates 500 paraphrases using GPT-4, synonym replacement, and a BERT similarity filter. These questions and answers fine-tune nine culture-specific GPT-3.5-0613 models, Arabic, Bengali, Chinese, US English, German, Korean, Portuguese, Spanish, and Turkish, and one unified CultureLLM-One. The appendix applies the same idea with LoRA on Llama-2-70B. The main evaluation uses 59 zero-shot classification datasets covering hate, offense, abuse, toxicity, threat, bias, stance, and spam; a 65-question generative set brings the stated total to 60.

The paper reports that culture-specific models improve on average by 8.1% over GPT-3.5 and 9.5% over Gemini Pro, with results close to GPT-4. CultureLLM-One also improves the average but trails the specific models; Korean shows no clear gain. On the generative task, Gemini Pro prefers CultureLLM over GPT-3.5 in eight of nine cultures, while Turkish has a WinRate of -.062. The evidence supports that small supervised fine-tunes change classification behavior and reduce neutral responses, but it cannot causally attribute improvement to “cultural understanding”: the tasks measure content moderation, and fine-tuning can also improve formatting, instruction following, or calibration.

Validity and reproducibility are limited. The paper mixes macro-F1 with the CValues metric, normalizes and averages 59 heterogeneous tasks, calls differences “significant” without defining a test, unit, interval, or repeated-run protocol, and releases no numeric result artifacts. The official code audited at dcdb64d7289a8ee88c7e718fb5e8641bf97c441e contains data and scripts but no results, release, license, locked environment, CI, or tests. Experimental model IDs, endpoints, and keys are replaced with xxx. The fine-grained hate parser maps HS2–HS6 to HS1; test.sh passes china although the code only recognizes chinese; generation uses temperature and unseeded randomness. Open-ended evaluation uses Gemini Pro as the sole judge and has no human validation. The similarity study does include 50 English-major students aged 22–26, but the human score is 4.60/5; the 4.84 aggregate and its conversion to about 96.5% mix humans with GPT-4 and Gemini Pro. The released corpus itself contains unnatural paraphrases, qualifying the near-perfect equivalence claim.

CultureLLM is relevant to how cultural fine-tuning can alter apparent model behavior, and it also exposes a central risk: turning national averages into a cultural voice can fix stereotypes and reproduce harmful opinions. It does not measure psychometric traits, stable personality, persistent identity, or within-culture individual differences.

Español

CultureLLM no crea una base de conocimiento cultural ni evalúa personalidad. Parte de 50 preguntas seleccionadas de la World Values Survey (WVS), asigna a cada cultura la respuesta media de uno o varios países y genera 500 paráfrasis mediante GPT-4, sustitución de sinónimos y un filtro de similitud BERT. Con esas preguntas y respuestas afina nueve GPT-3.5-0613 específicos, árabe, bengalí, chino, inglés de Estados Unidos, alemán, coreano, portugués, español y turco, y un CultureLLM-One unificado. El apéndice replica la idea con LoRA sobre Llama-2-70B. La evaluación principal usa 59 conjuntos de clasificación zero-shot sobre odio, ofensa, abuso, toxicidad, amenaza, sesgo, postura y spam; un conjunto generativo de 65 preguntas eleva el total declarado a 60.

El artículo informa que los modelos específicos mejoran en promedio un 8,1 % frente a GPT-3.5 y un 9,5 % frente a Gemini Pro, con resultados próximos a GPT-4. CultureLLM-One también mejora el promedio, pero queda por debajo de los modelos específicos; en coreano no se observa una mejora clara. En la tarea generativa, Gemini Pro prefiere CultureLLM a GPT-3.5 en ocho de nueve culturas, mientras turco obtiene WinRate -0,062. La evidencia sostiene que un pequeño ajuste supervisado cambia la conducta de clasificación y reduce respuestas neutrales, pero no permite atribuir causalmente la mejora a “comprensión cultural”: las tareas miden moderación de contenido y el ajuste también puede mejorar formato, obediencia o calibración.

La validez y reproducibilidad son limitadas. El paper mezcla macro-F1 con la métrica de CValues, normaliza y promedia 59 tareas heterogéneas, llama a las diferencias “significativas” sin definir test, unidad, intervalos o repeticiones, y no publica los resultados numéricos en el repositorio. El código oficial auditado en dcdb64d7289a8ee88c7e718fb5e8641bf97c441e contiene datos y scripts, pero no resultados, release, licencia, entorno bloqueado, CI ni tests. Los modelos, endpoints y claves de los experimentos están sustituidos por xxx. El script multiclase de odio asigna HS2–HS6 a HS1; test.sh usa china aunque el código solo reconoce chinese; la generación usa temperatura y aleatoriedad sin semilla. La evaluación abierta usa Gemini Pro como único juez y no publica prueba humana. El estudio de similitud sí reúne 50 estudiantes de inglés de 22–26 años, pero la nota humana es 4,60/5; el 4,84 agregado y su conversión a aproximadamente 96,5 % mezclan humanos con GPT-4 y Gemini Pro. El propio corpus publicado contiene paráfrasis antinaturales que matizan la afirmación de equivalencia casi perfecta.

CultureLLM es relevante para estudiar cómo el ajuste cultural puede alterar el comportamiento aparente de un modelo, y también alerta sobre un riesgo central: convertir promedios nacionales en una voz cultural puede fijar estereotipos y reproducir opiniones dañinas. No mide rasgos psicométricos, estabilidad de una personalidad, identidad persistente ni diferencias individuales dentro de cada cultura.

Research question

Can an economical fine-tuning based on 50 WVS questions and synthetic paraphrases produce specific or unified models that perform better on multilingual tasks associated with different cultures?

Method

50 WVS questions from seven topics are manually selected and the mean response of representative countries is used as the cultural label. GPT-4 generates five templates per question; a BERT similarity filter retains paraphrases above 0.8 and a second stage substitutes synonyms until forming 500 examples. Specific and unified GPT-3.5-0613, and Llama-2-70B via LoRA, are fine-tuned. They are compared with GPT-3.5, GPT-4-1104, Gemini Pro, RAG and cultural models over 59 classification datasets, a generative evaluation judged by Gemini, ablations, BBH/GSM8K and a human study of similarity.

Sample: The main evaluation gathers 59 datasets and 68,607 examples; when adding 65 generative questions the text states 60 sets and 68,672 examples. Cultures are approximated through nine languages and representative countries. The tuning uses 50 WVS questions plus 500 augmented ones, although the published fine-tuning files normally contain 1,100 messages per culture because they combine two blocks of 550. The human study includes 25 men and 25 women, all students with high exposure to English, from 22 to 26 years old; 26 hold a bachelor's degree and 24 a master's.

Findings

  • The paper reports a mean improvement of 8.1% over GPT-3.5, 9.5% over Gemini Pro and 7.94% over RAG, with aggregate performance close to GPT-4; it does not publish a statistical test supporting the term "significant".
  • Specific cultural models usually outperform the unified model. There is no clear improvement in Korean and the text itself attributes this exception, without proof, to lower exposure of the base model.
  • In the open evaluation judged by Gemini Pro, CultureLLM beats GPT-3.5 in eight cultures; the WinRates range from 0.215 to 0.615 except for Turkish, with -0.062.
  • Table 8 reports for CultureLLM-One an average of 0.597 versus 0.556 for GPT-3.5. WVS only reaches 0.569 and the partial stages 0.564 and 0.563, so the isolated components do not improve monotonically.
  • In Llama-2-70B, the average goes from 0.3909 to 0.3997 for the unified model and 0.4093 for the specific ones. There are no repetitions or intervals that allow separating signal from training variation.
  • The forgetting control keeps GSM8K practically constant, but CultureLLM-En lowers BBH from 51.2 to 48.8; therefore, the claim that there is no negative effect has at least one published exception.
  • Humans score semantic equivalence at 4.60 ± 0.28 out of 5; GPT-4 gives 4.99 and Gemini 4.93. The mean 4.84 mixes human and automatic evaluators and is not a human approval rate.

Limitations

  • Language, country and culture are used almost interchangeably, although the appendix acknowledges that language is only one part of culture and that cultural boundaries are fluid.
  • Averaging WVS responses from one or several countries erases individual, regional, generational and social variation. Converting that average into the voice of a chatbot incurs the risk of ecological fallacy and stereotype.
  • The 59 main tasks evaluate moderation or content classification, not cultural knowledge or direct alignment with WVS values. The improvement may be due to format compliance or calibration.
  • Macro-F1, accuracy and tasks with different numbers of classes, difficulty and size are normalized and averaged. The weighting is not clearly defined and a complete readable numerical table is not published for each run.
  • The term "significantly" is not accompanied by hypotheses, test, unit of analysis, p-values, intervals, multiple correction or repetitions. The bars in Figure 3 are not defined.
  • Closed APIs and GPT-3.5-0613, GPT-4-1104 and Gemini Pro snapshots have changed. The paper admits that results may vary with ChatGPT versions.
  • Generation uses temperature 0.7, random substitution and sets without seeds. No augmentation, tuning or evaluation replicas are reported.
  • The manual selection of 50 questions and representative countries has no reproducible selection protocol, sensitivity analysis or independent WVS set to validate that values were learned.
  • The human study only measures paraphrase similarity in English with English students aged 22-26. It does not report inter-evaluator agreement, per-item distribution, cultural analysis, compensation or ethical review identifier.
  • The 96.5% is not the human result: the human score is 4.60/5 and the mean 4.84 incorporates two LLM judges. Treating an ordinal scale of 1-5 as a pass percentage is also not justified.
  • The published data contain grammatically or semantically defective paraphrases, such as "a essential", "a insight", "right single", "extremist tasks" and "only my religion meets the standard of consent".
  • The generative evaluation creates the questions with GPT-4 and uses Gemini Pro as the sole judge of GPT-3.5 versus CultureLLM. There are no human judges, order control, judge calibration or uncertainty.
  • The forgetting control uses only 100 examples per BBH task due to cost; CultureLLM-En falls on BBH. GSM8K changes by tenths, with no variance estimation.
  • The repository does not contain the paper's results, fine-tuned models, job IDs, logs or an experimental release. All CultureLLM endpoints and keys are replaced by xxx.
  • There is no requirements block, pyproject, environment, unit tests, CI, license or reproducible hardware/cost documentation. The test.py file aggregates results; it is not a test suite.
  • The fine-grained hate speech post-processing assigns categories HS2, HS3, HS4, HS5 and HS6 to HS1, corrupting that multiclass metric.
  • test.sh invokes the Chinese language, while getModel and getDataPath only implement chinese; the path remains unassigned. The script also does not test Gemini, RAG or CultureLLM-One and duplicates the Chinese block.
  • The code disables TLS verification for a Llama endpoint and disables Gemini safety filters in sensitive categories, decisions that require justification and isolation.
  • The repository redistributes about 87 MB of social network datasets with harmful language and possible personal data, but does not include licenses, data sheets, per-file provenance or a privacy protocol.
  • The six-dollar cost claim does not break down prices, tokens, retries, GPT-4 generation, evaluation, tariff date or the cost of the ten variants.

What the study does not establish

  • It does not demonstrate that CultureLLM understands a culture; it demonstrates output changes after supervised tuning and linguistic prompting on specific tasks.
  • It does not establish that a language or national average represents all people of a culture or that a chatbot should express those aggregated opinions.
  • It does not demonstrate a general reduction of bias or harm. Tuning with averages on gender, religion, immigration, surveillance or authority can also consolidate discriminatory opinions.
  • It does not statistically validate a universal superiority over GPT-4, GPT-3.5, Gemini, RAG or cultural models across all datasets, cultures and runs.
  • It does not allow reproducing the main tables from the current public repository without models, endpoints, results and historical configuration.
  • It does not measure Big Five, other psychometric traits, temporal stability, cross-situational consistency, persistent identity or an individual synthetic personality.

Traceability

Scope: Full text

Version: arXiv:2402.10946v3, submitted 9 February 2024, revised 3 December 2024; NeurIPS 2024, 40 pages

Consulted source: https://arxiv.org/pdf/2402.10946v3

Review: Codex full-text, bilingual-fidelity, 40-page visual, arXiv-v3, NeurIPS-2024, official-repository, construct-validity, heterogeneous-metric, statistical-claim, augmentation-quality, human-study, LLM-judge, code-correctness, data-governance, cultural-stereotype, forgetting, cost and reproducibility audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-3.5-0613 fine-tuned CultureLLM models
  • CultureLLM-One unified GPT-3.5 model
  • GPT-3.5 baseline
  • GPT-4-1104 baseline and augmentation model
  • Gemini Pro baseline and generative judge
  • Llama-2-70B and LoRA CultureLLM variants
  • SeaLLM
  • TaiwanLLM
  • CultureBank
  • BERT-base-uncased semantic filter

Instruments and metrics

  • World Values Survey seed questions and national response averages
  • GPT-4 semantic-template generation
  • GPT-4 synonym generation and random replacement
  • BERT cosine-similarity filter at 0.8
  • Macro F1 over content-moderation classifications
  • CValues automatic metric
  • Gemini Pro pairwise judgment and WinRate
  • Perplexity and diversity gain
  • Five-point human semantic-similarity rating
  • BBH and GSM8K forgetting checks
  • LoRA fine-tuning ablation

Data used

  • 50 selected World Values Survey questions across seven topics
  • 500 GPT-4 and synonym-augmented training samples
  • 59 classification datasets with 68,607 reported test items across nine languages
  • 65 GPT-4-generated open-ended WVS questions
  • CValues Chinese values benchmark
  • BIG-Bench Hard, 100 items from each of 21 tasks
  • GSM8K
  • 100 seed-generation pairs rated by 50 participants

Evidence and location

  • Scope, method and main claims: Paper, pp. 1-3, Abstract, Introduction and Contributions
  • WVS selection and semantic augmentation: Paper, pp. 4-5, Sections 3.1-3.3 and Figure 2
  • Fine-tuning, datasets, baselines and metrics: Paper, pp. 5-7, Sections 3.4 and 4.1-4.2
  • Open evaluation, ablation and synthetic quality: Paper, pp. 7-9, Sections 4.3-4.6 and Tables 1-2
  • Forgetting, Llama, impact and limitations: Paper, pp. 9-10, Sections 5-6 and Figure 5
  • Definition of culture, WVS and breakdown of 59 datasets: Paper, pp. 17-24, Appendices A-B and Tables 3-4
  • Prompts, aggregation and detailed results: Paper, pp. 25-31, Appendices C-E and Tables 5-10
  • Forgetting exception and human participants: Paper, pp. 32-33, Tables 11-12 and Appendix F
  • Reproducibility, statistics, ethics and assets statements: Paper, pp. 34-40, NeurIPS checklist
  • Code, data and evaluation defects: Official CultureLLM repository commit dcdb64d7289a8ee88c7e718fb5e8641bf97c441e, main, test_offensEval, test.sh, data and repository root
  • Comprehensive visual inspection: Paper, all 40 rendered pages, including every figure, table, appendix and checklist page