How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions

Society, culture, and collective behavior2025arXivApproved editorial review

Authors: Julia Kharchenko, Tanya Roosta, Aman Chadha, Chirag Shah

Keywords: Computation and Language, Artificial Intelligence, Computers and Society

Source: Open primary source (opens in a new tab)

4: Authors
6: Findings
16: Limitations
11: Evidence

Editorial summary

English

The paper studies whether five LLMs change advice according to national values represented by five Hofstede dimensions. It creates 250 binary dilemmas, 50 per dimension, and combines them with personas for 36 countries or NLLB-200 translations linked to 34 countries in the released data. It evaluates GPT-4, GPT-4o, Command R+, Gemma-7B-IT, and Llama-3-8B-Instruct. The models show broad response preferences, for example, long-term orientation and low competitive achievement motivation, and some country- or language-dependent differences, but alignment with countries' Hofstede scores is weak and inconsistent. Responses also contain national stereotypes and fabricated cultural sayings. The defensible conclusion is that national or language prompting can change outputs and elicit stereotypes, not that models understand or internalize cultural values.

The audit of the paper, all 22 pages, and the official repository finds problems that prevent acceptance of the reported significance. The notebooks first calculate a percentage per country and then expand it back to each of 1,700–9,000 rows before running Pearson correlation. This leaves the coefficient unchanged but creates an artificial sample size that makes tiny correlations significant. As a result, 40 of 50 tests appear to have p < .05; using country as the observational unit leaves 13 uncorrected results, and none survives a Bonferroni correction for 50 comparisons. This explains stars on values such as r=.0218 and conflicts with the prose, which says only one combination was significant. “Highest accuracy” is obtained by sweeping 100 thresholds on the same data and selecting the maximum, with no held-out test set.

The generation and labeling code also does not cleanly reproduce the study. In all ten scripts, selection between two answers and “Inconclusive” uses Python and expressions inside numeric comparisons; and returns one operand rather than comparing all three similarities, making classification asymmetric and incorrect. All ten scripts truncate “Low Power Distance” to “Low Power Distanc” when determining adherence, so part of the PDI condition cannot be marked correctly. Current scripts run two rounds while many frozen CSVs contain five and several models contain only one. The pipeline has no seeds, locked environment, tests, CI, license, or experimental release, and its BLEU script directly compares translations with English source text, which does not validate cross-language translation quality. The work is therefore a useful exploratory artifact about cultural prompting and stereotyping risk, but its p-values, accuracies, and cultural-understanding claims need corrected reanalysis and replication. It neither measures psychometric personality traits nor demonstrates persistent cultural identity.

Español

El artículo estudia si cinco LLM modifican consejos según valores nacionales representados por cinco dimensiones de Hofstede. Construye 250 dilemas binarios, 50 por dimensión, y los combina con personas de 36 países o con traducciones NLLB-200 vinculadas a 34 países en los datos publicados. Evalúa GPT-4, GPT-4o, Command R+, Gemma-7B-IT y Llama-3-8B-Instruct. Los modelos muestran preferencias globales, por ejemplo, orientación a largo plazo y baja motivación hacia el éxito competitivo, y algunas diferencias por país o idioma, pero la alineación con la puntuación Hofstede del país es débil e inconsistente. También aparecen respuestas estereotípicas o inventadas sobre países. La conclusión prudente es que el prompting nacional o lingüístico puede cambiar respuestas y activar estereotipos, no que el modelo comprenda ni interiorice valores culturales.

La auditoría del artículo, sus 22 páginas y el repositorio oficial revela problemas que impiden aceptar la significación publicada. Los notebooks calculan primero un porcentaje por país y después lo vuelven a expandir a cada una de las 1.700–9.000 filas antes de aplicar Pearson: el coeficiente no cambia, pero el tamaño muestral artificial convierte correlaciones diminutas en significativas. Así, 40 de 50 pruebas aparecen con p < 0,05; con el país como unidad quedan 13 sin corrección y ninguna supera Bonferroni para 50 comparaciones. Esto explica estrellas como r=0,0218 y contradice el texto, que dice que solo una combinación fue significativa. La “highest accuracy” se obtiene barriendo 100 umbrales sobre el mismo conjunto y eligiendo el máximo, sin test separado.

El código de generación y etiquetado tampoco reproduce limpiamente el estudio. En los diez scripts, la selección entre dos respuestas e “Inconclusive” usa expresiones Python con el operador and dentro de comparaciones numéricas; and devuelve uno de sus operandos y no compara las tres similitudes, por lo que el etiquetado es asimétrico e incorrecto. Los diez scripts truncaron “Low Power Distance” como “Low Power Distanc” al decidir adherencia, haciendo imposible marcar correctamente parte de PDI. Los scripts actuales ejecutan dos rondas mientras muchos CSV congelados contienen cinco; varios modelos solo tienen una. El pipeline carece de semillas, entorno bloqueado, tests, CI, licencia y release experimental, y el cálculo BLEU compara directamente traducciones con el texto inglés, lo que no valida la calidad de traducción entre idiomas. Por tanto, el trabajo aporta un conjunto exploratorio útil sobre prompting cultural y riesgo de estereotipos, pero sus p-valores, accuracies y afirmaciones de comprensión cultural requieren un nuevo análisis y una réplica corregida. No mide personalidad como rasgos psicométricos ni demuestra una identidad cultural persistente.

Research question

To what extent do LLMs distinguish the extremes of five Hofstede cultural dimensions and adapt their advice to the score attributed to a country when they receive a national persona or a prompt in an associated language?

Method

50 binary dilemmas are created for each of five Hofstede dimensions. Individualism uses 50 manual prompts; the other dimensions combine 10 manual and 40 expanded with GPT-3.5. Each dilemma requires choosing a single extreme. National personas in English or NLLB-200 translations are applied, and five LLMs are consulted. Another LLM step selects which of two canonical responses is closest to the output and SentenceTransformer assigns the final label. Percentages by country, Pearson correlations, and a separability based on the best threshold are aggregated, along with qualitative inspection.

Sample: The design declares 36 countries and 36 languages, grouped by web presence into high, medium, and low resources. The base CSVs contain 36 countries in persona (1,800 rows per dimension) but 34 in multilingual (1,700), without explicit reconciliation. There are 50 dilemmas per dimension. Results vary between one, five, or, in the current code, two rounds per model; per file, between 1,700 and 9,000 responses are published.

Findings

LLMs can lean toward distinct extremes of a dimension and vary by country or language, but the relationship with Hofstede national scores is generally weak, irregular, and sometimes inverse.
The paper highlights GPT-4o multilingual in individualism for high-resource languages with r=0.71 and p<0.05; this is a comparison selected among many and is not accompanied by correction for multiplicity.
When recalculating the 50 published correlations using the country as the unit, 13 have p<0.05 without correction and none reaches the Bonferroni threshold of 0.001. The notebooks, by pseudoreplicating national percentages for each response, mark 40 of 50 as significant.
Models prefer long-term orientation responses in more than 80% of many experiments and tend toward the low extreme of MAS, pointing to global response biases rather than specific cultural adaptation.
Qualitative outputs include stereotypical associations and fabrications: Soviet references for Belarus, invented Armenian or Russian sayings, vodka, and French accent caricatures.
Table 3 reports accuracy maxima between 0.58 and 0.86, but the notebook selects the best of 100 thresholds over the same countries and does not evaluate out-of-sample generalization.

Limitations

Hofstede scores are national aggregates, originally derived from workplace surveys, and do not necessarily describe an individual. Inferring individual advice from them incurs the risk of ecological fallacy and national stereotype.
The dilemmas force a choice between two options and eliminate nuance. Some mix culture with medical prudence, morality, family, or economics, so a choice does not cleanly identify a cultural dimension.
The notebooks expand each national percentage to all rows before Pearson. This preserves r but inflates n from 34–36 countries to 1,700–9,000 non-independent responses and produces artificially small p-values.
At least 50 global correlations are tested, in addition to resource strata, without correction for multiple comparisons. None of the 50 global correlations recalculated over countries exceeds Bonferroni.
Table 2 assigns significance to tiny correlations and most of the stars contradict the results text. The published p-values should not be interpreted as valid evidence.
Accuracy is the maximum of a sweep of 100 thresholds over the same set and in both directions. There is no train/test, cross-validation, intervals, or selection baseline.
The automatic classification does not publish a human reference set, inter-annotator agreement, prompt sensitivity, or tagger accuracy. It uses another LLM and embeddings to judge LLM responses.
The classifier of the ten scripts compares floats using expressions with the and operator, which return an operand in Python. This makes the decision between choice A, choice B, and Inconclusive asymmetric and incorrect.
The ten scripts compare the correct label with “Low Power Distanc”, truncated, so that low power distance responses cannot be marked adherent in that branch.
The current scripts generate two rounds, the GPT-4/GPT-4o CSVs and part of Command R+ contain five, and Gemma/Llama/other Command R+ results contain one. There is no configuration traceability per artifact.
No seeds are fixed and local generation uses sampling with temperature 0.6. The APIs, models, and prompt improver are not frozen with sufficient date or snapshot.
There is no requirements, pyproject, lockfile, test suite, CI, license, or tagged release. The README is a single line and does not document a reproduction procedure.
The BLEU script compares translations directly with references in English, rather than human translations in the target language or back-translation; that value does not demonstrate translation quality.
The published multilingual dataset covers 34 countries versus the 36 declared, and the Personas_or_Multilingual field is written as Personas also in the OpenAI multilingual script and other scripts.
Language and country do not have a one-to-one correspondence. Transnational languages and personas formulated as “proudly and thoroughly from” can induce clichés and confuse cultural knowledge with stereotype.
The ethical inspection recognizes overgeneralization, but the recommendation to adapt responses to a known country can cause stereotypical treatment if the individual's real values are not collected.

What the study does not establish

It does not demonstrate that LLMs understand, internalize, or causally represent cultural values; it only observes choices under specific prompts and defective automatic labeling.
It does not establish that nationality or language allow inferring a person's values or that it is desirable to personalize advice from that proxy.
It does not provide reliable statistical evidence for the stars in Table 2 or out-of-sample predictive accuracy for Table 3.
It does not demonstrate better alignment in non-Western cultures or low-resource languages; patterns vary by model, dimension, and approach and are often inverse.
It does not measure personality traits with psychometric instruments, temporal stability, persistent identity, coherence across contexts, or a complete synthetic personality.
It does not allow reproducing the published experiment end to end from a frozen release without correcting the code, reconstructing environments, and clarifying the drift between scripts and CSVs.

Traceability

Scope: Full text

Version: arXiv:2406.14805v2, submitted 21 June 2024, revised 5 August 2025; KDD 2025, 22 pages

Consulted source: https://arxiv.org/pdf/2406.14805v2

Review: Codex full-text, bilingual-fidelity, 22-page visual, arXiv-v2, KDD-2025, official-repository, statistical-reanalysis, pseudoreplication, multiple-comparison, threshold-selection, classification-code, translation-validity, cultural-construct, stereotype-risk, reproducibility and scope-fit audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

GPT-4
GPT-4o
Command R+
Gemma-7B-IT
Llama-3-8B-Instruct
GPT-3.5 for prompt augmentation
Anthropic prompt improver for paraphrasing
NLLB-200-3.3B for translation
SentenceTransformer for response-label similarity

Instruments and metrics

Five Hofstede cultural dimensions
Forced-choice advice dilemmas
Country-persona prompting
Multilingual prompting with NLLB-200 translations
LLM-based canonical-answer selection
SentenceTransformer similarity classification
Pearson correlation
Post-selected threshold accuracy
Qualitative stereotype inspection

Data used

250 base advice dilemmas, 50 per Hofstede dimension
Persona datasets: 1,800 rows per dimension across 36 countries
Multilingual datasets: 1,700 rows per dimension across 34 countries in the repository
Released result CSVs for five models and two prompting approaches

Evidence and location

Question, contribution, and stated conclusion: Paper, pp. 1–2, Abstract and Introduction
Construction of prompts, personas, translations, and models: Paper, pp. 3–4, Section 3 and Figure 1
Correlations, significance, and accuracy: Paper, pp. 4–6, Tables 2–3 and Section 4
Preferences, stereotypes, and qualitative examples: Paper, pp. 6–7, Sections 4.2–4.3
Limitations and recognized ethics: Paper, p. 7, Sections 6–7
Prompts, metadata, and selected result by resources: Paper, pp. 10–11, Appendices E–F and Table 7
Complete distributions and correlations: Paper, pp. 12–22, Tables 8–15
Pearson pseudoreplication and post-selected accuracy: Official repository commit bfc069e0b9a881e10f40c2fe7ad7225a85c04d32, ten visualization notebooks
Classification errors, PDI adherence, and round drift: Official repository commit bfc069e0b9a881e10f40c2fe7ad7225a85c04d32, ten experimentation scripts
Country mismatch, artifacts, and reproducibility: Official repository commit bfc069e0b9a881e10f40c2fe7ad7225a85c04d32, data, results, README and repository root
Comprehensive visual inspection: Paper, all 22 rendered pages, including every table, plot, appendix and qualitative example