Synthetic Personality Review

Exploring the Impact of Language Switching on Personality Traits in LLMs

Evaluation and psychometric validity2025ACL AnthologyApproved editorial review

Authors: Jacopo Amidei, Jose Gregorio Ferreira De Sá, Rubén Nieto Luna, Andreas Kaltenbrunner

Keywords: Large Language Models, Personality Traits, Language Switching, GPT-4o, Eysenck Personality Questionnaire-Revised (EPQR-A), Cross-language analysis

Source: Open primary source (opens in a new tab)

4
Authors
27
Findings
82
Limitations
14
Evidence

Editorial summary

English

This COLING 2025 paper studies multilingual response patterns from one reproducible version, gpt-4o-2024-05-13, with temperature=1 and top-p=1. In each condition the model completes the EPQR-A one hundred times: 24 binary questions yielding 0–6 scores for Extraversion (E), Neuroticism (N), Psychoticism (P), and Lie/social desirability (L). The first experiment uses published English, Hebrew, Brazilian Portuguese, Slovak, Spanish, and Turkish adaptations, always with English instructions. Repetitions describe prompt-conditioned output distributions, but they are neither people nor one hundred independent models.

Across the original versions, means differ modestly in absolute scale but several pairwise comparisons are significant. E ranges from 3.05 in English to 4.45 in Slovak; N from 1.19 in Turkish to 3.11 in English; P from .57 in Turkish to 1.59 in Slovak; and L from 4.79 in Spanish to 5.54 in Brazilian Portuguese. The paper interprets these differences as a possible language-switching effect. Direct evidence, however, establishes only that the same snapshot produces different EPQR-A distributions under different questionnaires and languages; it identifies no internal personality and does not reproduce human Cultural Frame Switching.

The second experiment tries to separate language from translation by comparing published adaptations with Google Translate versions produced from English. The prose concludes that differences are minimal and that E and N point more to language than translation. Table 1 itself requires more caution: 10 of 20 original-versus-translation contrasts are marked significant at p≤.01. They cover three Hebrew scales, P in Brazilian Portuguese, P and L in Slovak, E, N, and L in Spanish, and P in Turkish. Without human translation validation, measurement equivalence, or correction for the large number of tests, the design cannot rule out translation or firmly separate language, wording, and instrument effects.

The third experiment uses the English EPQR-A and directs GPT-4o to impersonate a “typical” native of the United States, Australia, the United Kingdom, Canada, or Ireland, reflect cultural norms, and explain its choices. Relative to generic English, E rises from 3.05 to between 4.75 and 5.97; N ranges from 1.05 in Australia to 3.70 in the UK; P from .70 in the UK to 1.83 in Ireland; and Canada has the highest L score, 5.22. Explanations draw on clichés about American individualism, Australian fair go, British politeness, Canadian harmony, or Irish sociability. The robust finding is sensitivity to an explicitly induced national role; descriptive comparisons with historical human studies do not establish alignment between human and model personalities.

Internal consistency is uneven. In language conditions, E and N usually exceed alpha .70, but P never does and reaches -.049 in English and -.258 in Slovak; L ranges from .073 to .861. Under national roles, N exceeds .70 in all five countries, while P lies between .323 and .380 and L reaches -.100 in Canada. The paper publishes no data, code, effect sizes, intervals, exact p-values, or multiple-comparison correction, and tests no factor structure, invariance, or external behavior. Its useful contribution is showing that language, translated version, and cultural persona alter GPT-4o output profiles; “personality” here should mean a prompt-dependent psychometric score.

Español

Este trabajo de COLING 2025 estudia patrones de respuesta multilingües de una única versión reproducible, gpt-4o-2024-05-13, con temperatura=1 y top-p=1. En cada condición el modelo completa cien veces el EPQR-A: 24 preguntas binarias que producen puntuaciones de 0 a 6 en Extraversión (E), Neuroticismo (N), Psicoticismo (P) y Mentira o deseabilidad social (L). El primer experimento usa adaptaciones publicadas en inglés, hebreo, portugués brasileño, eslovaco, español y turco, siempre con instrucciones en inglés. Las repeticiones permiten describir la distribución de salidas condicionadas por esos prompts, pero no son personas ni cien modelos independientes.

En las versiones originales, las medias difieren poco en escala absoluta pero varias comparaciones por pares son significativas. E va de 3,05 en inglés a 4,45 en eslovaco; N, de 1,19 en turco a 3,11 en inglés; P, de 0,57 en turco a 1,59 en eslovaco; y L, de 4,79 en español a 5,54 en portugués brasileño. El artículo interpreta estas diferencias como un posible efecto de cambio de idioma. La evidencia directa, sin embargo, solo establece que el mismo snapshot genera distribuciones EPQR-A distintas bajo cuestionarios y lenguas distintas; no identifica una personalidad interna ni reproduce Cultural Frame Switching humano.

El segundo experimento intenta separar idioma y traducción comparando las adaptaciones publicadas con traducciones automáticas desde inglés mediante Google Translate. La conclusión del texto es que las diferencias son mínimas y que E y N apuntan más al idioma que a la traducción. La propia Table 1 exige más cautela: 10 de los 20 contrastes original-traducción están marcados como significativos a p≤0,01. Incluyen tres escalas en hebreo, P en portugués brasileño, P y L en eslovaco, E, N y L en español y P en turco. Sin validación humana de las traducciones, equivalencia de medida o corrección por la gran cantidad de pruebas, el diseño no permite descartar la traducción ni separar con firmeza lengua, redacción e instrumento.

El tercer experimento usa el EPQR-A inglés y ordena a GPT-4o personificar a un nativo “típico” de Estados Unidos, Australia, Reino Unido, Canadá o Irlanda, reflejar normas culturales y explicar sus decisiones. Frente al inglés genérico, E sube de 3,05 a entre 4,75 y 5,97; N va de 1,05 en Australia a 3,70 en Reino Unido; P, de 0,70 en Reino Unido a 1,83 en Irlanda; y Canadá alcanza el mayor L, 5,22. Las explicaciones recurren a clichés sobre individualismo estadounidense, el fair go australiano, cortesía británica, armonía canadiense o sociabilidad irlandesa. El hallazgo sólido es sensibilidad al rol nacional explícitamente inducido; las comparaciones descriptivas con estudios humanos históricos no demuestran alineamiento entre personalidades humanas y del modelo.

La consistencia interna es desigual. En las condiciones de idioma, E y N suelen superar alfa 0,70, pero P nunca lo hace y llega a -0,049 en inglés y -0,258 en eslovaco; L oscila entre 0,073 y 0,861. En los roles nacionales, N supera 0,70 en los cinco países, mientras P queda entre 0,323 y 0,380 y L llega a -0,100 en Canadá. El artículo no publica datos, código, efectos, intervalos, p exactas ni corrección por comparaciones múltiples, y no prueba estructura factorial, invariancia o comportamiento externo. Su aportación útil es mostrar que idioma, versión traducida y persona cultural alteran los perfiles de salida de GPT-4o; “personalidad” debe entenderse aquí como puntuación psicométrica dependiente del prompt.

Research question

Do the EPQR-A score distributions of GPT-4o change when the language changes, how much of that difference can be attributed to translation, and what happens when a national role is induced within the same language?

Method

Three experiments with gpt-4o-2024-05-13, temperature 1 and top-p 1. The model produced one hundred complete questionnaires per condition. SRQ1 compared published EPQR-A adaptations in six languages; SRQ2 compared five of those adaptations with Google Translate translations from English; SRQ3 compared generic English with five anglophone national roles induced through a system role and asked for explanations. Means, deviations, two-sided pairwise Mann–Whitney U tests and Cronbach's alpha were calculated. The audit read and rendered the nine pages, verified Tables 1–2, complete prompts, Tables A1–A2, notes and bibliography, and contrasted metadata and abstract with ACL Anthology.

Sample: There are no human participants. The sample consists of one hundred stochastic generations from the same snapshot for each language, translation or national role condition: six original conditions, five translated and five national, approximately 1,600 complete profiles and 38,400 binary responses. These generations can characterize the output distribution under the fixed protocol, but do not represent a population of persons or a sample of independently trained models.

Findings

  • The paper was published in the main proceedings of COLING 2025, pages 2370–2378, with license CC BY 4.0.
  • The study identifies exactly gpt-4o-2024-05-13 and sets temperature and top-p to 1.
  • Each condition was repeated one hundred times with the 24-item EPQR-A.
  • SRQ1 used six published adaptations and kept the instructions in English.
  • In original questionnaires, E ranged from 3.05±2.26 in English to 4.45±2.04 in Slovak.
  • In original questionnaires, N ranged from 1.19±1.41 in Turkish to 3.11±2.37 in English.
  • In original questionnaires, P ranged from 0.57±0.73 in Turkish to 1.59±0.74 in Slovak.
  • In original questionnaires, L ranged from 4.79±0.82 in Spanish to 5.54±0.95 in Brazilian Portuguese.
  • Table 1 marks numerous pairwise differences between languages on the four scales.
  • SRQ2 compared five published adaptations with machine translations from English.
  • Ten of twenty original-translation contrasts are marked as significant at p≤0.01.
  • The Hebrew translation differed from the original in N, P and L.
  • The Spanish translation differed from the original in E, N and L.
  • The Slovak translation differed from the original in P and L.
  • The Brazilian and Turkish translations differed from their originals in P.
  • The claim of minimal differences by translation is stronger than the evidence in Table 1.
  • SRQ3 explicitly induced five anglophone national characters and asked to explain the responses.
  • Extraversion of the national roles ranged from 4.75 to 5.97 versus 3.05 in generic English.
  • Australia had N=1.05±1.62 and United Kingdom N=3.70±1.72.
  • United Kingdom had P=0.70±0.66 and Ireland P=1.83±0.47.
  • Canada reached the highest L, 5.22±0.61.
  • The generated explanations use national stereotypes, a fact acknowledged by the authors.
  • In original conditions, alpha of E was 0.798–0.909 and alpha of N 0.711–0.893.
  • P showed low consistency in all conditions and negative alpha in English (-0.049) and Slovak (-0.258).
  • In the national roles, alpha of N was 0.757–0.862, P was 0.323–0.380 and L reached -0.100 in Canada.
  • Table A1 shows acceptable alphas of L in some conditions and of E in United Kingdom and Canada that the reliability summary in the text does not mention.
  • The study demonstrates sensitivity of the output profile to language, translation and induced persona, not a stable internal personality.

Limitations

  • Only one provider and one model are evaluated.
  • Only the snapshot gpt-4o-2024-05-13 is evaluated.
  • It is not replicated with other versions of GPT-4o.
  • It is not replicated with models from other families.
  • No open models that allow inspecting weights or training are used.
  • The exact dates of the API calls are not reported.
  • No region, account or service endpoint is reported.
  • No SDK version or execution code is reported.
  • Generation seeds are not recorded.
  • Architecture, cutoff and prompt date placeholders are not filled in the published appendix.
  • The code is not published.
  • The raw responses are not published.
  • A table with exact p-values is not published.
  • Effect sizes for the comparisons are not published.
  • Confidence intervals are not published.
  • No preregistration is reported.
  • No prior directional hypothesis is formulated for each language and scale.
  • The use of one hundred repetitions is not justified by power.
  • The profiles are repetitions of the same model and not independently trained models.
  • Inference can only refer to the conditional distribution of this configuration.
  • Technical independence between calls or cache and rate limiting effects is not tested.
  • Scores are bounded between 0 and 6 and contain numerous ties.
  • It is not explained how ties were handled in Mann–Whitney U.
  • Hierarchical models that separate item, language and repetition are not applied.
  • Dozens of pairwise comparisons are performed in each table.
  • No correction for multiple comparisons is declared.
  • At p≤0.05 false positives can be expected even under absence of effect.
  • Significance is coded through many flags and underlines without complete statistics.
  • A global multivariate criterion before pairwise contrasts is not reported.
  • Significant differences may be small on a six-point scale.
  • The EPQR-A was validated for human self-report, not for generative models.
  • The four-scale factorial structure is not validated in the model responses.
  • Measurement invariance between languages is not tested.
  • Invariance between original questionnaires and machine translations is not tested.
  • Temporal test-retest with separate dates is not studied.
  • Alpha measures covariation between items, not stability of model personality.
  • Alpha is estimated over stochastic outputs conditioned by the same prompt.
  • P presents very low or negative alpha in all families of conditions.
  • L presents highly variable alpha and becomes negative.
  • The reliability prose omits some alphas greater than 0.70 present in Table A1.
  • Convergent validity with another inventory is not evaluated.
  • Discriminant validity is not evaluated.
  • External behavior is not used as a criterion.
  • It is not tested whether profiles predict responses in free conversation.
  • The L scale does not directly equate to honesty or integrity.
  • The qualitative interpretation treats L as honesty and conformity in a simplified way.
  • Published adaptations may differ in wording and psychometric properties.
  • Language and the specific version of the questionnaire are confounded in SRQ1.
  • SRQ2 uses only Google Translate.
  • No human translators are involved in the evaluation of the machine versions.
  • No back-translation or quality scoring is performed.
  • Ten of twenty original-translation contrasts are significant, in tension with the description of minimal variation.
  • Instructions remain in English although the questionnaire changes language.
  • It is not tested whether translating the instructions as well alters the results.
  • Tokenization, length or lexical frequency between languages is not controlled.
  • Robustness to paraphrasing of items is not tested.
  • Robustness to question order is not tested.
  • Robustness to different temperature or top-p is not tested.
  • Memorization or prior exposure to EPQR-A and its adaptations is not ruled out.
  • SRQ3 explicitly instructs the model to produce typical national behavior.
  • National differences are induced by the role and are not spontaneous.
  • The prompt asks for coherence with cultural norms and expressions, favoring stereotypes.
  • The prompt combines country, culture, personality and user expectations.
  • There is no neutral national condition with the same prompt length.
  • Whole countries are treated as homogeneous cultures.
  • Regions, classes, ethnicities, ages or individual variation are not modeled.
  • Only five Western anglophone countries are included.
  • Dialects or regional varieties of English are not tested.
  • Explanations are generated after the responses and may be post hoc rationalizations.
  • The qualitative evaluation of explanations does not describe a coding scheme.
  • Evaluators, blinding or inter-evaluator agreement are not reported.
  • Table A2 publishes only two examples per country.
  • Comparisons with humans are descriptive and come from distinct historical studies.
  • No contemporary human sample participates under the same protocol.
  • Direct tests of equivalence between model outputs and human data are not applied.
  • The highlighted human similarities are partial and discrepancies also exist that are not formalized.
  • Cultural Frame Switching is a theory about human cognition and its transfer to the LLM is not validated.
  • The study does not identify internal mechanisms that explain the differences.
  • It does not separate multilingual training, alignment, tokenization and prompt obedience.
  • The results are a 2024 snapshot and do not describe current GPT-4o.
  • There is no independent replication.
  • The article is presented as preliminary and acknowledges the need for more languages, countries, tests and models.

What the study does not establish

  • It does not demonstrate that GPT-4o has an internal human personality.
  • It does not demonstrate cognitive Cultural Frame Switching in a model.
  • It does not demonstrate that changing language causes an internal psychological change.
  • It does not rule out that the differences come from translation or wording.
  • It does not demonstrate equivalence of the EPQR-A between languages for LLMs.
  • It does not validate EPQR-A as a measure of artificial personality.
  • It does not demonstrate adequate reliability on the four scales.
  • It does not demonstrate statistical alignment with human populations.
  • It does not demonstrate that national roles represent people from those countries.
  • It does not demonstrate that stereotypes appear without an instruction that requests them.
  • It does not demonstrate stability across snapshots, dates or configurations.
  • It does not demonstrate that profiles predict conversational behavior.
  • It does not identify whether the effect comes from training, tokenization or alignment.
  • It does not allow generalization to other models, questionnaires or cultures.
  • It does not justify interpreting each scored profile as a persistent identity.

Traceability

Scope: Full text

Version: COLING 2025 main conference paper, ACL Anthology 2025.coling-main.162, pp. 2370–2378; CC BY 4.0

Consulted source: https://aclanthology.org/2025.coling-main.162/

Review: Codex full-text, visual, bilingual-fidelity, multilingual-method, psychometric-validity, multiple-testing, prompt-induction, stereotype-claim and source-transparency audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • OpenAI gpt-4o-2024-05-13 with temperature 1 and top-p 1

Instruments and metrics

  • Eysenck Personality Questionnaire-Revised Abbreviated, EPQR-A: 24 binary items
  • EPQR-A Extraversion scale: six items, score 0–6
  • EPQR-A Neuroticism scale: six items, score 0–6
  • EPQR-A Psychoticism scale: six items, score 0–6
  • EPQR-A Lie or social-desirability scale: six items, score 0–6
  • Published English, Hebrew, Brazilian Portuguese, Slovak, Spanish and Turkish EPQR-A adaptations
  • Google Translate versions from English into five languages
  • Five country-persona prompts for USA, Australia, UK, Canada and Ireland
  • Two-sided Mann–Whitney U pairwise comparisons
  • Cronbach alpha across 100 generated response profiles per condition
  • Qualitative inspection of generated country explanations

Data used

  • Six original-language conditions with 100 generated EPQR-A profiles per condition
  • Five Google-translated conditions with 100 generated EPQR-A profiles per condition
  • Five English-speaking country-persona conditions with 100 generated EPQR-A profiles per condition
  • Approximately 1,600 generated 24-item profiles across the 16 reported conditions
  • Aggregate means and standard deviations in Tables 1–2
  • Condition-level Cronbach alpha values in Table A1
  • Two selected generated explanations per country in Table A2
  • No public raw-response or analysis dataset identified

Evidence and location

  • Bibliographic identity and license: ACL Anthology 2025.coling-main.162; COLING 2025; pp. 2370–2378; CC BY 4.0
  • Complete PDF inspected: .cache/editorial-sources/article-081/source.pdf; ACL Anthology; sha256 ef29b16f278ad051e4600d806b2062c9fda8beb1f3fcf3e733b9b6c0dbdbf90f
  • Model and generation parameters: Footnote 1, paper p. 2370: gpt-4o-2024-05-13, temperature=1, top-p=1
  • Design of three experiments and one hundred trials: Methods, Experimental Setups, paper p. 2372
  • Means, deviations and differences of languages and translations: Table 1 and SRQ1–SRQ2 Results, paper pp. 2372–2373
  • Ten significant original-translation contrasts: Table 1 dagger markers, paper p. 2372; arithmetic audit 15 Jul 2026
  • Results of five national roles: Table 2 and SRQ3 Results, paper p. 2373
  • Acknowledged use of stereotypes: SRQ3 qualitative discussion, paper p. 2373; Abstract and Conclusions
  • Language and national role prompts: Appendix A, paper p. 2377
  • Complete alpha values: Table A1, paper p. 2377
  • Examples of national explanations: Table A2, paper p. 2378
  • Recognized limits and future extensions: Limitations and future work, paper pp. 2374–2375
  • Absence of published reproducible artifacts: ACL Anthology record and complete paper inspected 15 Jul 2026; no code or data link reported
  • Comprehensive reading and visual verification: All 9 PDF pages rendered and inspected, including Tables 1–2 and Appendices A–C; checked 15 Jul 2026