Synthetic Personality Review

Large Language Models as Superpositions of Cultural Perspectives

Trait induction and control2024arXivApproved editorial review

Authors: Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, Pierre-Yves Oudeyer

Keywords: cultural perspectives, personality stability, perspective shift, context dependency, value expression, psychological assessment, perspective controllability

Source: Open primary source (opens in a new tab)

6
Authors
32
Findings
43
Limitations
32
Evidence

Editorial summary

English

The paper proposes replacing the LLM-as-individual metaphor with an LLM-as-superposition-of-perspectives metaphor: prompts activate different behaviors, values, and traits, so a score observed in one context should not be interpreted as a stable model attribute. It evaluates forced-choice PVQ, VSM, and IPIP answers, one question per prompt, using greedy decoding and 50 permutations of answer order. For GPT-3.5-turbo-0301, five simulated conversations, five textual formats, and six Wikipedia paragraphs about music genres significantly change many scores even without explicit value instructions; the paper calls this the unexpected perspective shift effect. It also defines controllability as the normalized target-value score minus the mean non-target score. In the main comparison, the best reported settings reach 0.681 for GPT-3.5-0301 on PVQ, 0.265 for Upstage LLaMA 66B Instruct on VSM, and 0.400 for GPT-3.5-0613 on IPIP. The robust finding is that format, context, and wording substantially alter questionnaire outputs and that no induction method dominates across models and instruments. The statistical quantification, however, is not equivalent to a study of people: the 50 observations are answer-order permutations of the same deterministic model, shared across conditions, not participants or independent replications. Treating them as independent groups in ANOVA, Tukey, and Welch tests, and equating them with people for Cohen's d, rank-order stability, and ipsative stability, creates pseudoreplication and makes direct human effect-size comparisons non-equivalent. The VSM manual also explicitly says that the instrument is not for comparing individuals or a single group, which is the use made here. The preprint provides an important warning against anthropomorphizing isolated scores, but it does not establish an internal ontology of perspectives or that LLMs change more than humans. Reproducibility is partial: code, questionnaires, scripts, and an MIT license are public, but the paper-era commit cannot start without a missing module, hard-codes internal paths, declares contradictory dependencies, omits results, and depends on retired endpoints. The ICLR record was a rejected submission; the correct reference is the 2023 arXiv v3 preprint, not 2024 proceedings.

Español

El trabajo propone sustituir la metáfora del LLM como individuo por la del LLM como superposición de perspectivas: el prompt activa conductas, valores y rasgos distintos, por lo que una puntuación aislada no debería interpretarse como atributo estable del modelo. Evalúa respuestas de opción forzada a PVQ, VSM e IPIP, una pregunta por prompt, mediante decodificación codiciosa y 50 permutaciones del orden de las opciones. En GPT-3.5-turbo-0301, cinco conversaciones simuladas, cinco formatos textuales y seis párrafos de Wikipedia sobre géneros musicales cambian significativamente muchas puntuaciones, incluso sin instrucciones explícitas sobre valores; el artículo denomina a esto cambio inesperado de perspectiva. También define controlabilidad como la diferencia entre la puntuación normalizada de los valores inducidos y la media de los no inducidos. En la comparación principal, GPT-3.5-0301 alcanza 0,681 en PVQ, Upstage LLaMA 66B Instruct 0,265 en VSM y GPT-3.5-0613 0,400 en IPIP con sus mejores prompts. El hallazgo robusto es que el formato, el contexto y la formulación alteran fuertemente las respuestas de cuestionario de estos modelos, y que ningún método de inducción domina en todos. Sin embargo, la cuantificación estadística no equivale a estudiar personas: las 50 observaciones son reordenaciones de respuesta del mismo modelo determinista, compartidas entre condiciones, no participantes ni réplicas independientes. Tratarlas como grupos independientes en ANOVA, Tukey y Welch, y equipararlas con personas para calcular Cohen d, estabilidad de rangos e ipsativa, introduce pseudorreplicación y hace que las comparaciones directas con cambios humanos no sean válidas como magnitudes equivalentes. Además, el VSM advierte explícitamente que no sirve para comparar individuos ni un único grupo, justo el uso realizado. El preprint aporta una advertencia importante contra antropomorfizar puntuaciones puntuales, pero no demuestra una ontología interna de perspectivas ni que los LLM cambien más que las personas. La reproducibilidad es parcial: existe código MIT, cuestionarios y scripts, pero el commit contemporáneo al artículo no puede arrancar sin un módulo ausente, fija rutas internas, publica dependencias contradictorias, no incluye resultados y depende de endpoints retirados. El registro de ICLR fue una submission rechazada; la referencia correcta es arXiv v3 de 2023, no proceedings 2024.

Research question

To what extent do apparently unrelated contexts alter the values and traits expressed by an LLM in psychological questionnaires, and with what efficacy do different combinations of model, message, and grammatical person allow deliberately inducing a target perspective?

Method

The study administers PVQ for ten Schwartz values, VSM for six Hofstede cultural dimensions, and IPIP for the Big Five. Each item is queried separately and the model greedily chooses one letter among the options. The questionnaire is repeated with 50 fixed permutations of the answer order. The unexpected effect is studied in GPT-3.5-turbo-0301 by varying conversations, formats, and musical paragraphs; ANOVA, Tukey, and comparisons of mean change, rank order, and ipsative are applied. Controllability is calculated as the normalized mean of explicitly induced dimensions minus the mean of the remaining ones, for four prompts: second or third person in system or user message. The table compares proprietary and open models, adds Welch tests, an intensity ablation, fictional characters, and robustness to option order. The audit replicates the reading of the full PDF and reviews the repository at the commit immediately after arXiv v3.

Sample: There are no human participants or a population sample. The unit of inference is a deterministic execution of a model under a context and an option permutation. For each condition, up to 50 permutations selected with a fixed seed are used; they are transformations of the same questionnaire and not independent replicates. The main unexpected experiment uses GPT-3.5-turbo-0301; the appendix compares six models across formats, and the controllability table contains 17 model rows although text and abstract state 16.

Findings

  • The article formalizes a perspective as the context from which a model simulates a behavior and uses superposition as a metaphor, not as a mathematical mechanism of the model.
  • PVQ measures ten personal values grouped into four categories; VSM six cultural dimensions; IPIP five Big Five domains.
  • Each question is presented independently, so the model does not see previous questionnaire answers.
  • Responses are restricted to one letter through greedy decoding and bias toward the option tokens.
  • 50 fixed permutations of the option order are reused to estimate a score distribution per context.
  • Simulated conversations cover chess, history, poem, grammar, and joke, with GPT-4-0613 representing the human interlocutor.
  • The formats are chat, TOML, Python, C++, and LaTeX; the Wikipedia paragraphs cover classical, heavy metal, hip-hop, jazz, reggae, and gospel.
  • In simulated conversations, the ANOVA reports significant change in the ten PVQ values and the six VSM dimensions.
  • In formats, the analysis reports significant change in all PVQ and VSM values.
  • In musical paragraphs, all PVQ values except power change significantly; in VSM, power distance and masculinity are excluded.
  • The appendix presents analogous variations in the five IPIP scores, with exceptions noted by condition.
  • The largest standardized difference reported for GPT-3.5 is |d|=5.86, compared to |d|=0.53 in the selected human studies.
  • The lowest ipsative stability of ChatGPT is r=0.05 between chat and C++, compared to a selected human minimum mean of r=0.59.
  • Musical paragraphs produce smaller ipsative changes, with a minimum of r=0.77, than formats or conversations.
  • Across six models and five formats, the most stable model depends on the metric: OpenAssistant in mean change, GPT-3.5-0613 in rank order, and Upstage LLaMA-66B in ipsative.
  • Zephyr-7B-beta is the least stable of those six in rank order and ipsative, but not in mean change.
  • Controllability subtracts the normalized mean of non-target traits from the mean of target traits; it can vary between negative and positive values.
  • The best PVQ at 50 permutations is GPT-3.5-0301 with 0.681 using second person in system.
  • The best VSM at 50 permutations is Upstage LLaMA-66B-Instruct with 0.265 using third person in user.
  • The best IPIP is GPT-3.5-0613 with 0.400 using third person in system.
  • No combination of second or third person and system or user dominates across all models and questionnaires.
  • Models almost unable to follow the induction, such as StableLM and several older GPT-3, have controllability near zero.
  • The main table contains 17 model rows, although abstract and method speak of 16; GPT-4 and Davinci are evaluated first with only ten permutations.
  • For GPT-4, the most controllable technique is selected after ten permutations and only that one is extended to 50, while Davinci is not extended.
  • In GPT-3.5, controllability increases monotonically with intensity in PVQ, VSM, and IPIP; in OpenAssistant and StableVicuna only in PVQ and IPIP.
  • Tolkien characters produce qualitatively stereotypical profiles, but are evaluated only once and without human contrast.
  • In PVQ there appears to be an association between higher controllability and lower variance by option order; it is not clearly reproduced in VSM or IPIP.
  • The public code selects only the male PVQ form despite also including a female file.
  • The repository contemporary to arXiv v3 compiles syntactically and contains questions, scripts, license, and analysis, but no results or logs.
  • The public version of evaluate.py imports an absent module, requires OPENAI_API_KEY on loading, and contains absolute paths from the authors' infrastructure.
  • The requirements.txt simultaneously pins torch 2.0.0 and 1.13.1, duplicates packages, and installs Transformers from a branch without a fixed commit.
  • The OpenReview record corresponds to an ICLR 2024 submission with a rejection decision; arXiv classifies it as a 2023 preprint.

Limitations

  • The 50 observations are option permutations of the same deterministic system, not participants, sampling seeds, or independent replicates.
  • The same permutations are applied across perspectives, so the design is paired or repeated-measures, but is analyzed as independent groups.
  • The one-way ANOVA, Tukey, and Welch tests do not model the dependency by question and permutation; the p-values may be anti-conservative.
  • The comparison between models first chooses the best prompt for each model and then contrasts it, without correcting for the selection of the maximum.
  • GPT-4 and Davinci receive fewer permutations than the majority, and GPT-4 is extended after selecting its winning condition.
  • The Bonferroni correction covers dimensions within a questionnaire, but not the entire family of experiments, metrics, models, and exploratory comparisons.
  • Cohen d effect sizes depend on the variation across answer orders; they are not commensurable with the variation across individuals in longitudinal studies.
  • Equating permutations with participants for rank and ipsative stability lacks substantive equivalence between units of analysis.
  • The human studies are selected from different designs, ages, durations, interventions, and samples; they do not constitute a matched control.
  • Claiming that LLM changes are larger than human changes exceeds what non-comparable scales, denominators, and units allow.
  • The VSM 2013 was designed to compare means of matched cultural samples and its manual indicates that it is not suitable for comparing individuals or a single group.
  • Applying VSM formulas to a single model as if it expressed national culture compromises the construct validity of six dimensions and of its controllability results.
  • Internal reliability, factor structure, invariance, convergent, discriminant, or criterion validity of PVQ, VSM, or IPIP in LLMs is not evaluated.
  • The study uses only the male PVQ form and does not analyze the effect of gendered wording.
  • All tests are in English and do not study linguistic or cultural diversity despite the cultural scope of the argument.
  • The 'orthogonal' contexts were chosen by the authors and are not validated with judges who lack implicit associations with values.
  • Musical genres, formats, and topics have distinct content and pragmatics; the effect cannot be attributed to a specific latent perspective.
  • Each conversational topic uses a single generated and stored conversation; topic, specific wording, and trajectory are confounded.
  • Five topics, five formats, and six genres are a small convenience sample, not preregistered, of the universe of contexts.
  • Greedy decoding with a single token and logit bias measures local preference among letters, not a natural response from the model.
  • Each item is administered in isolation; the design does not evaluate coherence within a session or conversational persistence.
  • The controllability score can increase by reducing non-target dimensions even if the target trait barely rises.
  • Averaging non-target traits assumes comparability and independence between scales and hides very different profiles under the same scalar.
  • No confidence intervals for controllability are provided that reflect prompt, item, context, and model selection.
  • The inference that RLHF shifted control from user to system compares opaque OpenAI snapshots; other data, architecture, or serving changes are not controlled.
  • The main table mixes families, sizes, interfaces, and system prompt availability, so the differences do not identify causal model effects.
  • The main unexpected analysis concentrates on GPT-3.5-0301; the comparison of six models is limited to format changes.
  • OpenAI models and endpoints are withdrawn or modified, preventing exact prospective reproduction.
  • The quantum metaphor of superposition is illustrative and may suggest a formal structure that the study does not estimate.
  • The experiments observe conditioned outputs; they do not inspect activations, representations, circuits, or internal distribution of perspectives.
  • Extrapolation from value questionnaires to general knowledge, capabilities, and benchmarks is not tested experimentally.
  • There are no human participants to judge whether the induced profiles are coherent, realistic, culturally appropriate, or stereotypical.
  • The ethics section raises pluralism, rights, and alignment, but does not measure harm, stereotypes, bias, or differential effects.
  • The manuscript lacks a limitations section that acknowledges pseudoreplication, individual use of VSM, or non-equivalence with humans.
  • The code commit near arXiv v3 is not tagged as a release and there was a correction of correlations one month earlier, making it important to fix the exact revision.
  • evaluate.py imports evaluate_political_compass_csv.py, a file absent from the tree, so the entrypoint fails before executing even if that task is not used.
  • evaluate.py requires OPENAI_API_KEY on import and displays a partially masked version, an unnecessary credential exposure practice.
  • Cache and checkpoint paths are hardcoded for two of the authors' machines and require manual editing.
  • The environment is not resolvable as is due to incompatible torch versions, duplicated dependencies, and an unfixed Transformers installation.
  • No outputs, alignments.json, complete conversations for all conditions, logs, costs, or a single script that regenerates each table are published.
  • There is no CI or end-to-end testing; compileall only confirms syntax and does not detect broken imports, dependencies, APIs, or paths.
  • The MIT license retains only the original copyright of Dan Hendrycks, a reflection of the MMLU origin, without explicitly clarifying the authorship of the study's modifications.
  • The submission was rejected at ICLR; it should be read as an open preprint, not as an accepted article in proceedings.

What the study does not establish

  • It does not demonstrate that an LLM has personality, values, identity, subjective experience, or preferences of its own.
  • It does not demonstrate that an internal superposition of perspectives exists in a mathematical or neurocomputational sense.
  • It does not identify how many perspectives exist, how they are represented, or whether they are discrete, continuous, or compositional.
  • It does not demonstrate that a questionnaire score corresponds to the same construct in humans and LLMs.
  • It does not demonstrate that VSM measures culture in an individual model.
  • It does not demonstrate that LLMs change more than humans on a comparable scale.
  • It does not establish test-retest stability under sampling, independent sessions, or different dates.
  • It does not establish generalization to other languages, cultures, domains, open tasks, or deployments.
  • It does not demonstrate that the chosen contexts are psychologically orthogonal to values or personality.
  • It does not demonstrate that the effects are due to perspectives and not to tokenization, format, lexical associations, or option biases.
  • It does not demonstrate coherence across responses within a conversation, because each item is queried separately.
  • It does not demonstrate that the most controllable prompt produces a complete, stable, or humanly recognizable persona.
  • It does not demonstrate that higher controllability is beneficial, safe, or desirable.
  • It does not demonstrate that RLHF causes the differences between GPT-3.5 snapshots.
  • It does not demonstrate that one induction method is universally superior.
  • It does not establish fair comparisons between models with different interfaces, sizes, and numbers of permutations.
  • It does not validate general knowledge or capability on benchmarks other than the questionnaires studied.
  • It does not demonstrate the absence of bias, stereotypes, or harm when inducing cultural perspectives.
  • It does not allow exactly reproducing the current tables with the published entrypoint and artifacts without repairs and historical services.
  • It does not constitute evidence of acceptance at ICLR 2024 or of favorable peer review.
  • It does not authorize interpreting the figures as a psychological, cultural, clinical, or occupational diagnosis.

Traceability

Scope: Full text

Version: arXiv 2307.07870v3, 7 Nov 2023, 34 pages; paper-era code snapshot value_stability commit 70611f2b97bc7784b4882981ad23a04f60a4b08d

Consulted source: https://arxiv.org/pdf/2307.07870v3

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, venue-status, psychometric, VSM-level-of-analysis, experimental-design, statistical-independence, code-artifact, reproducibility and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4-0314
  • GPT-3.5-turbo-0301
  • GPT-3.5-turbo-0613
  • text-davinci-003
  • GPT-3.5-turbo-instruct-0914
  • Upstage LLaMA-2-70B-Instruct
  • Upstage LLaMA-66B-Instruct
  • Zephyr-7B-beta
  • OpenAssistant RLHF LLaMA-30B
  • StableLM 7B
  • LLaMA-65B
  • StableVicuna 13B
  • RedPajama INCITE 7B Chat
  • RedPajama INCITE 7B Instruct
  • Curie
  • Babbage
  • Ada
  • GPT-4-0613 como interlocutor simulado

Instruments and metrics

  • Portrait Values Questionnaire (PVQ)
  • Values Survey Module 2013 (VSM)
  • Goldberg IPIP representation of NEO-PI-R Big Five domains
  • Perspective controllability score
  • One-way ANOVA with Bonferroni threshold
  • Tukey HSD post-hoc tests
  • Welch independent-samples t-tests
  • Cohen's d
  • Pearson rank-order stability
  • Ipsative value-profile correlation
  • Variance over answer-order permutations

Data used

  • PVQ male-form questionnaire items bundled in the public repository
  • VSM 2013 24-item questionnaire
  • IPIP 50-item Big Five questionnaire
  • Five simulated-conversation topics
  • Five textual renderings of the questionnaire
  • First Wikipedia paragraphs for six music genres
  • Published aggregate results from five human longitudinal or priming studies

Evidence and location

  • Authorship, year, and current version: arXiv 2307.07870: six named authors, submitted 15 Jul 2023, v3 dated 7 Nov 2023, classified as Preprint
  • Editorial status: OpenReview forum 1FWDEIGm33: ICLR 2024 submission 9417, modified 11 Feb 2024; decision recorded as Reject
  • Full audited source: .cache/editorial-sources/article-091/source.pdf; arXiv 2307.07870v3; 34 pages; sha256 b73ac3e601b216016551a5b81cc0c1d73795b0498eb7a422419a1dab9f95ef8f
  • Metaphor and research questions of the work: Full text pp. 1-3, Abstract and Introduction
  • PVQ, VSM, and IPIP: Full text pp. 3-4, Section 3; pp. 16-18, Appendix A
  • One question per prompt and 50 permutations: Full text pp. 4-5, Section 3 and Figure 1
  • Definition of controllability: Full text p. 5, Equation 1 and Figure 2
  • Conversations, formats, and musical paragraphs: Full text pp. 5-7, Section 4.1 and Figure 3
  • Reported significance of contextual change: Full text p. 6, Section 4.1: ANOVA, Bonferroni and Tukey results
  • Unexpected IPIP results: Full text p. 32, Appendix Figure 13
  • Main controllability results: Full text pp. 8-9, Section 4.2 and Table 1
  • Welch comparisons: Full text p. 28, Appendix Table 9
  • Cohen d comparison with human studies: Full text pp. 18-20, Appendix C.1 and Table 3
  • Analogy between participants and permutations: Full text p. 19, Appendix C.1; p. 21, Appendix C.3
  • Rank stability: Full text pp. 19-21, Appendix C.2 and Table 4
  • Ipsative stability: Full text pp. 21-23, Appendix C.3 and Table 5
  • Unexpected comparison of six models: Full text pp. 22-23, Appendix D and Table 6
  • Fictional characters: Full text pp. 24-25, Appendices E.1-E.2 and Figure 4
  • Gradualness of induction: Full text pp. 25-26, Appendix E.4 and Table 7
  • Sensitivity to option order: Full text pp. 26-28, Appendix E.5, Table 8 and Figure 7
  • Declared ethics and reproducibility: Full text pp. 9-10, Ethics Statement and Reproducibility
  • VSM not valid for individuals: Official VSM 2013 Manual, Section 2: not for comparing individuals; official Research and VSM guidance: requires matched samples from two or more societies
  • Repository and article commit: https://gitlab.inria.fr/gkovac/value_stability commit 70611f2b97bc7784b4882981ad23a04f60a4b08d, 8 Nov 2023; checked 15 Jul 2026
  • Identical permutations and fixed seed: Paper-era public code evaluate.py lines 1633-1668: random.seed(1), shared sampled permutations
  • Male PVQ form: Paper-era public code evaluate.py lines 1420-1428; bar_viz.py lines 536-579
  • ANOVA over permutations as groups: Paper-era public code visualization_scripts/bar_viz.py lines 742-803
  • Independent Welch over shared results: Paper-era public code models_stat_test.py lines 237-261
  • Absent import and credential: Paper-era public code evaluate.py lines 109-113; evaluate_political_compass_csv.py absent from git tree
  • Internal paths: Paper-era public code evaluate.py lines 15-21 and 120; README asks user to edit llama_dir
  • Contradictory dependencies: Paper-era public code requirements.txt lines 1-103: torch 2.0.0 and 1.13.1, duplicated packages; README installs Transformers from unfixed git HEAD
  • Absence of reproducible results: Paper-era public repository tree: no results directories, alignments, generated conversations, CI or end-to-end test artifacts
  • Comprehensive reading and visual verification: All 34 pages rendered and inspected, including Figures 1-16, Tables 1-9, equations, appendices, prompts and references; checked 15 Jul 2026