Synthetic Personality Review

Personality Traits in Large Language Models

Evaluation and psychometric validity2023arXivApproved editorial review

Authors: Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, Maja Matarić

Keywords: Computation and Language, Artificial Intelligence, Computers and Society, Human-Computer Interaction

Source: Open primary source (opens in a new tab)

9
Authors
6
Findings
6
Limitations
7
Evidence

Editorial summary

English

The paper develops a psychometric protocol for deciding whether a model's Big Five questionnaire responses are internally consistent, converge across instruments, remain distinguishable from other traits, and relate to external criteria; it also tests whether those patterns can be prompted and observed in open-ended text. Eighteen variants from the PaLM, Llama 2, Mistral, Mixtral, and GPT families are evaluated. The measurement study separately administers the 300 IPIP-NEO items, the 44-item BFI, and 11 external-criterion subscales. Every item is combined with 50 PersonaChat biographical descriptions, five item instructions, and five response formats, yielding 1,250 paired profiles and 523,750 responses per model. Reliability is assessed with Cronbach's alpha, Guttman's lambda-6, and McDonald's omega; convergent and discriminant validity use a multitrait-multimethod matrix, while criterion validity uses affect, aggression, values, and creativity measures. Base models generally produce unreliable responses with negligible convergent validity. Instruction tuning is the most consistent source of improvement, and larger instruction-tuned variants usually perform better. Flan-PaLM 540B and GPT-4o reach mean convergent correlations of 0.90 and mean discriminant gaps of 0.51 and 0.48. For trait induction, the authors create 104 lexical markers and nine intensity levels. Single-trait shaping is strongly monotonic in eleven of twelve tested models, although small models cover a much narrower effective score range. Simultaneously shaping all five traits is harder: Flan-PaLM 540B and GPT-4o achieve the largest average median separations, about 2.5 scale points, and openness is the most resistant domain. In a downstream task, four models generate social-media updates conditioned on the same profiles. Questionnaire scores correlate 0.67 on average with language-based personality estimates, while requested levels correlate about 0.68–0.82 with observed textual estimates. These findings support controllable output regularities under the study's protocol, not an internal human-like personality. External validity remains limited by English-only tests, structured persona prompts, instrument selection, constrained item scoring, and a single family of generative tasks.

Español

El trabajo propone un protocolo psicométrico para decidir si las respuestas de un modelo a inventarios Big Five son consistentes, convergen entre instrumentos, se distinguen de otros rasgos y se relacionan con criterios externos; además prueba si esos patrones pueden inducirse mediante prompts y aparecer en texto abierto. Evalúa 18 variantes de PaLM, Llama 2, Mistral, Mixtral y GPT. Para la medición administra por separado los 300 ítems del IPIP-NEO, los 44 del BFI y 11 subescalas externas. Cada ítem se combina con 50 descripciones biográficas de PersonaChat, cinco instrucciones y cinco formatos de respuesta, lo que produce 1.250 perfiles emparejados y 523.750 respuestas por modelo. La fiabilidad se contrasta con alfa de Cronbach, lambda-6 de Guttman y omega de McDonald; la validez convergente y discriminante se examina con una matriz multirrasgo-multimétodo y la validez de criterio con afecto, agresión, valores y creatividad. Los modelos base ofrecen respuestas generalmente no fiables y sin validez convergente, mientras que el ajuste de instrucciones es el factor más consistente de mejora; dentro de modelos instruidos, el tamaño suele mejorar los resultados. Flan-PaLM 540B y GPT-4o alcanzan una correlación convergente media de 0,90 y diferencias discriminantes de 0,51 y 0,48. Para inducir rasgos, los autores construyen 104 pares de marcadores léxicos y nueve niveles de intensidad. La inducción de un solo rasgo funciona con gran monotonicidad en once de doce modelos, pero el rango efectivo es mucho menor en sistemas pequeños. Inducir los cinco rasgos simultáneamente es más difícil: Flan-PaLM 540B y GPT-4o logran las mayores separaciones medias, alrededor de 2,5 puntos, y apertura es el dominio más resistente. En una tarea posterior, cuatro modelos generan actualizaciones de redes sociales condicionadas por los mismos perfiles; las puntuaciones del cuestionario correlacionan en promedio 0,67 con estimaciones lingüísticas de personalidad, y los niveles pedidos correlacionan entre 0,68 y 0,82 con el texto observado. Es evidencia de regularidades de salida bajo este protocolo, no de personalidad humana interna. La validez externa queda limitada por pruebas en inglés, prompts estructurados, elección de instrumentos, puntuación restringida de ítems y una única familia de tarea generativa.

Research question

Can LLM-synthesized Big Five traits be measured reliably, validly, and with practical meaning; can they be induced to desired levels; and do questionnaire measures predict the expression of those traits in generated text?

Method

Psychometric and experimental study in three phases. First, 18 variants from five families respond to IPIP-NEO and BFI under 1,250 reproducible combinations of 50 biographies, five instructions, and five postambles; three reliability indices, convergent/discriminant validity, and correlations with 11 external criteria are calculated. Second, 104 adjective markers and nine intensities induce each domain separately in 2,250 profiles and all five domains simultaneously in 1,600 profiles; ordinal correlations and median separation are evaluated. Third, Flan-PaLM 540B, Llama 2-Chat 70B, Mixtral 8x7B Instruct, and GPT-4o generate social network updates, scored by Apply Magic Sauce and linked to their IPIP-NEO scores.

Sample: Eighteen variants from five families. The psychometric phase uses 1,250 prompt profiles and 419 items, equivalent to 523,750 responses per model. Induction uses 2,250 profiles for individual traits and 1,600 for five concurrent traits. The open task reuses 2,250 profiles across four models; it does not recruit new human participants or new human annotators.

Findings

  • The base models PaLM, Llama 2, Mistral, and Mixtral fail in reliability and convergent validity; their instructed variants improve markedly, which associates post-training with the ability to follow the psychometric format.
  • Flan-PaLM 540B and GPT-4o achieve the highest mean IPIP-NEO–BFI convergence, r = 0.90; their mean differences between convergent and discriminant correlations are 0.51 and 0.48.
  • Induction of a single domain shows very strong correlations between requested level and observed score in eleven of twelve models, but the separation between extremes grows substantially with capability; Mistral 7B Instruct only achieves a mean separation of 0.78.
  • Simultaneous induction is less controllable: Flan-PaLM 540B and GPT-4o achieve mean separations of 2.53 and 2.52, compared to 0.62 in Flan-PaLM 8B and 0.21 in Mistral 7B Instruct; openness is the domain with the lowest separation in several models.
  • Across four models, IPIP-NEO scores correlate 0.67 on average with trait estimates in generated social text; correlations between requested level and observed text range approximately between 0.68 and 0.82 per model.
  • Factorial exploration is imperfect: even Flan-PaLM 540B and GPT-4o load items onto the expected human factors in only just over 72% of cases, although their factorial and classical scores correlate strongly.

Limitations

  • The inventories are human instruments administered only in English and the models were trained primarily on Western data; cultural or linguistic invariance is not established.
  • The 50 biographies are duplicated deterministically to create variance and do not represent random individuals, so the authors consider the factorial evidence exploratory, not confirmatory.
  • Each item is scored in isolation using probabilities or restricted decoding; this procedure improves reproducibility, but does not reproduce a conversation, a complete test, or the usual generative mode of use.
  • Criterion validity remains based on other questionnaires and the external test uses a single type of text, an automated personality API, and independent turns; it does not cover longitudinal behavior or multi-turn dialogue.
  • Size, architecture, training volume, instruction tuning, alignment, and quantization are not fully separated, so the study does not identify a single cause of the differences between models.
  • The size and quantization details of the GPT models are not public and the selection of versions is fixed to 2024 snapshots, which limits extrapolation to later systems.

What the study does not establish

  • It does not demonstrate that an LLM possesses personality, motivations, affects, agency, or a stable identity comparable to humans; it measures patterns synthesized in responses and text.
  • It does not prove that Big Five scores generalize to any prompt, language, culture, task, or prolonged interaction.
  • It does not allow attributing the improvements solely to size or solely to instruction tuning, because the variants differ in more dimensions.
  • It does not demonstrate that inducing profiles produces safer, fairer, or more beneficial systems; the article itself identifies risks of persuasion, anthropomorphization, and malicious use.
  • It does not validate all human instruments for models nor establish metric equivalence between an LLM score and a human score.

Traceability

Scope: Full text

Version: arXiv:2307.00184v4

Consulted source: https://arxiv.org/pdf/2307.00184

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • PaLM 62B
  • Flan-PaLM 8B
  • Flan-PaLM 62B
  • Flan-PaLM 540B
  • Flan-PaLMChilla 62B
  • Llama 2 7B
  • Llama 2 13B
  • Llama 2 70B
  • Llama 2-Chat 7B
  • Llama 2-Chat 13B
  • Llama 2-Chat 70B
  • Mistral 7B v0.1
  • Mistral 7B Instruct v0.1
  • Mixtral 8x7B v0.1
  • Mixtral 8x7B Instruct v0.1
  • GPT-3.5 Turbo (gpt-3.5-turbo-0125)
  • GPT-4o mini (gpt-4o-mini-2024-07-18)
  • GPT-4o (gpt-4o-2024-08-06)

Instruments and metrics

  • IPIP-NEO (300 items)
  • Big Five Inventory (44 items)
  • Positive and Negative Affect Schedule
  • Buss-Perry Aggression Questionnaire
  • Revised Portrait Values Questionnaire
  • Short Scale of Creative Self
  • Apply Magic Sauce personality prediction API
  • Cronbach's alpha, Guttman's lambda-6 and McDonald's omega
  • Multitrait-multimethod validity matrix

Data used

  • PersonaChat true-cased dataset
  • Public Personality in LLMs response dataset
  • Model-generated social-media status updates

Evidence and location

  • Question, scope, and protocol of 18 models: Paper, pp. 1–7, Summary and sections 2.1–2.2; Table 2
  • Reliability and validity by tuning and size: Paper, pp. 7–10 and 29–39; Tables 10–12 and Figure 8
  • Design and results of individual and simultaneous induction: Paper, pp. 10–14 and 37–45; Tables 14–21
  • Validation in generated social text: Paper, pp. 15–17 and 42–50; Figure 4 and Table 4
  • Cultural, administration, and external validity limits: Paper, pp. 17–19, section 5.1, Limitations and Future Work
  • Exploratory factorial evidence and methodological caveats: Paper, pp. 29–34, Appendix I and Figure 6
  • Ethical implications, data, code, and conflicts: Paper, pp. 18–19 and 52–53, sections 5.2–5.3 and appendices P–S