Synthetic Personality Review

A psychometric framework for evaluating and shaping personality traits in large language models

Evaluation and psychometric validity2025NatureApproved editorial review

Authors: Gregory Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, Maja Matarić

Keywords: Large Language Models, Synthetic Personality, Big Five, IPIP-NEO, Big Five Inventory, Psychometric Reliability, Construct Validity, Convergent Validity, Discriminant Validity, Criterion Validity, Personality Shaping, Prompt-conditioned Measurement, Instruction Tuning

Source: Open primary source (opens in a new tab)

9
Authors
10
Findings
15
Limitations
11
Evidence

Editorial summary

English

This study develops one of the most comprehensive empirical frameworks available for evaluating synthetic personality in LLM outputs. Rather than treating one questionnaire as proof of personality, it administers two Big Five inventories, the 300-item IPIP-NEO and 44-item BFI, to 18 PaLM, Llama 2, Mistral, Mixtral, and GPT variants. For each model, 50 fictional PersonaChat biographies are crossed with five item instructions and five postambles to create 1,250 paired prompt profiles. The framework assesses internal consistency with Cronbach's alpha, Guttman's lambda-6, and McDonald's omega; convergence between inventories; discrimination among traits; relations with eleven criterion scales; and exploratory factor structure. Its central result is conditional: many base models fail, while larger instruction-tuned variants, especially Flan-PaLM 540B and GPT-4o, show much stronger reliability and convergence. The study also induces nine trait levels using 104 adjectives and finds that eleven models preselected for measurement quality generally follow single-trait instructions well. In four large models, instructed levels are also reflected in synthetic social-media text scored by Apply Magic Sauce. This is an important contribution because it applies substantially stronger psychometric standards, states limitations, and releases extensive code and data. Its scope nevertheless requires precision. The 1,250 rows are not independent people: they deterministically cross only 50 reused biographies with fixed prompt variants, yet reported p-values use n=1,250 without clustered or crossed-factor correction. Every measurement tells the model to follow a biography, so the study validates a prompt-conditioned response distribution rather than an autonomous or persistent model personality. Shaping explicitly names Big Five traits, and both questionnaires and the text classifier reward semantically aligned language; survey scores and posts also share the same manipulation. The correlations therefore demonstrate strong instruction following and lexical transfer, but do not show that a latent personality state causes real behavior. Factor structure is only partial, and test-retest stability, cultural invariance, live users, and quantitative safety evaluation are absent. The artifact audit reproduced 17 of the 18 published convergence/discrimination rows. Llama-2-Chat 70B differs: the current public pickle yields about 0.82/0.43/0.39, versus 0.80/0.39/0.42 in the table. The repository supports meaningful aggregate checks, but has no locked environment, CI, or single end-to-end run; its public bucket is about 43.74 GiB of large checksum-free pickles. The defensible conclusion is that some instruction-tuned LLMs generate coherent and steerable Big Five response patterns under specific prompt configurations. It does not establish stable human personalities in models or that synthetic posts constitute real-world behavior.

Español

Este trabajo desarrolla uno de los marcos empíricos más completos disponibles para evaluar personalidad sintética en salidas de LLM. En lugar de interpretar un cuestionario aislado como prueba de personalidad, administra dos inventarios Big Five, IPIP-NEO de 300 ítems y BFI de 44, a 18 variantes de PaLM, Llama 2, Mistral, Mixtral y GPT. Para cada modelo cruza 50 biografías ficticias de PersonaChat con cinco instrucciones de ítem y cinco postámbulos, creando 1.250 perfiles de prompt emparejados. Evalúa consistencia interna con alfa de Cronbach, lambda-6 de Guttman y omega de McDonald; convergencia entre ambos inventarios; discriminación entre rasgos; relaciones con once escalas criterio; y, de forma exploratoria, estructura factorial. El resultado central es condicional: varios modelos base fallan, mientras las variantes grandes e instruction-tuned, especialmente Flan-PaLM 540B y GPT-4o, muestran patrones mucho más fiables y convergentes. El estudio también induce nueve niveles de rasgo mediante 104 adjetivos y observa que once modelos previamente seleccionados por su fiabilidad siguen bien las instrucciones de rasgo. En cuatro modelos grandes, el nivel indicado también se refleja en textos sintéticos de redes sociales puntuados por Apply Magic Sauce. Es una contribución importante porque aplica estándares psicométricos más exigentes, explicita limitaciones y publica código y datos. Pero su alcance debe expresarse con precisión. Las 1.250 filas no son personas independientes: son el cruce determinista de solo 50 biografías reutilizadas con variantes fijas del prompt, aunque los p-valores usan n=1.250 sin corrección por clúster o factores cruzados. Toda medición ordena seguir una biografía, por lo que valida una distribución condicionada por el prompt, no una personalidad autónoma o persistente del modelo. El shaping nombra directamente rasgos Big Five y tanto el cuestionario como el clasificador textual premian lenguaje semánticamente alineado; además, encuesta y posts comparten la misma manipulación. Por eso las correlaciones prueban instrucción y transferencia léxica fuertes, pero no que un estado latente de personalidad cause conducta real. La estructura factorial solo es parcial, no hay test-retest, invariancia cultural, usuarios reales ni evaluación cuantitativa de seguridad. La auditoría del artefacto reprodujo 17 de las 18 filas de convergencia y discriminación publicadas. Llama-2-Chat 70B no coincide: el pickle actual produce aproximadamente 0,82/0,43/0,39, frente a 0,80/0,39/0,42 en la tabla. El repositorio permite comprobar análisis agregados, pero carece de entorno bloqueado, CI y una ejecución integral; el bucket ocupa unos 43,74 GiB en grandes pickles sin checksums. La conclusión defendible es que algunos LLM instruction-tuned producen respuestas Big Five coherentes y dirigibles bajo configuraciones de prompt concretas. No demuestra que posean personalidades humanas estables ni que los textos sintéticos sean comportamiento real.

Research question

Can synthetic Big Five traits be measured in LLM outputs with psychometric reliability and validity, which model properties favor that measurement, and to what extent can the expressed traits be shaped and transferred to another textual task?

Method

The study evaluates 18 variants from five families. Each IPIP-NEO and BFI item is presented independently under a prompt that combines a fixed instruction to follow a biography, one of 50 PersonaChat biographies, one of five item instructions, and one of five postambles. The 1,250 combinations per model generate matched scores for internal reliability, convergence, discrimination, and relations with eleven criterion scales. An exploratory factor analysis checks structure. Then, eleven models that achieve at least neutral reliability receive prompts with 104 Big Five adjectives across nine levels: 2,250 profiles for individual shaping and 1,600 for concurrent extreme configurations. The four largest models generate synthetic posts from the same profiles and Apply Magic Sauce estimates traits in those texts. The audit read and visually reviewed the 30 main pages and 34 supplementary pages, examined official commit c3fea44, reconstructed the minimal test environment, inventoried the 43.74 GiB published, and recalculated convergence and discrimination from the 18 aggregated files.

Sample: Per model, 1,250 profiles result from crossing 50 biographies x 5 item instructions x 5 postambles; across 18 models there are 22,500 IPIP-NEO/BFI pairs. These are deterministic reusable conditions, not independent participants. Individual shaping uses 45 level/trait profiles x 50 biographies = 2,250 per model; concurrent shaping uses 32 extreme configurations x 50 = 1,600. The article reports 523,750 prompt-continuation scores in validity, 675,000 in individual shaping, 480,000 in concurrent, and 56,250 downstream records; those counts also do not equate to persons.

Findings

  • Base variants frequently fail reliability and validity thresholds; instruction tuning is the clearest predictor and size helps within fine-tuned families.
  • Flan-PaLM 540B and GPT-4o show some of the strongest patterns of consistency and convergence between IPIP-NEO and BFI.
  • Seven published configurations achieve a mean convergent-discriminant difference of at least 0.40, although that validity is specific to model and prompt.
  • Only 13 datasets pass the Bartlett/KMO requirements for exploratory factor analysis; items with expected loading exceed 72 % in Flan-PaLM 540B and GPT-4o, but fall below 45 % in Llama-2-Chat 7B and Mixtral 8x7B Instruct.
  • In the eleven selected models, all but one achieve mean Spearman correlations above 0.80 between indicated level and measured trait in individual shaping; concurrent control is more difficult.
  • In the posts task, the mean survey-text convergence is r=0.67 and the indicated level-text association averages rho=0.68-0.82; Flan-PaLM openness drops to 0.47.
  • The 18 aggregated pickles contain 1,250 rows and 27 consistent columns; 17 reproduce the published convergence/discrimination table upon rounding.
  • Llama-2-Chat 70B does not reproduce the table: current public data recalculate 0.822 convergence, 0.434 absolute discrimination, and 0.388 delta, not 0.80/0.39/0.42.
  • The two included Python tests pass after manually combining dependencies, but the declared PsyBORGS environment omits tqdm and is not self-containedly installable.
  • The artifact offers valuable data and analysis, but not a fixed integral reproduction: there is no lockfile, CI, publication release, or data checksums.

Limitations

  • The 1,250 observations per model reuse 50 biographies in a deterministic factorial design; p-values with n=1,250 without modeling clusters or crossed factors overestimate independence and precision.
  • Every measurement contains the instruction to follow a biography; there is no neutral condition demonstrating stable personality without that induction.
  • Alpha, lambda-6, and omega evaluate internal consistency, not test-retest, between-session stability, paraphrases, provider updates, or contexts.
  • Criterion scales are also LLM responses under the same method; they share semantics and format, and human effects come from distinct bibliographic samples.
  • Shaping uses Big Five adjectives and is evaluated with instruments and a lexically-based classifier conceptually aligned, which permits circularity or vocabulary propagation.
  • Survey and posts receive the same explicit manipulation; their correlation does not identify a causal effect of a measured latent personality on the subsequent text.
  • Posts are synthetic one-turn generations scored by an API, not real, longitudinal, multi-turn behavior, nor validated by independent human judges.
  • The factorial structure does not fully coincide with Big Five and the supplement itself warns that deterministic biographies do not offer conventional random individual variance.
  • Shaping is evaluated only in eleven models preselected for reliability, so it does not generalize to the 18.
  • The study is in English only, with 50 fictional biographies mainly North American or European; there is no cultural, linguistic, or demographic invariance.
  • Extreme examples include hate, sexism, violence, depression, and self-harm, but there are no quantitative metrics of toxicity, refusals, safety, or harm by subgroup.
  • PaLM, historical GPT snapshots, and Apply Magic Sauce are proprietary dependencies that limit exact current reproduction.
  • The bucket occupies approximately 43.74 GiB and uses executable pickles up to 4.27 GiB without checksums, an insecure and poorly portable format for untrusted users.
  • Most notebooks do not preserve runs or outputs, contain template paths and ad hoc installations; the R analysis installs packages dynamically.
  • There are only two narrow tests; inference, statistics, tables, and paper-data concordance are not tested, and the README retains the old preprint title.

What the study does not establish

  • It does not demonstrate an autonomous, persistent, or human-equivalent personality within the model.
  • It does not convert 1,250 prompt combinations into 1,250 independent persons nor justify population inference without a dependency structure.
  • It does not demonstrate test-retest stability across sessions, prompts, snapshots, or deployments.
  • It does not test that the human Big Five latent structure is uniformly recovered across the 18 models.
  • It does not fully distinguish a latent personality change from instruction following and trait vocabulary repetition.
  • It does not establish that the measured personality causes the downstream text; both outcomes share the same shaping prompt.
  • It does not validate synthetic posts as real behavior or as an effect on users.
  • It does not establish cultural, linguistic, demographic, or multi-turn generalization.
  • It does not demonstrate that inducing trait extremes is safe, fair, or beneficial.
  • It does not offer an exact and integral reproduction from a versioned environment; one tabular result also does not match the current public artifact.

Traceability

Scope: Full text

Version: Nature Machine Intelligence 7, 2063-2073 (2025), DOI 10.1038/s42256-025-01115-6; 30-page article plus 34-page supplement

Consulted source: https://www.nature.com/articles/s42256-025-01115-6

Review: Codex complete bilingual fidelity pass using the published article and supplement, all-64-page visual inspection, official repository and commit audit, isolated dependency and unit-test reconstruction, complete public-bucket inventory, independent recalculation of all 18 aggregate score files, psychometric design and construct-boundary review, causal-interpretation and safety analysis; summaries written from full evidence rather than abstract keywords, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • PaLM 62B
  • Flan-PaLM 8B
  • Flan-PaLM 62B
  • Flan-PaLM 540B
  • Flan-PaLMChilla 62B
  • Llama 2 7B
  • Llama 2 13B
  • Llama 2 70B
  • Llama 2 Chat 7B
  • Llama 2 Chat 13B
  • Llama 2 Chat 70B
  • Mistral 7B
  • Mistral 7B Instruct
  • Mixtral 8x7B
  • Mixtral 8x7B Instruct
  • GPT-3.5 Turbo
  • GPT-4o mini
  • GPT-4o

Instruments and metrics

  • 300-item IPIP-NEO
  • 44-item Big Five Inventory
  • Cronbach alpha
  • Guttman lambda-6
  • McDonald omega
  • Multitrait-multimethod convergent and discriminant analysis
  • Exploratory factor analysis
  • Positive and Negative Affect Schedule
  • Buss-Perry Aggression Questionnaire
  • Portrait Values Questionnaire
  • Creative Self-Efficacy and Creative Personal Identity scales
  • Apply Magic Sauce personality estimates

Data used

  • PersonaChat biographies
  • 22,500 paired model-level construct-validity profiles
  • Public personality_in_llms Google Cloud Storage bucket
  • 18 public scores_joined_pr01 aggregate files
  • Synthetic social-media status updates
  • Official google-deepmind/personality_in_llms repository at commit c3fea44

Evidence and location

  • Question, scope, main results, and declared limits: Main PDF pages 1-11, Abstract, Main, Results, Discussion and Methods
  • Models, snapshots, quantizations, and experimental counts: Main PDF pages 12-18, Methods and Table 1
  • Reliability, convergence, discrimination, and criterion: Main PDF pages 4-7 and 14-18; Supplement pages 10-13; Extended Data Tables 1-4
  • Individual, concurrent, and downstream shaping: Main PDF pages 7-10 and 18-22; Supplement pages 19-31; Extended Data Tables 5-7
  • Exploratory factor structure and warning about deterministic variance: Supplement pages 11-18, Supplementary Note A.5 and Supplementary Tables 5-6
  • Extreme content examples and ethical considerations: Main PDF pages 10-11 and 22; Supplement pages 28-31
  • Repository, dependencies, tests, and inference: Official repository commit c3fea44, complete 44-file audit, requirements files, nine notebooks, three inference scripts and two tests
  • Bucket inventory and reproduction of 18 files: Official public Google Cloud Storage bucket audited 15 July 2026; all 18 scores_joined_pr01 files recalculated
  • Llama-2-Chat 70B discrepancy: Public scores_joined_pr01_Llama-2-70b-chat-hf.pkl versus published Extended Data Table 3
  • Integral validity and artifact audit: reports/verification/article-197-psychometric-validity-and-artifact-audit.json
  • Complete visual inspection: All 30 main-PDF and 34 supplement pages rendered and visually inspected on 15 July 2026