Synthetic Personality Review

The Power of Personality: A Human Simulation Perspective to Investigate Large Language Model Agents

Personas, identity, and agents2025arXivApproved editorial review

Authors: Yifan Duan, Yihong Tang, Xuefeng Bai, Kehai Chen, Juntao Li, Min Zhang

Keywords: Computation and Language

Source: Open primary source (opens in a new tab)

6
Authors
16
Findings
39
Limitations
20
Evidence

Editorial summary

English

For each of three open models, the study creates 32 agents covering all 2^5 high/low combinations of the Big Five traits. Each agent receives a prompt built from scale phrases associated with high or low neuroticism, agreeableness, conscientiousness, extraversion, and openness, then answers the 60-item BFI-2. Qwen2.5-32B-Instruct, Qwen2.5-14B-Instruct, and Llama 3.1-8B-Instruct run through vLLM at temperature zero. Paired high/low tests report p<.001 for all 15 model-trait combinations and Cohen’s d from 1.49 to 23.17. This confirms that the prompt changes questionnaire answers, but the validation is semantically circular: models are instructed with descriptions that directly express the same constructs later queried. There is no behavior-matched non-personality control, test-retest analysis, or human personality rating. On closed tasks, the same 32 prompts are applied to MMLU, MMLU-Pro, SciQ, ARC-Challenge, and ARC-Easy. The gap between the highest and lowest mean profile is 4.85 points for Qwen-32B (86.23 vs 81.38), 2.77 for Qwen-14B (80.96 vs 78.19), and 3.75 for Llama-8B (71.45 vs 67.70). The authors correlate the 32 BFI-2 scores with accuracy and find model-dependent patterns: Qwen-32B openness correlates .50–.65 with all five benchmarks; Qwen-14B does not show a stable pattern, and Llama only reaches .52 on MMLU-Pro. Neuroticism is usually negative, but correlates +.50 with Llama MMLU-Pro. The induction phrases include “am quick to understand,” “like to solve complex problems,” “am not interested in other people’s problems,” and “waste my time,” directly altering claimed ability, effort, and cooperation. The design therefore cannot attribute performance changes to personality rather than obedience to those instructions. For creativity, each profile answers AUT, INSTANCES, SCIENTIFIC, and SIMILARITIES. GPT-4o-mini, without an exact snapshot and at temperature zero, assigns 1–10 originality/elaboration scores and counts fluency/flexibility. Means and standard deviations are published, but not item counts per task, raw outputs, human validation, judge reliability, or sensitivity analysis. At least 120 trait-creativity correlations are added to 75 trait-accuracy correlations without multiplicity correction. Openness and agreeableness often correlate positively with the same judge’s scores, neuroticism negatively, and extraversion inconsistently; this does not establish human creativity or transfer beyond those prompts. The multi-agent section manually selects only nine three-member teams: six homogeneous, two diverse, and one containing a single +---- profile labeled extreme. After initial answers, discussion, and revision, accuracy or creativity is averaged across the three final member responses. There are no position permutations, repeated teams, same-capability non-persona controls, or inferential tests. For Qwen-32B, for example, Team 7 scores 47.25 on GPQA versus 44.93 for Team 4, but their MMLU and MMLU-Pro differences are only −.05 and −.12. The multi-agent creativity table repeats exactly the Teams 1–9 SIMILARITIES columns across all three models, which requires data or code to rule out a copy error. No separate collective decision is evaluated: members are scored and averaged. The results therefore do not establish collective intelligence distinct from individual ability. The arXiv v2 PDF also has visible publication failures: appendix tables lose headers, profile signs, columns, or nearly their entire body on pages 18, 22, 24, 28, and 30. No code, outputs, full per-profile prompts, or evaluation data are linked. The defensible conclusion is narrower: Big Five-linked instructions change BFI-2 answers and, for some models, accuracy and automated creativity scores; nine debate compositions yield small or variable descriptive differences. The study does not show psychological human simulation or identify personality as the causal mechanism.

Español

El estudio crea, para cada uno de tres modelos abiertos, 32 agentes que cubren las 2^5 combinaciones binarias de los rasgos Big Five. Cada agente recibe un prompt formado por frases de una escala asociadas al polo alto o bajo de neuroticismo, amabilidad, responsabilidad, extraversión y apertura, y después responde el BFI-2 de 60 ítems. Qwen2.5-32B-Instruct, Qwen2.5-14B-Instruct y Llama 3.1-8B-Instruct se ejecutan con vLLM y temperatura 0. Los contrastes emparejados entre polos del rasgo muestran p<0,001 en las 15 combinaciones modelo-rasgo y tamaños d de Cohen de 1,49 a 23,17. Esto confirma que el prompt cambia las respuestas del cuestionario, pero la validación es semánticamente circular: se instruye al modelo con descripciones que expresan directamente los mismos constructos que luego se preguntan. No hay condición con instrucciones conductuales equivalentes pero sin etiqueta de personalidad, ni test-retest, ni evaluación humana de personalidad. Para tareas cerradas, los mismos 32 prompts se aplican a MMLU, MMLU-Pro, SciQ, ARC-Challenge y ARC-Easy. El promedio máximo y mínimo difieren 4,85 puntos en Qwen-32B (86,23 frente a 81,38), 2,77 en Qwen-14B (80,96 frente a 78,19) y 3,75 en Llama-8B (71,45 frente a 67,70). Los autores correlacionan los scores BFI-2 de los 32 agentes con su exactitud y encuentran patrones dependientes del modelo: en Qwen-32B apertura correlaciona 0,50–0,65 con los cinco benchmarks; en Qwen-14B no lo hace de forma estable, y en Llama sólo aparece 0,52 en MMLU-Pro. Neuroticismo es generalmente negativo, pero en Llama correlaciona +0,50 con MMLU-Pro. Las frases usadas para inducir rasgos incluyen «comprendo rápido», «me gusta resolver problemas complejos», «no me interesan los problemas de otros» o «desperdicio mi tiempo»; por ello el tratamiento altera de manera directa capacidad declarada, esfuerzo y cooperación. El diseño no permite atribuir los cambios a personalidad en lugar de a seguimiento de esas órdenes. Para creatividad, cada perfil responde AUT, INSTANCES, SCIENTIFIC y SIMILARITIES. GPT-4o-mini, sin snapshot declarado y con temperatura 0, asigna originalidad y elaboración de 1 a 10 y cuenta fluidez y flexibilidad. Se publican medias y desviaciones, pero no el número de ítems por tarea, resultados crudos, validación humana, fiabilidad del juez ni análisis de sensibilidad. Se calculan al menos 120 correlaciones rasgo-creatividad además de 75 rasgo-exactitud, sin corrección por multiplicidad. Apertura y amabilidad correlacionan frecuentemente con las puntuaciones del mismo juez, neuroticismo de forma negativa y extraversión de forma inconsistente; no se prueba creatividad humana ni transferencia fuera de esos prompts. La parte multiagente selecciona sólo nueve equipos manuales de tres miembros: seis homogéneos, dos diversos y uno con un único perfil +---- calificado de extremo. Tras respuesta inicial, discusión y revisión, se promedia la exactitud o creatividad de las tres respuestas finales. No hay permutaciones de posición, réplicas, equipos de control con las mismas capacidades y sin persona, ni inferencia estadística. En Qwen-32B, por ejemplo, Team 7 obtiene 47,25 en GPQA frente a 44,93 de Team 4, pero en MMLU y MMLU-Pro ambos difieren sólo −0,05 y −0,12. La tabla de creatividad multiagente repite exactamente las columnas SIMILARITIES de Teams 1–9 en los tres modelos, lo que exige datos o código para descartar un error de copia. Tampoco evalúa una decisión colectiva separada: el protocolo puntúa a cada miembro y promedia. Por tanto, no establece inteligencia colectiva distinta de la capacidad individual. El PDF arXiv v2 presenta además fallos visibles: tablas del apéndice pierden encabezados, signos, columnas o casi todo el cuerpo en las páginas 18, 22, 24, 28 y 30. No se enlazan código, outputs, prompts completos por perfil ni datos de evaluación. La conclusión defendible es más estrecha: instrucciones semánticamente ligadas a Big Five cambian respuestas BFI-2 y, en algunos modelos, también exactitud y scores automáticos de creatividad; nueve composiciones de debate producen diferencias descriptivas pequeñas o variables. No se demuestra que el modelo simule psicológicamente a humanos ni que la personalidad sea el mecanismo causal.

Research question

How do BFI-2 responses, benchmark accuracy, automatic creativity scores, and debate outcomes change when three LLMs receive binary combinations of instructions associated with the five Big Five traits?

Method

Full factorial design 2^5 of 32 prompts per model. The high/low pole phrases are taken from items linked to Big Five and inserted into the context; the BFI-2 evaluates whether the responses follow the profile. The 32 agents run on five closed benchmarks and four open tasks, with Pearson correlations between BFI-2 scores and outcomes. GPT-4o-mini judges creativity. Nine selected teams of three agents perform initial response, discussion, and revision; the final responses of the members are averaged. The generating models use vLLM and temperature 0.

Sample: Thirty-two binary configurations per each of three models, with one BFI-2 administration and one execution of each benchmark per configuration. The correlation analyses use n=32 profiles per model. The collaboration uses nine preselected teams of three members per model. No generation replicates, seeds, or the number of items/stimuli applied in each creative task are reported.

Findings

  • The 15 model-trait contrasts of the BFI-2 yield p<0.001 and Cohen's d between 1.49 and 23.17, evidencing strong semantic obedience to the prompt.
  • For neuroticism, d is 23.17 in Qwen-32B, 18.47 in Qwen-14B, and 3.37 in Llama-8B; extraordinary magnitudes that reflect the proximity between induction and questionnaire.
  • Qwen-32B goes from a minimum closed mean of 81.38 for +---- to 86.23 for -++++, a range of 4.85 points.
  • Qwen-14B covers 78.19–80.96 and Llama-8B 67.70–71.45; the optimal profile for Llama is ++--+, not -++++.
  • In Qwen-32B, openness correlates 0.62 with MMLU, 0.60 with MMLU-Pro, 0.50 with ARC-Challenge, 0.65 with ARC-Easy, and 0.51 with SciQ.
  • In Qwen-14B, openness ranges from −0.23 to 0.32 and none of those five correlations reaches p<0.05.
  • In Llama-8B, neuroticism correlates +0.50 with MMLU-Pro but −0.70 with ARC-Easy, showing task dependence.
  • The conscientiousness of Llama-8B remains near zero across the five benchmarks, despite the proposed human analogies.
  • Openness and agreeableness frequently correlate positively with originality and elaboration scored by GPT-4o-mini; extraversion is small and inconsistent.
  • The creative correlations are calculated on scores from a single type of automatic judge and are not validated against humans.
  • Among Qwen-32B teams, Team 7 obtains the maximum GPQA of 47.25, but does not improve MMLU or MMLU-Pro compared to Team 4.
  • The nine Qwen-14B teams only cover 77.89–78.25 on MMLU and 60.24–61.73 on MMLU-Pro.
  • Team 9 does not always fall below Team 4: in Qwen-14B it slightly exceeds MMLU and MMLU-Pro, and in Llama it exceeds GPQA.
  • In the multiagent creativity table, the eighteen SIMILARITIES values of Teams 1–9 are identical for Qwen-32B, Qwen-14B, and Llama-8B.
  • Visual inspection confirms that five pages of the appendix do not fully show the results that the textual layer purports to contain.
  • No official repository, code, outputs, or raw data linked from arXiv, PDF, OpenReview, or targeted searches by title and prompts were found.

Limitations

  • The prompts do not manipulate only personality: they include explicit statements of understanding, interest in problems, effort, order, social distance, and ability to solve complex tasks.
  • The high openness induction tells the model that it understands quickly and enjoys complex problems, directly confusing it with ability and motivation.
  • Low agreeableness orders disinterest in others' problems, which can reduce cooperation with the user by definition.
  • The BFI-2 validation is semantically circular because the prompt contains descriptions of the same construct that is then measured via self-report.
  • There is no placebo condition with equal length, tone, and difficulty but without Big Five content.
  • There is no comparison with independent narrative profiles, blind human evaluation, or external behavior not suggested by the induction phrases.
  • The traits are discretized into only two poles, although the paper itself recognizes their continuity and multidimensional complexity.
  • The positive/negative notation can be confused with desirable/undesirable; for neuroticism the positive sign represents more neuroticism.
  • It is not reported whether the 32 conditions are executed in random order or whether they share caches or context.
  • Temperature 0 does not by itself guarantee bit-by-bit determinism; no seeds, exact versions of vLLM, kernels, quantization, or replicates are published.
  • A single administration per profile and model is done, without test-retest or generation uncertainty.
  • The BFI-2 t-tests treat 16 pairs of configurations as units, but do not discuss independence, normality, or multiple correction.
  • The extreme d sizes are interpreted as simulation capacity, although the independent variable directly includes the contents that the questionnaire predicts.
  • The performance correlations use n=32 profiles and many variables derived from the same factorial manipulation.
  • No factorial model or regression is estimated to separate main effects, interactions between traits, and collinearity among the realized BFI-2 scores.
  • There are at least 75 trait-benchmark tests and 120 trait-creativity tests, without FDR/Bonferroni correction or preregistration.
  • Selected patterns are compared with heterogeneous human literature without meta-analysis, task matching, or formal test of equivalence.
  • The benchmarks may be present in training data and contamination is not evaluated.
  • Versions, splits, subsets, prompts per benchmark, parsing, context limits, or handling of invalid responses are not detailed.
  • No accuracy for a comparable no-personality condition for the 32 closed-task profiles is published.
  • The format forces emitting all reasoning, which can alter performance and is not comparable with standard protocols of all models.
  • The GPT-4o-mini snapshot, API date, repetition, seed, or judge stability is not reported.
  • A single automatic judge scores originality and elaboration without humans, a second judge, agreement, calibration, or bias analysis by style or length.
  • The creativity metrics favor volume and textual elaboration; the openness/agreeableness prompts may change precisely those superficial traits.
  • The number of creative stimuli is not reported and individual responses are not published, so the deviations have no verifiable unit of analysis.
  • The nine multiagent teams are manual selections and not a systematic sample of the enormous space of possible compositions.
  • There is only one team per hypothesis, without replicates, permutation of positions, or random selection of members.
  • Team 9 uses a single +---- profile as a synonym for extreme personality; no other extreme profiles or positions are tested.
  • Teams 7 and 8 are not sufficient to identify a general diversity effect.
  • There is no debate control with the same three models and neutral prompts, nor a control of mere multiple generation without discussion.
  • The team score is an average of individual post-discussion outcomes, not an independent collective decision.
  • No intervals or tests for team differences are reported; terms such as significant are used without statistics in that section.
  • The closed differences between teams are often less than one point and may depend on few items in GPQA.
  • The exactly repeated SIMILARITIES values among three models in Table 4 are an unexplained anomaly.
  • There is no code or raw data to check tables, correlations, copy errors, or parsing.
  • The v2 PDF presents serious typesetting errors: pages 18, 22, 24, 28, and 30 cut or make parts of tables disappear.
  • Figure 3 is mistakenly titled as correlation with Closed Tasks although it shows creativity metrics.
  • The Limitations heading and its first lines appear duplicated in the textual layer of the PDF.
  • The ethics section only claims to use non-sensitive public datasets; it does not consider risks of anthropomorphization, use of traits for selection, stigmatization, or invalid psychological inferences.

What the study does not establish

  • It does not demonstrate that the models have human personality, psychological states, or simulated human intelligence.
  • It does not causally separate personality from direct instructions about ability, motivation, effort, and cooperation.
  • It does not validate the BFI-2 as a psychometric instrument for LLMs.
  • It does not demonstrate temporal or between-run stability of the profiles.
  • It does not show that the traits are expressed without their descriptions being present in the context.
  • It does not establish equivalence between model correlations and human psychological relationships.
  • It does not demonstrate human creativity; it reports scores from an LLM judge on textual tasks.
  • It does not test that temperature 0 makes the entire pipeline deterministic.
  • It does not establish that the traits predict performance in models, tasks, or prompts not evaluated.
  • It does not identify reliable interactions among the five traits.
  • It does not demonstrate a general benefit of personality diversity in teams.
  • It does not demonstrate that a member with extreme personality generally harms the team.
  • It does not demonstrate collective intelligence distinct from individual capabilities.
  • It does not evaluate human users, personality perception, trust, safety, fairness, or product utility.
  • It does not allow reproducing the complete results with the published artifacts.

Traceability

Scope: Full text

Version: arXiv:2502.20859v2, revised 27 May 2025, 31 pages

Consulted source: https://arxiv.org/pdf/2502.20859v2

Review: Codex full-text, bilingual-fidelity, visual, construct-validity, semantic-circularity, treatment-contamination, factorial-design, test-retest, multiplicity, correlation, automated-judge, creativity, multi-agent, collective-outcome, table-integrity, reproducibility, ethics and evidence-level audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • Qwen2.5-32B-Instruct
  • Qwen2.5-14B-Instruct; also written Qwen-14B-Instruct in the appendix
  • Llama 3.1-8B-Instruct
  • GPT-4o-mini automated creativity judge; exact snapshot not reported
  • GPT-4 used by the authors to modify evaluation prompts; exact snapshot not reported

Instruments and metrics

  • Big Five high/low prompt phrases based on DeYoung et al. scale items
  • BFI-2, 60 items, five domains and 15 facets
  • Paired t-tests and Cohen d for high/low BFI-2 scores
  • Pearson correlations between BFI-2 domain scores and benchmark outcomes
  • TTCT-inspired originality, elaboration, fluency and flexibility rubric
  • GPT-4o-mini automated creativity scoring
  • Three-stage multi-agent debate: initial response, discussion, final response
  • Mean of three member-level final results as team score

Data used

  • MMLU
  • MMLU-Pro
  • SciQ
  • ARC-Challenge
  • ARC-Easy
  • GPQA for multi-agent closed-task evaluation
  • Alternative Uses Task (AUT)
  • INSTANCES
  • SCIENTIFIC
  • SIMILARITIES

Evidence and location

  • Identity, authors, version, and complete abstract: arXiv:2502.20859v2 metadata and PDF p. 1
  • Thirty-two binary combinations of traits: PDF p. 3, section 3.1 and equation y_k
  • 60-item BFI-2: PDF pp. 3, 15, sections 3.2 and C.1
  • Models, vLLM, L20, and temperature: PDF pp. 4, 13, sections 3.5 and B.3
  • BFI-2 contrasts, p, and d: PDF pp. 4–5, section 4.1 and Table 1
  • Accuracy correlations: PDF pp. 5–6, section 4.2 and Figure 2
  • Creativity correlations and evaluation: PDF pp. 5–7, section 4.3 and Figure 3
  • Nine teams and selected profiles: PDF pp. 6–7, section 5.1 and Table 2
  • Multiagent closed results: PDF p. 7, Table 3 and section 5.2
  • Multiagent open results and repeated columns: PDF p. 8, Table 4 and section 5.3
  • Declared limitations and ethics: PDF p. 9, Limitations and Ethics Statement
  • Exact induction phrases and treatment contamination: PDF p. 12, Appendix B.1
  • Team calculation as mean of members: PDF p. 13, Appendix B.3
  • Complete automatic judge rubrics: PDF pp. 13–14, Appendix B.5 and Table 5
  • BFI-2 scores per profile and model: PDF pp. 15–16, Tables 6–7
  • Complete accuracy of the 32 profiles: PDF pp. 17–19, Tables 8–10
  • Creative results per profile: PDF pp. 20–31, Tables 11–22
  • Visual inspection and layout failures: All 31 pages of arXiv v2 rendered and visually inspected; pages 18, 22, 24, 28 and 30 visibly lose table content; checked 15 Jul 2026
  • Publication status: arXiv v2, OpenReview forum/PDF and author publication page checked; OpenReview is an anonymous submission and no final proceedings record was found; checked 15 Jul 2026
  • Absence of reproducible artifacts: arXiv abstract/PDF, OpenReview record and targeted GitHub searches by exact title, arXiv ID and distinctive prompt checked; no official code or data repository found; checked 15 Jul 2026