Synthetic Personality Review

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

Personas, identity, and agents2024ACL AnthologyApproved editorial review

Authors: Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, Jad Kabbara

Keywords: LLM personas, Big Five personality model, personality expression, text generation, psycholinguistic patterns, human evaluation, personalized chatbots

Source: Open primary source (opens in a new tab)

6
Authors
25
Findings
46
Limitations
19
Evidence

Editorial summary

English

PersonaLLM studies whether GPT-3.5-turbo-0613, GPT-4-0613, and, in appendices, LLaMA 2 70B express five prompt-assigned Big Five traits. The authors construct all 32 binary combinations of extroverted/introverted, agreeable/antagonistic, conscientious/unconscientious, neurotic/emotionally stable, and open/closed to experience. They generate ten personas per combination and model: each receives the explicit adjectives in its system prompt, completes the 44-item BFI, and, with that exchange retained in context, writes an approximately 800-word personal story without naming the traits. GPT-3.5 and GPT-4 BFI scores separate every assigned pole at p<.001 with very large effect sizes; for GPT-4, d ranges from 4.22 to 6.30. This establishes strong prompt and expected-self-report compliance, not internal personality or psychometric validity for LLMs. The LIWC-22 analysis correlates 81 categories with binary assignments and compares significant directions with 2,467 human Essays-corpus texts. GPT-4 shares more human associations than GPT-3.5 for conscientiousness and openness, but the human and synthetic tasks use different prompts and hundreds of tests are run without a reported multiplicity correction. Human evaluation excludes stories containing explicit personality lexemes: the filter removes 96.56% of GPT-3.5 stories and 31.87% of GPT-4 stories, leaving only 32 selected GPT-4 stories, one per profile. Thirty-nine US-based, English-first-language Prolific workers, paid $15/hour, rate stories under informed and uninformed AI-authorship conditions. Each story receives five ratings per condition on six qualities and five traits. Under majority vote, the uninformed group identifies extraversion at 0.84 and agreeableness at 0.69; the other traits range from 0.56 to 0.59. Informing raters of AI authorship lowers these figures to 0.72, 0.53, 0.53, 0.41, and 0.53, respectively. Inter-rater agreement is nevertheless nearly absent: personality Krippendorff alpha values run roughly from -0.03 to 0.12, so majority vote stabilizes highly discordant judgments. GPT-4 as a judge also strongly favors GPT-4 stories, assigning near-perfect scores with minimal variance. The paper provides useful evidence that extreme binary prompts leave recognizable linguistic signals, especially for extraversion, and that authorship disclosure changes human attribution; it does not show that all traits are robustly perceptible or that the stories model human people. The official repository preserves 320 BFI outputs and 320 stories per model, texts, and LIWC results, but an audit of branch v2.0 found neither the promised human annotations nor a dependency manifest. It also found method-code discrepancies: the paper says paired t-tests, whereas the script selects ANOVA or Mann-Whitney; and the published story generator iterates from personality combination index 14, so it cannot reconstruct a clean full run without modification.

Español

PersonaLLM estudia si GPT-3.5-turbo-0613, GPT-4-0613 y, en apéndices, LLaMA 2 70B expresan cinco rasgos Big Five impuestos por prompt. Los autores construyen las 32 combinaciones binarias de extroversión/introversión, amabilidad/antagonismo, responsabilidad/falta de responsabilidad, neuroticismo/estabilidad emocional y apertura/cierre a la experiencia. Para cada combinación generan diez personas por modelo: cada una recibe los adjetivos explícitos en el system prompt, completa el BFI de 44 ítems y, conservando ese intercambio en el contexto, escribe una historia personal de unas 800 palabras sin nombrar los rasgos. Los BFI de GPT-3.5 y GPT-4 separan todos los polos con p<.001 y tamaños de efecto muy grandes; en GPT-4, d va de 4,22 a 6,30. Esto demuestra fuerte cumplimiento del prompt y del autoinforme esperado, no personalidad interna ni validez psicométrica en LLM. El análisis LIWC-22 correlaciona 81 categorías con las etiquetas binarias y compara los signos significativos con 2.467 ensayos humanos del corpus Essays. GPT-4 comparte más asociaciones con humanos que GPT-3.5 en responsabilidad y apertura, pero las tareas humanas y sintéticas no usan el mismo prompt y se ejecutan cientos de pruebas sin corrección por multiplicidad publicada. Para la evaluación humana se descartan historias que contienen lexemas explícitos de personalidad: el filtro elimina el 96,56 % de las historias GPT-3.5 y el 31,87 % de las GPT-4, por lo que solo se estudian 32 historias GPT-4 seleccionadas, una por perfil. Treinta y nueve trabajadores estadounidenses de Prolific, anglófonos y pagados a 15 dólares/hora, valoran historias en condiciones informada y no informada sobre la autoría de IA. Cada historia recibe cinco valoraciones por condición sobre seis cualidades y cinco rasgos. Al agregar por mayoría, el grupo no informado acierta extroversión en 0,84 y amabilidad en 0,69; las demás quedan entre 0,56 y 0,59. Informar de la autoría reduce esas cifras a 0,72, 0,53, 0,53, 0,41 y 0,53, respectivamente. Sin embargo, el acuerdo entre anotadores es casi nulo: los alfa de Krippendorff de personalidad van aproximadamente de -0,03 a 0,12, de modo que la mayoría estabiliza votos muy discordantes. GPT-4 como evaluador muestra además preferencia extrema por historias GPT-4, con puntuaciones casi perfectas y varianza mínima. El trabajo aporta evidencia valiosa de que prompts binarios extremos dejan señales lingüísticas reconocibles, sobre todo para extroversión, y de que revelar autoría cambia la atribución humana; no prueba que todas las personalidades sean perceptibles de forma robusta ni que las historias modelen personas humanas. El repositorio oficial conserva 320 BFI y 320 historias por modelo, textos y resultados LIWC, pero la auditoría de la rama v2.0 no encontró las anotaciones humanas prometidas ni un manifiesto de dependencias. También detectó discrepancias entre método y código: el paper dice paired t-tests, mientras el script elige ANOVA o Mann–Whitney; y el generador publicado recorre las combinaciones desde el índice 14, por lo que no reconstruye una ejecución completa sin modificación.

Research question

To what extent do LLMs conditioned with the 32 binary combinations of the Big Five produce BFI self-reports and narratives coherent with the assigned profile; what LIWC patterns appear compared to human writing; how do humans and LLMs evaluate those narratives; and can they infer their traits, with or without knowing the AI authorship?

Method

For GPT-3.5, GPT-4 and LLaMA 2, five Big Five adjective pairs are crossed into 32 profiles and ten replicas per profile are generated at temperature 0.7. Each persona completes the 44-item BFI and then writes an 800-word personal story within the same conversation history. BFI scores are calculated per trait and differences between poles with means tests and Cohen's d. LIWC-22 extracts 81 psychological and extended vocabulary categories from 320 narratives per model; point-biserial correlations with the assigned label and Spearman correlations with the BFI score are used, compared with 2,467 human essays. A lexicographic filter excludes explicit mentions of traits and one GPT-4 story is taken for each of the 32 profiles. On Prolific, five raters per story and condition rate six narrative properties and five traits on a 1-5 Likert scale; another condition reveals AI authorship. Individual and majority-vote accuracy, Spearman correlation with BFI and Krippendorff's alpha are calculated. GPT-3.5 and GPT-4 perform parallel evaluations at temperature 0, with five repetitions.

Sample: The generative core contains 32 binary factorial profiles and ten replicas per model. The main analyses use 320 GPT-3.5 and 320 GPT-4; LLaMA 2 appears in appendices. The human evaluation uses only 32 GPT-4 narratives that survive a word filter and represent one per profile, with 16 positive and 16 negative labels per trait. 39 Prolific workers residing in the United States participate, with English as their first language and 99-100% approval; 37 were born in the USA and two in Nigeria. The stories are divided into four batches of eight and each receives five ratings per condition. Conversations, chatbot users or simulated human behaviors outside a written task are not studied.

Findings

  • The system prompt directly lists five adjectives, one per dimension, and covers all 32 possible binary combinations.
  • Each story is generated after the same persona has answered the BFI; the code retains BFI questions and answers in the narrative context.
  • In GPT-3.5, Cohen's d between BFI poles are 7.81, 5.93, 1.56, 1.83 and 2.90 for extraversion, agreeableness, conscientiousness, neuroticism and openness.
  • In GPT-4, the corresponding d values are 5.47, 4.22, 4.39, 5.17 and 6.30, all with p<.001 according to the paper.
  • LLaMA 2 also separates all poles in BFI, but with smaller effects, from d=1.16 to 2.86.
  • The BFI measures responses conditioned by explicit labels and does not constitute an independent test of latent traits.
  • Assigned labels correlate with stereotypical LIWC categories: extraversion with positive tone and affiliation; neuroticism with anxiety, negative tone and mental health; openness with curiosity.
  • GPT-4 shares with the human corpus 11 of 31 significant conscientiousness associations and 17 of 36 openness associations; GPT-3.5 shares 1 of 31 and 2 of 36, respectively.
  • Some associations diverge from humans: responsible LLM writing is linked to achievement when human writing is not, and sadness correlates negatively with neuroticism in GPT-3.5 but positively in GPT-4 and humans.
  • The LIWC analysis uses non-equivalent writing tasks and treats the human corpus as an approximate reference, a caution acknowledged by the authors.
  • 96.56% of GPT-3.5 stories and 31.87% of GPT-4 stories contain some explicit lexicographic mention of traits despite the prohibition.
  • Due to that instruction failure and to budget, the human evaluation is limited to 32 filtered GPT-4 stories.
  • Humans rate the stories near or above 4 on readability, cohesion, credibility and personal character, with likability somewhat lower.
  • When AI authorship is revealed, the human average for personal character drops from 4.32 to 3.99; the other narrative dimensions change little.
  • GPT-4 as evaluator gives 4.84-5.00 on almost all qualities and very small deviations, a preference toward GPT-4 content that the article itself acknowledges.
  • In uninformed individual prediction, humans reach 0.68 on extraversion, 0.51 on agreeableness and 0.41-0.47 on the other three traits.
  • With uninformed majority vote, human accuracy is 0.84, 0.69, 0.59, 0.59 and 0.56 for extraversion, agreeableness, conscientiousness, neuroticism and openness.
  • With authorship revealed, the human vote drops to 0.72, 0.53, 0.53, 0.41 and 0.53.
  • GPT-4 predicts extraversion with 0.97 and agreeableness and conscientiousness with approximately 0.68-0.69; neuroticism and openness remain around 0.56-0.59.
  • In the uninformed condition, human perception and the persona's BFI correlate r=.64, .33, .26, .23 and .22; informed, r=.42, .32, .20 and .17, while openness ceases to be significant.
  • Human agreement is extremely low: for the six qualities alpha ranges from approximately -0.04 to 0.11 and for personality from -0.03 to 0.12.
  • Informed comments have a mean sentiment of 0.45 versus 0.16 for uninformed ones according to the automatic classifier, although no test of that difference is reported.
  • The study received exempt status from the IRB, reports consent, collects no identifiers and pays 15 dollars per hour.
  • The official repository publishes 320 BFI and 320 stories for each of the three models, along with texts, scores and LIWC results.
  • The audited official branch v2.0 does not contain the human annotations or comments promised in the appendix, nor a versioned dependency environment.

Limitations

  • The study measures expression and compliance with personality prompts, not possession of psychological traits by the model.
  • The five continuous traits are reduced to extreme binary adjectives and all gradation, uncertainty and context are lost.
  • Some poles have a moral or normative loading, such as agreeable/antagonistic and conscientious/unconscientious.
  • The BFI is administered immediately after revealing to the model the labels that determine the expected responses.
  • The narrative retains the 44 previous BFI answers in the context, so the narrative signals do not come only from the system prompt and the task is not independent.
  • The enormous effect sizes may reflect experimental demand and literal compliance, not a stable psychological structure.
  • Test-retest consistency with new prompts, distractor contexts, long conversations or unrelated tasks is not tested.
  • There is no condition without prior BFI, without explicit labels, with equivalent behavioral descriptions or with continuous profiles.
  • The paper states paired t-tests, but the audited official code applies a normality test and then one-way ANOVA or Mann-Whitney, without pairing the factorial replicas.
  • No correction for the five BFI tests is reported, although the p<.001 and large effects reduce the practical relevance of that omission in that block.
  • LIWC evaluates 81 variables by five traits and several corpora with a p<=.05 threshold; the paper and scripts do not apply correction for hundreds of comparisons.
  • Without correction, part of the overlap of significant associations may arise from false positives.
  • Counting significant coincidences does not measure agreement of magnitude, stability or equivalence of distributions between humans and LLMs.
  • The Essays corpus uses human stream-of-consciousness and the LLM personas write 800-word personal stories; length, era, topic and prompt differ.
  • The human labels of Essays are binarized self-reports, whereas those of the LLM are explicit experimental assignments.
  • LIWC is proprietary and the repository publishes derived results, not an autonomous workflow that can be executed without a license.
  • The explicit mention filter only searches for concrete roots and may leave synonyms, paraphrases or equally transparent stereotypical descriptions.
  • Post-result selection excludes 96.56% of GPT-3.5 and almost a third of GPT-4, favoring stories that obey better and altering the evaluated population.
  • The human evaluation does not include GPT-3.5 or LLaMA 2, so it does not allow comparing human perception between models.
  • Only one story per profile is evaluated; effects of story, topic and combination are confounded with the trait.
  • No seed or selection rule among the eligible stories of each profile is published.
  • Thirty-two stories imply accuracy increments of 1/32=0.03125, so figures such as 0.84 have coarse resolution and wide unreported intervals.
  • The maximum of up to 80% corresponds to majority vote for extraversion in a filtered subset, not to general accuracy per persona or across the five traits.
  • The article describes the drop when revealing authorship as significant, but publishes no direct test or interval for the difference in accuracies.
  • Informed and uninformed raters may differ by composition; no model controlling for story, participant or batch is presented.
  • Near-zero or negative agreement shows that individual perception is unstable and limits the interpretation of majority vote as consensus.
  • Five votes per story are few to estimate a robust majority when the Likert labels include a neutral category.
  • Converting 1-2 to negative, 3 to neutral and 4-5 to positive and then evaluating against a binary label does not clearly explain how neutral is treated in accuracy.
  • Accuracy against 0.5 ignores possible neutral responses and is not accompanied by sensitivity, specificity, confusion matrix or calibration.
  • The multiple perception-BFI correlations do not document correction for multiplicity or intervals.
  • The BFI scores and the narratives come from the same context, so their correlation does not validate two independent measures.
  • The human sample is limited to English-speaking US residents with high approval on Prolific; 37 of 39 were born in the USA.
  • The effect of age, sex, ethnicity or annotator personality is not analyzed despite showing their distributions.
  • Five repetitions of the same LLM evaluator at temperature 0 are not independent judges and produce artificially low variance.
  • GPT-4 evaluates GPT-4 stories and shows scores of 5.00 with zero deviation, so it cannot replace an impartial human evaluation.
  • The comment sentiment difference is summarized with a mean of -1/0/1 labels without human validation or statistical testing.
  • LLaMA 2 was excluded from human evaluation due to repetition and explicit mentions, which creates selection by model performance.
  • Only 2023 models and endpoints, an English writing task and Big Five traits are studied; there is no dialogue, planning or real behavior.
  • The GPT-3.5-turbo-0613 and GPT-4-0613 endpoints no longer offer a stable surface for an identical contemporary replication.
  • The repository contains no requirements, environment, pyproject or lockfile, and uses an old OpenAI interface without library versions.
  • The published story script iterates list(product(...))[14:], so an execution from scratch omits the first 14 profiles although the complete outputs are versioned.
  • The repository does not include the human annotations, condition assignments or comments that the paper declares it will publish, preventing recalculation of tables 2 and 5-7.
  • The MIT license of the repository does not by itself clarify the redistribution rights of Essays, LIWC or other third-party artifacts included or derived.
  • There is no audit of stereotypes, toxicity or harm produced by antagonistic, irresponsible, neurotic or closed profiles.
  • The safety evaluation is limited to checking that the 32 selected stories do not contain harmful or offensive text.
  • There are no chatbot users, repeated interactions, trust, manipulation, emotional dependence or measured social impact.

What the study does not establish

  • It does not demonstrate that GPT-3.5, GPT-4 or LLaMA 2 have an internal personality.
  • It does not demonstrate that the BFI is psychometrically valid as a self-report of an LLM instructed with the expected profile.
  • It does not demonstrate trait stability outside the context that contains labels and BFI answers.
  • It does not demonstrate that the narratives are equivalent to human behavior or to realistic social agents.
  • It does not demonstrate equivalence between LIWC patterns of humans and LLMs; only partial overlap of selected associations.
  • It does not demonstrate that GPT-4 is more human in personality by sharing more significant correlations.
  • It does not demonstrate that 80% applies to the five traits, to individual raters or to unfiltered stories.
  • It does not demonstrate robust perception of neuroticism, openness or conscientiousness at the individual level.
  • It does not demonstrate human consensus, given the Krippendorff's alpha near zero.
  • It does not causally demonstrate that informing authorship reduces accuracy without a direct analysis of difference and control for participants.
  • It does not demonstrate that revealing authorship worsens the overall narrative rating; most of the means change little.
  • It does not demonstrate that the near-perfect scores of GPT-4 as evaluator are valid or impartial.
  • It does not demonstrate generalization to current models, other prompts, languages, cultures, modalities or interactive tasks.
  • It does not demonstrate that an induced personality improves personalization, utility, safety or user well-being.
  • It does not allow fully reproducing the human evaluation from the audited official repository.
  • It does not allow running a clean contemporary replication without reconstructing dependencies, correcting the generator and replacing retired endpoints.
  • It does not establish that the risks of personified agents are mitigated by disclosure of authorship alone.

Traceability

Scope: Full text

Version: Findings of NAACL 2024 final paper, pp. 3605-3627, DOI 10.18653/v1/2024.findings-naacl.229, 23 pages; official PersonaLLM repository v2.0 audited at commit 286c149fc52363127857ef2863091a4c377f065e

Consulted source: https://aclanthology.org/2024.findings-naacl.229.pdf

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, psychometric, LIWC, human-evaluation, inter-rater, multiple-testing, code-artifact, reproducibility, bias and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-3.5-turbo-0613 como generador y evaluador
  • GPT-4-0613 como generador y evaluador
  • LLaMA 2 70B Chat mediante Replicate, revisión 2d19859030ff705a87c746f7e96eea03aefb71f166725aee39692f1476566d48
  • cardiffnlp/twitter-roberta-base-sentiment-latest para comentarios humanos
  • LIWC-22 como software de análisis léxico

Instruments and metrics

  • Big Five Inventory (BFI), 44 ítems y puntuación Likert 1-5
  • Cinco pares binarios de adjetivos Big Five definidos en el system prompt
  • Linguistic Inquiry and Word Count 2022 (LIWC-22), 81 métricas psicológicas y de vocabulario extendido
  • Seis escalas narrativas: legibilidad, carácter personal, redundancia, cohesión, agrado y credibilidad
  • Cinco escalas de personalidad percibida con descriptores de facetas
  • Filtro lexicográfico de menciones explícitas de personalidad
  • Exactitud individual y por voto mayoritario
  • Correlación punto-biserial y Spearman
  • d de Cohen y pruebas de diferencia entre polos
  • Alfa de Krippendorff para acuerdo entre anotadores
  • Clasificador de sentimiento RoBERTa para comentarios opcionales

Data used

  • 960 personas sintéticas: 320 por GPT-3.5, GPT-4 y LLaMA 2
  • 960 respuestas BFI, diez por cada una de las 32 combinaciones y modelo
  • 960 historias personales generadas, 320 por modelo
  • 32 historias GPT-4 filtradas, una por combinación, para evaluación humana y automática
  • Essays dataset: 2.467 ensayos stream-of-consciousness humanos de 1997-2004 con Big Five binario
  • Comentarios opcionales: 104 válidos en condición informada y 122 en condición no informada
  • Repositorio oficial con salidas, textos, puntuaciones BFI, resultados LIWC y scripts; sin fichero localizable de anotaciones humanas en la revisión auditada

Evidence and location

  • Bibliographic record and official abstract: ACL Anthology 2024.findings-naacl.229: 6 authors, Findings of NAACL 2024, pp. 3605-3627, DOI 10.18653/v1/2024.findings-naacl.229
  • Full audited source: .cache/editorial-sources/article-098/source.pdf; official ACL PDF; 23 pages; sha256 4aebad9530ccb1ab3a4efc6d9b52e774689dd23474101460718a22dd4511c56f
  • Research questions and flow: Full text pp. 3605-3606, Introduction, Figure 1 and RQ1-RQ4
  • Models, profiles and prompts: Full text pp. 3606-3607, sections 2.1.1-2.1.4
  • BFI and effect sizes: Full text p. 3608, Figure 2 and section 3.1; appendix p. 3625, Table 9
  • LIWC and human comparison: Full text pp. 3607 and 3609-3610, sections 2.2.1 and 3.2, Table 1; appendix pp. 3625-3627
  • Filter and story selection: Full text p. 3607, section 2.2; appendix pp. 3620-3621, section D.1
  • Human and automatic evaluation design: Full text pp. 3608-3610, sections 2.2.2-2.3 and 3.3, Table 2
  • Individual and majority accuracy: Full text pp. 3611-3612, section 3.4.1 and Figures 3-4
  • Correlations between perception and BFI: Full text p. 3611, section 3.4.2
  • Examples and comments: Appendix pp. 3617-3620, Tables 3-4 and sections A-B
  • Inter-annotator agreement: Appendix pp. 3620-3623, Tables 6-7 and section D.3
  • Prolific sample and procedure: Appendix pp. 3621-3625, sections D.2-D.4 and Figures 5-18
  • LLM rater bias and temperature: Full text p. 3610, section 3.3; appendix p. 3625, Table 8
  • Acknowledged limitations and ethics: Full text pp. 3613-3614, Limitations and Ethical Considerations
  • Official repository and published outputs: Official https://github.com/hjian42/PersonaLLM branch v2.0, commit 286c149fc52363127857ef2863091a4c377f065e: 320 BFI and 320 writing outputs for each of GPT-3.5, GPT-4 and LLaMA 2
  • Statistical test discrepancy: Official repository run_llm_liwc_analysis.py at audited commit: personality_t_test selects scipy.stats.f_oneway or mannwhitneyu, while paper p. 3608 states paired t-tests
  • Reproducible code limitations: Official repository at audited commit: no dependency manifest; run_creative_writing.py iterates product profiles with [14:]; no human annotation dataset or comments located
  • Integral visual inspection: All 23 PDF pages rendered and visually inspected, including nine tables, eighteen figures, prompts, examples, human-study materials, demographics, limitations, ethics and appendices; checked 15 Jul 2026