Synthetic Personality Review

Is persona enough for personality? Using ChatGPT to reconstruct an agent's latent personality from simple descriptions

Evaluation and psychometric validity2024arXivApproved editorial review

Authors: Yongyi Ji, Zhisheng Tang, Mayank Kejriwal

Keywords: HEXACO personality framework, personality reconstruction, persona modeling, socio-demographic factors, personality consistency, latent dimensions, agent simulation

Source: Open primary source (opens in a new tab)

3
Authors
24
Findings
40
Limitations
19
Evidence

Editorial summary

English

The paper asks whether GPT-3.5-Turbo and GPT-4-Turbo can recover a HEXACO profile from a textual persona. Each persona combines one demographic sentence, age, binary gender, marital status, income, and number of children, with ten personality sentences: five of the six HEXACO dimensions are selected, each is assigned a high or low pole, and the sixth is deliberately omitted. GPT-3.5 first generates two sentences for each pole from HEXACO descriptions and also reorders the resulting eleven sentences. The evaluated model then answers all 60 HEXACO items on a 1-5 scale at temperature 0; each dimension's mean is dichotomized as high when above 2.5 and low otherwise. The study tests 1,000 personas with GPT-3.5 and a 100-person subset with GPT-4, yielding 5,000 and 500 comparisons on described dimensions. GPT-3.5 matches the assigned pole in 3,594 of 5,000 cases, 71.88%, but the aggregate conceals extreme variation: 78.51% for Honesty-Humility, 97.81% for Emotionality, 47.96% for Extraversion, 94.88% for Agreeableness, 60.87% for Conscientiousness, and 51.92% for Openness. Of 1,406 errors, 1,393-99.08%, change a low assignment to high. GPT-4 reaches 381 of 500-76.20%, with per-dimension rates of 92.68%, 100%, 79.22%, 60.49%, 55.06%, and 71.26%; 87 of its 119 errors are low-to-high, while 32 are high-to-low, all for Agreeableness. When a dimension is absent from the persona, GPT-3.5 classifies it as high in 986 of 1,000 cases, 98.6%, and GPT-4 in 89 of 100. This documents a strong protocol default rather than reliable reconstruction of latent information. The design itself favors high: the prompt forces an answer despite uncertainty, and the >2.5 cutoff classifies a neutral response of 3 as high. The manipulation is also contaminated before testing: supposedly high descriptions for Emotionality, Extraversion, and Agreeableness contain a second sentence characteristic of the low pole. High Extraversion, for example, combines social enjoyment with awkwardness under attention and a preference for being reserved; high Agreeableness combines cooperation with grudges, criticism, and anger. Some inconsistency therefore measures GPT-3.5-generated contradictions, not failure to infer a well-specified trait. The 77 ANOVA tests in Table 2 report only p-values, without effect sizes or multiplicity correction; they apply univariate ANOVA to dichotomized outcomes, and the aggregate concatenates six dimensions without explaining dependence. Age and number of children are significant in the aggregate, but demographic variables were not randomized independently: construction rules couple age, marriage, and children. There are no real participants, human validation of personas, repetitions, psychometric validation of HEXACO for LLMs, code, data, or exact model snapshots. The defensible conclusion is that 2024 models follow explicit behavioral instructions unevenly and tend to answer HEXACO toward high poles, not that they reconstruct latent human personality or psychologically valid demographic associations.

Español

El trabajo pregunta si GPT-3.5-Turbo y GPT-4-Turbo pueden recuperar un perfil HEXACO a partir de una persona textual. Cada persona combina una frase sociodemográfica, edad, género binario, estado civil, ingresos y número de hijos, con diez frases de personalidad: se eligen cinco de las seis dimensiones HEXACO, se asigna a cada una un polo alto o bajo y se omite deliberadamente la sexta. GPT-3.5 genera previamente dos frases para cada polo a partir de descripciones HEXACO y también reordena las once frases. Después el modelo evaluado responde los 60 ítems HEXACO en escala 1-5 con temperatura 0; la media de cada dimensión se dicotomiza como alta si es mayor que 2,5 y baja en caso contrario. Se prueban 1.000 personas con GPT-3.5 y una submuestra de 100 con GPT-4, lo que produce 5.000 y 500 comparaciones en dimensiones descritas, respectivamente. GPT-3.5 coincide con el polo asignado en 3.594 de 5.000 casos, 71,88 %, pero el agregado oculta diferencias extremas: 78,51 % en honestidad-humildad, 97,81 % en emocionalidad, 47,96 % en extraversión, 94,88 % en amabilidad, 60,87 % en responsabilidad y 51,92 % en apertura. De sus 1.406 errores, 1.393, 99,08 %, convierten un polo bajo en alto. GPT-4 alcanza 381 de 500, 76,20 %, con 92,68 %, 100 %, 79,22 %, 60,49 %, 55,06 % y 71,26 % por dimensión; 87 de sus 119 errores son bajo→alto y 32 alto→bajo, todos estos últimos en amabilidad. Cuando la dimensión no aparece en la persona, GPT-3.5 la clasifica alta en 986 de 1.000 casos, 98,6 %, y GPT-4 en 89 de 100. Esto documenta un fuerte valor por defecto del protocolo, no una reconstrucción fiable de información latente. El propio diseño favorece “alto”: el prompt obliga a responder aun con incertidumbre y el umbral >2,5 clasifica una respuesta neutral de 3 como alta. Además, la manipulación está contaminada antes del test: las descripciones supuestamente altas de emocionalidad, extraversión y amabilidad incluyen una segunda frase propia del polo bajo. La de extraversión alta, por ejemplo, combina disfrute social con incomodidad al ser el centro de atención y preferencia por ser reservado; la de amabilidad alta combina cooperación con rencor, crítica e ira. Por tanto, parte de la inconsistencia mide contradicciones generadas por GPT-3.5, no incapacidad para inferir un rasgo bien especificado. Las 77 pruebas ANOVA de la Tabla 2 informan solo p-valores, sin tamaños de efecto ni corrección por multiplicidad; aplican ANOVA univariante a resultados dicotómicos y el agregado concatena seis dimensiones sin explicar la dependencia. Edad y número de hijos son significativos en el agregado, pero las variables demográficas no fueron aleatorizadas de forma independiente: las reglas de construcción vinculan edad, matrimonio e hijos. No hay personas reales, validación humana de las personas, repetición, análisis psicométrico de HEXACO en LLM, código, datos ni instantáneas exactas de los modelos. La conclusión defendible es que modelos de 2024 siguen instrucciones conductuales explícitas de forma desigual y tienden a contestar el HEXACO hacia polos altos; no que reconstruyan una personalidad humana latente ni que las asociaciones demográficas sean psicológicamente válidas.

Research question

With what consistency do GPT-3.5-Turbo and GPT-4-Turbo reproduce explicit HEXACO poles from person descriptions, what value do they assign to an omitted dimension, and what sociodemographic or personality variables are associated with the reconstructed dimensions?

Method

Synthetic persons are randomly generated with five sociodemographic attributes subject to three restrictions and five of six HEXACO dimensions in binary poles. GPT-3.5-Turbo converts official descriptions of each pole into two sentences and reorders one demographic sentence plus ten personality sentences. With that person as the system message, the model responds to the 60 HEXACO items with values 1-5 at temperature 0. After inverting the corresponding items, ten items per dimension are averaged and classified as high if the mean is >2.5. 1,000 persons are evaluated in GPT-3.5 and 100 of them in GPT-4. Assigned and reconstructed poles are compared, the omitted dimension is studied, and univariate ANOVAs are run to relate eleven input variables to six outputs and one aggregated output.

Sample: The experiment does not include persons. Each of the 1,000 synthetic persons contributes five described dimensions, for 5,000 comparisons in GPT-3.5; GPT-4 uses 100 persons and 500 comparisons. The omitted dimension produces another 1,000 and 100 classifications. The age intervals are 18-29, 30-49, 50-64, and 65-80; gender is limited to male/female; marital status to single/married/divorced; income to three intervals; and children to zero, one, or more than one.

Findings

  • GPT-3.5 matches the described pole in 3,594 of 5,000 dimensions, 71.88 %.
  • Its consistency varies from 97.81 % in emotionality and 94.88 % in agreeableness to 47.96 % in extraversion.
  • The remaining GPT-3.5 rates are 78.51 % in honesty-humility, 60.87 % in conscientiousness, and 51.92 % in openness.
  • Of 1,406 GPT-3.5 errors, 1,393 are low->high and only 13 high->low; these thirteen occur in agreeableness.
  • For low extraversion, GPT-3.5 reconstructs 433 as high and only 3 as low; the accuracy of the low pole is 0.69 %.
  • For low conscientiousness, GPT-3.5 gets 95 of 426 correct; for low openness, 39 of 439.
  • GPT-4 matches in 381 of 500 dimensions, 76.20 %, only 4.32 points above the GPT-3.5 aggregate.
  • GPT-4 reaches 100 % in emotionality, 92.68 % in honesty-humility, and 79.22 % in extraversion.
  • GPT-4 drops to 60.49 % in agreeableness, 55.06 % in conscientiousness, and 71.26 % in openness.
  • In GPT-4, 87 errors are low->high and 32 high->low; the 32 high->low belong to agreeableness.
  • With the omitted dimension, GPT-3.5 responds high in 986 of 1,000 cases and GPT-4 in 89 of 100.
  • The high default appears in all six dimensions and produces only 14 omitted low poles in GPT-3.5 and 11 in GPT-4.
  • The threshold >2.5 classifies the neutral response 3 as high and the prompt forbids abstaining, two decisions that push the result toward high.
  • The high descriptions of emotionality, extraversion, and agreeableness contain sentences semantically proper to the low pole.
  • The high description of agreeableness mixes forgiveness and cooperation with resentment, criticism, and anger; GPT-4 reconstructs 32 of 51 high cases as low.
  • The high description of extraversion mixes sociability with discomfort under attention and preference for reserve.
  • The high description of emotionality mixes seeking support with absence of fear and emotional detachment.
  • Table 2 reports 77 p-values corresponding to eleven input variables by seven outputs.
  • In the aggregate, age has p=0.00842 and number of children p=9.96e-06; the other demographic variables are not significant in that column.
  • Income and gender are associated with emotionality; gender also with conscientiousness; marital status and number of children with some individual traits.
  • All input personality dimensions except extraversion show association with the aggregated output.
  • The cell age->honesty-humility shows p=0.03009 but lacks the asterisk that the legend would assign to p<0.05.
  • The arXiv record confirms a single version, accepted at the ICML 2024 Workshop on Large Language Models and Cognition.
  • No official publication of code, persons, responses, results, or seeds was identified.

Limitations

  • The experiment measures adherence to instructions that describe explicit HEXACO behaviors, not inference from a minimal or indirect person.
  • The input sentences and the output items share narrow semantic content, so conceptual circularity exists.
  • GPT-3.5 generates the manipulations that GPT-3.5 then evaluates, without human review of clarity, polarity, or contradiction.
  • Three of six high descriptions contain low-pole content, invalidating a clean manipulation.
  • The inconsistency per dimension mixes model capacity with stimulus construction errors.
  • The continuous HEXACO traits are reduced to high/low and intensity, facets, and uncertainty are lost.
  • The 2.5 cutoff is not psychometrically justified and places neutral=3 in the high pole.
  • Forcing a response even when the model is not confident prevents measuring absence of information or calibration.
  • Calling the completion of an omitted dimension a hallucination confuses a forced response to a questionnaire with an invented factual statement.
  • There is no condition without personality description that estimates the model baseline under the same questionnaire.
  • The threshold is not compared with alternatives such as 3, neutral intervals, or continuous scores.
  • Internal reliability, factorial structure, invariance, convergent validity, and test-retest of the HEXACO applied to the LLM are not evaluated.
  • Temperature 0 does not guarantee reproducibility of a remote API nor does it replace replications.
  • Exact snapshots of GPT-3.5-Turbo and GPT-4-Turbo are not published.
  • There are no replications per person, confidence intervals, or sensitivity analyses to order and formulation.
  • Reordering with GPT-3.5 adds an unaudited transformation and may modify contextual associations despite preserving sentences.
  • The random selection of dimensions, poles, ages, and income does not include seeds or code.
  • GPT-4 uses only a tenth of the persons and the article does not demonstrate that it is a sample stratified by poles and demographics.
  • The aggregate rates weight dimensions with different sizes due to the randomly omitted dimension.
  • The 71.88 % hides performance near or below chance in three dimensions and nearly zero for some low poles.
  • The high bias may jointly stem from the model, the cutoff, the items, the forced response, and the contaminated sentences; the design does not separate them.
  • The matrices do not include intervals or tests of difference between models or dimensions.
  • The 77 ANOVA tests do not apply correction for multiple comparisons.
  • Only p-values are published, without effect sizes, means, variances, degrees of freedom, or assumptions.
  • ANOVA with dichotomous output may violate normality and homoscedasticity; a binomial model would have been more appropriate.
  • The aggregated output concatenates six correlated dimensions and multiple observations per person, violating independence if treated as a single variable.
  • The tests are univariate and do not simultaneously control for the other variables.
  • Age, marital status, and children are linked by explicit rules, so their associations are confounded by construction.
  • Income is generated in three intervals without region, contextualized currency, or representative population distribution.
  • Gender is restricted to male/female and family structure to simple normative categories.
  • The sociodemographic variables are fictitious and there is no human evidence that allows interpreting their associations as real psychology.
  • Interaction between demographics and the personality sentence is not studied, despite the article recommending complete sociodemographic information.
  • The asterisk inconsistency for p=0.03009 shows an editorial error in the significance table.
  • The article does not include a specific limitations section, demographic bias assessment, or harm analysis.
  • Languages, cultures, open models, non-GPT models, or later versions are not evaluated.
  • There are no conversations, memory, actions, or decisions that prove behavioral stability of the agents.
  • No comparison is made with human evaluation of the same persons or with known human HEXACO profiles.
  • No code, data, persons, responses, logs, or ANOVA results are published for independent audit.
  • The single arXiv version does not document subsequent revisions or corrections of the contradictions of Table 1.
  • The Impact Statement is limited to describing relevance for agents and does not address personification, stereotypes, or profiling uses.

What the study does not establish

  • It does not demonstrate that GPT-3.5 or GPT-4 have a latent personality.
  • It does not demonstrate that they reconstruct human personality from simple sociodemographic descriptions.
  • It does not demonstrate psychometric validity of HEXACO for LLM agents.
  • It does not demonstrate high consistency across all dimensions or both poles.
  • It does not demonstrate that GPT-4 robustly outperforms GPT-3.5; there is no statistical test and several dimensions remain weak.
  • It does not demonstrate that the omitted dimension is inferred; it mainly shows a high default under mandatory response.
  • It does not demonstrate that the high default is solely a model bias and not an artifact of the cutoff.
  • It does not demonstrate that a well-constructed textual person causes the errors, because three high manipulations are contradictory.
  • It does not demonstrate independent causal effects of age or number of children.
  • It does not demonstrate valid demographic associations in real persons.
  • It does not demonstrate that the p-values survive correction for 77 comparisons.
  • It does not demonstrate stability across replications, API versions, orders, or prompts.
  • It does not demonstrate generalization to other cultures, languages, models, or personality instruments.
  • It does not demonstrate that these profiles produce consistent behavior in conversations or external tasks.
  • It does not allow reproducing the results from published official artifacts.
  • It does not establish that agents built this way are realistic, safe, or suitable for social simulation.

Traceability

Scope: Full text

Version: arXiv:2406.12216v1, submitted 18 Jun 2024, 9 pages; accepted to the ICML 2024 Workshop on Large Language Models and Cognition; no official code or data release identified in the paper, arXiv/OpenReview records, or targeted author/project search

Consulted source: https://arxiv.org/pdf/2406.12216

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, HEXACO, prompt-contamination, dichotomization, per-pole consistency, omitted-dimension, ANOVA, demographic-confounding, reproducibility, release, bias and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-3.5-Turbo, instantánea no especificada, como generador, reordenador y modelo evaluado
  • GPT-4-Turbo, instantánea no especificada, como modelo evaluado en 100 personas
  • OpenAI API a temperatura 0

Instruments and metrics

  • HEXACO-60, diez ítems por dimensión y escala 1-5
  • Seis dimensiones HEXACO: honestidad-humildad, emocionalidad, extraversión, amabilidad, responsabilidad y apertura
  • Descripciones binarias alto/bajo generadas por GPT-3.5 desde definiciones oficiales
  • Umbral de dicotomización: media >2,5 alta; media <=2,5 baja
  • Matrices de consistencia por dimensión
  • ANOVA de una vía y p-valores sin tamaños de efecto

Data used

  • 1.000 personas sintéticas para GPT-3.5-Turbo
  • Submuestra de 100 de esas personas para GPT-4-Turbo
  • 60.000 respuestas HEXACO de GPT-3.5 y 6.000 de GPT-4 implícitas por el protocolo, no publicadas
  • Cinco atributos sociodemográficos sintéticos por persona
  • Diez frases de cinco polos HEXACO por persona y una dimensión omitida

Evidence and location

  • Record, version, and acceptance: arXiv:2406.12216v1, submitted 18 Jun 2024; accepted to ICML 2024 Workshop on Large Language Models and Cognition
  • Full audited source: .cache/editorial-sources/article-101/source.pdf; official arXiv PDF; 9 pages; sha256 3030a6e640dadbc7ead0f4d5425e459f552cea5584a3617fb0250ca59a9af469
  • Objective and claims: Full text pp. 1-2, Abstract and Introduction
  • Sociodemographic variables and restrictions: Full text p. 2, section 2
  • HEXACO sentence construction and contradictions: Full text pp. 2-3, section 2 and Table 1
  • HEXACO-60, prompt, and samples: Full text p. 3, section 3
  • Aggregate consistency and low-high bias: Full text p. 3 and appendix p. 6, section 4 and Figure 1
  • ANOVA and associations: Full text p. 4, Table 2 and section 4
  • Authors' interpretation: Full text p. 4, Discussion and Impact Statement
  • Cutoff and scoring: Appendix p. 6 and p. 9, section A.1 and Table 8
  • Sentence generation prompt: Appendix p. 6, Table 3
  • Omitted dimensions GPT-3.5: Appendix p. 7, Figure 2: 986 high and 14 low across 1,000 omitted dimensions
  • Reordering prompt: Appendix p. 7, Table 4
  • Complete items and matrices per dimension: Appendix p. 8, Tables 5-7
  • Omitted dimensions GPT-4: Appendix p. 9, Figure 3: 89 high and 11 low across 100 omitted dimensions
  • Independent recalculation GPT-3.5: Table 6 cross-check: per-dimension consistency 78.51%, 97.81%, 47.96%, 94.88%, 60.87%, 51.92%; total 3,594/5,000
  • Independent recalculation GPT-4: Table 7 cross-check: per-dimension consistency 92.68%, 100%, 79.22%, 60.49%, 55.06%, 71.26%; total 381/500
  • Unidentified artifacts: Paper, arXiv/OpenReview records and targeted official author/project search checked 15 Jul 2026; no official code or data release located
  • Comprehensive visual inspection: All 9 PDF pages rendered and visually inspected, including eight tables, three figures, all prompts, HEXACO items, scoring rules, matrices, ANOVA and Impact Statement; checked 15 Jul 2026