Synthetic Personality Review

Evaluating the Efficacy of LLMs to Emulate Realistic Human Personalities

Evaluation and psychometric validity2024AAAI ProceedingsApproved editorial review

Authors: Lawrence J. Klinkert, Steph Buongiorno, Corey Clark

Keywords: personality emulation, NPC behavior, game AI, personality alignment, human-like traits, LLM evaluation, interactive agents

Source: Open primary source (opens in a new tab)

3
Authors
32
Findings
50
Limitations
31
Evidence

Editorial summary

English

The paper proposes a benchmark for testing whether seven LLMs answer the IPIP-50 consistently with one of twenty target Big Five profiles, motivated by later use in video-game characters. It starts from 1,015,342 human questionnaires from the Open-Source Psychometrics Project; after requiring complete responses, a minimum response time, and the first submission per IP, the paper reports 596,956 cases. Five trait scores are normalized, each person is assigned by Euclidean distance to the nearest profile, and a balanced reference of 50,500 people, 2,525 per profile, is constructed. The twenty prototypes do not come from a taxonomy validated in the study; they are taken from a 2018 web page and include disorder-derived names and proposed opposites that function here only as non-diagnostic labels. For each profile, the prompt supplies the exact OCEAN vector and asks the model to answer all 50 items separately on a 1–5 Likert scale. The main table reports nearest-centroid accuracies of 5.00% for Flan-T5-XL, 5.00% for Flan-T5-XXL, 15.10% for text-davinci-003, 31.90% for GPT-3.5, 74.00% for GPT-4, 7.11% for Dolphin-2.7-Mixtral, and 6.33% for Mixtral-8x7B-Instruct. The 100% highlighted in the abstract is not global accuracy: Table 4 shows that, for particular profiles, every GPT-4 response lies inside the convex hull of the corresponding human cloud, which is a different metric. Auditing the GitLab artifact yields recomputed accuracies of approximately 5.00%, 5.00%, 15.08%, 31.90%, 71.64%, 7.11%, and 5.16%, so its GPT-4 and Mixtral figures do not match the paper. Artifact sample sizes also range from 3 to 64 or 128 repetitions per profile even though the paper states 64 for every model. The implementation filters human response times above 1,000 ms rather than the paper's 300 ms and computes five separate squared one-dimensional Mahalanobis distances, not one five-dimensional distance or an average number of standard deviations. The full human data and 50,500-person reference are not released; only a 4,500-person subset is present. The evidence does show substantial model differences under these prompts and that GPT-4 is closest to the chosen prototypes under the selected metrics. It does not establish human personality, realistic decision-making, stable behavior, or convincing NPCs: the game example is illustrative and includes no players, longitudinal dialogue, or behavioral evaluation. Reproducibility is further weakened by missing data, local paths, untraceable cleaned outputs, a script that fails to compile, weak dependency control, and an exposed API credential in the code that should be revoked and rotated.

Español

El artículo propone un benchmark para comprobar si siete LLM responden el IPIP-50 de forma compatible con uno de veinte perfiles Big Five objetivo, pensando en su uso posterior para personajes de videojuegos. Parte de 1.015.342 cuestionarios humanos del Open-Source Psychometrics Project; tras exigir respuestas completas, un tiempo mínimo y la primera participación por IP, el texto declara 596.956 casos. Normaliza los cinco rasgos, asigna cada persona por distancia euclídea al perfil más cercano y forma una referencia equilibrada de 50.500 personas, 2.525 por perfil. Los veinte prototipos no proceden de una taxonomía validada en el estudio, sino de una página web de 2018, e incluyen nombres tomados de trastornos y supuestos opuestos que aquí funcionan solo como etiquetas no diagnósticas. Para cada perfil, el prompt entrega al modelo el vector OCEAN exacto y pide responder por separado los 50 ítems con una escala Likert de 1 a 5. La tabla principal publica exactitudes de clasificación al centroide de 5,00 % para Flan-T5-XL, 5,00 % para Flan-T5-XXL, 15,10 % para text-davinci-003, 31,90 % para GPT-3.5, 74,00 % para GPT-4, 7,11 % para Dolphin-2.7-Mixtral y 6,33 % para Mixtral-8x7B-Instruct. El 100 % destacado en el abstract no es exactitud global: la Tabla 4 muestra que, en perfiles concretos, todas las respuestas de GPT-4 caen dentro de la envolvente convexa de la nube humana; es una métrica diferente. La auditoría del artefacto GitLab permite recomputar aproximadamente 5,00 %, 5,00 %, 15,08 %, 31,90 %, 71,64 %, 7,11 % y 5,16 %, por lo que las cifras públicas de GPT-4 y Mixtral no coinciden con el artículo. También aparecen tamaños de muestra de 3, 64 o 128 repeticiones por perfil, pese a que el texto afirma 64 para todos. La implementación filtra tiempos humanos por encima de 1.000 ms, no 300 ms como dice el paper, y calcula cinco distancias de Mahalanobis unidimensionales al cuadrado, no una única distancia en cinco dimensiones ni un promedio de desviaciones estándar. Además, los datos humanos completos y la referencia de 50.500 no están publicados: solo aparece un subconjunto de 4.500. La evidencia sí muestra que los modelos producen distribuciones de respuestas muy diferentes bajo estos prompts y que GPT-4 es el más cercano a los prototipos según las métricas elegidas. No demuestra personalidad humana, toma de decisiones realista, conducta estable ni NPC convincentes: el caso de juego es ilustrativo, sin jugadores, diálogo longitudinal ni evaluación conductual. La reproducibilidad queda debilitada por datos ausentes, rutas locales, resultados limpiados sin trazabilidad, un script que no compila, dependencias poco controladas y una credencial de API expuesta en el código, que debe revocarse y rotarse.

Research question

To what extent do different LLMs, conditioned with Big Five vectors of twenty target profiles, respond to the 50 IPIP items in a manner comparable to human distributions assigned to those same profiles, and can this procedure serve as an initial benchmark for designing NPCs with personality?

Method

Comparative study without new participants. The authors download 1,015,342 human responses to the IPIP-50, apply completeness, time and IP filters, normalize OCEAN and assign each case to the nearest of twenty fixed centroids using Euclidean distance. They build a balanced reference of 50,500 cases. Seven LLMs receive in each prompt the numeric vector of the profile and a randomized IPIP item; they return a Likert option from 1 to 5. Responses are scored with the IPIP key and compared via classification to the centroid, squared error, membership in the human convex hull and an implementation called Mahalanobis. The public artifact and its NPY matrices are additionally audited to contrast samples, figures, cleaning and reproducibility with the final text.

Sample: The article declares 1,015,342 initial human questionnaires, 596,956 after cleaning and a final balanced sample of 50,500, with 2,525 people in each of twenty profiles. No new participants are collected. The text states 64 synthetic repetitions per profile and model, equivalent to 1,280 evaluated profiles and 64,000 item responses per model, but the artifacts contain 60 cases for Flan-XL, 1,280 for Flan-XXL, Dolphin and Mixtral, 2,560 for Davinci and GPT-4, and 2,558 for GPT-3.5. Only a human subset of 4,500 cases is published, not the three declared human samples.

Findings

  • The study evaluates synthetic responses to questionnaires under an imposed profile; it does not observe a spontaneous personality of the model.
  • The 1,015,342 questionnaires come from a preexisting public dataset and not from participants recruited for this work.
  • The text declares 596,956 cases after requiring the 50 complete items, applying a time threshold and keeping the first response per IP.
  • Each human response is normalized to five OCEAN scores between zero and one.
  • The human profile label is obtained by assigning the score to the nearest of twenty centroids by Euclidean distance.
  • The baseline artificially balances the classes with 2,525 cases per profile, up to 50,500.
  • The twenty centroids are taken from a 2018 web page and their values are not estimated or validated with the dataset of this study.
  • The names of several profiles come from personality disorder terminology, but the experiment uses them as geometric labels, not as diagnoses.
  • The prompt communicates to the LLM the exact OCEAN vector of the profile before each item.
  • The 50 items are asked separately, in random order and with a closed Likert response from 1 to 5.
  • The accuracy task checks whether the synthetic score returns to the same centroid that was supplied in the prompt.
  • Table 2 publishes 5.00% accuracy for Flan-T5-XL and 5.00% for Flan-T5-XXL.
  • Table 2 publishes 15.10% for text-davinci-003 and 31.90% for GPT-3.5.
  • Table 2 publishes 74.00% for GPT-4, the best result in the main comparison.
  • Table 2 publishes 7.11% for Dolphin-2.7-Mixtral and 6.33% for Mixtral-8x7B-Instruct.
  • The bands accompanying the accuracies are described as 95% intervals, but the article does not provide their formula or resampling unit.
  • Inclusion in the convex hull of Table 4 is a metric distinct from classification to the centroid of Table 2.
  • GPT-4 reaches 1.000 convex inclusion for Pronoid and People person, not 100% global accuracy.
  • The average of the twenty GPT-4 values in Table 4 is approximately 0.729, consistent with a strong but not perfect pattern.
  • The Flan models do not place points within the human hull in the majority of profiles and only recover one profile in global classification.
  • The repository allows recomputing 5.00%, 5.00%, 15.08%, 31.90%, 71.64%, 7.11% and 5.16% for the seven comparable artifacts.
  • The recomputed results for GPT-4 and Mixtral differ materially from the published 74.00% and 6.33%; the others match by rounding.
  • The Flan-XL artifacts contain only three cases per profile, not the 64 claimed repetitions.
  • Davinci and GPT-4 contain 128 cases per profile; GPT-3.5 almost 128, with two missing rows.
  • The implementation filters human response times of up to 1,000 ms, whereas the article says it eliminates times below 300 ms.
  • The R code computes Mahalanobis trait by trait with one-dimensional inputs and averages the squared results.
  • For this reason Table 5 does not represent the joint five-dimensional Mahalanobis distance that the prose describes.
  • The R package returns squared Mahalanobis distance, so it also does not directly equate to a mean number of standard deviations.
  • The raw matrices of some models contain zeros derived from parsing failures; the clean variants substitute or remove them without documenting the procedure in the article.
  • The video game example illustrates how to use a personality vector for prompting, but it does not contain an experiment with NPCs, players or interactive behavior.
  • The comparative evidence favors GPT-4 within the defined benchmark, not a general equivalence between GPT-4 outputs and human personality.
  • The final article appears in AIIDE 2024, volume 20(1), pages 65-75, DOI 10.1609/aiide.v20i1.31867.

Limitations

  • The human assignment and the synthetic evaluation depend on the same twenty fixed centroids, which introduces circularity in the definition of success.
  • The model receives exactly the vector it must recover; the task measures adherence to a numeric condition rather than autonomous discovery of a personality.
  • The source of the twenty profiles is a non-peer-reviewed web page and the study provides no psychometric validation of the taxonomy.
  • Labels with disorder names may suggest clinical validity or reinforce stereotypes even though here they only represent OCEAN vectors.
  • It is not demonstrated that the supposed opposite profiles are valid or symmetric psychological constructs.
  • Balancing at 2,525 cases per profile removes natural prevalence and may hide that some centroids describe much more frequent regions.
  • Euclidean distance assumes comparable scales, independence and spherical geometry of the five traits without justifying those assumptions.
  • Stability of the human label against alternative centroids, different scaling or uncertainty near boundaries is not evaluated.
  • IPIP responses are self-reported and online; no identity verification, diagnosis or population representativeness is provided.
  • Filtering by IP address may remove cohabitants or shared networks and does not guarantee unique persons.
  • The time threshold contradicts the code: 300 ms in the article versus 1,000 ms in the implementation.
  • The raw human data, the 596,956-case clean set and the 50,500-case baseline used in the tables are not published.
  • The only public human subset has 4,500 cases, less than a tenth of the declared baseline.
  • The claim of 64 repetitions per model contradicts public sizes equivalent to 3, 64 or 128 per profile.
  • It is not explained why Davinci, GPT-3.5 and GPT-4 have approximately twice as many repetitions as those described.
  • The two missing rows of GPT-3.5 require constructing labels by position and truncating them, a decision that is not documented.
  • Temperature, top-p, seeds, decoding parameters and retries are not fully specified for all providers.
  • The API models are fixed to 2023 versions and some can no longer be queried, so exact regeneration depends on the artifacts.
  • Contamination is not controlled: the LLMs may have seen IPIP, Big Five explanations or even the profile vectors during training.
  • Asking each item in a separate call eliminates questionnaire context, memory, fatigue and interdependence among responses.
  • A forced Likert scale measures format compliance and not richness of dialogue, action or decision in a game.
  • The parser searches for textual fragments of the options; a malformed response is silently converted to zero.
  • Zeros may be interpolated to the lower extreme instead of being flagged as errors, altering scores and profiles.
  • The clean variants of Mistral, Dolphin and Mixtral do not document which responses were substituted, by what rule or from which retry.
  • A raw Mistral artifact contains almost all zeros due to format failures and only the clean result approximates the presented figure.
  • The GPT-4 and Mixtral differences between table and artifact have no explanation or traceable generation script.
  • The article claims that 64 repetitions guarantee statistical significance, but size alone does not guarantee significance or validity.
  • It is not defined how the confidence intervals in Table 2 were calculated or whether the unit is profile, repetition or item.
  • No adjustment is made for multiple comparisons across seven models, twenty profiles, five traits and several metrics.
  • The convex hull is very sensitive to the size, dimensionality and dispersion of the balanced human sample.
  • Belonging to a broad convex hull does not imply high density, individual resemblance or psychological plausibility.
  • Table 3 calls MSPE a measure based on Euclidean distance without offering a derivation sufficient to reproduce it from the text.
  • The Mahalanobis implementation does not match the joint five-dimensional description.
  • The R output is squared distance and cannot be directly interpreted as a mean of standard deviations.
  • The PCA visualization of the final notebook is fit on GPT-4 synthetic data after overwriting the human variable.
  • That PCA is then applied to the other models and may visually favor the structure of GPT-4 over the expected human reference.
  • The partition used for PCA does not fix random_state, so the figures are not deterministic.
  • The accuracy table in the notebook is manually written in Markdown, not calculated in the shown execution.
  • The repository contains absolute Windows paths and references to missing files, so it does not run end to end in another environment.
  • There is no root README, license, automated tests or continuous integration that explain or verify the artifact.
  • A Python file contains invalid syntax and compileall fails, in addition to warnings for paths with invalid escapes.
  • requirements.txt lacks versions for most packages, duplicates dependencies and retains a legacy OpenAI API.
  • The repository mixes IDE files, R state, caches, binaries, PDFs and results, without a reproducible separation between source and artifacts.
  • There is an OpenAI credential with an active appearance written directly in the code; it must be revoked, rotated and also removed from the history.
  • Consent, privacy and the implications of grouping responses by IP and reusing psychometric data are not documented.
  • Demographic, cultural, linguistic or accessibility differences in the human sample are not evaluated.
  • The study is limited to IPIP-50 and does not contrast longer instruments, interviews, observers or real behavior.
  • There is no human evaluation by psychologists, game designers or players.
  • Consistency across multiple turns, sessions, memories or context changes is not tested.
  • The NPC prototype is a conceptual demonstration without measures of immersion, engagement, credibility, safety or fun.

What the study does not establish

  • It does not demonstrate that an LLM has personality, emotions, intentions, identity or subjective experience.
  • It does not demonstrate that responding as an IPIP profile equates to making realistic human decisions.
  • It does not establish that the twenty profiles are a valid psychological taxonomy.
  • It does not turn labels with clinical names into diagnoses of persons or models.
  • It does not demonstrate that GPT-4 reaches 100% global accuracy; the maximum figure is specific to profiles and to the convex hull.
  • It does not prove that all GPT-4 responses are indistinguishable from human responses.
  • It does not prove that a point within the convex hull is probable, typical or individually human.
  • It does not establish that 74% is reproducible with the public artifact, which yields approximately 71.64%.
  • It does not establish the 6.33% for Mixtral from the comparable artifact, which produces approximately 5.16%.
  • It does not demonstrate statistical significance merely by repeating prompts 64 times.
  • It does not allow interpreting Table 5 as a joint five-trait Mahalanobis distance.
  • It does not allow interpreting its values as a mean number of standard deviations.
  • It does not demonstrate that differences between models are due solely to size or to being local versus frontier models.
  • It does not separate instruction-following ability, knowledge of the test, response format and personality representation.
  • It does not demonstrate robustness to alternative prompts, languages, question order or different scales.
  • It does not demonstrate generalization to HEXACO, MBTI, IPIP-120, IPIP-300, emotion or other constructs.
  • It does not demonstrate consistency of personality in a prolonged conversation.
  • It does not demonstrate transfer from questionnaire responses to actions, decisions or social relationships of an NPC.
  • It does not demonstrate that the resulting characters improve player immersion or engagement.
  • It does not evaluate safety, toxicity, biases or stereotypes induced by the profiles.
  • It does not provide a reproducible package that regenerates the published tables from scratch.
  • It does not justify using the benchmark as clinical validation, personnel selection or inference of real personality.

Traceability

Scope: Full text

Version: AIIDE 2024 final proceedings paper, volume 20 issue 1, pp. 65-75, DOI 10.1609/aiide.v20i1.31867, 11 pages; code artifact commit 143dc5f71b81ff0a2746befd87d400909ec6f624

Consulted source: https://ojs.aaai.org/index.php/AIIDE/article/download/31867/34034

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, psychometric, statistical, artifact-reproduction, code-quality, security, privacy and claim-scope audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • Flan-T5-XL
  • Flan-T5-XXL
  • text-davinci-003
  • GPT-3.5 Turbo (gpt-3.5-turbo-0613)
  • GPT-4 (gpt-4-0613)
  • Dolphin 2.7 Mixtral
  • Mixtral 8x7B Instruct

Instruments and metrics

  • International Personality Item Pool Big-Five Factor Markers (IPIP-50)
  • Cinco Grandes / OCEAN normalizados a [0,1]
  • Veinte perfiles prototipo atribuidos a Van Mensvoort (2018)
  • Asignación al centroide más cercano por distancia euclídea
  • Exactitud de recuperación del perfil objetivo
  • Mean Squared Prediction Error (MSPE)
  • Inclusión en la envolvente convexa de datos humanos
  • Distancias denominadas Mahalanobis en el artículo
  • PCA para visualización

Data used

  • Open-Source Psychometrics Project IPIP Big-Five Factor Markers responses: 1.015.342 registros iniciales
  • Conjunto humano filtrado declarado: 596.956 respuestas completas
  • Baseline humano equilibrado declarado: 50.500 casos, 2.525 por perfil
  • Subconjunto público clean_subset_225: 4.500 casos, 225 por perfil
  • Matrices NPY de respuestas y evaluaciones sintéticas del repositorio GitLab humin-game-lab/llm-personality

Evidence and location

  • Official bibliographic record: AAAI AIIDE article 31867: Lawrence J. Klinkert, Steph Buongiorno and Corey Clark; volume 20(1), pp. 65-75; DOI 10.1609/aiide.v20i1.31867; published 15 Nov 2024
  • Complete audited source: .cache/editorial-sources/article-093/source.pdf; official AIIDE 2024 PDF; 11 pages; sha256 469f46e154a9c6454ec48643d35b6f21069586986b0900c19b65e5c1428605f9
  • Objective, scope and stated conclusion: Full text pp. 65-66, Abstract and Introduction
  • Origin and cleaning of the human set: Full text pp. 68-69, Human Baseline; 1,015,342 initial and 596,956 retained records
  • Time threshold discrepancy: Full text p. 68 states 300 ms; GitLab commit 143dc5f, fall_23/scripts/main_zoo.py lines 88-103 applies 1,000 ms
  • Normalization and assignment to profiles: Full text pp. 67-69, Personality Representation and Human Baseline
  • Provenance of the twenty prototypes: Full text p. 67, Figure 1 and citation Mensvoort 2018; GitLab notebook_commons.py lines 372-392 contains fixed OCEAN vectors
  • Balanced baseline of 50,500: Full text p. 69, Human Baseline and Table 1; 2,525 cases for each of 20 profiles
  • Prompt design and Likert scale: Full text pp. 69-70, Generate Synthetic Data and displayed Laissez-faire prompt
  • Declared repetition of 64 times: Full text p. 70, Generate Synthetic Data
  • Accuracy per published model: Full text p. 69, Table 2: 5.00, 5.00, 15.10, 31.90, 74.00, 7.11 and 6.33 percent
  • Definition of classification to profile: Full text pp. 70-71, Synthetic Data and Personality Labels
  • MSPE and per-trait results: Full text pp. 69-71, Table 3 and Results
  • Convex hull as a separate metric: Full text pp. 71-72, Synthetic Data and Human Data and Table 4
  • Specific cases of 100% inclusion: Full text p. 71, Table 4: GPT-4 equals 1.0000 for Pronoid and People person
  • Published description of Mahalanobis: Full text pp. 72-73, paragraph before Table 5 and Table 5 caption
  • Actual implementation of Mahalanobis: GitLab commit 143dc5f, fall_23/r/r_llm_personality/test.R lines 1022-1070: separate R mahalanobis calls for O, C, E, A and N
  • Video game use case only illustrative: Full text pp. 73-74, LLM and Personality Use Case and Figures 4-5
  • Limitations acknowledged by the authors: Full text p. 74, Limitations
  • Conclusion and proposed scope: Full text pp. 74-75, Conclusion
  • Audited code artifact: GitLab humin-game-lab/llm-personality, sole commit 143dc5f71b81ff0a2746befd87d400909ec6f624 dated 6 Sep 2024
  • Public synthetic sizes: GitLab NPY artifacts: n=60 Flan-XL; 1,280 Flan-XXL, Dolphin and Mixtral; 2,560 Davinci and GPT-4; 2,558 GPT-3.5
  • Recomputed accuracies from artifacts: Nearest-centroid recomputation over public evaluation matrices: .0500, .0500, .15078125, .318999, .71640625, .07109375 and .0515625
  • Missing human data: GitLab tree lacks data-final-commas.csv, clean_set, the 596,956-case clean data and 50,500-case baseline; only clean_subset_225 with 4,500 rows is present
  • Parsing and silent treatment of failures: GitLab commit 143dc5f, fall_23/scripts/main_zoo.py lines 325-343 and TraitBasedPersonalityTest interpolation path
  • Raw and clean results without traceability: GitLab result directories include raw and _clean arrays; raw Dolphin and Mixtral contain about 4.6% zero responses and raw Mistral about 99.8%
  • PCA fit on GPT-4: GitLab final-report notebook code overwrites the human object with GPT-4 data before train_test_split and PCA fit; split has no fixed random_state
  • Portability and compilation failure: GitLab repository uses hard-coded Windows paths; python -m compileall fails at fall_23/scripts/test_file.py line 3
  • Absence of reproducible packaging: GitLab root has no README, LICENSE, CI or tests; requirements.txt is largely unpinned, duplicates langchain and pins legacy openai 0.28
  • Exposed credential: GitLab commit 143dc5f, fall_23/scripts/notebook_commons.py lines 491-492 contains a hard-coded OpenAI credential and organization identifier; secret value intentionally omitted
  • Comprehensive reading and visual verification: All 11 pages rendered and inspected, including Figures 1-5, Tables 1-5, prompts, limitations, conclusion and references; checked 15 Jul 2026