Synthetic Personality Review

Rediscovering the Latent Dimensions of Personality with Large Language Models as Trait Descriptors

Evaluation and psychometric validity2024NeurIPS WorkshopApproved editorial review

Authors: Joseph Suh, Suhong Moon, Minwoo Kang, David M. Chan

Keywords: personality assessment, Big Five traits, singular value decomposition, latent dimensions, trait descriptors, personality probing, LLM evaluation

Source: Open primary source (opens in a new tab)

4
Authors
26
Findings
29
Limitations
28
Evidence

Editorial summary

English

The paper asks whether an LLM's next-token probabilities, when completing a personal-story description with adjectives, contain axes corresponding to the Big Five without administering a questionnaire. It builds a 208-story by 100-trait-adjective matrix, centers the log-probabilities, and applies SVD; it then uses the signs of five components for binary classification and, in a supervised variant, fits Lasso regression. The stories come from PersonaLLM: GPT-4-0613 generated ten narratives for each of the 32 binary Big Five combinations, and 208 remained after filtering stories with explicit trait lexicons. The first five components explain 74.3% of variance. The best SVD result has 0.793 mean accuracy, below fine-tuned DeBERTaV3 at 0.867; the best Lasso reaches 0.912, an absolute gain of 4.5 points over DeBERTaV3 and 21.4 over the published 0.698 GPT-4 prompting result from PersonaLLM. The defensible conclusion is that log-probabilities over a closed adjective list can provide a useful predictive representation on this synthetic benchmark. This is not an independent rediscovery of personality: the Big Five is injected both into the instructions used to generate the stories and into Goldberg's 100 adjectives, which are already assigned to dimensions and poles. Components are also named by inspecting those labels and matched to traits with training-set labels, while an SVD component's sign is mathematically arbitrary. There is no validation with people, external corpora, confidence intervals, significance testing, random seeds, stability analysis, or open code and results. The complete Mixtral SVD and Lasso rows exactly duplicate the Llama-3.1-70B-Instruct rows without explanation. The study offers a promising lexical probe and a controlled demonstration, but it does not establish psychological personality in the model, an internal latent personality structure, or psychometric validity in real populations.

Español

El trabajo estudia si las probabilidades de siguiente token de un LLM, al completar con adjetivos una descripción de una historia personal, contienen ejes que se correspondan con los Big Five sin administrar un cuestionario. Construye una matriz de 208 historias por 100 adjetivos descriptivos de rasgos, centra las log-probabilidades y aplica SVD; después usa el signo de cinco componentes para clasificación binaria y, en una variante supervisada, ajusta una regresión Lasso. Las historias proceden de PersonaLLM: GPT-4-0613 generó diez relatos para cada una de las 32 combinaciones binarias Big Five y, tras filtrar relatos con léxico explícito de rasgo, quedaron 208. Los cinco primeros componentes explican el 74,3 % de la varianza. El mejor SVD obtiene 0,793 de exactitud media, por debajo del DeBERTaV3 ajustado (0,867); el mejor Lasso llega a 0,912, una mejora absoluta de 4,5 puntos sobre DeBERTaV3 y de 21,4 sobre el 0,698 publicado para el prompting GPT-4 de PersonaLLM. La conclusión defendible es que las log-probabilidades de una lista cerrada de adjetivos pueden servir como representación predictiva en este benchmark sintético. No es un redescubrimiento independiente de personalidad: el Big Five está inyectado tanto en las instrucciones que generaron los relatos como en los 100 adjetivos de Goldberg, ya clasificados por dimensión y polo. Además, los componentes se nombran inspeccionando esas etiquetas y se emparejan con rasgos mediante etiquetas del conjunto de entrenamiento; el signo de una SVD es matemáticamente arbitrario. No hay validación con personas, corpus externos, intervalos de confianza, pruebas de significación, semillas, análisis de estabilidad ni código o resultados abiertos. Dos filas completas de SVD y Lasso para Mixtral coinciden exactamente con Llama-3.1-70B-Instruct sin explicación. El estudio aporta una sonda léxica prometedora y una demostración controlada, pero no demuestra personalidad psicológica del modelo, estructura latente interna ni validez psicométrica en poblaciones reales.

Research question

Do the correlations between the log-probabilities that different LLMs assign to 100 descriptive adjectives when evaluating personal stories produce five principal components alignable with the Big Five and allow predicting binary personality labels better than direct prompting or a fine-tuned text classifier?

Method

208 synthetic stories from PersonaLLM are taken, generated by GPT-4-0613 from the 32 binary configurations of the Big Five. For each story and each of the 100 unipolar adjectives of Goldberg, five families of LLMs compute the sum of log-probabilities of the adjective tokens under a fixed prompt in English. The 208 by 100 matrix X is centered by columns and SVD is applied with k=5. The extreme loadings and the known Big Five labels of the adjectives are used to interpret the components; a training-set accuracy matrix pairs components and traits. The sign of U is evaluated as a binary predictor. A supervised Lasso regression uses the 100 log-probabilities as variables. The comparisons are a DeBERTaV3 Large fine-tuned with BCE and a GPT-4-0613 prompting result taken from the PersonaLLM article.

Sample: PersonaLLM attempted to generate 320 stories in English: ten for each of the 32 combinations of five binary traits. After removing stories that contained explicit trait-related lexicon, 208 remained (65 %; 112 were excluded, 35 %). The article does not report the final balance by trait or combination, nor does it provide stories from real people.

Findings

  • The analyzed matrix has 208 rows of stories and 100 columns of trait-descriptive adjectives.
  • The first five SVD components jointly explain 74,3 % of the variance of the centered log-probabilities.
  • The authors describe a drop between the fifth and sixth singular values, but do not apply parallel analysis, bootstrap, or any other statistical criterion for factor retention.
  • The declared correspondence is component 1-extraversion, 2-openness, 3-agreeableness, 4-neuroticism, and 5-conscientiousness.
  • In the training matrix of Table 5, the maximum accuracies per component are 0,899 for extraversion, 0,798 for openness, 0,817 for agreeableness, 0,726 for neuroticism, and 0,803 for conscientiousness.
  • The best SVD result is 0,793 mean accuracy with Llama-3-70B-Instruct; SVD results range from 0,667 to 0,793.
  • The best Lasso result is 0,912 with Llama-3.1-70B; Lasso results range from 0,860 to 0,912.
  • Fine-tuned DeBERTaV3 Large reaches 0,867 mean accuracy, thus outperforming all reported unsupervised SVD variants.
  • The direct PersonaLLM baseline with GPT-4-0613 has 0,698 mean accuracy, a figure imported from the previous work and not re-estimated under a common protocol.
  • The maximum improvement of Lasso over DeBERTaV3 is 0,045, that is, 4,5 absolute percentage points; over the GPT-4 baseline it is 0,214, or 21,4 points.
  • The best accuracies by trait in the Lasso rows are 1,000 in extraversion, 0,964 in agreeableness, 0,881 in conscientiousness, 0,893 in neuroticism, and 0,905 in openness.
  • Performance does not improve monotonically with model size or with instruction tuning; some Instruct versions clearly worsen compared to the base model.
  • The five SVD figures for Mixtral-8x22B exactly match those for Llama-3.1-70B-Instruct.
  • The five Lasso figures for Mixtral-8x22B also exactly match those for Llama-3.1-70B-Instruct; the manuscript does not comment on this double duplication.
  • The mean log-probability varies greatly across adjectives: the text gives -5,7 for 'introverted' and -20,0 for 'unenvious'.
  • The method sums log-probabilities of all tokens of each adjective; for example, 'sophisticated' is split into four tokens with the Llama 3 tokenizer.
  • The first dimension is not purely extraversion: among its highest loadings appear 'careless', 'disorganized', 'haphazard', and 'unsystematic', all markers of negative conscientiousness.
  • The second dimension also does not show a clean openness: its highest loadings include 'anxious', 'introverted', 'withdrawn', 'distrustful', 'inhibited', and 'reserved'.
  • Table 4 acknowledges four sign exceptions in openness: 'complex', 'introspective', 'deep', and 'intellectual'.
  • The fifth dimension mixes conscientiousness markers with 'shy', 'relaxed', 'introverted', 'unadventurous', 'bashful', 'harsh', 'unemotional', and 'assertive'.
  • Table 5 establishes the component-trait correspondence by comparing signs with Big Five labels in the training set.
  • The future directions section acknowledges that convergent validity, effects of degree of acquaintance, and test-retest reliability are missing.
  • The article was published as a poster at the Workshop on Behavioral Machine Learning of NeurIPS 2024, not as an article in the main conference.
  • The official OpenReview record contains four authors; the PDF identifies David M. Chan and does not include John Canny as an author.
  • The source does not link a repository, derived dataset, log-probability matrices, splits, fine-tuned models, or result files.
  • The work tests a probe on output probabilities conditioned by a prompt; it does not inspect internal activations or hidden representations of the LLM.

Limitations

  • Circularity of the design: the stories are explicitly generated from the five binary Big Five variables and afterwards recovering five axes is celebrated.
  • Circularity of the vocabulary: the 100 input variables are Goldberg Big Five markers already grouped by trait and pole, not free descriptors discovered by the model.
  • The phrase 'without imposing predefined taxonomies' does not faithfully describe the protocol, which fixes a Big Five taxonomy both in the dataset and in the adjective set.
  • The texts are synthetic and their labels are generation conditions, not verified psychological evaluations of human authors.
  • It is not validated that each story actually expresses the five prescribed traits after filtering.
  • Of the 320 planned stories, 112 are removed, but complete operational criteria, examples, counts per rule, or an audit of the filter are not published.
  • The balance by trait or by the 32 combinations after removing 35 % of the stories is not reported; filtering may break the factorial design.
  • Only one corpus of English stories generated by a single model and a single personality template is tested.
  • There is no external corpus, real autobiographies, observer judgments, self-reports, or cross-cultural replication.
  • The choice of k=5 is guided by the Big Five hypothesis; a 74,3 % cumulative variance does not by itself demonstrate that exactly five constructs exist.
  • The name of each component is assigned through inspection of previously labeled adjectives; there is no independent metric of factorial congruence, rotation, or confirmation in another sample.
  • For SVD prediction, the component-trait correspondence uses an accuracy matrix with training labels, which qualifies the claim that the entire procedure needs no labels.
  • The sign of the singular vectors is indeterminate; the article does not specify a reproducible rule to orient each component before measuring accuracy.
  • It is not explained in sufficient detail how test stories are projected onto the learned SVD basis, nor what data are used to center, fit, and match components.
  • The 80/10/10 split is described only for DeBERTaV3; the splits for SVD and Lasso are not clearly documented.
  • No seeds, number of repetitions, standard deviations, confidence intervals, or significance tests are reported.
  • Many accuracies are not compatible with a single test of approximately 21 stories, suggesting averaging or multiple splits that the text does not document.
  • The identical rows for Mixtral and Llama-3.1-70B-Instruct across two complete methods may be a coincidence or a table error; without artifacts it cannot be resolved.
  • Summing log-probabilities penalizes adjectives with more tokens; no normalization by length or robustness to alternative tokenizers is tested.
  • Centering removes per-adjective means, but does not equalize variances or correct for other lexical, frequency, length, or tokenization biases.
  • Only one evaluation prompt and temperature 1,0 are used; there is no sensitivity to wording, language, format, or temperature.
  • Lasso is supervised and uses the same 100 semantically Big Five variables; its superiority does not validate the unsupervised discovery of factors.
  • The comparison with the 0,698 from PersonaLLM mixes a published GPT-4 result with different models, variables, and possibly different splits.
  • The DeBERTaV3 sweep comprises 128 configurations over a very small validation set and is not accompanied by cross-validation.
  • The text calls a BCE-trained output for five binary labels a '5-way classifier', an ambiguous description between multiclass and multilabel classification.
  • There is no analysis of reliability, convergent, discriminant, criterion, or incremental validity, nor measurement invariance.
  • No code, environment, dependencies, exact checkpoint versions, hardware, cost, matrices, splits, predictions, or fine-tuned models are published.
  • There is no substantive limitations or ethics section on privacy, non-consensual inference, discrimination, or use of personality labels.
  • The conclusion extrapolates toward ethical values, cultural norms, and social behaviors without experiments on those domains.

What the study does not establish

  • It does not demonstrate that an LLM possesses psychological personality, identity, stable traits, or subjective experience.
  • It does not demonstrate that the obtained components reside in the internal latent space of the model; they are components of an external matrix of output probabilities.
  • It does not demonstrate an independent rediscovery of the Big Five from the taxonomy, because the design uses stories and adjectives constructed from the Big Five.
  • It does not demonstrate that five is the intrinsic number of personality dimensions in the models.
  • It does not demonstrate equivalence between LLM perception and human perception.
  • It does not establish psychometric, diagnostic, clinical, or occupational validity of the probe.
  • It does not establish accuracy for real people, spontaneous conversations, social media texts, or deployment contexts.
  • It does not establish generalization beyond PersonaLLM, beyond GPT-4-0613 as generator, or beyond English.
  • It does not establish invariance across cultures, languages, genders, demographic groups, or writing styles.
  • It does not establish temporal stability, test-retest reliability, or robustness across prompts.
  • It does not establish that the accuracy differences between models or baselines are statistically significant.
  • It does not establish that Lasso generally outperforms fine-tuned text models or direct prompting under comparable protocols.
  • It does not establish that the SVD method needs no labels to produce named predictions, since the evaluated matching uses training labels.
  • It does not establish that the loadings reflect personality semantics and not tokenization, frequency, template, or synthetic generation regularities.
  • It does not allow causal attribution of results to model size, instruction tuning, or a specific architecture.
  • It does not demonstrate that the generation labels are ground truth for the retained stories.
  • It does not demonstrate safety, fairness, privacy, or acceptability of inferring personality without consent.
  • It provides no empirical evidence on ethical norms, cultural norms, or social behaviors despite mentioning them as a potential extension.
  • It does not prove that the results are reproducible from the published materials.

Traceability

Scope: Full text

Version: arXiv 2409.09905v1, 16 Sep 2024, 18 pages; publication metadata cross-checked against OpenReview forum HmtCakUGHj and the official NeurIPS 2024 Behavioral ML workshop record

Consulted source: https://arxiv.org/pdf/2409.09905v1

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, venue-status, psychometric-validity, factor-analysis, metric, internal-consistency, reproducibility and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4-0613 como generador del dataset y baseline publicado de prompting
  • Llama-3.1-70B
  • Llama-3.1-70B-Instruct
  • Llama-3-70B
  • Llama-3-70B-Instruct
  • Llama-3.1-8B
  • Llama-3.1-8B-Instruct
  • Llama-3-8B
  • Llama-3-8B-Instruct
  • Mixtral-8x22B
  • DeBERTaV3 Large

Instruments and metrics

  • Goldberg 100 unipolar trait-descriptive adjectives
  • Big Five binary trait labels
  • Log-probability matrix of adjective completions
  • Column mean centering
  • Singular value decomposition with k=5
  • Lasso regression
  • Binary accuracy
  • Explained variance ratio
  • Five-output DeBERTaV3 classifier with binary cross-entropy

Data used

  • PersonaLLM synthetic personal stories
  • Goldberg 1992 100-adjective Big Five marker set

Evidence and location

  • Publication and venue type: Official NeurIPS 2024 virtual record 102146: poster in Workshop on Behavioral Machine Learning; title and four authors
  • Final editorial record: OpenReview forum HmtCakUGHj: published 10 Oct 2024, modified 18 Oct 2024, submission 42, CC BY 4.0
  • Full audited source: .cache/editorial-sources/article-089/source.pdf; arXiv 2409.09905v1; 18 pages; sha256 134f67be18e14ad51a4e36f26398dcf13141439aeaa32baa188639a683ebfa8f
  • Question and main promise: Full text pp. 1-2, Abstract and Introduction
  • Prompt and log-probability matrix: Full text pp. 2-3, Figure 2 and Section 2
  • Centering, SVD, and use of U and V: Full text p. 3, Sections 2.1-2.2
  • Evaluated models: Full text p. 3, Section 3; p. 4, Table 1
  • Synthetic sample and filtering: Full text p. 12, Appendix B: 10 stories x 32 combinations, 208 retained
  • Five components and 74,3 %: Full text p. 3, Section 3.1; p. 13, Figure 5 and Appendix C.1
  • Complete SVD, Lasso, and baseline results: Full text p. 4, Table 1
  • Row duplication between models: Full text p. 4, Table 1: Mixtral-8x22B exactly matches Llama-3.1-70B-Instruct for all five SVD and all five Lasso trait accuracies
  • Matching with training labels: Full text p. 17, Appendix C.3 and Table 5
  • Extreme loadings and mixing between traits: Full text pp. 13-16, Appendix C.2 and Tables 3-4
  • Exceptions in openness: Full text p. 16, Table 4 caption and Openness loadings
  • Adjective frequency bias: Full text pp. 8-10, Appendix A.1 and Figures 3-4
  • Tokenization and sum of probabilities: Full text p. 12, Appendix A.3
  • Single temperature: Full text p. 12, Appendix A.4: main results at T=1.0
  • Predefined Big Five list: Full text pp. 8 and 11, Appendix A.2 and Table 2: Goldberg 100 unipolar adjectives grouped by trait and pole
  • DeBERTa baseline design: Full text p. 17, Appendix D.1: 80/10/10 split and 128 hyperparameter combinations
  • Imported prompting baseline: Full text p. 18, Appendix D.2: published PersonaLLM accuracy used as baseline
  • Pending validity and reliability: Full text p. 4, Section 4: convergent validity, acquaintance and test-retest reliability listed as future directions
  • Final extrapolation: Full text pp. 4-5, Conclusion: potential extension to ethical and cultural norms and social behaviors
  • Absence of real people: Full text pp. 3 and 12: all 208 evaluated stories originate from GPT-4-0613 PersonaLLM generation
  • Absence of internal inference: Full text pp. 2-3, Figure 1 and Section 2: X consists of output adjective log-probabilities, not hidden activations
  • Absence of statistical uncertainty: Full text pp. 3-4 and 13-18: point accuracies and loadings only; no seeds, error bars for accuracy, confidence intervals or significance tests
  • Absence of reproducible artifacts: Full text pp. 1-18 and official author/publication pages checked 15 Jul 2026: no code, result archive or study repository linked; focused GitHub repository search returned no project
  • Absence of ethical evaluation: Full text pp. 1-18: no ethics, privacy, consent, fairness or misuse analysis
  • Integral reading and visual verification: All 18 pages rendered and inspected, including Figures 1-5, Tables 1-5, equations, appendices and references; checked 15 Jul 2026