Synthetic Personality Review

On the emergent capabilities of ChatGPT 4 to estimate personality traits

Evaluation and psychometric validity2025Frontiers (Artificial Intelligence)Approved editorial review

Authors: Marco Piastra, Patrizia Catellani

Keywords: Artificial Intelligence, Personality Traits, Large Language Models, Big Five, Text Analysis, ChatGPT 4

Source: Open primary source (opens in a new tab)

2
Authors
9
Findings
15
Limitations
14
Evidence

Editorial summary

English

This brief report tests whether GPT-4o can estimate Big Five traits from text without scored examples and whether its stated confidence identifies reliable estimates. The authors compare model outputs with self-reports for 2,347 psychology-student essays and Twitter messages from 294 PAN15 participants. On the essays, baseline correlations are modest and fairly even (r = .239–.283); on PAN15 they are weaker and heterogeneous (r = −.042–.259), including a negative association for Openness. Correlation also hides substantial level bias: for example, mean Neuroticism shifts from −.011 in self-report to .375 in the essay predictions, and PAN15 Openness from .253 to .375. Adding facet descriptions or asking for confidence changes little. Stated confidence remains high, tracks the magnitude of predictions rather than their error, and does not fall enough when the available text is sharply reduced. The study supplies a useful comparison and supplementary code, but it does not validate individual assessment: the criterion is self-report, no human or computational comparator is run on the same cases, the main results lack uncertainty intervals or reported inferential tests, and the released configuration differs from the paper on some parameters.

Español

Este informe breve evalúa si GPT-4o puede estimar los cinco grandes rasgos a partir de textos sin ejemplos puntuados y si la confianza que declara sirve para detectar estimaciones fiables. Los autores comparan las salidas del modelo con autoinformes en 2.347 ensayos de estudiantes de Psicología y en mensajes de Twitter de 294 participantes de PAN15. En los ensayos, las correlaciones de la configuración base fueron modestas y bastante homogéneas (r = 0,239–0,283); en PAN15 fueron menores y heterogéneas (r = −0,042–0,259), incluida una asociación negativa para apertura. A la vez, aparecen sesgos de nivel que una correlación no captura: por ejemplo, neuroticismo pasa de una media de autoinforme de −0,011 a 0,375 en los ensayos, y apertura de 0,253 a 0,375 en PAN15. Añadir descripciones de facetas o pedir confianza apenas cambia el cuadro. La confianza autodeclarada permanece alta, se relaciona con la magnitud de las predicciones y no con su error; tampoco cae de forma suficiente cuando se reduce drásticamente el texto. El trabajo aporta una prueba comparativa útil y código suplementario, pero no valida una evaluación individual: el criterio es autoinforme, no hay comparador humano o computacional aplicado a los mismos casos, faltan intervalos o pruebas inferenciales en los resultados principales y el material reproducible discrepa del artículo en algunos parámetros.

Research question

To what degree of agreement can GPT-4o estimate in zero-shot the Big Five personality traits from two types of text, in comparison with self-report, and to what extent does the numerical confidence that the model itself declares reflect the accuracy of those estimates or the sufficiency of the text?

Method

Two English-language datasets are reused. The Essays corpus starts from 2,467 students who wrote for 20 minutes in a stream-of-consciousness style; after removing incomplete records, 2,347 remain. Their scores, derived from questionnaires and partially different scoring schemes between 1997 and 2004, are converted to per-item means, normalized to [−0.5, 0.5], and rounded to tenths. PAN15 gathers 294 authors and between 26 and 100 tweets per person (81% with 100); its BFI-10 receives the same normalization and rounding. Using the Batch API, gpt-4o-2024-05-13 is queried with temperature 0 and seed 123, requesting JSON with five scores on an eleven-value scale. A base prompt, psychometric extensions, and versions that ask for confidence 0–1 are compared; the article reports mainly on the base and a facet description. Each experiment is run four times and the outputs are averaged. Pearson correlations and 1 − NRMSE are computed, predictions are filtered by a confidence threshold, and the number of tweets is progressively reduced to study text sufficiency.

Sample: 2,347 complete essays from Psychology students, drawn from an original sample of 2,467 and collected between 1997 and 2004 except 2001; and Twitter messages from 294 participants in PAN15, with 26–100 messages per author and exactly 100 in 81% of cases.

Findings

  • In Essays, the base prompt reaches correlations r = 0.276 in extraversion, 0.283 in neuroticism, 0.239 in agreeableness, 0.261 in conscientiousness, and 0.269 in openness; these correspond approximately to 5.7–8.0% of shared variance per trait, not to strong individual prediction.
  • In PAN15, the base correlations are 0.130, 0.259, 0.087, 0.196, and −0.042 respectively. The facet description improves agreeableness and conscientiousness to 0.159 and 0.264, but openness remains negative (−0.025).
  • The predictions are compressed and biased toward particular bands of the scale. The clearest case is Essays-neuroticism: self-report mean −0.011 versus 0.375 for the base model, with a model standard deviation of only 0.11 versus 0.20.
  • In PAN15 the base model overestimates especially extraversion (0.321 versus 0.170) and openness (0.375 versus 0.253); in openness, a 1 − NRMSE of 0.759 coexists with a negative correlation, showing that average proximity and inter-person ordering respond to distinct properties.
  • Asking for confidence slightly alters the scores, but does not make them systematically better. In Essays, correlations with confidence range between 0.244 and 0.287; in PAN15 between −0.030 and 0.273.
  • The mean declared confidence is high even for traits with very little signal: in PAN15 it reaches 0.797 for extraversion and 0.793 for openness, although their base correlations with confidence are 0.104 and −0.030.
  • When retaining only estimates that exceed confidence thresholds, 1 − NRMSE does not improve and generally worsens; moreover, confidence grows with higher predicted scores, indicating a dependence on the estimated level and not calibration against error.
  • When PAN15 is trimmed from all messages down to the first two tweets, correlations tend toward zero, while confidence remains approximately between 0.56 and 0.67 by trait even with two messages.
  • The four results per condition are averaged to dampen residual non-determinism, but the article does not show the variation across runs.

Limitations

  • The pattern is not robust across domains: the performance on essays does not replicate in PAN15 and openness even correlates negatively with self-report.
  • The Essays associations are small in terms of explained variance. The text calls them significant, but the main table provides no p-values, confidence intervals, or correction for the numerous comparisons.
  • Self-report is an imperfect criterion for observational inference. Neither human judges nor other models are evaluated on exactly the same texts, so the comparison with previous meta-analyses is indirect.
  • The Essays corpus combines years, questionnaires, and partially different scoring procedures; the conversion to per-item means, normalization, and rounding does not eliminate the lack of measurement equivalence.
  • Both the labels and the model responses are quantized in steps of 0.1 over a narrow range. This discretization conditions the correlations, the RMSE, and the histograms.
  • 1 − NRMSE can appear high when predictions concentrate near a frequent mean, even if they do not correctly rank individuals; PAN15-openness offers the clearest example.
  • The requested confidence is a textual declaration by the model, not a calibrated probability. No calibration curves, expected calibration error, Brier score, prediction intervals, or coverage are presented.
  • The confidence analysis by thresholds changes the number of observations and averages four runs, which produces fractional counts in the figure; no dispersions or tests quantifying the uncertainty of those curves are shown.
  • Only one snapshot of a model, one temperature, and one seed is studied. Performance cannot be extrapolated to other models, versions, languages, or configurations.
  • There is a reproducible discrepancy: the article declares gpt-4o-2024-05-13 and seed 123, while chatgpt_config.py from the supplement configures the alias gpt-4o and seed 1234. The repository also does not include the already preprocessed Essays file without additional permission.
  • The term zero-shot means that no scored examples are given, but the prompt explicitly identifies the Big Five, adopts the role of a social psychologist, and, in some conditions, incorporates facet descriptions; it is not an inference free of prompt-induced knowledge.
  • The domains are narrow and dated: essays from Psychology students and English-language Twitter. No temporal, cultural, linguistic, or generalization analysis to conversational and clinical texts is conducted.
  • No results by age, gender, or other groups are reported, nor are biases or error differences across subpopulations studied.
  • The design is predictive and cross-sectional; it does not allow attributing the results to stable latent traits or distinguishing personality from topic, style, state, cohort, or platform context.
  • Although existing datasets are reused and the article declares that no new ethical approval was required, the risks of privacy, contextual consent, or secondary use when deploying personality inference on real texts are not evaluated in this work.

What the study does not establish

  • It does not demonstrate that GPT-4o can reliably assess the personality of a specific individual.
  • It does not demonstrate that the confidence declared by the model is calibrated or that it detects insufficient text.
  • It does not demonstrate superiority or equivalence relative to human judges, supervised models, or other LLMs on the same data.
  • It does not demonstrate that the correlations are stable across domains, languages, populations, or model versions.
  • It does not validate clinical, occupational, educational, forensic, or selection decisions based on these scores.
  • It does not allow interpreting the outputs as causal or direct measures of latent traits, independent of the content and context of the text.

Models evaluated

  • GPT-4o (gpt-4o-2024-05-13)

Instruments and metrics

  • Big Five Inventory-10 (BFI-10)
  • Big Five self-report measures in the Essays corpus (heterogeneous by collection year)
  • Pearson correlation
  • 1 − normalized root mean square error (1 − NRMSE)
  • Model-stated confidence score (0–1)

Data used

  • Essays / stream-of-consciousness corpus (Pennebaker and King, 1999)
  • PAN 2015 Author Profiling English dataset

Evidence and location

  • Research questions and zero-shot scope: Publisher PDF, pp. 1–2, abstract and sections 1–2
  • Composition, harmonization, and rounding of Essays: Publisher PDF, pp. 2–3, section 2.1.1
  • Composition and BFI-10 of PAN15: Publisher PDF, pp. 2–3, section 2.1.2
  • Prompts, scale, confidence, and additional descriptions: Publisher PDF, pp. 3–4, section 2.2
  • Model, API, temperature, seed, and four runs: Publisher PDF, p. 4, section 2.3
  • Complete correlations, means, standard deviations, and 1 − NRMSE: Publisher PDF, pp. 4–5, section 3 and Table 1
  • Compression and bias of the distributions: Publisher PDF, pp. 5–6, Figure 1 and accompanying text
  • Confidence associated with scores but not with accuracy: Publisher PDF, p. 6, Figure 2 and accompanying text
  • Reduction of the number of tweets and persistence of confidence: Publisher PDF, pp. 6–7, Figure 3 and accompanying text
  • Indirect comparison, self-report criterion, and acknowledged limitations: Publisher PDF, p. 7, section 4
  • Data access and supplementary material: Publisher PDF, pp. 7–8, data availability and supplementary material statements
  • Implementation of correlation, 1 − RMSE, and confidence filtering: Supplementary Data Sheet 1, chatgpt_utils.py, compute_statistics and correlation
  • Discrepancy of model and seed relative to the article: Supplementary Data Sheet 1, chatgpt_config.py, model and request_input_template
  • Restriction of access to the preprocessed Essays: Supplementary Data Sheet 1, README, section 0.1