Synthetic Personality Review

Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation

Evaluation and psychometric validity2023ACL AnthologyApproved editorial review

Authors: Adithya V Ganesan, Yash Kumar Lal, August Håkan Nilsson, H. Andrew Schwartz

Keywords: personality estimation, GPT-3, zero-shot learning, Big Five traits, social media analysis, human-level NLP, psychometrics

Source: Open primary source (opens in a new tab)

4
Authors
20
Findings
35
Limitations
20
Evidence

Editorial summary

English

The paper evaluates whether text-davinci-003 can classify Big Five traits zero-shot from 20 Facebook posts per person. It starts with 202 US participants who consented to share posts and complete a psychological battery; requiring exactly 20 posts from the final year leaves 142 people with continuous Big Five scores from 1 to 5. The authors turn those scores into two or three classes using quantiles and compare four prompts: a basic prompt and three variants adding a textbook definition, correlated word lists, or two questionnaire items. The trivial baseline predicts the most frequent class (MFC), while the strong comparator is WT-LEX, a supervised ridge regression over n-grams and LDA topics previously trained on Facebook data. In binary classification, GPT-3 obtains mean macro-F1 of 0.400 with the basic prompt, 0.419 with a definition, 0.441 with word lists, and 0.454 with questionnaire items; MFC reaches 0.379 and WT-LEX 0.518. GPT-3’s best prompt exceeds WT-LEX on conscientiousness (0.521 versus 0.393) and extraversion (0.569 versus 0.516), but trails it on openness, agreeableness, and neuroticism. Moving from two to three classes lowers the item-prompt mean macro-F1 from 0.454 to 0.230, close to the three-class MFC reported as 0.212; this supports the paper’s explicit conclusion that the model is unsuitable for fine-grained or continuous estimation. Changing the item pairs yields averages from 0.430 to 0.454, with no stable advantage for one formulation. The LIWC analysis suggests that GPT-3 handles social language for extraversion better, but misses associations between social/affective language and openness. The evidence concerns one closed model snapshot, a small sample, and classification relative to this dataset; it does not validate individual psychometric assessment or clinical use. Inspection of the public repository confirms the prompts and text-davinci-003 calls but does not support complete reproduction: data, predictions, metric code, the baseline, statistical analyses, and a runnable configuration are absent; the PDF also sets frequency_penalty to 0.1 while the public code sets it to 0.0.

Español

El artículo evalúa si text-davinci-003 puede clasificar en zero-shot los cinco grandes rasgos a partir de 20 publicaciones de Facebook por persona. Parte de 202 participantes estadounidenses que habían consentido compartir publicaciones y responder una batería psicológica; tras exigir exactamente 20 posts del último año quedan 142 personas, con scores Big Five continuos de 1 a 5. Los autores convierten esos scores en dos o tres clases mediante cuantiles y comparan cuatro prompts: uno básico y tres que añaden una definición de manual, listas de palabras correlacionadas o dos ítems del cuestionario. El baseline trivial predice la clase más frecuente (MFC) y el comparador fuerte es WT-LEX, una regresión ridge supervisada sobre n-gramas y temas LDA entrenada previamente con datos de Facebook. En clasificación binaria, GPT-3 obtiene macro-F1 medio de 0,400 con el prompt básico, 0,419 con definición, 0,441 con palabras y 0,454 con ítems; MFC logra 0,379 y WT-LEX 0,518. El mejor prompt de GPT-3 supera a WT-LEX en responsabilidad (0,521 frente a 0,393) y extraversión (0,569 frente a 0,516), pero queda por debajo en apertura, amabilidad y neuroticismo. Al pasar de dos a tres clases, el macro-F1 medio del prompt de ítems cae de 0,454 a 0,230, cerca del MFC de tres clases citado como 0,212; esto respalda la conclusión explícita del trabajo de que el modelo no es adecuado para estimación fina o continua. Cambiar las parejas de ítems produce promedios de 0,430–0,454, sin una ventaja estable de una formulación. El análisis LIWC sugiere que GPT-3 capta mejor lenguaje social para extraversión, pero falla en asociaciones de procesos sociales y afecto con apertura. La evidencia corresponde a una única instantánea cerrada, una muestra pequeña y clasificación relativa al propio conjunto; no valida evaluación psicométrica individual ni uso clínico. La revisión del repositorio público confirma los prompts y las llamadas a text-davinci-003, pero no permite reproducción completa: faltan datos, predicciones, código de métricas, baseline, análisis estadístico y configuración lista para ejecutar; además, el PDF usa frequency_penalty=0,1 y el código público 0,0.

Research question

What knowledge about traits improves zero-shot personality estimation with GPT-3, how does it compare with a supervised lexical model, how much does it worsen as granularity increases, and does it maintain its predictions when external prompt items are changed?

Method

Zero-shot classification study on 142 U.S. users with 20 anonymized Facebook posts and five Big Five scores. The continuous scores [1,5] are discretized by quantiles into two and three classes. text-davinci-003 receives the 20 chronological posts and a BASIC, TEXTBOOK, WORDLIST, or ITEMDESC prompt; generates a single token at temperature zero. Macro-F1 is calculated per trait and on average and compared with MFC and WT-LEX. Four positive/negative item pairs are permuted to analyze sensitivity. Finally, LIWC categories are compared through standardized differences and log-odds in cases where GPT-3 and WT-LEX disagree. The appendix documents prompts and parameters; the independent review additionally inspected the public repository.

Sample: From 202 participants with psychological results and Facebook posts, 142 U.S. users with exactly 20 posts from the last year are retained. The text reports a gender ratio of 79:18:3 (woman:man:others), ages from 21 to 66 years, and a median of 37. Each person contributes five Big Five scores in [1,5], later discretized by quantiles. No racial, educational, socioeconomic, or regional distribution is reported, nor the resulting exact thresholds.

Findings

  • In binary classification, the macro-F1 averages are 0.379 for MFC, 0.518 for WT-LEX, 0.400 for BASIC, 0.419 for TEXTBOOK, 0.441 for WORDLIST, and 0.454 for ITEMDESC.
  • ITEMDESC is the best GPT-3 variant on average, but remains 0.064 points below WT-LEX.
  • By trait, ITEMDESC obtains 0.342 in openness, 0.521 in conscientiousness, 0.569 in extraversion, 0.488 in agreeableness, and 0.349 in neuroticism.
  • WT-LEX obtains 0.492 in openness, 0.393 in conscientiousness, 0.516 in extraversion, 0.609 in agreeableness, and 0.578 in neuroticism.
  • GPT-3 numerically outperforms WT-LEX only in conscientiousness and extraversion; its largest deficit appears in neuroticism.
  • WORDLIST is the best GPT-3 variant for openness, agreeableness, and neuroticism, while ITEMDESC is best for conscientiousness and extraversion.
  • GPT-3 predictions are strongly biased toward high openness and low neuroticism, according to the authors' analysis.
  • Combining all types of knowledge reduces performance below BASIC; the article offers no table or systematic configuration to order or compose that combination.
  • When moving from two to three classes, ITEMDESC drops from an average macro-F1 of 0.454 to 0.230.
  • In three classes, the macro-F1 per trait are 0.141, 0.288, 0.240, 0.160, and 0.320 for openness, conscientiousness, extraversion, agreeableness, and neuroticism.
  • The three-class average remains close to the three-class MFC cited in the text as 0.212, so utility degrades as granularity increases.
  • Changing the ITEMDESC item pairs yields averages of 0.454, 0.448, 0.430, and 0.448; the overall differences are small.
  • Variations by trait are not identical: for example, the best openness pair reaches 0.374 and the agreeableness pair 0.523, while neuroticism remains between 0.333 and 0.364.
  • The factor loadings of the alternative pairs, computed on an external sample N=741, vary little for most traits and are used as an explanation for the overall similarity.
  • In extraversion, cases where only GPT-3 is correct contain more NUMBER, SOCIAL, AFFILIATION, and YOU categories than the other cases compared.
  • In openness, errors exclusive to GPT-3 contain more THEY, FOCUSPAST, AFFECT, ACHIEVE, and SOCIAL than the contrast set.
  • The article interprets these patterns as a contextual advantage for social lexis and a shortcoming in relating affective/social language to openness; it is an associative error analysis, not a test of the internal mechanism.
  • The paper reports approval from an institutional committee, consent, researcher certification, and anonymization of posts; recommends clinical supervision and disparity analysis before any clinical use.
  • The public repository contains a prompt generator, OpenAI client, and templates, with Python 3.9.6 recommended and dependencies openai 0.26.0 and pandas 1.5.2.
  • The repository contains no data, outputs, evaluation, baseline, or table analysis, so it reproduces the query issuance only if private inputs and configuration are reconstructed, not the published end-to-end results.

Limitations

  • The effective sample is small: 142 people, with only 20 posts each.
  • All participants are from the United States and the gender ratio 79:18:3 is highly imbalanced; cultural or linguistic generalization is not evaluated.
  • The text does not characterize race, education, region, socioeconomic level, language, platform of use, or exact collection period of the effective sample.
  • Requiring exactly 20 posts and limiting to the last year may introduce selection bias and discard relevant longitudinal information.
  • The five traits are continuous, but the main evaluation reduces them to yes/no via quantiles; the limitations section itself acknowledges that this contradicts the descriptive purpose of the Big Five.
  • The quantile thresholds are relative to this sample, not clinical or normative psychometric cutoff points.
  • The drop to three classes shows that the binary result does not transfer to fine estimation; the continuous regression corresponding to the construct is not executed.
  • Only text-davinci-003 is evaluated, a single closed snapshot, with no comparison between sizes, families, or versions from the same provider.
  • Temperature zero is presented as reproducible, but queries are not repeated to measure stability nor is an immutable backend identifier documented.
  • The output is restricted to one token and two cases required manually adding line breaks to produce a valid label; the total rate of invalid outputs is not systematically reported.
  • The post hoc manual intervention in two prompts reduces the purity of the uniform protocol and is not automated in the repository.
  • The PDF sets frequency_penalty=0.1, while src/gpt.py in the public repository uses 0.0; this discrepancy makes it impossible to know which configuration exactly produced the tables.
  • The repository published in a single commit in 2023 includes no license, data, example configuration, complete environment file, outputs, prompt hashes, or detailed experimental instructions.
  • There is no discretization code, macro-F1 calculation, significance tests, WT-LEX, LIWC, factor loadings, or table and figure generation in the repository.
  • The README only indicates creating an environment and installing dependencies; gpt.py imports a config.py that is absent and requires reconstructing undocumented paths and inputs.
  • Sensitive data are not published, understandably, but neither is a synthetic dataset with the same schema, sufficient aggregate statistics, or cached responses that would allow verifying the pipeline.
  • The comparison with WT-LEX depends on a model previously trained on different Facebook data; the article does not document in detail domain compatibility, possible overlap, recalibration, or provenance of each artifact.
  • WT-LEX produces continuous scores and is then discretized for comparison, while GPT-3 predicts labels directly; the systems do not solve exactly the same source task.
  • Significance against WT-LEX is reported with symbols, but the main text does not identify the statistical test, resampling unit, confidence intervals, or correction for multiple comparisons.
  • Macro-F1 is the only main metric; confusion matrices, precision/recall by class, calibration, balanced accuracy, and uncertainty of differences are missing.
  • Multiple traits, prompts, and formulations are tested; no preregistered protocol or separation between exploration and confirmation is offered.
  • ITEMDESC incorporates into the prompt items from the same instrument that defines the target; it is a reasonable choice to describe the construct, but limits attributing the result to spontaneous understanding of personality.
  • It is not compared with human estimators reading the same 20 posts, so the 'human-level' task label does not imply human parity.
  • The LIWC analysis selects cases based on accuracy discrepancies and then describes categories; it does not demonstrate causality or that those categories are represented in a hypothetical embedding space of the API.
  • The article infers that language patterns are encoded in the GPT-3 embedding space, but does not access or analyze embeddings or activations of text-davinci-003.
  • The description attributes anonymization to a SciPy NER model, although SciPy is not a NER library; it is not clarified whether spaCy was meant, nor is the effectiveness of de-identification validated.
  • The public code does not contain the anonymization pipeline, so that privacy component cannot be audited.
  • Although approval from an institutional committee is reported, the institution, protocol, date, withdrawal management, or residual risk of re-identification of 20 concatenated posts are not identified.
  • Disparities by gender, age, or other groups are not evaluated, despite the Ethics Statement recommending doing so before applied use.
  • Privacy attacks, memorization, sensitive attribute inference, or accidental exposure from sending posts to a commercial API are not tested.
  • Exact cost, latency, service error rate, or dependence on keys provided by different people are not analyzed.
  • Cost and window restrictions prevented exploring few-shot, longer history, sensitivity to the number of posts, or in-context learning.
  • The WORDLIST lists include potentially offensive terms and dated cultural associations; whether they induce stereotypes or demographic bias is not examined.
  • Validation is limited to Facebook and English, with no transfer to conversation, other textual genres, other platforms, or time periods.
  • There is no clinical, diagnostic, or decision-making validation, nor study of the harm that false positives and false negatives would produce in real use.

What the study does not establish

  • It does not demonstrate that GPT-3 measures continuous personality validly; the best evidence is binary classification relative to the sample.
  • It does not validate text-davinci-003 as a substitute for a Big Five inventory administered to people.
  • It does not demonstrate parity with humans or with the state of the art: the average remains below WT-LEX.
  • It does not demonstrate robust performance in three classes, regression, or fine profiles; it precisely observes a drop near the trivial baseline.
  • It does not prove that GPT-3 possesses personality, internal traits, self-knowledge, or human psychological understanding.
  • It does not prove that the model infers causes, life history, or clinical states from posts.
  • It does not demonstrate stability across executions, provider updates, broad paraphrases, history lengths, or languages.
  • It does not demonstrate generalization beyond 142 U.S. Facebook users.
  • It does not establish factorial validity, invariance, test-retest reliability, incremental validity, or psychometric calibration of the model's predictions.
  • It does not prove clinical utility, safety, therapeutic benefit, or suitability for hiring, advertising, or high-impact decisions.
  • It does not demonstrate absence of bias by gender, age, culture, dialect, race, or socioeconomic level.
  • It does not convert LIWC correlations into a causal explanation of the GPT-3 mechanism.
  • It does not demonstrate that the associations are found in the embedding space, because the study observes output labels and not internal representations.
  • It does not prove that prompts with items are universally better; per-trait variants produce different advantages and the average changes little.
  • It does not demonstrate that combining more knowledge improves: the reported attempt worsens compared to the basic prompt.
  • It does not establish that the repository allows reproducing the complete tables or verifying all analyses.
  • It does not demonstrate that de-identification eliminates the risk of re-identifying sensitive social texts.
  • It does not authorize interpreting the scores as diagnosis, therapeutic recommendation, or a stable label of a person.

Traceability

Scope: Full text

Version: WASSA 2023 proceedings, pp. 390–400; code repository commit 231fdebfbaba17ecd3a1dd08f1a47b151fce3025

Consulted source: https://aclanthology.org/2023.wassa-1.34.pdf

Review: Codex full-text, bilingual-fidelity, visual, metadata, dataset, Big-Five operationalization, prompt, baseline, metric, statistical-claim, privacy, ethics, code-reproducibility and evidence-level audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • OpenAI text-davinci-003, described as GPT-3
  • WT-LEX: pretrained ridge regression over dimension-reduced n-grams and LDA topics from Park et al. (2015)
  • Most Frequent Class baseline
  • BASIC zero-shot prompt
  • TEXTBOOK trait-definition prompt
  • WORDLIST correlated-unigram prompt
  • ITEMDESC questionnaire-item prompt
  • Combined-knowledge prompt attempted but reported only narratively

Instruments and metrics

  • Mini-IPIP / four Big Five questionnaire items per trait in the source battery
  • Quantile discretization into binary and ternary labels
  • Macro-F1 per trait and mean across traits
  • LIWC language categories for error analysis
  • Cohen’s d for category-distribution differences
  • Log odds ratio with informative Dirichlet prior
  • External factor loadings for alternate item pairs

Data used

  • Consenting Facebook-post and psychological-assessment dataset described by Jose et al. (2022): 202 participants before filtering
  • Evaluation subset: 142 US users with exactly 20 posts from the final year and five Big Five scores
  • Kosinski et al. (2013) Facebook data used to train WT-LEX and an external N=741 subset used for item factor loadings
  • Trait-correlated unigram lists from Schwartz et al. (2013)
  • Private Facebook text and labels are not included in the public code repository

Evidence and location

  • Identity, authors, abstract, and code availability: ACL proceedings PDF p. 1 (printed p. 390), title and abstract
  • Study questions and Big Five framework: PDF pp. 1–2, Introduction and Background
  • Initial sample, filter to 142, demographics, posts, and scores: PDF p. 2 (printed p. 391), section 3 Dataset
  • Binary/ternary discretization and four prompt types: PDF p. 2, section 4 Experimental Design
  • WT-LEX baseline and macro-F1 metric: PDF p. 3 (printed p. 392), Baseline and Evaluation
  • Results by trait and by knowledge type: PDF p. 3, Table 1 and section 5
  • Bias toward high openness and low neuroticism and knowledge combination: PDF p. 3, Results discussion
  • Drop from two to three classes: PDF p. 4 (printed p. 393), Table 2 and Problem Framing
  • Sensitivity to item pairs: PDF p. 4, Table 3 and Consistency with Survey Items
  • SOCIAL/AFFECT error analysis: PDF pp. 4, 9–10, Figure 1 and Appendix C Tables 5–6
  • Explicit conclusion on dimensional estimation: PDF pp. 4–5, Conclusion
  • Consent, IRB, anonymization, and clinical warnings: PDF p. 5 (printed p. 394), Ethics Statement
  • Limitations of discretization, cost, and context window: PDF p. 5, Limitations
  • Model and parameters published in the paper: PDF p. 9 (printed p. 398), Appendix A.1
  • Complete binary and ternary templates: PDF p. 9, Appendix A.2
  • Definitions, word lists, and injected items: PDF p. 10 (printed p. 399), Figure 2
  • Alternative items, factor loadings, and macro-F1: PDF p. 11 (printed p. 400), Tables 7–8
  • Scope of public code and frequency_penalty discrepancy: GitHub humanlab/gpt3-personality-estimation commit 231fdeb, README.md, requirements.txt and src/gpt.py; inspected 15 Jul 2026
  • Absence of data and complete pipeline in the repository: Public repository file inventory: README.md, requirements.txt and three Python files only; inspected 15 Jul 2026
  • Visual inspection: All 11 PDF pages rendered and visually inspected, including Figures 1–2 and Tables 1–8; checked 15 Jul 2026