Synthetic Personality Review

Can Large Language Models Understand You Better? An MBTI Personality Detection Dataset Aligned with Population Traits

Evaluation and psychometric validity2025ACL AnthologyApproved editorial review

Authors: Bohan Li, Jiannan Guan, Longxu Dou, Yunlong Feng, Dingzirui Wang, Yang Xu, Enbo Wang, Qiguang Chen, Bichen Wang, Xiao Xu, Yimeng Zhang, Libo Qin, Yanyan Zhao, Qingfu Zhu, Wanxiang Che

Keywords: Computation and Language, Computers and Society, MBTI Personality Detection, Large Language Models, Personality Traits

Source: Open primary source (opens in a new tab)

15
Authors
13
Findings
24
Limitations
19
Evidence

Editorial summary

English

The paper introduces MBTIBench, a 286-profile benchmark drawn from the test sets of Kaggle MBTI, PANDORA, and Twitter. It selects roughly six users per self-reported 16-type class and source, manually removes sentences considered noise or label leakage, and asks three English-degree annotators to assign one of four intensity labels on each MBTI dimension. A Dawid–Skene-inspired procedure ranks annotation combinations and converts them, through normalized cumulative frequencies, into scores between 0 and 1. Six LLMs and four prompting methods are compared; zero-shot is often competitive, but no model consistently beats the mean-label baseline and predictions show strong model- and prompt-specific concentrations. In an auxiliary stress task on 300 Dreaddit cases, adding soft labels raises mean accuracy from 72.13% without MBTI and 73.37% with hard labels to 74.00%, although the gain over hard labels is only 0.63 points, reverses in three of ten runs, and the paper does not report the statistical test result. The defensible contribution is its attention to noise, label leakage, disagreement, and output bias in MBTI detection benchmarks. It does not establish that 29.58% of self-reports are psychologically incorrect: that figure records disagreement with three annotators' text-based inferences, not validation against an independent instrument. Population alignment is also unproven because sampling is balanced by type, soft scores are constructed from frequencies within the same dataset, and an audit of the release finds explicit MBTI or related-theory references in at least 67 of 286 records. Scale and discretization inconsistencies, moderate agreement, and privacy and licensing concerns make MBTIBench a useful but still fragile experimental resource rather than psychometric ground truth about people.

Español

El artículo presenta MBTIBench, un banco de 286 perfiles tomado de los test sets de Kaggle MBTI, PANDORA y Twitter. Selecciona aproximadamente seis usuarios por cada uno de los 16 tipos autodeclarados y por fuente, elimina manualmente frases consideradas ruido o fuga de etiqueta, y pide a tres anotadores con formación universitaria en inglés que asignen a cada eje MBTI una de cuatro intensidades. Un procedimiento inspirado en Dawid–Skene ordena las combinaciones de anotaciones y las transforma, mediante frecuencias acumuladas normalizadas, en puntuaciones entre 0 y 1. Se comparan seis LLM y cuatro prompts; zero-shot suele ser competitivo, pero ningún modelo supera de forma consistente al baseline de predecir la media y las distribuciones presentan fuertes concentraciones dependientes del modelo y del prompt. En una prueba auxiliar de estrés sobre 300 casos de Dreaddit, añadir etiquetas blandas eleva la accuracy media de 72,13% sin MBTI y 73,37% con etiquetas duras a 74,00%, aunque la ventaja sobre etiquetas duras es solo 0,63 puntos, se invierte en tres de diez ejecuciones y el artículo no publica el resultado del test estadístico. La aportación defendible es señalar ruido, fuga de etiquetas, desacuerdo y sesgos de salida en los benchmarks de detección MBTI. No queda demostrado que el 29,58% de autoinformes sea psicológicamente incorrecto: esa cifra expresa desacuerdo con inferencias de tres anotadores, no comparación con un instrumento independiente. Tampoco se valida el supuesto alineamiento poblacional; la muestra se balancea por tipo, las etiquetas blandas se construyen a partir de las frecuencias del propio conjunto y una auditoría del release encuentra referencias explícitas a MBTI o teorías afines en al menos 67 de 286 registros. Inconsistencias entre escala, discretización y código, junto con acuerdo interanotador moderado y problemas de privacidad/licencia, obligan a interpretar MBTIBench como un recurso experimental todavía frágil, no como verdad psicométrica sobre las personas.

Research question

Can a cleaned MBTI detection test set, reannotated with soft intensities, offer a more realistic evaluation of personality inference from text, what patterns and biases do six LLMs show under four prompting strategies, and do those labels provide useful information in an auxiliary stress detection task?

Method

The test sets of Kaggle MBTI, PANDORA and Twitter are reconstructed by selecting six profiles for each of the 16 types in each source, except four ESFJ available in PANDORA, for a total of 286. Sentences of fewer than 100 words, illegible text, links, and direct or indirect mentions of personality theories are manually removed. Three annotators with a degree or master's in English, trained with examples elaborated under the guidance of doctoral students and psychology experts, independently assign E+/E−/I−/I+ and equivalents on the four axes. After three pilot rounds, voting, human review, and reannotation of less than 15%, Fleiss kappas between 0.4622 and 0.5517 are obtained. An algorithm inspired by Dawid–Skene estimates probabilities per annotator combination and then rescales them using cumulative frequencies to create soft labels. gpt-4o-mini, gpt-4o, Qwen2-7B/72B-Instruct and Llama-3.1-8B/70B-Instruct are evaluated with zero-shot, step-by-step, few-shot and PsyCoT, temperature 0, and truncation of each post to 80 tokens. Continuous predictions are compared via RMSE/MAE and segmented versions; hard predictions, with accuracy and macro-F1. Finally, gpt-4o infers MBTI from each Dreaddit post and Llama-3.1-70B uses that information as auxiliary input to classify stress in ten runs. The review also cross-checks the dataset, the SQLite logs, and the public code from commit b33ec40.

Sample: MBTIBench contains 286 profiles: 96 from Kaggle, 96 from Twitter, and 94 from PANDORA. Sampling starts from six cases per MBTI type and source, with only four ESFJ available in PANDORA; therefore it is deliberately balanced by self-declared labels and not a population sample. Each profile receives four dimension annotations by three annotators. Open-model experiments are repeated five times and the two GPT models once. The auxiliary validation uses 300 cases from Dreaddit over ten runs per condition.

Findings

  • The authors count useless noise in 15.92% of Kaggle, 43.13% of Twitter, and 2.65% of PANDORA, and label leakage in 31.21%, 0.62%, and 16.56%, respectively. These figures show that the original benchmarks contain important shortcuts.
  • The 29.58% highlighted in the abstract aggregates disagreements between self-declared labels and the annotator majority: 29.17% in Kaggle, 29.17% in Twitter, and 29.79% in PANDORA. The study does not have an independent psychological measure to determine which of the two is correct.
  • Final agreement is moderate: Fleiss kappa 0.4779 on E/I, 0.4686 on S/N, 0.5517 on T/F, and 0.4622 on J/P. The best-agreed dimension remains far from near-perfect agreement.
  • The release contains 38, 41, 42, and 43 unique soft values for E/I, S/N, T/F, and J/P. They are not individualized continuous measurements: each value depends on one of the observed combinations of three ordinal labels and on its frequency in the same set.
  • The soft label distribution has more central than extreme cases, but this shape does not constitute an independent population validation because the mapping uses cumulative frequencies from the benchmark itself and the sample was selected by type quotas.
  • The baseline of predicting the mean obtains S-RMSE of 2.66, 2.70, 2.82, and 3.04 on E/I, S/N, T/F, and J/P. Numerous model and prompt combinations worsen it, so LLMs do not demonstrate a consistent advantage over a text-free rule.
  • Zero-shot achieves the best rank within gpt-4o-mini and Llama-3.1-8B and is usually among the best methods, but it does not dominate all backbones: PsyCoT has a better rank in Qwen2-7B and there are ties or per-dimension advantages for other prompts.
  • In hard classification, the best overall result in the supplementary table is gpt-4o with step-by-step, 66.70% accuracy and 65.75 macro-F1. Some configurations show large gaps between accuracy and F1, a sign of predictions concentrated in majority classes.
  • The figures show discrete and strongly polarized distributions: Qwen2 often concentrates at the midpoint, PsyCoT can push responses to extremes, and hard predictions favor P in J/P. These are output biases under this protocol, not a general measurement of fairness.
  • In Dreaddit, the published means are 72.13/71.95 accuracy/F1 without MBTI, 73.37/72.40 with hard label, and 74.00/73.73 with soft label. The released logs reproduce these means.
  • The advantage of soft over hard in Dreaddit is small: 0.63 points of mean accuracy and the soft labels fall below the hard labels in three of ten rounds. The article claims to have performed a t-test, but publishes no statistic, p-value, interval, or effect size.
  • The SQLite logs and the evaluator allow reproducing the main tables with nine bins. This provides partial traceability, although it exposes a contradiction with the proceedings text, which claims to use ten intervals.
  • The COLING 2025 proceedings version confirms the main results of the preprint and publishes the work on pages 5071–5081; the methodological and ethical appendices are only available in arXiv v1.

Limitations

  • Disagreement with a self-declared label does not prove that the self-report is incorrect. Annotators infer linguistic signals according to a guideline; they do not administer MBTI, do not observe longitudinal behavior, and do not have clinical history or informant assessment.
  • The three annotators have training in English, not professional accreditation in psychometrics. Doctoral students and psychology experts guide the guidelines and reviews, but their number, independence, specialization, and exact role in each adjudication are not identified.
  • The guidelines may turn linguistic stereotypes into supposed truth: they associate social anxiety with introversion, formal writing and verb tenses with J, casual grammar with P, and expressed emotion with F. These signals may reflect situation, language, health, or platform.
  • Kappas of 0.46–0.55 indicate only moderate agreement. The acceptance criterion of 0.45 is low for a resource presented as high-quality and is not accompanied by intervals, agreement by source, confusions, or reliability of the +/− intensity.
  • The review of 'unreasonable' cases is not described as blind or independent. Problems are communicated to the same annotator and reinserted for reannotation; it is not reported how many exact cases change or how persistent disagreements are resolved.
  • Sampling six cases of each MBTI type in each source does not represent any natural population. There are no population weights, demographics, country, age, gender, educational level, or comparison with a probabilistic sample.
  • The claim of population alignment is circular: the algorithm orders annotator combinations, accumulates their frequency within MBTIBench, and normalizes both sides of 0.5. Observing more central values afterward in that distribution does not validate an external latent structure.
  • Soft labels are not responses to a continuous instrument or individual estimates with uncertainty. There are only 38–43 distinct values per axis, shared by all profiles with the same combination of three labels.
  • The released dataset does not include the original self-declared labels, so the 29.58% figure and the before/after shifts cannot be reproduced directly from the published artifacts.
  • A conservative audit of the JSONL finds at least 67 of 286 profiles with explicit terms such as MBTI, enneagram, introvert, extrovert, or four-letter types: 34 in Kaggle and 33 in PANDORA. Several entries contain extensive explanations of functions and types, contradicting the declared removal of direct leakage and cross-theory.
  • The selection of few examples is done manually and its sensitivity is not studied. Prompts do not receive equivalent information: PsyCoT adds a complete questionnaire and few-shot may induce the range of the chosen demonstrations.
  • The method text says to predict values 1–9, while the templates request 0–10. The parser accepts 0 and 10 but transforms with (score−1)/8, producing −0.125 or 1.125, outside the declared range.
  • The evaluator's range assertions use y_true.all() and y_pred.all(), so they only check booleans and do not detect predictions outside [0,1]. The logs contain some extreme responses 0 or 10 in Llama-3.1-8B.
  • The proceedings article claims to discretize into ten intervals; the appendix and code use nine. The table figures match nine bins, so the published metric definition does not reproduce the result without consulting the repository.
  • The arXiv v1 preprint contains numerical examples of baseline and Llama-3.1-70B that do not match its main table. The proceedings version reorganizes those research questions, but does not explicitly document the correction.
  • The aliases gpt-4o and gpt-4o-mini do not fix snapshot or date. Open models lack checkpoint and seed review; the code uses rigid local paths and 70B/72B tokenizers even for 8B/7B variants.
  • Open models are repeated five times, but GPT only once. There are no confidence intervals or tests per model, dimension, or prompt; near-zero deviations with temperature 0 do not replace an evaluation on new samples.
  • The Dreaddit test uses the same post to infer MBTI and to detect stress. The auxiliary label may act as a generated reformulation of signals from the text itself, without demonstrating that it represents personality or that it causally mediates the improvement.
  • The t-test code uses ttest_ind on paired rounds and publishes no results. Soft beats hard by only 0.63 points of mean accuracy and loses in three rounds, so the claim of superiority needs a paired analysis and an effect interval.
  • The term 'bias' is applied to concentrations of scores across models and prompts. No protected groups, harm, calibration, false positives by demographics, or consequences of use are evaluated; it is not a social equity audit.
  • The repository does not include automated tests, a fully portable hardware manifest, or a legal license, although the README claims to allow academic and commercial research without restrictions. Reproducibility depends on internal paths and hosts, fixed APIs, and CUDA packages.
  • The release redistributes social text with names, ages, locations, diagnoses, medication, depression, panic attacks, and references to suicidal ideation. Having taken public data or with adaptation permission does not equate to consent for a new psychological inference; deidentification audit, withdrawal procedure, and clear per-source license are missing.
  • The ethics statement claims that privacy is maintained because there is no interaction with the accounts, but the publication of sensitive messages may allow reidentification by textual search. The risk of using inferred labels in employment, education, health, or persuasive personalization is also not discussed.
  • The study is limited to English social text, MBTI, and 2024 models. It does not validate temporal stability, multi-turn conversation, other languages/cultures, Big Five, private settings, or real decisions about users.

What the study does not establish

  • It does not prove that 29.58% of people reported their personality incorrectly or that the annotator labels are the psychological truth.
  • It does not prove that MBTIBench reproduces the personality distribution of a human population or that its scores are equivalent to a validated continuous psychometric scale.
  • It does not prove that LLMs understand a person better, infer stable internal traits, or consistently outperform a baseline that ignores the text.
  • It does not prove that zero-shot is universally superior; the order changes by backbone, dimension, and metric.
  • It does not prove that soft labels generally improve other psychological tasks: it only provides a small stress experiment, with the same reused input and a reduced mean advantage over hard labels.
  • It does not evaluate demographic fairness or prove that output concentrations causally come from the training corpora.
  • It does not justify using these inferences in clinical, employment selection, educational, credit, surveillance, advertising, or individualized high-impact decisions.

Traceability

Scope: Full text

Version: arXiv:2412.12510v1 frozen source; COLING 2025 proceedings version, pp. 5071–5081, and linked repository commit b33ec40af3eba25f0ebec5e719da8784c714be30 also reviewed

Consulted source: https://arxiv.org/pdf/2412.12510

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • gpt-4o-mini (API alias; snapshot unspecified)
  • gpt-4o (API alias; snapshot unspecified)
  • Qwen2-7B-Instruct
  • Qwen2-72B-Instruct
  • Meta-Llama-3.1-8B-Instruct
  • Meta-Llama-3.1-70B-Instruct

Instruments and metrics

  • Myers–Briggs Type Indicator four dichotomies
  • Four-level polarity annotation per MBTI dimension
  • Fleiss’ kappa and annotator accuracy
  • Dawid–Skene-inspired EM and cumulative-frequency soft-label mapping
  • RMSE, MAE, segmented RMSE and segmented MAE
  • Accuracy and macro-F1
  • Zero-shot, step-by-step, few-shot and PsyCoT prompts

Data used

  • MBTIBench release (286 profiles)
  • Kaggle MBTI / PersonalityCafe test set
  • PANDORA test set
  • Twitter MBTI test set
  • Dreaddit stress-detection test subset (300 profiles)

Evidence and location

  • Publication, objective, and declared contributions: COLING 2025 proceedings, pp. 5071–5072, abstract and introduction
  • Sources, balanced sampling, and filtering criteria: COLING proceedings, pp. 5072–5073, sections 2.1 and 3.1–3.2
  • Annotator training, pilots, and quality control: COLING proceedings, pp. 5073–5074, sections 3.3–3.4
  • Final kappas and reannotation of less than 15%: COLING proceedings, p. 5074, section 3.4.2 and footnote 6
  • EM algorithm, accumulation, and normalization of trends: COLING proceedings, pp. 5074–5075, section 3.5 and Algorithm 1
  • Noise, leakage, and 29.58% disagreement distributions: arXiv:2412.12510v1, pp. 7–8, section 4 and Tables 3–4
  • Population alignment claim and central distribution: COLING proceedings, pp. 5075–5076, sections 4.2–4.3 and Figures 5–6
  • Models, prompts, truncation, and evaluation: COLING proceedings, pp. 5076–5077, section 5
  • Baseline and segmented results of the six models: COLING proceedings, p. 5077, Table 2 and section 6.1
  • Polarization and concentration of predictions: COLING proceedings, pp. 5077–5078, Figure 7 and RQ1–RQ3
  • Stress validation design and means: COLING proceedings, p. 5078, section 6.2 and Figure 8
  • Templates, requested scale, and metrics: arXiv v1, pp. 20–22, Appendix E and Tables 7–8
  • Complete continuous and hard results: arXiv v1, pp. 23–28, Appendix F and Tables 9–10
  • Privacy statements and expert participation: arXiv v1, pp. 17–18, Appendices A–C
  • Size, distribution by source, and leakage residues in the release: Linked repository commit b33ec40af3eba25f0ebec5e719da8784c714be30, dataset/mbtibench.jsonl audit
  • Discrete construction of soft labels and absence of original labels: Linked repository commit b33ec40, dataset/em_softlabel.py and dataset/mbtibench-nolabel.jsonl
  • Scale, parser, bins, and range checks: Linked repository commit b33ec40, mbtibench/prompt.py, mbtibench/evaluator.py and reproduced SQLite metrics
  • Ten Dreaddit rounds, means, and unreported statistical test: Linked repository commit b33ec40, downstream/results-reproduce and downstream/evaluate.py
  • Portability, aliases, paths, and absence of license: Linked repository commit b33ec40, README.md, requirements.txt, mbtibench/llm.py and repository root