Synthetic Personality Review

Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts

Evaluation and psychometric validity2025arXivApproved editorial review

Authors: Yifan Lyu, Liang Zhang

Keywords: Large Language Models, Personality, MBTI, Psychometrics, Persona

Source: Open primary source (opens in a new tab)

2
Authors
29
Findings
50
Limitations
13
Evidence

Editorial summary

English

ROME is a supervised classifier for the four binary MBTI dimensions from user posts. Its Ask stage gives GPT-4o-2024-08-06 each training user's posts and true MBTI label and asks it to answer 60 “MBTI-style” questions on a −3 to +3 scale. The Answer stage trains a question-conditioned Mixture-of-Experts to reproduce those synthetic answers from posts; Detect combines predicted answers with a text representation to classify I/E, S/N, T/F, and P/J. The available source remains the 13-page arXiv v1; no published version or code repository was located.

The intermediate targets are not independent psychometric measurements. The prompt tells the model to use the MBTI type to resolve ambiguous cases or reinforce its judgment, while the method calls answers never supplied by participants “ground truth.” The auxiliary signal is therefore explicitly conditioned on the same target class the classifier must learn. This may be a useful form of supervised distillation, but it does not show that GPT-4o reconstructs real questionnaire responses or that the resulting vector is psychological evidence. The full 60-item inventory is not released, and its instrument, license, source, or validation study is not identified. The paper alternates among “standardized,” “validated,” and “MBTI-style” without reporting reliability, factor structure, scoring rules, or agreement with human responses.

Evaluation uses 8,675 Kaggle/PersonalityCafe users with their 50 most recent posts and 9,067 PANDORA/Reddit users with dozens to hundreds of posts. Labels are self-declared types from personality-centered communities, not assessments administered by this study or objective ground truth. The paper states a user-disjoint 60/20/20 split but releases no seed, indices, or repeated runs. An audit of the exact cited Kaggle download found explicit MBTI codes in 8,220/8,675 rows (94.76%) and the row's own target type in 7,576 (87.33%). A parameter-free contamination control that predicts the most frequently mentioned type reaches 64.20% exact 16-type accuracy and 79.70% mean macro-F1 across the four dimensions. The paper does not report removing these tokens or evaluating a cleaned corpus, so Kaggle partly measures label recovery and MBTI-discussion language rather than personality inference from ordinary text alone.

ROME reports 89.78% mean macro-F1 on Kaggle versus 77.79% for ETM, a 15.41% relative gain, and 69.04% on PANDORA versus 65.77%, a 4.97% relative gain. It does not outperform the best baseline on every dimension: relative results are −9.24% for Kaggle P/J, −28.47% for PANDORA I/E, and −3.79% for PANDORA P/J. No confidence intervals, between-seed variation, significance tests, or hyperparameter-selection correction are reported, so “consistently” and “significantly” go beyond the published evidence. The limited-data experiment compares ROME trained on 40% of the data with ETM trained on 100%, rather than retraining baselines at the same fractions, and every retained user still requires a target label to generate 60 auxiliary targets.

Ablations support that the synthetic signal contributes to classification, not that it is psychometrically valid. Removing auxiliary evidence lowers Kaggle mean macro-F1 to 65.77%; using only that evidence reaches 75.12%. Expert routing receives an explicit one-hot MBTI-axis assignment produced by another unidentified LLM, so apparent axis specialization is not wholly emergent. The interpretability case selects one correctly classified user, computes item-removal impact, and then asks GPT-4o post hoc to select compatible snippets; this does not trace the model's actual decision or validate faithfulness. One displayed rationale even uses a sentence about “INFPs,” illustrating the risk of treating type discussion as evidence about the author. The prose also swaps the random- and minimum-removal labels around Table 6.

The paper presents a plausible architecture for distilling label-conditioned synthetic supervision and reports high averages on two familiar MBTI benchmarks. The defensible conclusion is narrow: on an unreproducible split of self-identifying communities with heavy type vocabulary, GPT-4o-generated auxiliary targets improve a BERT/MoE classifier. It does not establish human questionnaire responses, psychometric validity, real-personality inference, faithful explanations, out-of-domain robustness, fairness, clinical safety, or fitness for recommendation or mental-health decisions.

Español

ROME es un clasificador supervisado de las cuatro dimensiones binarias MBTI a partir de publicaciones de usuarios. Su etapa Ask entrega a GPT-4o-2024-08-06 los posts y la etiqueta MBTI verdadera de cada usuario de entrenamiento y le pide contestar 60 preguntas “MBTI-style” en una escala de −3 a +3. La etapa Answer entrena un Mixture-of-Experts condicionado por pregunta para reproducir esas respuestas sintéticas desde los posts; Detect combina las respuestas predichas con la representación textual para clasificar I/E, S/N, T/F y P/J. La fuente disponible sigue siendo arXiv v1 de 13 páginas; no se localizó versión publicada ni repositorio de código.

La intermediación no constituye una medición psicométrica independiente. El prompt ordena usar el tipo MBTI para resolver casos ambiguos o reforzar el juicio, y el método llama “ground truth” a respuestas que nunca fueron dadas por las personas. Así, la señal auxiliar está condicionada explícitamente por la misma clase objetivo que el modelo debe aprender. Esto puede ser una forma de destilación supervisada útil, pero no demuestra que GPT-4o reconstruya respuestas reales ni que el vector resultante sea evidencia psicológica. Las 60 preguntas no se publican completas, no se identifica su instrumento, licencia, fuente o estudio de validación y el texto alterna “standardized”, “validated” y “MBTI-style” sin aportar fiabilidad, estructura factorial, reglas de puntuación o comparación con respuestas humanas.

La evaluación usa 8.675 usuarios del corpus Kaggle/PersonalityCafe, con sus 50 posts más recientes, y 9.067 usuarios de PANDORA/Reddit, con decenas o cientos de posts. Las etiquetas son tipos autodeclarados en comunidades centradas en personalidad, no resultados administrados por el estudio ni una verdad objetiva. El paper declara splits de usuario 60/20/20 sin solapamiento, pero no publica seed, índices o repeticiones. Una auditoría de la descarga exacta de Kaggle citada encontró códigos MBTI explícitos en 8.220/8.675 filas (94,76%) y el tipo objetivo en 7.576 (87,33%). Un control determinista sin entrenamiento que elige el tipo más mencionado obtiene 64,20% de exactitud de 16 tipos y 79,70% de macro-F1 medio en las cuatro dimensiones. El artículo no informa haber eliminado esos tokens ni evalúa un corpus limpio, por lo que Kaggle mide en parte recuperación de etiquetas y lenguaje sobre MBTI, no solo inferencia de personalidad desde texto ordinario.

ROME reporta macro-F1 medio de 89,78% en Kaggle frente a 77,79% de ETM, una mejora relativa de 15,41%, y 69,04% en PANDORA frente a 65,77%, una mejora relativa de 4,97%. Sin embargo, no supera al mejor baseline en todas las dimensiones: cae 9,24% relativamente en P/J de Kaggle, 28,47% en I/E de PANDORA y 3,79% en P/J de PANDORA. No hay intervalos, desviaciones entre seeds, tests de significación o corrección por selección de hiperparámetros, por lo que “consistently” y “significantly” exceden la evidencia publicada. La prueba de escasez compara ROME con 40% de entrenamiento contra ETM con 100%, no contra baselines reentrenados con la misma fracción, y cada usuario reducido sigue requiriendo su etiqueta para generar las 60 dianas auxiliares.

Las ablaciones respaldan que la señal sintética contribuye al clasificador, no que sea psicométricamente válida. Sin evidencia auxiliar el promedio de Kaggle baja a 65,77%; usando solo esa evidencia llega a 75,12%. El análisis de expertos está condicionado por un one-hot del eje MBTI asignado por otro LLM no identificado, de modo que la especialización por eje no es enteramente emergente. El caso interpretativo elige un único usuario correctamente clasificado, calcula impacto por eliminación y después pide a GPT-4o seleccionar snippets compatibles; no rastrea la decisión real del modelo ni valida fidelidad. Incluso incluye como evidencia una frase sobre “INFPs”, ilustrando el riesgo de confundir discusión de tipos con rasgos del autor. Además, el texto intercambia las etiquetas random y minimum en la descripción de la Tabla 6.

El trabajo propone una arquitectura plausible para destilar supervisión sintética condicionada por etiqueta y obtiene promedios altos en dos benchmarks MBTI conocidos. La conclusión defendible es limitada: bajo un split no reproducible de comunidades con autodeclaraciones y fuerte vocabulario de tipos, el auxiliar generado por GPT-4o mejora un clasificador BERT/MoE. No establece respuestas de cuestionario humanas, validez psicométrica, inferencia de personalidad real, explicaciones fieles, robustez fuera de dominio, fairness, seguridad clínica ni aptitud para recomendación o salud mental.

Research question

Can an auxiliary signal of 60 synthetic MBTI responses, generated by an LLM that receives posts and the user's label, improve classification of the four MBTI dimensions through a question-conditioned Mixture-of-Experts?

Method

Supervised study on Kaggle/PersonalityCafe and PANDORA. For each training user, GPT-4o receives posts and the MBTI label and generates T samples of 60 Likert responses; T and the full schedule of temperatures are not specified. A router conditioned on posts, question, and MBTI axis combines 32 MLP experts to imitate those responses with L1 loss. The importance of each item is computed with the separation of synthetic responses between classes and the reliability with variance across samples. The predicted responses are masked by axis, fused with BERT, and optimized alongside four binary losses. Macro-F1 is reported on a single 60/20/20 split and ablations are added on Kaggle.

Sample: Kaggle contains 8,675 users and PANDORA 9,067. The counts in Table 1 correctly sum those totals for each dimension and reflect strong imbalances, especially S/N. The declared split is 60/20/20 per user with no overlap. The units are not participants recruited and evaluated by the study, but forum accounts with self-declared types. Kaggle comes from PersonalityCafe and retains abundant MBTI vocabulary; PANDORA obtains labels from introductions that mention the type. No age, culture, language, location, consent, deletions, activity, or evaluation by subgroups are reported.

Findings

  • arXiv v1 of 9 December 2025 remains the latest and only version located.
  • The architecture has three stages: synthetic Ask generation, MoE Answer imitation, and fused Detect classification.
  • GPT-4o-2024-08-06 explicitly receives the posts and the true MBTI label of each training user.
  • The prompt allows using the label to resolve ambiguities or reinforce judgment.
  • The auxiliary responses were not emitted by users nor verified against human responses.
  • The complete list of 60 items, their provenance, and their validation are not published.
  • The model uses 32 MLP experts, BERT-base-uncased, 100 epochs of L1 pretraining, and joint fine-tuning with classification lambda 0.05.
  • Kaggle has 8,675 users and PANDORA 9,067, with declared split 60/20/20 with no user overlap.
  • ROME obtains 89.78 mean macro-F1 on Kaggle and 69.04 on PANDORA.
  • The mean relative improvements over ETM are 15.41% on Kaggle and 4.97% on PANDORA.
  • ROME worsens compared to the best baseline on Kaggle P/J and on PANDORA I/E and P/J.
  • There are no repetitions, intervals, or tests supporting statistical significance.
  • The exact Kaggle download contains some MBTI code in 94.76% of the rows.
  • The exact target label appears in the text of 87.33% of the Kaggle rows.
  • In 67.85% of the rows, the target type ties as the most mentioned MBTI code.
  • A no-training rule based on the most mentioned type achieves 64.20% accuracy of 16 types.
  • The same control achieves 78.81/80.16/80.70/79.13 macro-F1 for I/E, S/N, T/F, and P/J; mean 79.70%.
  • The paper does not report cleaning of MBTI codes or an evaluation with those tokens removed.
  • Without questionnaire evidence, ROME drops to 65.77; with evidence only it reaches 75.12; the full model reaches 89.78.
  • The ablation without weighting drops only 1.31 points, from 89.78 to 88.47.
  • With 40% of the training, ROME obtains 78.01, but no baselines trained with the same 40% are reported.
  • GPT-3.5, GPT-mix, and GPT-4o as Ask generators produce averages 86.20, 87.45, and 89.78 on Kaggle.
  • BERT, Llama, and RoBERTa as encoders produce 89.78, 92.71, and 85.93, respectively.
  • The apparent specialization of experts receives as input an explicit assignment of the MBTI axis.
  • The interpretive case uses post-hoc selection of snippets by GPT-4o, not a causal trace of the classifier.
  • The only case shown is chosen among users correctly classified on the four dimensions.
  • One shown explanation uses a post about INFPs to support the classification of an ISTJ user.
  • The description of Table 6 swaps the names of random-removal and minimum-weight-removal.
  • No code repository or artifacts linked from arXiv or through exact searches were found.

Limitations

  • The target label is introduced into the generator of the auxiliary targets, so they are not independent supervision.
  • The method does not separate how much the text contributes and how much the stereotyped regularity induced by the label in GPT-4o contributes.
  • There is no control with responses generated without label, with permuted label, or with real human responses.
  • The claim of a validated questionnaire lacks name, version, source, license, and validation citation of the 60-item inventory.
  • The complete 60 items are not published, preventing auditing of content, pole balance, redundancy, and scoring.
  • No reliability, convergent validity, factor structure, invariance, test-retest, or measurement error are reported.
  • The seven-point format from -3 to +3 is not linked to an official MBTI administration.
  • The synthetic responses are called ground truth even though they were not observed.
  • T, exact temperatures, seed, top-p, system prompt, error handling, and parsing are not specified.
  • The figure suggests several temperatures, but the number and procedure of samples are not documented.
  • Another unidentified LLM assigns each question to an axis without publishing prompt, human validation, or error rate.
  • The item importance directly uses the separation between true classes, so it is a supervised statistic, not independent psychometric evidence.
  • The router receives the one-hot of the axis and afterwards the activation is interpreted as emergent psychological specialization.
  • BERT has limited length, but truncation, segmentation, or pooling of 50 posts or hundreds of posts is not explained.
  • Nor is it explained how a single user representation is constructed for BERT, RoBERTa, or Llama.
  • No code, weights, synthetic responses, outputs, questions, splits, or executable configuration are published.
  • The random split has no seed or indices and only one run is reported.
  • There is no cross-validation, standard deviations, bootstrap, intervals, or hypothesis tests.
  • The hyperparameters, number of experts, and questionnaire length are selected on the same benchmark without correction for search.
  • The implementations and configurations of the eleven baselines are not sufficiently documented.
  • It is not clarified whether the baseline numbers were reproduced with the same split or taken from previous works.
  • The claim consistently outperforms is false at the dimension level in three columns of Table 2.
  • The word significantly is used without a statistic, p-value, interval, or inferential effect size.
  • The few-shot experiment does not compare all methods with the same fraction of training.
  • Generating 60 responses still requires one label per user, so it does not solve label scarcity.
  • Calls, tokens, cost, offline latency, or computational footprint of GPT-4o and eight RTX 5880 Ada are not accounted for.
  • The text says GPT-4o is queried once, but the method requires T samples per user; the meaning of once is not clarified.
  • Kaggle contains explicit MBTI codes in almost the entire corpus and their removal is not reported.
  • The control rule by mentions surpasses the best ETM baseline on average on Kaggle, showing that the benchmark admits trivial shortcuts.
  • There is no ablation with the 16 codes, type names, URLs, and related vocabulary removed.
  • PersonalityCafe is a community centered on personality, so its language does not represent general everyday text.
  • PANDORA also derives labels from self-declarations; they are not administered assessments or external observations.
  • It is not evaluated whether the posts used include the self-declaration comment, flairs, or type references.
  • The classes are highly imbalanced and no calibration, precision-recall curves, or behavior by full type are reported.
  • The per-dimension results hide errors of 16-type combinations.
  • There is no comparison with an explicit label recovery rule, majority, logistic regression over type tokens, or cleaned corpus.
  • The interpretability case is a single one, correctly classified, not a blind or representative sample.
  • The snippets are selected by GPT-4o after knowing the influential items and do not prove model fidelity.
  • The paper does not verify that the snippets describe the author rather than third parties, quotes, or discussion of stereotypes.
  • The removal analysis tests sensitivity of the aggregate, not psychological causality or reasoning correctness.
  • The random/minimum nomenclature of Table 6 is contradicted between text and variant names.
  • There is no evaluation on Big Five, continuous measures, real questionnaires, or participants outside MBTI communities.
  • Temporal stability of the label, repeated self-report, disagreement between tests, or type change are not evaluated.
  • Age, gender, culture, language, region, subcommunity, or fairness are not analyzed.
  • The potential use in mental health is mentioned without clinical validation, harm analysis, or safety threshold.
  • There is no specific section on limitations, ethics, privacy, or consent.
  • The processing of sensitive posts through a closed API is not accompanied by a retention, anonymization, or governance policy.
  • Attacks, post manipulation, adversarial type mentions, or robustness to paraphrasing are not evaluated.
  • There is no external, temporal, cross-platform, or out-of-distribution validation.
  • The high performance on Kaggle cannot be separated from label contamination and domain-specific language.

What the study does not establish

  • It does not establish that users would have responded this way to a questionnaire.
  • It does not validate the 60-item inventory as official MBTI or as a psychometric instrument.
  • It does not demonstrate that the generated responses are psychological evidence and not encoding of the label.
  • It does not demonstrate personality inference from free text without explicit type mentions.
  • It does not demonstrate consistent superiority across the eight dimension-dataset combinations.
  • It does not demonstrate statistical significance of the improvements.
  • It does not demonstrate data efficiency compared to baselines under equal label budget.
  • It does not demonstrate faithful or causal interpretability of the MoE.
  • It does not demonstrate generalization to different users, platforms, languages, or cultures.
  • It does not demonstrate clinical validity, fairness, privacy, or safety for decisions about people.
  • It does not demonstrate that self-declared MBTI is true or objective personality.
  • It does not offer a reproducible result without questions, splits, synthetic targets, code, and seeds.

Traceability

Scope: Full text

Version: arXiv:2512.08814v1, submitted 9 December 2025; 13 pages; latest version confirmed through the arXiv API on 15 July 2026; no later proceedings or journal version located

Consulted source: https://arxiv.org/pdf/2512.08814v1

Review: Codex complete bilingual full-text fidelity pass, all-page PDF visual inspection, arXiv latest-version reconciliation, prompt and label-conditioning audit, exact cited Kaggle-source contamination audit, metric and table audit, interpretability-fidelity review, reproducibility and psychometric-validity assessment; summaries written from the full paper and verified source data rather than abstract keywords, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o-2024-08-06 as the label-aware offline role-play generator
  • BERT-base-uncased as the default post and question encoder
  • Question-conditioned Mixture-of-Experts with 32 MLP experts and hidden dimension 1024
  • Meta-Llama-3-8B-Instruct as a frozen encoder ablation
  • RoBERTa-base as an encoder ablation
  • GPT-3.5 as an Ask-stage ablation; exact model snapshot not reported
  • GPT-mix averaging GPT-3.5 and GPT-4o synthetic answers
  • ChatGPT-3.5 and GPT-4o zero-shot and three-shot personality-prediction baselines

Instruments and metrics

  • Undocumented 60-item MBTI-style questionnaire with integer responses from −3 to +3
  • Label-aware prompt that explicitly permits using the true MBTI type
  • T stochastic GPT samples averaged per user and item; T and full decoding schedule not reported
  • LLM-generated one-hot assignment of each item to one MBTI dimension; model and prompt not reported
  • Question-conditioned router over post, item and explicit dimension embeddings
  • Robust L1 answer-prediction objective and binary cross-entropy personality objective
  • Reliability weight from across-sample variance and importance weight from true-label class separation
  • Macro-F1 per binary dimension and unweighted mean over four dimensions
  • Component, data-fraction, generator, encoder, questionnaire-length and expert-count ablations
  • Single-user item-removal case study with post-hoc GPT-4o snippet selection

Data used

  • Kaggle datasnaek/mbti-type: 8,675 PersonalityCafe users, target type plus 50 recent posts per row
  • PANDORA MBTI subset: 9,067 Reddit users labeled from self-introductions, with dozens to hundreds of posts
  • User-level 60/20/20 train/validation/test partitions with no user overlap; split seed and indices unavailable
  • Locally verified Kaggle archive hash f0a190405e43c76c9d86ad6a2d09177567976fc56ef6e951a55f9e56de033fc1 and CSV hash 8b068e734efaed7a7905a1434b1f76eab2fc2f8c943247b5af78103e96940bdb
  • No released synthetic-answer table, questionnaire inventory, split files, trained weights, outputs, or source code

Evidence and location

  • Version, authorship, and date of arXiv: arXiv API record 2512.08814v1 checked 15 July 2026 and paper page 1
  • Prompt that receives label and 60 questions: Paper Section 3.3 and Figure 3, page 4
  • Ask–Answer–Detect architecture and objectives: Paper Sections 3.2–3.5, pages 3–6
  • Construct assignment by LLM and class weighting: Paper Equations 2 and 5–8, pages 5–6
  • Sizes, provenance, and splits: Paper Section 4.1 and Table 1, page 6
  • Baselines, implementation, and training: Paper Sections 4.2–4.3, pages 6–7
  • Results and improvements per dimension: Paper Table 2 and Section 4.4, page 7
  • Ablations and comparison with LLM: Paper Tables 3–4 and Sections 4.5–4.8, pages 8–9
  • Questionnaire length and number of experts: Paper Figures 6–7 and Section 4.9, page 10
  • Case, snippets, and random/minimum contradiction: Paper Table 5, Table 6 and Section 4.10, pages 10–11
  • MBTI code contamination and deterministic control: reports/verification/article-177-kaggle-label-leakage-audit.json; exact cited Kaggle CSV audited 15 July 2026
  • Absence of subsequent publication and code: arXiv API plus exact-title, arXiv-ID, GitHub and Hugging Face searches checked 15 July 2026
  • Visual inspection: All 13 pages of arXiv:2512.08814v1 rendered and visually inspected on 15 July 2026