Synthetic Personality Review

Identifying Multiple Personalities in Large Language Models with External Evaluation

Evaluation and psychometric validity2024arXivApproved editorial review

Authors: Xiaoyang Song, Yuta Adachi, Jessie Feng, Mouwei Lin, Linhao Yu, Frank Li, Akshat Gupta, Gopala Anumanchipalli, Simerjot Kaur

Keywords: Computation and Language, Artificial Intelligence, Large Language Models, Personality Assessment, Machine Learning

Source: Open primary source (opens in a new tab)

9
Authors
10
Findings
23
Limitations
13
Evidence

Editorial summary

English

The preprint proposes an “external” evaluation: rather than asking an LLM to complete a questionnaire, it LoRA-fine-tunes Llama2-7B to predict one of 16 MBTI labels from 50 texts and applies that classifier to posts and comments generated by ChatGPT and three Llama2-chat models. The detector reaches 81.0% accuracy when aggregating four binary classifiers and 81.7% as a direct 16-class classifier on an internal split of the Kaggle PersonalityCafe dataset. For each model and role, the study resamples 100 sets of 50 texts with replacement. Llama2-7B and 13B shift from a modal ESTJ label on posts to INFP on comments, Llama2-70B from ESTJ to INFJ, while ChatGPT remains modally INFJ but has a substantially different distribution. Eight celebrities form a human comparison: seven retain the modal label across posts/comments, but only six retain the second-most-frequent label as well. The study shows that a style/topic classifier assigns different MBTI distributions to two output types. It does not demonstrate multiple LLM personalities: role, prompt, content, and discourse act change together; the detector is transferred from a personality forum to Twitter and synthetic text without out-of-domain validation; the 100 samples overlap; and there is no statistical test, celebrity MBTI ground truth, or control for topic and length. The strongest conclusion is methodological: text-inferred personality labels depend strongly on domain and situation and should be treated as classifier output patterns, not internal traits.

Español

El preprint propone una evaluación «externa»: en vez de pedir a un LLM que conteste un cuestionario, entrena un Llama2-7B con LoRA para predecir una de 16 etiquetas MBTI a partir de 50 textos y aplica ese clasificador a publicaciones y comentarios generados por ChatGPT y tres Llama2-chat. El predictor alcanza 81,0% de accuracy al agregar cuatro clasificadores binarios y 81,7% como clasificador directo de 16 clases en un split interno del dataset Kaggle PersonalityCafe. Para cada modelo y rol se remuestrean 100 conjuntos de 50 textos con reemplazo. En los resultados, Llama2-7B y 13B pasan de una moda ESTJ en posts a INFP en comentarios, Llama2-70B de ESTJ a INFJ, y ChatGPT mantiene INFJ como moda pero cambia mucho su distribución. Ocho celebridades sirven como contraste: siete conservan la moda entre posts/comentarios, pero solo seis conservan también la segunda moda. El estudio muestra que un clasificador de estilo/tema asigna distribuciones MBTI distintas a dos tipos de salida. No demuestra que los LLM tengan múltiples personalidades: rol, prompt, contenido y acto discursivo cambian a la vez; el detector se traslada de un foro de personalidad a Twitter y texto sintético sin validación fuera de dominio; las 100 muestras se solapan; no hay test estadístico, verdad MBTI de celebridades ni control por tema/longitud. La conclusión más sólida es metodológica: las etiquetas de personalidad inferidas desde texto dependen fuertemente del dominio y la situación, por lo que deben interpretarse como patrones de salida del clasificador, no como rasgos internos.

Research question

Can an MBTI classifier trained on sets of 50 human texts serve as an external evaluator of open LLM outputs, and do the label distributions change when the same model generates posts versus comments on Twitter compared with human authors?

Method

The Kaggle MBTI dataset of 8,675 PersonalityCafe users is split into 81% training, 9% validation, and 10% test. Each entry contains 50 posts/comments and one MBTI label. Four binary Llama2-7B models and one 16-class Llama2-7B are fine-tuned with LoRA for five epochs (r = 16, learning rate 1e-4, batch 8, warm-up 100 steps, linear decay). For the external study, news from November 2023 is summarized and 5,000 tweets are collected across ten topics; ChatGPT and Llama2-chat 7B/13B/70B generate 4,500 posts from news and 5,000 comments to tweets using two distinct system/user prompts, temperature 0.2, top_p 0.95, and maximum 200. For each model and role, 100 samples of 50 outputs are drawn with replacement and classified. The resampling is repeated with real posts and comments from eight anonymous celebrities as a human contrast.

Sample: 8,675 entries to train/evaluate the detector, with an 81:9:10 split. For each combination of four LLMs and two roles, 100 resamples of 50 texts are drawn with replacement from the generated pools. The human contrast uses eight celebrities and 100 resamples of 50 posts or comments per person. The identities and true MBTI labels of the celebrities are not provided.

Findings

  • The aggregated binary detector achieves 93.3% mean accuracy per axis but 81.0% on the full 16-class label; the direct detector achieves 93.5% per axis and 81.7% on 16 classes. The difference illustrates that averaging axes overestimates full-type performance.
  • The cited baselines fall from 61.0–78.2% mean accuracy per axis to 14.0–37.6% on 16 classes. However, their results are taken from prior works and no homogeneous reevaluation on the same split is reported.
  • In posts, ESTJ dominates in Llama2-7B (73%), Llama2-13B (43%), and Llama2-70B (53%). In comments, INFP dominates in 7B and 13B (80% each), while 70B is split mainly between INFJ (45%) and INFP (37%).
  • ChatGPT keeps INFJ as the mode in both posts and comments, but the distribution changes: in posts INFJ/INFP/ISTJ account for approximately 30/29/19%, while in comments INFJ reaches 73.7%.
  • The mode change is not universal: ChatGPT retains INFJ. Therefore the general result is a role-assigned distribution change, not necessarily a categorical change for all models.
  • Among the eight celebrities, seven retain the most frequent type; Celebrity VI changes INFJ→ENTJ. Six also retain the second type; Celebrity VIII keeps INFJ but changes ISFJ→ENFJ as the second mode.
  • Human distributions are not identical either: for example, Celebrity II goes from 67% to 56% INFJ and Celebrity III from 79% to 54%. Greater relative stability does not equal invariance.
  • The LLM post/comment differences are larger than those observed for most celebrities under this detector, but the design does not separate the effect of role from the effect of topic, prompt, length, or domain.
  • The authors acknowledge that the human definition of personality should not be transferred naively to LLMs and that the work offers no validated alternative definition or measure.
  • The work shows the usefulness of reporting the full distribution: an equal mode, as in ChatGPT, can hide large changes, and a single label loses the classifier's uncertainty.

Limitations

  • The detector is validated within PersonalityCafe and then applied without validation to Twitter, celebrities, and LLM-generated text. The change in domain, platform, length, audience, and authorship may alter predictions.
  • PersonalityCafe is a forum about personality types. The article does not document removing MBTI mentions, type names, signatures, URLs, or other direct cues, so there is a risk of label leakage.
  • No distribution of the 16 classes, confusion matrix, per-class performance, or definition of whether F1/precision/recall are macro, micro, or weighted is presented. High accuracy can hide minority classes.
  • The claim of significantly outperforming the state of the art includes no statistical test. The baselines are cited from their original works rather than retrained with exactly the same split, preprocessing, and protocol.
  • An MBTI label from a PersonalityCafe user is self-reported and categorical; it is not verified with an administered instrument, reliability is not measured, and MBTI has its own psychometric limitations.
  • The classifier may learn topic, community vocabulary, and platform style rather than personality. The selection of ten topics and the prompts themselves induce signals of that type.
  • Posts and comments do not constitute an isolated manipulation: they use different system prompts, different inputs (news summaries versus tweets), different speech acts, and probably different lengths/topics.
  • The study does not pair the same content expressed as a post and a comment, does not counterbalance prompts, and does not use a role-blind classifier. Therefore it cannot causally attribute the difference to role-dependent personality.
  • The 100 observations are samples of 50 with replacement from the same pools. They overlap heavily and are not 100 independent model runs or 100 authors; the graphs provide no intervals adjusted for that dependence.
  • Sampling with replacement may duplicate texts within a set of 50, whereas the detector was trained on 50 distinct posts from one person. This discrepancy may increase confidence on repeated features.
  • The text uses "significantly different" for the post/comment distributions, but reports no distance, chi-square, divergence, permutation, interval, or p-value.
  • The classification uses a Llama2-7B to evaluate outputs from the same Llama2 family. Shared tokenizer, pretraining, or style artifacts may affect ChatGPT and Llama2 differently.
  • The ChatGPT snapshot, exact API date, and seed are not specified. The claim that November 2023 content was not in training cannot be verified for a changing API alias.
  • The pipeline that summarizes thousands of news items does not identify the summarization system or evaluate fidelity. Possible summary biases pass into the prompt and the subsequent label.
  • It is not unequivocally clarified whether the figures 4,500 posts and 5,000 comments correspond to each model or to the total, nor are the generated pools, news, tweets, or code published.
  • It is not documented how 50 texts are concatenated and truncated for Llama2-7B, whose window may be smaller than 50 outputs of up to 200 tokens. Order and truncation could dominate the classification.
  • The human validation does not know the true personality of the celebrities. Obtaining similar labels across two text types proves detector consistency, not accuracy or validity on persons.
  • Eight celebrities are a small, selective, and anonymous sample. There is no information on total number of posts per role, period, language, automated activity, or inclusion criteria.
  • The human results are not completely stable: one celebrity changes the mode and another the second mode; several change their proportions substantially. The claim that human personality is consistent rests on an undefined descriptive threshold.
  • Comparing celebrity posts and comments may mix personal writing, communication teams, promotional content, and professional management. Authorship is not verified.
  • No consent, privacy, platform terms, or ethical treatment of tweets and celebrities is discussed, and identities are hidden for proprietary reasons, preventing independent audit.
  • Only English, Twitter, four 2023 models, and MBTI are studied. There is no generalization to multi-turn conversation, other languages/platforms, Big Five, current models, or high-impact tasks.
  • Even a perfectly precise text classification would not demonstrate an internal state: it may reflect behavior conditioned by the prompt, which is precisely what the model was designed to adapt.

What the study does not establish

  • It does not demonstrate that ChatGPT or Llama2 possess a human personality or multiple internal personalities.
  • It does not demonstrate that role causes the observed change, because prompt, content, topic, and format also change.
  • It does not validate the MBTI detector outside PersonalityCafe or demonstrate accuracy on Twitter, celebrities, or synthetic text.
  • It does not demonstrate that humans maintain a true and invariant MBTI across posts and comments; it only observes greater similarity of predictions in eight cases.
  • It does not provide a validated definition of LLM personality or a measure suitable for clinical, workplace, or security decisions.
  • It does not allow inferring traits, intentions, motivations, or future stability from a modal label of the classifier.

Traceability

Scope: Full text

Version: arXiv:2402.14805v1 (22 Feb 2024)

Consulted source: https://arxiv.org/pdf/2402.14805

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • Llama2-7B base fine-tuned as MBTI detector
  • ChatGPT (unspecified API snapshot)
  • Llama2-7B-chat
  • Llama2-13B-chat
  • Llama2-70B-chat

Instruments and metrics

  • Myers-Briggs Type Indicator (16 categorical types)
  • Four binary Llama2-7B MBTI classifiers
  • Direct 16-class Llama2-7B MBTI classifier
  • Accuracy, F1, precision and recall
  • Modal and resampled MBTI label distributions

Data used

  • Kaggle MBTI / PersonalityCafe dataset (8,675 users, 50 texts each)
  • November 2023 news-derived topics
  • 5,000 collected tweets across Bitcoin, NFL, Music, Oscars, Travel, Fashion, Food, Fitness, Gaming and Technology
  • 4,500 LLM-generated posts and 5,000 LLM-generated comments reported
  • Posts and comments from eight anonymized celebrities

Evidence and location

  • Motivation, external evaluation, and stated conclusion: arXiv v1, pp. 1–2, abstract and introduction
  • PersonalityCafe dataset and previous baselines: arXiv v1, pp. 3–4, sections 2.3 and 3.1
  • Detector architecture, LoRA, and hyperparameters: arXiv v1, p. 4, section 3.1
  • Performance per axis and on 16 classes: arXiv v1, pp. 4–5, Tables 1–2
  • News/tweets, two roles and distinct prompts: arXiv v1, pp. 5–6, section 3.2 and Table 3
  • Pool sizes and resampling of 50 with replacement: arXiv v1, p. 6, sections 3.2–3.3
  • LLM distributions by role: arXiv v1, pp. 6–7, Figure 3 and Table 4
  • Contrast with eight celebrities: arXiv v1, pp. 7–8, Figure 4, Table 5 and Validation with Human Counterpart
  • Conceptual caution and classifier limitation: arXiv v1, p. 8, Conclusions and Limitation
  • 81:9:10 split and complete per-axis results: arXiv v1, p. 11, Appendix A.3, Tables 6–7
  • Generation, topics, temperature, top_p, and length: arXiv v1, p. 11, Appendix A.4
  • Distributions of the eight celebrities and exceptions: arXiv v1, pp. 11–12, Appendix A.5 and Figure 5
  • Examples and possible identifying content from PersonalityCafe: arXiv v1, p. 12, Table 8