Synthetic Personality Review

From Post To Personality: Harnessing LLMs for MBTI Prediction in Social Media

Evaluation and psychometric validity2025arXivApproved editorial review

Authors: Tian Ma, Kaiyu Feng, Yu Rong, Kangfei Zhao

Keywords: Computation and Language, Social and Information Networks, CIKM 2025

Source: Open primary source (opens in a new tab)

4
Authors
7
Findings
18
Limitations
10
Evidence

Editorial summary

English

PostToPersonality (P2P), published as a CIKM 2025 short paper, separately classifies the four MBTI dichotomies from the 50 most recent posts of each of 8,675 PersonalityCafe users. A local DeepSeek-R1-Distill-Llama-8B is QA-LoRA-fine-tuned for ten epochs and generates a textual personality description. To address imbalance, the paper uses SMOTE to interpolate hidden representations for the 15 less frequent types. The original text and generated description are encoded with Sentence-BERT, five labeled training users are retrieved, and a DeepSeek-V3 API returns four letters through in-context learning. On one 60/20/20 split, P2P reports 93.21% I/E, 94.75% N/S, 93.06% T/F, and 88.58% J/P accuracy, outperforming ten conventional and neural baselines. It does not report joint 16-type accuracy. The central threat is label leakage intrinsic to the corpus. Data come from an MBTI-focused forum, while preprocessing only lowercases, lemmatizes, and removes stopwords, special characters, links, URLs, and whitespace; it does not remove type names or MBTI vocabulary. The appendix itself includes an input literally stating “I knew I was an INTJ”, matching its INTJ ground truth, and another containing ENTP/ESTP/EXTP abbreviations. The model and retriever can therefore learn self-labels, quotations, and forum jargon rather than infer traits from language behavior. Hallucination is not measured either: the paper concludes that RAG mitigates hallucination solely because some classification metrics improve. P2P AUC is not computed from its prediction probabilities; it is approximated using the ground-truth label distribution among the five retrieved neighbors. It is therefore not the final classifier’s AUC and is not equivalent to baseline AUC. The fine-tuning pipeline is also underspecified: the local LLM is trained to predict a label but later used to generate a descriptive explanation without a published target for that explanation; SMOTE creates continuous hidden vectors, but the paper does not explain how these become QA-LoRA training examples. No official code or artifact is linked, and seeds, API snapshot, decoding parameters, costs, repeated runs, intervals, and statistical tests are absent. The results demonstrate benchmark performance under severe leakage risk, not reliable psychometric personality inference.

Español

PostToPersonality (P2P), publicado como short paper en CIKM 2025, clasifica por separado las cuatro dicotomías MBTI a partir de las 50 publicaciones recientes de cada uno de 8.675 usuarios de PersonalityCafe. Un DeepSeek-R1-Distill-Llama-8B local se ajusta con QA-LoRA durante diez épocas y genera una descripción textual de la personalidad. Para combatir el desbalance, el artículo interpola con SMOTE las representaciones ocultas de los quince tipos menos frecuentes. El texto original y la descripción generada se codifican con Sentence-BERT, se recuperan cinco usuarios de entrenamiento con sus etiquetas y una API DeepSeek-V3 produce las cuatro letras finales mediante aprendizaje en contexto. En un único split 60/20/20, P2P informa exactitudes de 93,21 % en I/E, 94,75 % en N/S, 93,06 % en T/F y 88,58 % en J/P, por encima de diez baselines clásicos y neuronales. Sin embargo, no se publica exactitud conjunta de los 16 tipos. La principal amenaza es una fuga de etiqueta inherente al corpus. El conjunto procede de un foro dedicado a MBTI y el preprocesado solo pasa a minúsculas, lematiza y elimina stopwords, caracteres, enlaces y espacios; no elimina nombres de tipos ni vocabulario MBTI. El propio anexo muestra un caso cuya entrada contiene literalmente “I knew I was an INTJ”, coincidiendo con su etiqueta real INTJ, y otro con abreviaturas como ENTP/ESTP/EXTP. El modelo y la recuperación pueden, por tanto, aprender autoetiquetas, citas y jerga del foro en lugar de inferir rasgos desde conducta lingüística. Tampoco se mide la alucinación: el artículo concluye que RAG la mitiga únicamente porque mejora algunas métricas de clasificación. El AUC de P2P no se calcula con probabilidades de su predicción, sino que se aproxima mediante la distribución de etiquetas reales entre los cinco vecinos recuperados; por ello no es el AUC del clasificador final ni una comparación equivalente con los baselines. La tubería de ajuste también está insuficientemente definida: se entrena al LLM local para predecir la etiqueta, pero después se usa para generar una explicación descriptiva sin publicar el objetivo que supervisa esa explicación; SMOTE crea vectores ocultos continuos, pero no se explica cómo se convierten en ejemplos compatibles con QA-LoRA. No hay código ni artefacto oficial enlazado, semillas, snapshots de API, temperatura, costes, repeticiones, intervalos o tests estadísticos. Los resultados prueban rendimiento sobre este benchmark contaminado, no inferencia psicométrica fiable de personalidad.

Research question

Can an architecture with efficient fine-tuning of a local LLM, oversampling of minority types, retrieval of labeled examples, and an LLM API surpass baselines in the classification of the four MBTI dichotomies from PersonalityCafe posts?

Method

The 50 recent posts of each user are concatenated and the 8,675 records are split into 60% training, 20% validation, and 20% test. The text is normalized, lemmatized, and cleaned. DeepSeek-R1-Distill-Llama-8B is fine-tuned with QA-LoRA, AdamW, ten epochs, rate 1e-4, batch 8, early stopping, and dropout 5%. For all types except INFP, SMOTE interpolates representations of the last layer between users of the same label. The local LLM generates textual traits; Sentence-BERT separately encodes posts and traits, concatenates both vectors, and FAISS retrieves five neighbors from the training set with their labels. DeepSeek-V3 receives the text, the traits, and the demonstrations and returns four letters. Accuracy, F1, and AUC are compared with ten baselines, and ablations of PEFT, SMOTE, RAG, backbones, and k are performed.

Sample: The corpus contains 8,675 users from PersonalityCafe, each with a self-declared MBTI label and their 50 recent posts concatenated. Approximately 5,205 users are used for training, 1,735 for validation, and 1,735 for test if the 60/20/20 split is exact, but the article does not publish the split lists. The distribution is highly uneven: INFP and INFJ dominate, and ESFJ, ESTJ, ESFP, and ESTP are scarce. The authors treat the fifteen types other than INFP as minority. It is not reported how many synthetic vectors are generated, the final balance, or whether the partition was repeated.

Findings

  • P2P obtains accuracy 0.9321/0.9475/0.9306/0.8858 and F1 0.9224/0.8747/0.9342/0.8759 on I/E, N/S, T/F, and J/P, respectively.
  • Compared with the best baseline of each dichotomy, the mean calculable absolute accuracy advantage in Table 1 is about 6.2 points, not a personality validation outside the corpus.
  • P2P does not dominate all AUCs: on N/S it obtains 0.8849, below SVM with 0.9184. Moreover, its AUC uses labels of neighbors and not probabilities of the final output.
  • Without SMOTE, T/F reaches F1 0.9530 and AUC 0.9516, higher than the full system on those metrics; SMOTE improves other dichotomies, so its effect is not uniformly positive.
  • Removing RAG while keeping PEFT and SMOTE leaves accuracies of 0.8950/0.9061/0.9006/0.8674; RAG improves the result, but the study does not measure factual hallucination nor demonstrate that as the cause.
  • The configuration with DeepSeek-V3 outperforms GPT-4 in this table, although no API version or decoding parameters are fixed, and the explanation of consistency between models is speculative.
  • The qualitative cases show attention to stereotypes such as art for INFP and thinking for ENTP; they also reveal explicit MBTI terms in the inputs, including the user's own INTJ label.

Limitations

  • The corpus presents direct label leakage: forum posts include type names and self-identification statements. The appendix contains “I knew I was an INTJ” in a case with label INTJ and multiple MBTI acronyms in another.
  • The described preprocessing does not remove MBTI strings, type names, signatures, quotes, or test responses. There is no evaluation with those cues masked nor a set outside typology forums.
  • The distribution of a self-selected forum does not demonstrate the “natural distribution in the population”; that claim confuses dataset frequency with population prevalence.
  • The labels are types self-declared by users, without a verified instrument, date, reliability, independent agreement, or psychometric evaluation.
  • Only metrics per dichotomy are published. It is not reported what percentage of users obtain the four letters correctly, which would be the real accuracy of the 16 types.
  • The AUC of P2P is approximated from the distribution of true labels of the retrieved neighbors, not from probabilities or scores of the final predictor; it is not directly comparable with the AUC of the baselines.
  • It is not defined whether F1 is macro, micro, weighted, or of a positive class, nor which pole is considered positive; under strong imbalance, that omission changes the interpretation.
  • Hallucination has no operational definition, set of facts, human evaluation, or metric. A classification improvement from adding RAG does not demonstrate a reduction of hallucinations.
  • The published objective fine-tunes the local LLM to predict label c, but the pipeline then asks it to produce a textual description h. No targets, prompts, or loss that teach that explanation are documented.
  • SMOTE interpolates continuous representations of the last layer and says it adds them to the QA-LoRA fine-tuning set, but it does not explain where they are injected, how they align with token sequences, nor how they produce valid generative examples.
  • Class imbalance is called OOD, although class rarity and distribution shift are distinct problems; OOD generalization is not evaluated.
  • A single 60/20/20 split is used without seeds, repetitions, confidence intervals, significance tests, or sensitivity analysis to the partition.
  • Hardware, exact version of DeepSeek-V3, temperature, top-p, decoding seeds, retry format, invalid response rate, cost, or latency are not specified.
  • The “state of the art” comparison is limited to ten baselines implemented by the authors and some ablations; it does not include a clean test of direct prompting with contamination control nor other contemporary datasets.
  • Attention visualizations are not validated as faithful explanations, and the interpretation uses MBTI stereotypes. There is no human evaluation, causal perturbation, or expert agreement.
  • Privacy, consent, bias by age/gender/culture/language, harm from profiling, possibility of manipulation, or high-impact uses are not analyzed.
  • The paper does not link code, weights, complete prompts, predictions, or reproducible artifacts; the search for official materials did not identify a verifiable repository associated with the publication.
  • The generative AI section declares that GPT-4.1 polished part of the text, but it does not identify which passages; this does not invalidate the results, although it reinforces the need for code and experimental traces that are not published.

What the study does not establish

  • It does not demonstrate that P2P infers personality from spontaneous language; it may recognize MBTI self-labels, forum jargon, and correlations from a contaminated benchmark.
  • It does not establish a reduction of hallucinations, because it does not measure factuality or hallucination before and after RAG.
  • It does not demonstrate performance on the full 16 types, general populations, other platforms, other languages, or users who never mention MBTI.
  • It does not validate MBTI, the self-declared label, or the psychological explanations generated by the LLM.
  • It does not prove that SMOTE consistently improves all dichotomies or that its synthetic vectors correspond to plausible texts or persons.
  • It does not justify using the tool for education, employment, mental health, sensitive recommendation, segmentation, or decisions about individuals.
  • It does not allow reproducing the main figures with public materials linked by the article.

Traceability

Scope: Full text

Version: arXiv:2509.04461v1, submitted 28 August 2025, 8 pages; CIKM 2025 short paper, DOI 10.1145/3746252.3760813

Consulted source: https://arxiv.org/pdf/2509.04461v1

Review: Codex full-text, bilingual-fidelity, 8-page visual, CIKM-metadata, direct-label-leakage, forum-contamination, metric-equivalence, SMOTE-implementation, RAG-hallucination, 16-type-scope, statistical, interpretability, privacy, fairness and reproducibility audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • DeepSeek-R1-Distill-Llama-8B
  • DeepSeek-V3 API
  • Sentence-BERT
  • MindChat-7B
  • ChatGLM-6B
  • GPT-4
  • Multinomial Naive Bayes
  • Logistic Regression
  • Support Vector Machine
  • XGBoost
  • Multilayer Perceptron
  • Word2Vec
  • e5-base
  • ModernBERT-base
  • DeBERTa-v3-base

Instruments and metrics

  • Four binary MBTI dichotomy classifiers
  • QA-LoRA fine-tuning
  • SMOTE on local-LLM hidden representations
  • FAISS exact k-nearest-neighbor retrieval
  • Labeled in-context demonstrations
  • Accuracy and unspecified F1 variant
  • Neighbor-label proxy AUC
  • Final-layer attention visualization

Data used

  • Kaggle MBTI Personality Type Dataset / PersonalityCafe forum posts

Evidence and location

  • Architecture, contributions, and performance claims: Paper, pp. 1–2, Abstract, Introduction and Figure 1
  • Labeled retrieval, local fine-tuning, and hidden SMOTE: Paper, pp. 2–3, Sections 3.1–3.2 and Equations 1–3
  • Dataset, split, models, API, and approximate AUC: Paper, p. 3, Section 4.1
  • Main results and comparison with baselines: Paper, p. 4, Table 1
  • Ablations of PEFT, SMOTE, RAG, models, and k: Paper, p. 4, Table 2 and Section 4.2.2
  • Label leakage and examples with explicit MBTI acronyms: Paper, pp. 6–8, Appendix D, especially Case 2: “I knew I was an INTJ”
  • Attention, stereotypes, and absence of fidelity validation: Paper, p. 4, Figure 2 and Section 4.2.3
  • Use of generative AI and unlinked materials: Paper, p. 5, Usage of Generative AI; arXiv:2509.04461v1 metadata and paper references
  • CIKM 2025 publication: Crossref record for DOI 10.1145/3746252.3760813; CIKM 2025, pp. 5011–5015
  • Comprehensive visual inspection: Paper, all 8 rendered pages, including both result tables, three figures and complete case-study appendix