Synthetic Personality Review

LLM vs Small Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model

Evaluation and psychometric validity2024AAAI ProceedingsApproved editorial review

Original title: LLM vs Small Model? LLM-Based Text Augmentation for Personality Detection

Authors: Linmei Hu, Hongyu He, Duokang Wang, Ziwang Zhao, Yingxia Shao, Liqiang Nie

Keywords: personality detection, text augmentation, knowledge distillation, contrastive learning, psycho-linguistic analysis, social media, MBTI

Source: Open primary source (opens in a new tab)

6
Authors
28
Findings
37
Limitations
27
Evidence

Editorial summary

English

This paper does not evaluate an LLM's personality: it uses GPT-3.5 as an auxiliary-data generator to predict four MBTI dichotomies for people from their posts. TAE asks gpt-3.5-turbo-0301 at temperature zero for three analyses of every post, semantic, sentiment, and linguistic, and uses them as positive pairs in a contrastive loss that trains a BERT encoder. It also generates descriptions of each MBTI label and derives cosine-similarity soft targets; only the small model is required at inference. On 60/20/20 splits of Kaggle/PersonalityCafe with 8,675 users and PANDORA/Reddit with 9,067, TAE reports average macro-F1 of 72.07 and 63.05. Its gains over the best baseline in each table are 0.72 and 1.04 absolute points, corresponding to the 1.01% and 1.68% relative improvements stated in the prose; gains over BERTmean are 5.83 and 6.53 points. The evidence supports that representations trained with generated analyses outperform BERTmean and variants that concatenate those texts in this setup. It does not isolate the effect as psycholinguistic knowledge: generated analyses receive no human validation, ablations are single values without uncertainty, and removing label information changes the average only from 72.07 to 72.02. Nor does it support the general claim that reasoning or few-shot prompting degrades ChatGPT: CoT falls to 65.12, but three-shot rises from 66.89 to 67.74 and uses a different, longer-context snapshot. The study treats noisy, severely imbalanced MBTI declarations as reference labels and does not examine psychometric validity, minority-class behavior, bias, privacy, or cross-domain transfer. It releases no code, seeds, GPT outputs, cost accounting, confidence intervals, or statistical tests, limiting reproduction and confidence in small performance margins.

Español

Este trabajo no evalúa la personalidad de un LLM: usa GPT-3.5 como generador de datos auxiliares para predecir en personas las cuatro dicotomías MBTI a partir de sus publicaciones. El método TAE pide a gpt-3.5-turbo-0301, con temperatura cero, tres análisis por publicación, semántico, de sentimiento y lingüístico, y los usa como positivos de una pérdida contrastiva que entrena un codificador BERT. También genera descripciones de cada etiqueta MBTI para construir etiquetas suaves mediante similitud coseno; en inferencia solo queda el modelo pequeño. En particiones 60/20/20 de Kaggle/PersonalityCafe, con 8.675 usuarios, y PANDORA/Reddit, con 9.067, TAE obtiene macro-F1 medio de 72,07 y 63,05. Las mejoras frente al mejor baseline de cada tabla son 0,72 y 1,04 puntos absolutos, equivalentes a los 1,01% y 1,68% relativos que cita el texto; frente a BERTmean son 5,83 y 6,53 puntos. La evidencia apoya que, en esta configuración, las representaciones entrenadas con análisis generados superan a BERTmean y a las variantes que concatenan esos textos. No permite atribuir por separado el efecto a conocimiento psicolingüístico: no hay validación humana de los análisis, las ablaciones son ejecuciones puntuales sin incertidumbre y retirar la información de etiquetas apenas cambia 72,07 a 72,02. Tampoco sustenta la afirmación general de que el razonamiento o el few-shot empeoren ChatGPT: CoT baja a 65,12, pero tres ejemplos suben de 66,89 a 67,74 y además usan otro snapshot con ventana de contexto mayor. El trabajo trata declaraciones MBTI ruidosas y muy desequilibradas como referencia, no analiza validez psicométrica, clases minoritarias, sesgo, privacidad ni transferencia entre dominios. No publica código, semillas, salidas de GPT, coste, intervalos ni pruebas estadísticas, por lo que la reproducibilidad y la solidez de unas ventajas pequeñas quedan limitadas.

Research question

Can the knowledge generated by an LLM (semantic, sentiment, and linguistic analyses of posts, plus MBTI tag descriptions) improve a small BERT model to detect personality types declared on social media without adding LLM cost during inference?

Method

TAE generates with gpt-3.5-turbo-0301 and zero temperature three analysis texts per post. A BERT base uncased encodes the post and each augmentation; InfoNCE approximates them as positive pairs and uses other pairs from the minibatch as negatives. The representations of up to 50 posts per user are averaged. The same LLM drafts three explanations for each pole of the four MBTI dimensions; BERT represents them and its cosine similarity with the user produces a distribution that is combined with the one-hot label before a KL loss. The total function sums detection and contrast with lambda 1. The study compares macro-F1 per axis and average on Kaggle and PANDORA, runs ablations only on Kaggle, compares ChatGPT zero-shot, CoT, and 3-shot, and tunes lambda on validation.

Sample: The unit is the user, not the LLM. Kaggle contains 8,675 users divided into 5,205 training, 1,735 validation, and 1,735 test. PANDORA contains 9,067 users divided into 5,440, 1,813, and 1,814. Each user contributes at most 50 posts; each post and analysis is truncated to 70 words. Prevalences are highly uneven: for example, S/N is 727/4,478 in Kaggle training and 610/4,830 in PANDORA. Labels come from MBTI types declared on PersonalityCafe or extracted from Reddit self-descriptions, not from a clinical assessment administered by the study.

Findings

  • The problem is human personality classification from text; the LLM acts as an auxiliary supervision generator and not as an evaluated subject.
  • TAE generates three analyses for each post: semantic, sentiment, and linguistic.
  • The generation prompt asks to analyze characteristics of a user and to omit a possible truncated ending with ellipses.
  • The post and each analysis form positive pairs; other pairs from the minibatch serve as negatives in InfoNCE.
  • BERT uses the CLS state and an MLP projection with tanh for the contrast.
  • The user representation is the mean of the BERT representations of their original posts.
  • The LLM also describes semantics, sentiment, and linguistics of each MBTI pole to obtain label representations.
  • Soft labels combine user-label similarity with the one-hot label using alpha 4 and are optimized with KL.
  • The final loss is Ldet + lambda Lcl; it does not contain a separate term for LLM logit distillation.
  • At inference no analyses are generated: the LLM cost remains in training preparation.
  • Both datasets use four binary MBTI classifiers and show severe imbalance.
  • The declared split is 60/20/20 after shuffling, following D-DGCN.
  • TAE obtains 72.07 average macro-F1 on Kaggle and 63.05 on PANDORA.
  • On Kaggle the best baseline is D-DGCN+l0 with 71.35; on PANDORA it is D-DGCN with 62.01.
  • The prose swaps the names of those two baselines, although the values and the table allow reconstructing the correct comparison.
  • Gains over the best baseline are 0.72 and 1.04 absolute points; 1.01% and 1.68% are relative improvements, not macro-F1 points.
  • Against BERTmean, TAE improves 5.83 points on Kaggle and 6.53 on PANDORA.
  • Removing the linguistic augmentation produces the largest drop among the three types, from 72.07 to 71.07.
  • Removing the semantic augmentation leaves 71.40 and removing sentiment 71.25; the differences among the three are small and include no uncertainty.
  • Removing the label descriptions leaves 72.02, only 0.05 points below the full model.
  • Eliminating all components reproduces BERTmean with 66.24 on Kaggle.
  • TAEConcat and TAEWS reach 67.91 and 67.31: using the generated texts as input improves BERTmean, but less than contrastive training.
  • ChatGPT zero-shot obtains 66.89 on Kaggle, CoT 65.12, and 3-shot 67.74.
  • The textual claim that CoT and few-shot decrease performance contradicts the table: 3-shot improves 0.85 points over zero-shot.
  • The 3-shot comparison switches from gpt-3.5-turbo-0301 to gpt-3.5-turbo-16k-0613 because of context length.
  • Lambda is searched between 0.5 and 4 and reaches the best validation value at 1 for both datasets.
  • The article does not link a repository or publish code artifacts or generated outputs.
  • The final version was published in AAAI-24, volume 38(16), pages 18234-18242, with DOI 10.1609/aaai.v38i16.29782.

Limitations

  • No runs with different seeds, deviations, confidence intervals, or significance tests are reported.
  • The advantages of 0.72 and 1.04 points over the best baseline could fall within unmeasured training variation.
  • The prose confuses relative improvement with absolute points and swaps the names of the best baselines between datasets.
  • The ablations are limited to Kaggle and present a single figure per configuration.
  • The effect of label information is only 0.05 points and it is not shown to exceed experimental noise.
  • No manual validation is performed of whether the generated analyses are correct, relevant, consistent, or free of hallucinations.
  • No comparison is made of the quality of GPT-3.5 with human augmentations, non-generative rules, or different generator models.
  • The prompt generates three perspectives in a single response; it is not documented how they are separated, cleaned, or how malformed responses are handled.
  • Other examples from the minibatch may be semantic false negatives or belong to the same MBTI type.
  • The proximity between a post and its analytical paraphrase does not prove that the learned representation is specifically about personality.
  • The similarity between BERT embeddings of generated descriptions and users does not validate psychological relationships between MBTI poles.
  • The motivation for soft labels recognizes that dichotomies lose position on a scale, but the dataset contains no continuous scores with which to test that neutrality.
  • MBTI labels are self-declared or extracted from self-descriptions, not verified through a controlled administration of the instrument.
  • Label extraction error in PANDORA is not studied, nor are type changes declared by the same person.
  • MBTI is used as four binary labels without examining reliability, construct validity, stability, or suitability for automated inference.
  • The article cites Big Five evidence to support questionnaire errors, but the task and experimental labels are MBTI.
  • The datasets are highly imbalanced and only macro-F1 per axis is reported; per-class results, confusion matrices, and calibration are missing.
  • Averaging four macro-F1 values hides differences: on PANDORA TAE drops below D-DGCN on T/F and P/J despite winning on average.
  • No disaggregated results are reported by language, demographics, community, activity, history length, or other subgroups.
  • No fairness evaluation is performed, nor whether the system amplifies personality stereotypes present in GPT-3.5 or on the platforms.
  • There is no cross-evaluation: training on PersonalityCafe and testing on Reddit, or vice versa, would reveal domain generalization.
  • No comparison is made with a temporal split or with new users or platforms outside the benchmarks.
  • Although words matching labels are replaced with <type>, other explicit expressions that may leak the type are not audited.
  • The shuffling seed, the identifiers of each split, and duplicate or overlap controls are not published.
  • Truncating to 50 posts and 70 words may exclude information unevenly; there is no sensitivity analysis.
  • The study does not report how many calls to GPT were made, tokens, cost, failures, retries, or storage of responses.
  • Using a retired snapshot of GPT-3.5 and not publishing its outputs prevents exactly regenerating the augmented dataset.
  • The zero-shot, CoT, and 3-shot comparison is not clean because 3-shot uses another snapshot with a different window.
  • The few-shot selection randomly picks three examples, but reports no seed, class balance, or sensitivity to the demonstrations.
  • A single run at zero temperature does not measure variation by backend, service versions, or residual non-determinism.
  • Inference does not use an LLM, but training requires prior massive generation whose cost and footprint are not compared with the improvement.
  • No code, full configuration, checkpoints, classification prompts, generated outputs, or evaluation scripts are published.
  • The formulation of the contrastive loss and the handling of three positives per post require implementation decisions not resolvable solely from the equations.
  • It is not clarified whether all baselines were re-run under exactly the same splits, preprocessing, and seeds, or whether some figure was taken from prior work.
  • There is no limitations, impact, ethics, or privacy section despite inferring psychological attributes from forum and Reddit users.
  • Consent, contextual expectations, anonymization, retention, or secondary use of potentially sensitive posts are not discussed.
  • Application to dialogue or psychological treatment is mentioned as motivation, but safety or utility in those high-risk contexts is not evaluated.

What the study does not establish

  • It does not demonstrate that an LLM has personality, psychological traits, or an identity of its own.
  • It does not evaluate the personality expressed by GPT-3.5; it evaluates its utility as an auxiliary generator and classifier of people.
  • It does not demonstrate that MBTI is a complete, continuous, or clinically valid representation of personality.
  • It does not demonstrate that self-declared labels are objective truth about each user.
  • It does not establish that the generated analyses contain correct psycholinguistic knowledge.
  • It does not isolate whether the improvement comes from the LLM content, from a paraphrase, from contrastive regularization, or from more text.
  • It does not demonstrate that the label descriptions contribute materially to performance.
  • It does not demonstrate a statistically reliable improvement over the best baseline.
  • It does not establish a new state of the art robust to seeds, splits, or different datasets.
  • It does not demonstrate that CoT harms personality detection in general.
  • It does not demonstrate that few-shot harms; the table itself shows an improvement over zero-shot.
  • It does not allow a causal comparison of gpt-3.5-turbo-0301 with the 3-shot condition based on 16k-0613.
  • It does not demonstrate generalization between PersonalityCafe and Reddit or to other platforms.
  • It does not demonstrate validity in other languages, cultures, periods, or personality models.
  • It does not establish fair performance for minority classes or demographic groups.
  • It does not demonstrate absence of label leakage, memorization, duplicates, or bot-generated content.
  • It does not demonstrate that the prediction is suitable for diagnosis, job selection, education, health, or high-impact personalization.
  • It does not demonstrate that inferring personality without consent is ethical or compatible with user expectations.
  • It does not allow exactly reproducing the figures without the historical GPT outputs, the code, and the splits.
  • It does not demonstrate that removing the LLM at inference removes its cost, biases, or dependence on the training process.

Traceability

Scope: Full text

Version: AAAI-24 final proceedings paper, volume 38 issue 16, pp. 18234-18242, DOI 10.1609/aaai.v38i16.29782, 9 pages

Consulted source: https://ojs.aaai.org/index.php/AAAI/article/download/29782/31350

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, task-scope, psychometric, experimental-design, result-arithmetic, internal-consistency, reproducibility, privacy and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • TAE (Text Augmentation Enhanced personality detection model)
  • gpt-3.5-turbo-0301 como generador y clasificador zero-shot/CoT
  • gpt-3.5-turbo-16k-0613 para la condición 3-shot
  • bert-base-uncased como codificador de publicaciones, aumentos y etiquetas
  • SVM
  • XGBoost
  • BiLSTM
  • BERTconcat
  • BERTmean
  • AttRCNN
  • SN+Attn
  • Transformer-MD
  • TrigNet
  • D-DGCN
  • D-DGCN+l0
  • TAEConcat
  • TAEWS

Instruments and metrics

  • Cuatro dicotomías MBTI: I/E, S/N, T/F y P/J
  • Macro-F1 por dimensión y media no ponderada de las cuatro
  • Pérdida InfoNCE post-aumento con negativos en minibatch
  • Etiquetas suaves mediante similitud coseno, softmax y alfa 4
  • Pérdida KL multieje para detección
  • Ablaciones de aumentos semántico, de sentimiento, lingüístico y de etiquetas
  • Comparación zero-shot, chain-of-thought y 3-shot de ChatGPT
  • Barrido de lambda en validación

Data used

  • Kaggle MBTI / PersonalityCafe: 8.675 usuarios, 45-50 publicaciones por usuario
  • PANDORA / Reddit: 9.067 usuarios, decenas a cientos de publicaciones y etiquetas MBTI extraídas de autodescripciones

Evidence and location

  • Official bibliographic record: AAAI article 29782: authors, abstract, DOI 10.1609/aaai.v38i16.29782, published 24 Mar 2024, volume 38 issue 16, pp. 18234-18242
  • Full audited source: .cache/editorial-sources/article-092/source.pdf; official AAAI-24 PDF; 9 pages; sha256 a6a3b33927c4a1d4fcb7050c76325eae7432da1ec6046d94dd68d6085c2adc96
  • Stated objective and contributions: Full text pp. 18234-18235, Abstract and Introduction
  • The LLM as auxiliary generator: Full text pp. 18235-18236, Method and Generating Knowledgeable Post Augmentations
  • Post analysis prompt: Full text p. 18236, displayed instruction under Generating Knowledgeable Post Augmentations
  • Contrastive encoder: Full text pp. 18236-18237, Equations 1-6
  • Label descriptions and representations: Full text p. 18237, Equation 7
  • Soft labels and loss: Full text pp. 18237-18238, Equations 8-12
  • Dataset composition: Full text p. 18238, Datasets and Table 1
  • 60/20/20 split: Full text p. 18238, Datasets paragraph
  • Imbalance per dimension: Full text p. 18238, Table 1
  • Baselines: Full text p. 18238, Baselines section
  • Hyperparameters and truncation: Full text p. 18239, Implementation Details
  • Complete results: Full text p. 18239, Table 2
  • Absolute and relative gains: Full text p. 18239, Table 2 and Overall Results; 72.07-71.35=0.72 and 63.05-62.01=1.04
  • Swapping of baseline names in the prose: Full text p. 18239, Table 2 versus Overall Results lines naming D-DGCN and D-DGCN+l0
  • Component ablations: Full text pp. 18239-18240, Table 3 and Ablation Study
  • Minimal contribution of labels: Full text p. 18239, Table 3: TAE 72.07 versus TAE without label 72.02
  • Comparison of concatenation, sum, and contrast: Full text pp. 18239-18240, Table 3 and second half of Ablation Study
  • Direct ChatGPT results: Full text p. 18240, Table 4: zero-shot 66.89, CoT 65.12, 3-shot 67.74
  • Few-shot contradiction: Full text p. 18240, Table 4 versus LLM Performance statement that both CoT and few-shot decreased
  • Snapshot change in 3-shot: Full text p. 18240, LLM Performance: 0301 switched to 16k-0613 because input exceeded limit
  • Lambda tuning: Full text p. 18240, Figure 4 and Effect of Trade-Off Parameter
  • Stated conclusions: Full text p. 18240, Conclusion
  • Absence of limitations and ethics analysis: Full 9-page paper structure: no limitations, ethics, privacy, fairness or broader-impact section
  • Absence of linked code: Full paper footnotes 1-5 and official AAAI record: dataset/platform/ChatGPT links only; no repository or software artifact link identified
  • Comprehensive reading and visual verification: All 9 pages rendered and inspected, including Figures 1-4, Tables 1-4, equations, footnotes, conclusion and references; checked 15 Jul 2026