Synthetic Personality Review

Extroversion or Introversion? Controlling The Personality of Your Large Language Models

Trait induction and control2024arXivApproved editorial review

Authors: Yanquan Chen, Zhen Wu, Junjie Guo, Shujian Huang, Xinyu Dai

Keywords: Computation and Language, Computers and Society

Source: Open primary source (opens in a new tab)

5
Authors
7
Findings
15
Limitations
5
Evidence

Editorial summary

English

This preprint compares five ways to induce personality-associated responses: continual pretraining, supervised fine-tuning (SFT), PPO-based RLHF, system or user prompts, and PISF, which applies a personality prompt after SFT. It tests Llama-2-Chat-13B, Qwen-Chat-7B and ChatGLM2-6B; a preliminary prompt analysis adds other Llama and Qwen sizes. Evaluation uses 200 English forced-choice items, 50 per MBTI dichotomy, presented through five prompt variants. A Falcon-7B-Instruct classifier converts free-form outputs into 1-5 scores. The study measures the increase in target-trait proportion, the percentage of traits crossing 50%, changes in non-target traits and, for four-letter types, whether all four traits are induced together. On average, the paper orders immediate efficacy as Prompt > SFT > RLHF > continual pretraining, although SFT often achieves a higher success rate. Prompts are more vulnerable to an opposite-personality instruction, while combining SFT and prompting produces some of the strongest figures. For example, user-prompt PISF on Qwen reports TIE 24.89, ISR 100, PIE 18.10 and PISR 100. However, PISF does not dominate every reverse-prompt comparison, and methods are not matched for data, compute or objective. The experiment demonstrates control of MBTI-questionnaire responses, not an internal or stable personality. The same trait concepts and descriptions recur in data generation, training, prompts and evaluation, and GPT-3.5 generates much of the SFT/RLHF material. There are no behavioral tasks, capability or safety checks, confidence intervals, significance tests or seed replications. The artifact audit confirms that the public evaluation set has 125 files and 25,000 rows, but every file repeats the same ordered 200 items. The extractor is validated by a random split of 4,194 repeated responses over those same 200 indices, with no independent test set. Repository code compiles syntactically, but 83 scripts contain 554 private absolute paths; trained checkpoints, raw generations and the complete analysis are missing. The study therefore provides useful evidence that prompting and tuning can shift response style under this instrument, but its tables are not reproducible end to end and it does not support strong psychological claims.

Español

Este preprint compara cinco formas de inducir respuestas asociadas a rasgos de personalidad: preentrenamiento continuo, ajuste supervisado (SFT), RLHF con PPO, prompts de sistema o usuario y PISF, que aplica un prompt de personalidad después de SFT. Prueba Llama-2-Chat-13B, Qwen-Chat-7B y ChatGLM2-6B; el análisis preliminar de prompts añade otros tamaños de Llama y Qwen. La evaluación consiste en 200 ítems ingleses de elección forzada, 50 por cada dicotomía MBTI, presentados con cinco variantes de prompt. Un clasificador Falcon-7B-Instruct convierte las respuestas libres en puntuaciones 1-5. El trabajo mide cuánto aumenta la proporción del rasgo objetivo, qué porcentaje de rasgos supera el 50%, cuánto cambian rasgos no objetivo y, para tipos de cuatro letras, si se inducen simultáneamente sus cuatro rasgos. En promedio, el paper ordena la eficacia inmediata como Prompt > SFT > RLHF > preentrenamiento continuo, aunque SFT suele lograr mayor tasa de éxito. Los prompts son más vulnerables a una instrucción inversa; combinar SFT y prompt produce algunas de las cifras más altas. Por ejemplo, PISF con prompt de usuario en Qwen reporta TIE 24,89, ISR 100, PIE 18,10 y PISR 100. Sin embargo, PISF no domina uniformemente la prueba inversa y las comparaciones no igualan datos, cómputo u objetivos de entrenamiento. Lo observado es control de respuestas a un cuestionario MBTI, no evidencia de una personalidad interna o estable. Los mismos conceptos y descripciones de rasgo aparecen en generación de datos, entrenamiento, prompt y evaluación; GPT-3.5 genera buena parte de los datos SFT/RLHF. No hay tareas conductuales, pruebas de capacidad o seguridad, intervalos, tests ni réplicas por semilla. La auditoría confirma que el dataset de evaluación público tiene 125 archivos y 25.000 filas, pero todos repiten la misma secuencia de 200 ítems. El clasificador se valida con un split aleatorio de 4.194 respuestas repetidas sobre esos mismos 200 índices, sin test independiente. El repositorio compila sintácticamente, pero 83 scripts contienen 554 rutas absolutas privadas; faltan checkpoints, generaciones crudas y el análisis completo. Por tanto, el estudio aporta evidencia útil de que prompts y ajuste pueden desplazar el estilo de respuesta bajo este instrumento, pero no permite reproducir de extremo a extremo sus tablas ni sostener afirmaciones psicológicas fuertes.

Research question

What efficacy, side effects, and resistance to an opposite personality instruction do continuous pretraining, SFT, RLHF, prompting, and the PISF combination have for shifting LLM responses toward target MBTI traits or types?

Method

Descriptive comparison of methods on Llama-2-Chat-13B, Qwen-Chat-7B, and ChatGLM2-6B. Each condition answers 200 MBTI-style items in five prompt variants. A Falcon extractor classifies each output on a 1-5 scale; afterwards efficacy, success rate, change in non-target traits, and joint success of the four traits are calculated. Robustness is estimated by requesting the opposite personality through Reverse Personality Prompt Induction.

Sample: The base evaluation is 200 question-option pairs, 50 per E/I, S/N, T/F, and J/P dichotomy. They are repeated under five wordings and 25 targets, without personality, eight traits, and 16 types, forming 125 files and 25,000 correlated rows. SFT uses 2,500 examples per trait and 10,000 per type; PPO publishes the same number of questions. The extractor uses 4,194 labeled responses, with 15-21 repetitions per test index.

Findings

  • The paper summarizes mean efficacy as Prompt > SFT > RLHF > continuous pretraining, but SFT usually surpasses prompting in success rate.
  • Prompt induction tends to increase with model size, with relevant variation by family and trait.
  • SFT resists an inverse instruction better than prompt-only, although RPPI still induces opposite traits in many conditions.
  • PISF produces some of the highest efficacies: Qwen with user prompt obtains TIE 24.89, ISR 100, PIE 18.10, and PISR 100.
  • In Llama, PISF with user prompt obtains TIE 24.76, ISR 100, PIE 16.19, and PISR 93.75.
  • The robustness of PISF is not uniform: under RPPI, Qwen PISF clearly reduces inverse induction compared to SFT, while the Llama comparison is mixed depending on the metric.
  • The artifact confirms the shape of the datasets and the main code, but does not publish the assets needed to reconstruct all tables.

Limitations

  • MBTI offers discrete targets, but the paper does not validate its dichotomies as a psychometric measure of models.
  • Responses to a simulated self-report questionnaire do not demonstrate internal, persistent, or behavioral personality.
  • Data, prompts, and evaluation share trait descriptions and vocabulary, so lexical compliance and role-play are plausible explanations.
  • GPT-3.5 generates much of SFT and RLHF; the effect may include imitation of its induced style.
  • The four methods use different amounts of data, targets, updated parameters, and compute; the ranking is not a controlled causal comparison.
  • The authors acknowledge that the capacity and brief training of the actor/reward model may harm RLHF.
  • No intervals, tests, variance by seed, multiple correction, or uncertainty of the comparisons are published.
  • The 25,000 evaluation records repeat the same 200 items and are not independent observations.
  • The extractor randomly divides repeated responses from the same 200 items between train and validation and has no independent test.
  • Fifteen groups with identical question, options, and prediction have contradictory labels in the extractor data.
  • General capacity, truthfulness, toxicity, safety, or degradation in tasks unrelated to personality are not evaluated.
  • RPPI may measure robustness of the target, but also rigidity or lower instruction following; a generic conflict control is missing.
  • Checkpoints, Falcon adapter, raw generations, complete results, and a reproducible figure pipeline are missing.
  • Eighty-three scripts depend on private paths and have runtime defects, in addition to having no tests, CI, tags, or release.
  • The study is an arXiv v1 preprint and only covers English and 2023 models.

What the study does not establish

  • It does not demonstrate that LLMs possess human personality or a stable latent MBTI type.
  • It does not demonstrate that the change in responses generalizes outside the questionnaire.
  • It does not establish that PISF is universally the most effective and robust method.
  • It does not prove that the ranking of methods is causal under equivalent resources.
  • It does not prove that RLHF is inherently worse for controlling personality.
  • It does not validate the 95.95% of the extractor on truly unseen items, models, or styles.
  • It does not separate personality from lexical imitation, role-play, or instruction following.
  • It does not prove that resisting RPPI is different from ignoring conflicting instructions.
  • It does not demonstrate that control preserves capacity, safety, truthfulness, or utility.
  • It does not generalize to other languages, instruments, model generations, or behavioral contexts.
  • It does not allow reproducing the tables end to end with the current public repository.

Traceability

Scope: Full text

Version: arXiv:2406.04583v1, submitted 2024-06-07; official GitHub code and public Google Drive datasets audited separately

Consulted source: https://arxiv.org/abs/2406.04583

Review: Codex 21-page visual, arXiv-v1, official-code, public-evaluation-and-training-artifact, item-grain, answer-extractor, runtime, MBTI-construct, statistics, reproducibility and claim-boundary audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • Llama-2-Chat-13B
  • Qwen-Chat-7B
  • ChatGLM2-6B
  • Falcon-7B-Instruct answer extractor
  • GPT-3.5-turbo-1106 data generator
  • Llama-2-Chat 7B and 70B in prompt-size analysis
  • Qwen-Chat 1.8B, 14B and 72B in prompt-size analysis

Instruments and metrics

  • Two-hundred-item English MBTI-style forced-choice questionnaire
  • Five generated prompt phrasings per item bank
  • Trait Induction Efficacy, TIE
  • Induction Success Rate, ISR
  • Trait Induction Side Effect, TSE
  • Personality Induction Efficacy, PIE
  • Personality Induction Success Rate, PISR
  • Reverse Personality Prompt Induction, RPPI
  • Falcon-7B-Instruct five-class answer extractor

Data used

  • Public evaluate_datasets.zip: 125 JSON files, 25,000 rows and one repeated 200-item bank
  • Public SFT payload: 180,000 rows across 24 files, 20,000 exact records unique
  • Public PPO prompt payload: 180,000 rows across 24 files, 10,000 exact questions unique
  • Public answer-extractor v1.json: 4,194 labeled responses over 200 item indices
  • Public reward-augmentation archive listed at 269,197,495 bytes
  • Public continual-pretraining archive listed at 415,694,799 bytes

Evidence and location

  • Metadata, version, subjects, and exact abstract: arXiv:2406.04583v1
  • Method, formulas, configuration, tables, discussion, limitations, and appendices: arXiv PDF, 21 pages, sha256 6d59fe71f09269e76a6ac6deb6cba36bd5c69ff6bcc0b9a5aaf996003fd168c2
  • Code, scripts, paths, dependencies, defects, and missing assets: github.com/yqchen0205/PISF commit 9729b91fa61b0096269774023d57bd8009d40f80
  • Shape, repetition, uniqueness, and quality of datasets: Public Google Drive evaluate_datasets.zip and train_datasets.zip; evaluated payload sha256 a9e732e68447c12feda2d16d79f07b53116a6427865a22291aa4e37f5d2401ce
  • Audit of construct, results, code, data, reproducibility, and claim limits: reports/verification/article-244-arxiv-pisf-personality-control-mbti-code-data-and-claim-audit.json