Synthetic Personality Review

Evaluating and Inducing Personality in Pre-trained Language Models

Trait induction and control2023NeurIPS ProceedingsApproved editorial review

Authors: Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, Yixin Zhu

Keywords: Language Models, Personality Assessment, Machine Behavior, Psychometric Studies, Big Five, Machine Personality Inventory (MPI)

Source: Open primary source (opens in a new tab)

6
Authors
6
Findings
6
Limitations
7
Evidence

Editorial summary

English

The paper introduces the Machine Personality Inventory (MPI), which converts Big Five items from IPIP/IPIP-NEO and BFI-S into multiple-choice questions for describing model response patterns, and Personality Prompting (P²), a prompt chain intended to increase or decrease a target trait. Six systems are compared: BART-large-MNLI, GPT-Neo 2.7B, GPT-NeoX 20B, T0++ 11B, Alpaca 7B, and text-davinci-003, labelled GPT-3.5 175B. Each model answers 120-item and roughly 1,000-item versions at temperature zero using format-specific templates. For every trait, the study computes the mean OCEAN score and the standard deviation across items; it treats this deviation as “internal consistency” and descriptively compares it with 619,150 human IPIP-NEO-120 responses. Alpaca and GPT-3.5 yield dispersions and means closer to the human references than the three base models, but the study does not estimate alpha, omega, test-retest reliability, factor structure, or measurement equivalence. P² is tested primarily on GPT-3.5. It starts from a simple instruction, adds trait words from psychological literature, and asks the same model to write a more detailed portrait. On MPI it raises the target score relative to a neutral prompt, but it also shifts non-target traits and is not uniformly better than either baseline. External validation uses five vignettes and comparative judgments from 100 Prolific participants. P² receives high recognition rates for positive and negative openness and extraversion, but negative conscientiousness reaches only 0.45, below nominal chance. The appendix shows weaker trait separation on Alpaca and meaningful sensitivity to GPT-3.5 prompt paraphrases. The evidence supports prompt-controlled questionnaire and text patterns in particular model snapshots; it does not validate an internal, stable, or human-equivalent personality.

Español

El artículo propone el Machine Personality Inventory (MPI), una conversión de ítems Big Five de IPIP/IPIP-NEO y BFI-S en preguntas de opción múltiple para describir patrones de respuesta de modelos, y Personality Prompting (P²), una cadena de prompts para aumentar o reducir un rasgo objetivo. Se comparan seis sistemas: BART-large-MNLI, GPT-Neo 2.7B, GPT-NeoX 20B, T0++ 11B, Alpaca 7B y text-davinci-003, etiquetado como GPT-3.5 175B. Cada modelo responde versiones de 120 y aproximadamente 1.000 ítems a temperatura cero con plantillas adaptadas a su formato. Para cada rasgo se calcula la media OCEAN y la desviación estándar entre ítems; esta desviación se interpreta como «consistencia interna» y se contrasta descriptivamente con 619.150 respuestas humanas del IPIP-NEO-120. Alpaca y GPT-3.5 producen dispersiones y medias más próximas a las referencias humanas que los tres modelos base, aunque el estudio no aplica alfa, omega, test-retest, análisis factorial ni equivalencia de medida. P² se prueba principalmente sobre GPT-3.5: parte de una instrucción simple, incorpora palabras descriptivas de la literatura psicológica y pide al propio modelo redactar un retrato más detallado. En MPI eleva el rasgo objetivo respecto al prompt neutral, pero también altera rasgos no objetivo y no supera uniformemente a los dos baselines. La validación externa usa cinco viñetas y juicios comparativos de 100 participantes de Prolific. P² obtiene altas tasas de reconocimiento para apertura y extraversión positivas y negativas, pero la inducción de baja responsabilidad solo alcanza 0,45, por debajo del azar nominal. El apéndice muestra que Alpaca separa peor los rasgos y que para GPT-3.5 las paráfrasis del prompt cambian los resultados. La evidencia respalda que determinados prompts controlan patrones de cuestionario y texto en una instantánea concreta; no valida una personalidad interna, estable o equivalente a la humana.

Research question

Can Big Five inventories converted to multiple choice quantitatively describe LLM response patterns and can a prompting chain recognizably induce positive or negative levels of each trait?

Method

Two-stage study. MPI transforms IPIP/IPIP-NEO and BFI-S items into five-option questions, with versions of 120 and approximately 1,000 items; six models respond at temperature zero using specific templates and OCEAN means and standard deviations are compared with human statistics. Then, P² generates a portrait from a trait instruction, psychological descriptors, and self-prompting; it is compared in GPT-3.5 with naive prompting and three-word search. Transfer is evaluated with five vignettes, positive, neutral, and negative versions, and binary judgments from human participants. The appendix adds P² in Alpaca and five paraphrases generated by GPT-4.

Sample: Six models in the MPI evaluation. The main induction is run on text-davinci-003 and the appendix adds Alpaca 7B; GPT-4 only produces five paraphrases. The human evaluation gathers 100 participants from Prolific (67 women, mean age 42.8 years), divided into 50 response sets for P² and 50 for the word baseline.

Findings

  • In MPI-120, Alpaca and GPT-3.5 show standard deviations closer to human references across all five traits; BART, GPT-Neo, and GPT-NeoX show greater dispersion, while T0++ produces very extreme and consistent profiles.
  • In GPT-3.5, P² raises openness from 3.50 to 4.54, conscientiousness from 3.83 to 4.92, extraversion from 4.00 to 4.58, agreeableness from 3.58 to 5.00, and neuroticism from 3.12 to 3.75 compared to the neutral.
  • The induction is not isolated to the target factor: several prompts simultaneously increase conscientiousness and agreeableness or reduce neuroticism, so the control of dimensions is not fully disentangled.
  • P² clearly outperforms the baselines in openness and agreeableness, ties the extraversion score, falls slightly below the best baseline in conscientiousness, and produces higher neuroticism; therefore, its advantage is not uniform.
  • In the vignettes, the positive/negative P² recognition rates are 0.77/0.90 for openness, 0.73/0.45 for conscientiousness, 0.90/0.92 for extraversion, 0.88/0.84 for agreeableness, and 0.68/0.74 for neuroticism.
  • Alpaca responds worse to P² and mixes the dimensions more. The five paraphrases change the scores: they tend to match or worsen openness, conscientiousness, extraversion, and agreeableness, while some increase neuroticism.

Limitations

  • The standard deviation across items is called internal consistency, but it is not a psychometric reliability coefficient; alpha, omega, test-retest, factor structure, convergent validity, and uncertainty intervals are not calculated.
  • The templates are manually adapted to each architecture, so that capacity, format adherence, size, and alignment are confounded with the presumed trait.
  • The human comparison uses averages from a historical baseline and does not demonstrate invariance or equivalence between people's responses and models' tokens.
  • The main induction is limited to text-davinci-003, a closed snapshot, and P² uses the same model's knowledge to draft the prompt; testing in Alpaca shows considerably weaker generalization.
  • The external validation contains a single vignette per trait and relative judgments on few responses. The text reports 15 responses but 10 binary questions and does not report significance, intervals, or inter-rater agreement.
  • The study is conducted in English, at temperature zero, and without multi-turn dialogues, temporal repetition, operational tasks, or measurement of real effects on users.

What the study does not establish

  • It does not demonstrate that an LLM has personality, consciousness, emotions, identity, or internal traits; the data are responses conditioned on items and prompts.
  • It does not validate MPI as a psychometric instrument equivalent to IPIP-NEO in humans, nor does it make its scores directly comparable.
  • It does not prove that personality is an emergent capability or that size or alignment cause the observed patterns.
  • It does not demonstrate that P² is a universal or superior controller across all traits, models, and formulations.
  • It does not prove longitudinal stability, transfer to real behavior, safety, clinical utility, or absence of harm or manipulation.
  • It does not allow inferring that vignette responses predict the model's decisions outside those five scenarios.

Models evaluated

  • BART-large-MNLI
  • GPT-Neo 2.7B
  • GPT-NeoX 20B
  • T0++ 11B
  • Alpaca 7B
  • GPT-3.5 / text-davinci-003 (labelled 175B by the paper)
  • GPT-4 (prompt paraphrasing only)

Instruments and metrics

  • Machine Personality Inventory, 120-item version
  • Machine Personality Inventory, approximately 1,000-item version
  • Big Five OCEAN mean scores and cross-item standard deviations
  • Five personality-relevant vignette scenarios
  • Binary comparative human evaluation on Prolific

Data used

  • International Personality Item Pool and IPIP-NEO derivatives
  • BFI-S items
  • 619,150 human IPIP-NEO-120 responses reported by Johnson (2014)
  • Five vignette contexts adapted from Kwantes et al. (2016)
  • Model-generated vignette essays

Evidence and location

  • Definition of MPI, item sources, and OCEAN protocol: NeurIPS paper, pp. 4–5, section 3.1 and Table 1
  • Models, configuration, and comparison with 619,150 human responses: NeurIPS paper, pp. 5–6, section 3.2 and Table 2
  • Construction of P² and comparison with baselines: NeurIPS paper, pp. 7–8, sections 4.1–4.2 and Tables 3–4
  • Design and success rates of the human evaluation: NeurIPS paper, pp. 9–10, section 4.3 and Tables 5–6
  • Conceptual scope, risks, and recognized limitations: NeurIPS paper, p. 10 and appendix p. 14, section 5 and Appendix A
  • MPI-1K results, model details, and templates: NeurIPS paper, appendix pp. 15–17, Tables A1–A2 and sections B.1–B.4
  • Alpaca, paraphrase sensitivity, and full vignette texts: NeurIPS paper, appendix pp. 17–22, Tables A3–A6 and sections C.2–C.4