Synthetic Personality Review

Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects

Trait induction and control2025arXivApproved editorial review

Authors: Gunmay Handa, Zekun Wu, Adriano Koshiyama, Philip Treleaven

Keywords: Computation and Language, Large Language Models, Personality Manipulation, Big Five Traits, Machine Learning Evaluation

Source: Open primary source (opens in a new tab)

4
Authors
10
Findings
19
Limitations
18
Evidence

Editorial summary

English

This preprint compares three ways of inducing Big Five-associated expression in instruction-tuned models: exemplar-based in-context learning (ICL), LoRA adapters (PEFT), and activation-vector mechanistic steering (MS). Its main evidence consists of changes from a separate within-run baseline on an alignment task, seven MMLU subjects, 53 GAIA tasks, and BBQ's ambiguous subset. ICL yields strong alignment with small capability changes on both models; PEFT has the highest average alignment and also preserves MMLU/GAIA on LLaMA-3, while some Gemma traits deteriorate; MS, evaluated only on Gemma, aligns less strongly and can cause large MMLU losses and bias-score shifts up to ±29.7 points. The most defensible result is that effects depend strongly on method, trait, model, and benchmark, not that one universal hierarchy has been established. The paper provides configurations and a proposed 4,000-example contrastive dataset, but uses a single evaluation run per benchmark, reports no uncertainty intervals or all absolute baselines, applies MS to only one model, and does not yet release the promised code or data. Its claims about stability, representational depth, cognition, and interpretability therefore remain hypotheses rather than established conclusions.

Español

Este preprint compara tres formas de inducir expresiones asociadas a los Big Five en modelos instruct: aprendizaje en contexto con ejemplos (ICL), adaptadores LoRA (PEFT) y vectores de activación (mechanistic steering, MS). La evidencia central son cambios respecto a una línea base propia de cada ejecución en una tarea de alineación, siete materias de MMLU, 53 tareas de GAIA y el subconjunto ambiguo de BBQ. ICL obtiene alineación alta con cambios de capacidad pequeños en ambos modelos; PEFT logra la media de alineación más alta y en LLaMA-3 también conserva MMLU/GAIA, mientras que en Gemma algunos rasgos empeoran; MS, evaluado solo en Gemma, alinea menos y puede producir caídas grandes en MMLU y desplazamientos de sesgo de hasta ±29,7 puntos. El hallazgo más defendible es que el efecto depende mucho del método, el rasgo, el modelo y el benchmark, no que exista una jerarquía universal. El trabajo aporta configuraciones y una idea de dataset contrastivo de 4.000 ejemplos, pero la evaluación es de una sola ejecución por benchmark, no publica intervalos ni todos los baselines, aplica MS a un solo modelo y no libera todavía el código o dataset prometidos. Por ello, sus afirmaciones sobre estabilidad, profundidad representacional, cognición e interpretabilidad deben tratarse como hipótesis, no como conclusiones establecidas.

Research question

How do the expression of five traits, task performance, and demographic bias change when controlling declared personality through ICL, LoRA, or activation vectors, and what tradeoffs in efficacy, cost, and supposed stability does each technique present?

Method

Gemma-2-2B-IT and LLaMA-3-8B-Instruct are manipulated to separately express openness, conscientiousness, extraversion, agreeableness, and neuroticism. ICL shows examples of the five traits and requests one target; PEFT trains a trait-specific LoRA adapter (rank 64, alpha 16, dropout 0.1, two epochs, batch 2, learning rate 2e-4); MS computes in Gemma mean differences between activations of high/low responses in layers 5, 10, 15, and 20, calibrates layer and intensity, and composes two directions for openness. GPT-4.1 Mini generates the low-trait responses that duplicate the original dataset up to 4,000 examples and also extracts final responses from MMLU/GAIA as a judge. Expression is scored with the classifier of Jain et al.; capability is measured as accuracy delta against each method's own baseline on MMLU (7 subjects, 50 questions per subject) and GAIA Level 1 (N = 53), and bias as SAMB delta on ambiguous BBQ. A composite index combines variance, range, and absolute mean of the deltas to rank the so-called stability.

Sample: The declared evaluation uses 4,000 contrastive examples and 1,000 test samples for personality, 350 MMLU questions per run (7 subjects x 50), 53 GAIA tasks per run, and an ambiguous subset of BBQ whose N is not reported. The authors acknowledge a single run of each benchmark and partial subsets. MS is tested only on Gemma; ICL and PEFT, on Gemma and LLaMA-3.

Findings

  • ICL strongly induces several traits: in Gemma the alignment deltas range from +0.24 on openness to +0.97 on neuroticism; in LLaMA-3, from +0.17 on openness to +0.99 on neuroticism. Agreeableness is the weakest trait for ICL in both models (+0.50 and +0.32).
  • PEFT presents the highest mean alignment of the evaluated methods in each model, but not in all traits: openness remains low (+0.21 in Gemma and +0.06 in LLaMA-3), while agreeableness and neuroticism reach +0.95-1.00.
  • MS is only evaluated in Gemma and obtains lower alignment deltas (+0.10 to +0.64). The purification/mixing for openness chooses layer 15 and intensity 110, but the final improvement remains below ICL and PEFT.
  • In Gemma-MMLU, ICL loses approximately 0.06-0.08 of accuracy per trait. MS loses between 0.03 and 0.45, with the largest drops in agreeableness (-0.45) and conscientiousness (-0.43). PEFT ranges between +0.01 and -0.15.
  • In LLaMA-3, both ICL and PEFT show small MMLU deltas (-0.04 to +0.01) and PEFT even presents GAIA deltas from 0 to +0.04. This qualifies the general claim that PEFT maximizes alignment necessarily at the cost of performance.
  • In GAIA-Gemma, ICL improves between +0.06 and +0.09, while MS and PEFT drop between -0.04 and -0.13. With N = 53, changes of a few hundredths correspond to very few responses and are not accompanied by uncertainty.
  • BBQ shows large and non-monotonic effects: in Gemma-MS agreeableness and neuroticism change SAMB by -29.7, but conscientiousness by +22.1; in Gemma-PEFT openness increases +22.3. In LLaMA-3-PEFT all reported deltas are positive (+4.7 to +16.4).
  • The results do not support that a trait is intrinsically safe: the same trait can increase or reduce bias depending on method and model.
  • The proposed index ranks ICL (0.0366), PEFT (0.0363), and MS (0.0326), very small differences whose uncertainty is not estimated and whose scale is dominated by the chosen form to aggregate heterogeneous deltas.
  • Layer 15 is chosen for four of five Gemma vectors among the four inspected layers, but this selection procedure does not demonstrate that personality is causally consolidated in intermediate layers.

Limitations

  • The comparative matrix is incomplete: MS is not applied to LLaMA-3. The general conclusions across three methods therefore mix a comparison of two models for ICL/PEFT with one of a single model for steering.
  • Each benchmark is run a single time and with partial subsets. No seeds, confidence intervals, error bars, significance tests, or repetition are offered to allow separating real effects from sampling or generation variation.
  • The checklist claims that the stability analysis reports variance across runs, but the limitations section declares single benchmark evaluation runs and the metric uses dispersion across deltas of benchmarks/traits, not experimental replications.
  • The checklist marks open access to code and data, but its justification says they will be released after acceptance. The reviewed version provides no repository or artifacts to reproduce the tables.
  • The checklist refers to a supposed detail of GPU A100, times, and total number of runs in Appendix E; that appendix describes steering, but does not contain those compute data. Exact reviews of checkpoints, optimizer, seeds, and several partition details are also missing.
  • The methods do not receive equivalent information: MS uses synthetically generated high/low pairs, while ICL and PEFT use only high examples from the original dataset. This makes it difficult to attribute differences exclusively to the control mechanism.
  • The low-trait responses are generated by GPT-4.1 Mini without human validation or reported psychometric analysis. They may encode stereotypes of the generator or confuse low expression of a trait with its supposed opposite.
  • The same type of model GPT-4.1 Mini is involved in dataset generation and in the extraction of final responses. The accuracy of the judge or an audit of parsing errors is not reported.
  • Alignment depends on a classifier inherited from Jain et al. and on a sparsely described dedicated task. The table gives no N, absolute scores, classifier performance, intervals, or examples of false positives.
  • The scores measure traits expressed in text according to a classifier; they do not justify attributing human personality, internal states, cognition, or a psychological hierarchy to the model.
  • The absolute baselines per method are not published. The within-run deltas reduce one type of confounding, but make it impossible to know whether two methods start from the same performance or to compare final absolute utility.
  • SAMB changes in large directions, but the N of BBQ, the distribution by groups, and the uncertainty are not reported. Positive/negative signs are not sufficient to assess harm, equity, or generalization.
  • The stability metric combines MMLU/GAIA on an approximate accuracy scale with BBQ on a much larger scale. Although it normalizes variance and range through empirically chosen constants, the consistency factor uses mean_abs_deltas without explicit scale normalization, so BBQ may dominate the ranking.
  • The factors 10,000 and 1,000 of the stability metric are arbitrary and are not validated against temporal reliability, repetitions, prompt perturbations, or real production behavior.
  • Layers 5, 10, 15, and 20 are chosen arbitrarily, the best intensity is sought over the same criteria, and no independent confirmation set is offered; the conclusion about intermediate layers may be a selection artifact.
  • The evaluation is single-turn. It does not measure persistence after context changes, resistance to contradictory instructions, multi-turn interaction, trait combinations, or long-term coherence.
  • Academic benchmarks do not represent customer service conversations or agents. Utility for users, naturalness, deception, dependence, persuasion, or real harms are not evaluated either.
  • Only two relatively small open models and a Western Big Five framework are studied; there is insufficient linguistic, cultural, multimodal, demographic, or architectural variation to generalize.
  • Manipulation may amplify biases and facilitate non-transparent influence. The article discusses these risks, but does not implement or evaluate disclosure, consent, use limits, or technical safeguards.

What the study does not establish

  • It does not demonstrate that the models possess human personality, stable latent traits, or a cognitive architecture comparable to humans.
  • It does not demonstrate that ICL, PEFT, or MS is universally superior; the order depends on model, trait, benchmark, and metric.
  • It does not demonstrate that layer 15 is the causal locus where personality is represented or consolidated.
  • It does not demonstrate stability across runs, in long conversations, under prompt changes, or in production.
  • It does not demonstrate that the induction is safe, fair, culturally valid, or beneficial for real users.
  • It does not validate the composite stability index as a deployment criterion or as a psychometric measure.
  • It does not allow independently reproducing the tables with version v1, because the code and the contrastive dataset are promised for the future.

Traceability

Scope: Full text

Version: arXiv:2509.04794v1 (5 Sep 2025)

Consulted source: https://arxiv.org/pdf/2509.04794

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • Gemma-2-2B-IT
  • LLaMA-3-8B-Instruct
  • Azure OpenAI GPT-4.1 Mini (synthetic low-trait generation and answer extraction)

Instruments and metrics

  • Big Five trait framework
  • Personality classifier from Jain et al. (2025)
  • Dedicated trait-alignment task
  • MMLU AccuracyAvg
  • GAIA Level 1 accuracy
  • BBQ ambiguous-subset SAMB
  • Composite stability score proposed by the authors

Data used

  • Personality manipulation dataset from Jain et al. (2025)
  • GPT-4.1-Mini-generated high/low contrastive extension (4,000 examples; 1,000 test samples reported)
  • MMLU (7 selected subjects)
  • GAIA 2023 Level 1
  • BBQ ambiguous subset

Evidence and location

  • Declared contributions and scope of the study: arXiv v1, pp. 1-2, abstract, introduction and section 2
  • Contrastive generation and unequal use by method: arXiv v1, p. 2, Contrastive Dataset Generation
  • Design of ICL, PEFT, steering, and openness purification: arXiv v1, pp. 2-3, Methods
  • Joint table of alignment, MMLU, GAIA, and BBQ: arXiv v1, p. 3, Table 1
  • Interpretations of results and mechanistic claims: arXiv v1, p. 4, Results and Discussion
  • Acknowledged limitations, single run, and partial subsets: arXiv v1, p. 7, Appendix A
  • Prompt and ICL configuration: arXiv v1, pp. 10-11, Appendix C
  • PEFT hyperparameters and results: arXiv v1, pp. 11-12, Appendix D
  • Layers, intensities, and steering procedure: arXiv v1, pp. 12-13, Appendix E
  • MMLU and GAIA sizes and analysis by deltas: arXiv v1, pp. 14-15, Appendix F
  • Alignment deltas without reported uncertainty: arXiv v1, p. 15, Appendix G, Table 2
  • Complete results of MMLU, GAIA, and BBQ: arXiv v1, pp. 15-16, Appendix H, Tables 3-5
  • Classifier dependence and extended experimental limitations: arXiv v1, p. 17, Appendix J.1
  • Risks of manipulation, bias, and governance: arXiv v1, pp. 18-19, Appendix J.2
  • Benchmark definitions and absence of BBQ N: arXiv v1, pp. 19-20, Appendix K
  • Formula, constants, and stability ranking: arXiv v1, pp. 20-21, Appendix L and Table 6
  • Contradiction on open access to data and code: arXiv v1, p. 21, NeurIPS checklist item 5
  • Unsupported claims about significance and compute: arXiv v1, p. 22, NeurIPS checklist items 7-8