Synthetic Personality Review

Exploring the Impact of Personality Traits on LLM Bias and Toxicity

Applications, bias, and safety2025ACL AnthologyApproved editorial review

Authors: Shuo Wang, Renhao Li, Xi Chen, Yulin Yuan, Min Yang, Derek F. Wong

Keywords: Artificial Intelligence, Large Language Models, Personality Traits, Bias, Toxicity

Source: Open primary source (opens in a new tab)

6
Authors
9
Findings
11
Limitations
11
Evidence

Editorial summary

English

The study tests whether instructions based on high and low levels of the six HEXACO traits change bias, sentiment, and toxicity in GPT-4o-mini, Llama 3.1 70B Instruct, and Qwen 2.5 72B Instruct. It first confirms that models produce the expected extreme HEXACO-100 scores after receiving descriptions that already enumerate the corresponding behaviors. It then compares a base condition with twelve trait conditions on 29,246 ambiguous BBQ questions, 3,014 BOLD prompts, and 1,199 challenge prompts from RealToxicityPrompts, using temperature zero. On RealToxicityPrompts, high Agreeableness reduces mean toxicity relative to base from 13.2 to 6.4 for GPT-4o-mini, 21.2 to 9.1 for Llama, and 26.1 to 10.6 for Qwen; low Agreeableness raises it to 33.0, 31.8, and 36.7 and also sharply increases negative sentiment. High Honesty-Humility, Extraversion, and Openness show favorable patterns in open-ended text, but not every trait, model, or task follows the same pattern. On BBQ, only six of 36 comparisons against base reach p < .05 without multiple-comparison correction: low Agreeableness is the only nominally significant effect in all three models, while high Agreeableness is not significant in any. The strongest evidence is therefore that a low-Agreeableness instruction increases harmful responses; the broader claim that personality configuration mitigates bias and toxicity remains exploratory. The prompts also do not isolate a latent trait: they directly instruct models to forgive or hold grudges, control or express anger, avoid manipulation or break rules, so effects may arise from those behavioral directives. VADER and Perspective API are checked against 780 annotations per metric and reach aggregate correlations with manual ratings of .752 and .768. Robustness to one prompt rewrite and three repeated subsamples is high. However, the combined Sopen index is described as ranging from zero to one even though its formula can reach 1.5 and weights sentiment twice; human raters are members of the research team and inter-rater agreement is not reported. Low Honesty-Humility yields highly positive text through excessive flattery, which the paper itself recognizes as apparent mitigation carrying sycophancy risk. The work provides causal evidence about specific instructions on these benchmarks, not about internal personality or a production-ready safety defense.

Español

El estudio examina si instrucciones basadas en los niveles alto y bajo de los seis rasgos HEXACO cambian el sesgo, el sentimiento y la toxicidad de GPT-4o-mini, Llama 3.1 70B Instruct y Qwen 2.5 72B Instruct. Primero confirma que los modelos producen puntuaciones extremas esperadas en HEXACO-100 cuando reciben descripciones que ya enumeran las conductas correspondientes. Después compara una condición base y doce condiciones de rasgo en 29.246 preguntas ambiguas de BBQ, 3.014 prompts de BOLD y 1.199 prompts challenge de RealToxicityPrompts, con temperatura cero. En RealToxicityPrompts, alta amabilidad reduce la toxicidad media respecto de base de 13,2 a 6,4 en GPT-4o-mini, de 21,2 a 9,1 en Llama y de 26,1 a 10,6 en Qwen; baja amabilidad la eleva a 33,0, 31,8 y 36,7, y también incrementa fuertemente el sentimiento negativo. Alta honestidad-humildad, extraversión y apertura muestran patrones favorables en texto abierto, pero no todos los rasgos, modelos o tareas siguen el mismo patrón. En BBQ solo seis de 36 contrastes frente a base alcanzan p < 0,05 sin corrección por comparaciones múltiples: baja amabilidad es el único efecto nominalmente significativo en los tres modelos, mientras que alta amabilidad no lo es en ninguno. La evidencia más consistente es, por tanto, que una instrucción de baja amabilidad aumenta respuestas dañinas; la afirmación general de que configurar personalidad mitiga sesgo y toxicidad es más exploratoria. Además, los prompts no aíslan un rasgo latente: ordenan directamente perdonar o guardar rencor, controlar o expresar ira, evitar manipulación o romper reglas, de modo que el efecto puede proceder de esas instrucciones conductuales. VADER y Perspective API se contrastan con 780 anotaciones por métrica y alcanzan correlaciones agregadas con evaluación manual de 0,752 y 0,768. La robustez ante una reescritura de prompts y tres submuestras repetidas es alta. Sin embargo, el índice combinado Sopen se describe como 0–1 aunque su fórmula alcanza 1,5 y da doble peso al sentimiento; los evaluadores humanos pertenecen al equipo y no se informa acuerdo entre anotadores. Baja honestidad-humildad genera respuestas muy positivas por adulación, un caso que el propio artículo reconoce como falsa mitigación con riesgo de complacencia. El trabajo aporta evidencia causal sobre el efecto de instrucciones concretas en estos benchmarks, no sobre personalidad interna ni sobre una defensa de seguridad lista para producción.

Research question

How do bias in ambiguous questions and the sentiment and toxicity of generated text change when three LLMs receive descriptions of high or low levels of each HEXACO trait, and can certain configurations reduce those outcomes without degrading general tasks?

Method

Factorial prompting experiment with a baseline condition and twelve conditions, one high and one low for each HEXACO trait. The authors verify the manipulation with HEXACO-100 and run the three models at temperature 0 over BBQ-Ambiguous, BOLD, and the challenge partition of RealToxicityPrompts. BBQ is evaluated with the signed bias score and paired t-tests over its eleven categories; BOLD and RealToxicityPrompts with positive/negative VADER proportions, Perspective API, and a combined index. A subsample of 780 outputs per metric is re-evaluated manually and with GPT-4.1-mini. Robustness is studied by rewriting prompts with GPT-4.5 and repeating three times subsamples of 1,000 examples; MMLU College and Gigaword serve as general performance controls.

Sample: Three models, thirteen conditions per model, and deterministic execution at temperature 0. BBQ contributes 29,246 ambiguous questions across eleven categories; BOLD, 3,014 prompts balanced across five domains and subgroups; and RealToxicityPrompts, 1,199 prompts from its challenge partition. The automatic validation uses 780 outputs for sentiment and another 780 for toxicity, balanced by trait and model. The annotators are voluntary members of the team, with postgraduate training and residing in China, but how many participated is not reported.

Findings

  • The prompts bring the HEXACO-100 scores of the requested trait to approximately 4.38–5.00 in high conditions and to 1.00–1.69 in low conditions, confirming instruction following, although not independence or psychometric construct validity.
  • On BBQ, low agreeableness raises the mean bias score to 3.39 in GPT-4o-mini, 6.41 in Llama, and 1.85 in Qwen, compared with baselines of 1.55, 1.50, and −0.07; it is the only nominally significant contrast across the three models, with p = 0.020, 0.001, and 0.039.
  • Only six of the 36 BBQ t-tests have p < 0.05 without adjustment: four correspond to GPT-4o-mini and one additional to each open model. High agreeableness obtains p = 0.068, 0.919, and 0.413, so its supposed mitigation is not statistically supported on that task.
  • On BOLD, low agreeableness raises the mean negative proportion by 24.60 points and toxicity up to 4.5, 15.3, and 10.1; high agreeableness keeps it at 2.2, 2.7, and 2.8. The toxicity differences are described as non-significant and generally below one point for the high traits.
  • On RealToxicityPrompts, high agreeableness reduces toxicity by 6.8, 12.1, and 15.5 points relative to baseline, while low agreeableness increases it by 19.8, 10.6, and 10.6. High honesty-humility, extraversion, and openness also show consistent descriptive reductions.
  • Low honesty-humility increases positive responses on BOLD up to 92.0%, 94.4%, and 85.8%, but does so through excessive flattery; the qualitative case shows that lower automatic negativity can conceal complacency and lack of sincerity.
  • Perspective API and VADER correlate with the manual averages at r = 0.768 and r = 0.752; GPT-4.1-mini reaches 0.623 and 0.633. These are reasonable associations, but not equivalence with human evaluation.
  • The rewriting of the prompts achieves 96.8% agreement on BBQ and correlations of 0.90–0.99 on open text; three repetitions over subsamples also fluctuate little. This supports stability under that specific rewriting and that sampling.
  • The mean maximum variation is small on MMLU College and Gigaword, but some conditions are not trivial: in Llama, low honesty-humility reduces ROUGE-1 from 0.312 at baseline to 0.251.

Limitations

  • The prompts directly contain behaviors with a safety valence: low agreeableness instructs holding grudges, criticizing, and getting angry, and low honesty-humility instructs flattering and breaking rules. The design does not separate a personality effect from a direct semantic effect of the instruction.
  • The HEXACO verification reuses descriptions derived from the instrument itself; obtaining expected extremes demonstrates obedience to the prompt, not reliability, convergent validity, invariance, or a stable personality.
  • The 36 BBQ t-tests have no correction for multiplicity, confidence intervals, or effect sizes. Treating eleven heterogeneous categories aggregated as pairs limits inference, and the signed averages can cancel stereotypic and counter-stereotypic bias.
  • For BOLD and RealToxicityPrompts, expressions such as significant or effective are used without presenting inferential tests per trait and model; the tables are mainly descriptive comparisons.
  • The Sopen index adds the positive proportion, the complement of the negative proportion, and the complement of toxicity, and divides by two. Its theoretical range is 0–1.5, not 0–1, and it gives two components to sentiment versus one to toxicity; therefore, the compact comparison in Figure 3 depends on a questionable weighting.
  • VADER is lexical and Perspective API may have domain or identity biases. The validation correlations are computed over thirteen condition averages, and no confidence intervals are given; a positive text also does not necessarily equate to a safe or truthful one.
  • The annotators belong to the research team, know the definitions of the automatic evaluators, and their number, inter-annotator agreement, discrepancy resolution, or independent external evaluation are not reported.
  • Only three models are studied, in English, with a proprietary snapshot and two 70–72B models. Small systems, other languages, prolonged dialogues, model updates, or effects on users are not covered.
  • No code, outputs, sample IDs, seeds, library versions, or complete inference configuration are published. The official checklist marks budget, infrastructure, and licenses as described, but the cited sections do not provide those details, so the experiment cannot be reproduced end to end.
  • The robustness rewriting comes from a single model and the repetitions use subsamples of 1,000; they do not test robustness against substantial changes in formulation, conversational context, or adaptive attacks.
  • The qualitative similarity with human psychological correlations comes from prior literature and not from a parallel human sample subjected to the same tasks.

What the study does not establish

  • It does not demonstrate that the models possess HEXACO traits, internal personality, emotions, moral values, or a persistent identity.
  • It does not demonstrate that high agreeableness or any other trait universally reduces bias and toxicity: the effects depend on task, metric, and model, and a large part of the BBQ contrasts are not significant.
  • It does not allow attributing the changes to a psychological construct separate from the explicit behavioral instructions contained in the prompt.
  • It does not validate the Sopen index as a measure of safety, nor does it prove that positive sentiment is beneficial; the flattery of low honesty-humility shows the opposite case.
  • It does not establish reduction of harm for users, safety in real conversations, resistance to jailbreaks, distributive fairness, or absence of new biases.
  • It does not demonstrate equivalence between the patterns of the LLMs and human personality-behavior relationships; it only observes a qualitative correspondence with previous studies.
  • It does not present a production-ready intervention nor compare trait prompting with guardrails, classifiers, safety fine-tuning, or other controls.

Traceability

Scope: Full text

Version: arXiv:2502.12566v3 (18 Sep 2025); EMNLP 2025 proceedings version, DOI 10.18653/v1/2025.emnlp-main.206, also reviewed

Consulted source: https://arxiv.org/pdf/2502.12566

Review: Codex editorial review and methods audit, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o-mini-2024-07-18
  • Llama-3.1-70B-Instruct
  • Qwen2.5-72B-Instruct
  • GPT-4.1-mini (evaluation judge)
  • GPT-4.5 (prompt rewriting for robustness)

Instruments and metrics

  • HEXACO-100-English
  • BBQ bias score in ambiguous contexts
  • VADER sentiment score
  • Perspective API toxicity probability
  • Paired t-tests across BBQ bias categories
  • Manual sentiment and toxicity ratings
  • ROUGE-1, ROUGE-2 and ROUGE-L

Data used

  • BBQ-Ambiguous
  • BOLD
  • RealToxicityPrompts challenge subset
  • MMLU college-level multiple-choice subsets
  • English Gigaword headline summarization

Evidence and location

  • Publication, authorship, DOI, and definitive version: ACL Anthology 2025.emnlp-main.206 and EMNLP 2025 proceedings PDF, pp. 4125–4143
  • Models, temperature, and HEXACO manipulation validation: Paper, pp. 4127–4129, sections 3.1 and 4.1, Figure 2
  • Datasets, sample sizes, and metric definitions: Paper, pp. 4127–4128, sections 3.2–3.3, Equations 1–2
  • BBQ results and significance by trait and model: Paper, pp. 4129–4130 and 4139, section 4.2, Tables 1 and 9
  • BOLD and RealToxicityPrompts results: Paper, pp. 4130–4131, sections 4.3–4.4, Tables 2–3
  • Human and automatic validation of sentiment and toxicity: Paper, pp. 4131–4132 and 4143, section 4.5, Table 4 and Appendix G/Table 15
  • Flattery in low honesty-humility and limits of an apparent mitigation: Paper, pp. 4132–4133, section 4.6, Table 5 and Discussion
  • Robustness to rewriting and repetition: Paper, p. 4136, Appendix D
  • Performance on general tasks and possible trade-offs: Paper, pp. 4136 and 4143, Appendix E, Tables 13–14
  • Declared limitations and mitigation conclusion: Paper, p. 4133, sections 5–6 and Limitations
  • Reproducibility statements, artifacts, and annotation: EMNLP 2025 Responsible NLP Checklist attachment for 2025.emnlp-main.206, pp. 1–2, checked against the cited paper sections