LLMs Simulate Big Five Personality Traits: Further Evidence

Evaluation and psychometric validity2024arXivApproved editorial review

Authors: Aleksandra Sorokovikova, Natalia Fedorova, Sharwin Rezagholi, Ivan P. Yamshchikov

Keywords: Computation and Language, Artificial Intelligence, Big Five personality traits, Large Language Models

Source: Open primary source (opens in a new tab)

4: Authors
5: Findings
5: Limitations
5: Evidence

Editorial summary

English

This case study compares how GPT-4, Llama 2 70B Chat, and Mixtral 8x7B Instruct answer the IPIP-NEO-120 questionnaire and tests whether their scores change under a small prompt variation or different generation temperatures. Each of the 120 items is presented separately with a 1–5 response scale. The authors use two headers, one requests a questionnaire response and the other adds “answer as if you were a person”, combine each header with three model-specific temperatures, and repeat each treatment five times. Within this protocol, the mean profiles differ substantially across models. GPT-4 scores high on Agreeableness and Conscientiousness, relatively high on Extraversion, and low on Neuroticism; Llama 2 stays closer to the scale midpoint, with higher Neuroticism and lower Agreeableness, Conscientiousness, and Openness; Mixtral shows low Neuroticism and high Agreeableness and Conscientiousness. Temperature does not yield a conclusive general effect: the authors see some responsiveness in GPT-4 but no consistent pattern across all three systems. The minor header change, however, alters some scores for every model. Llama 2 refuses two items under both prompt variants. The paper provides a descriptive comparison of output patterns and limited stability evidence across six conditions, not a full psychometric validation. It explicitly uses “personality” to mean textual properties analogous to human traits rather than model agency. No human observers evaluated the profiles, and the results do not establish that a score makes a model better suited to creative, emotional, or care-related tasks.

Español

Este estudio de caso compara la forma en que GPT-4, Llama 2 70B Chat y Mixtral 8x7B Instruct responden al cuestionario IPIP-NEO-120 y comprueba si las puntuaciones cambian ante una pequeña modificación del prompt o de la temperatura de generación. Cada uno de los 120 ítems se presenta por separado con una escala de 1 a 5. Los autores usan dos cabeceras: una pide contestar el test y otra añade «responde como si fueras una persona»; combinan cada cabecera con tres temperaturas adaptadas a cada modelo y repiten cada tratamiento cinco veces. Las puntuaciones medias difieren claramente entre modelos dentro de este protocolo. GPT-4 obtiene valores altos en Agreeableness y Conscientiousness, Extraversion relativamente alta y Neuroticism bajo; Llama 2 se acerca más al centro de la escala, con mayor Neuroticism y menores Agreeableness, Conscientiousness y Openness; Mixtral presenta Neuroticism bajo y puntuaciones altas en Agreeableness y Conscientiousness. La variación de temperatura no muestra un efecto concluyente: los autores observan cierta respuesta en GPT-4, pero no una pauta general para los tres sistemas. En cambio, el pequeño cambio de cabecera modifica algunas puntuaciones en todos ellos. Llama 2 rehúsa dos ítems en ambas variantes. El trabajo aporta una comparación descriptiva de patrones de salida y de su estabilidad bajo seis condiciones, no una validación psicométrica completa. Los propios autores aclaran que «personalidad» designa aquí propiedades del texto parecidas a rasgos humanos y no agencia del modelo. Tampoco se evaluó si personas reales perciben esos perfiles, ni se demostró que una puntuación haga a un modelo más adecuado para tareas creativas, emocionales o asistenciales.

Research question

What Big Five profiles do GPT-4, Llama 2, and Mixtral produce when responding to the IPIP-NEO-120, and to what extent do they remain stable or change across two prompt headers and three generation temperatures?

Method

Descriptive study with three LLMs. The 120 items of the IPIP-NEO-120 were administered individually with a numeric response from 1 to 5. Two prompt headers were crossed, with and without the instruction to respond as a person, with three temperatures: 1, 1.5, and 2 for GPT-4; 0.3, 0.7, and 1 for Llama 2 and Mixtral. Each of the six conditions per model was repeated five times. Means were calculated for the five domains and standard deviations across repetitions; the article presents no inferential tests, convergent validity, or human evaluation.

Sample: Three models, six treatments per model, and five repetitions per treatment. Each treatment comprises the 120 items of the IPIP-NEO-120; there are no human participants or external annotators.

Findings

GPT-4 maintained approximately Neuroticism 2.18 to 2.29, Extraversion 3.68 to 3.85, Openness 3.42 to 3.46, Agreeableness 4.17 to 4.26, and Conscientiousness 4.19 to 4.28 across the six conditions.
Llama 2 showed a profile closer to the midpoint, with Neuroticism 3.33 to 3.50 and comparatively lower scores in Openness, Agreeableness, and Conscientiousness; it refused two items under both headers.
Mixtral obtained Neuroticism 2.04 to 2.21 and high values for Agreeableness and Conscientiousness, approximately 4.50 to 4.58 and 4.38 to 4.58.
No conclusive and generalizable effect of temperature appeared; GPT-4 was the only model that the authors describe as possibly sensitive to that variation.
The addition of "respond as if you were a person" produced changes in some dimensions of the three models, showing sensitivity even to a brief modification of the prompt.

Limitations

It is an empirical case with only three models and partially specified versions; the authors indicate that they cannot estimate how much the observations generalize.
Five repetitions per condition and the absence of inferential tests limit the interpretation of small differences and of the effect of temperature.
The study provides no construct, convergent, or predictive validity, nor does it compare responses with judgments from human observers or task behavior.
The temperatures differ between GPT-4 and the open models, so they do not constitute an identical manipulation across architectures.
The second header was introduced to avoid safety refusals, so its changes may mix compliance effects with the simulation of traits.

What the study does not establish

It does not demonstrate that the models possess personality, agency, emotional states, or stable human dispositions.
It does not validate the IPIP-NEO-120 as a psychometric instrument for LLMs nor establish equivalence with human scores.
It does not demonstrate that the measured profiles predict user experience or performance in creative, care, or emotional tasks.
It does not allow the conclusion that temperature has no effect in general; only that this design did not produce a conclusive pattern.

Traceability

Scope: Full text

Version: arXiv:2402.01765v1

Consulted source: https://arxiv.org/pdf/2402.01765

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

GPT-4 (version not specified)
Llama-2-70b-chat-hf
Mixtral-8x7B-Instruct-v0.1

Instruments and metrics

IPIP-NEO-120
Five-point Likert response scale

Evidence and location

Non-anthropomorphic definition of personality and objective of the study: Paper, pp. 1 to 2, Introduction
Instrument and two prompt headers: Paper, p. 2, section 2, Adopting the Big5 for LLMs
Design of six treatments, temperatures, and five repetitions: Paper, pp. 2 to 4, section 3, Experiments
Scores by model and condition: Paper, p. 3, Table 2
Interpretation, limits, and absence of conclusion on temperature: Paper, p. 4, Discussion, Conclusion and Limitations