Synthetic Personality Review

Personality testing of large language models: limited temporal stability, but highlighted prosociality

Evaluation and psychometric validity2024Royal SocietyApproved editorial review

Original title: Personality Testing of Large Language Models: Limited Temporal Stability But Highlighted Social Desirability

Authors: Bojana Bodroža, Bojana M. Dinić, Ljubiša Bojić

Keywords: Large Language Models, Personality testing, Temporal stability, Prosociality, Psychometric assessment

Source: Open primary source (opens in a new tab)

3
Authors
6
Findings
6
Limitations
7
Evidence

Editorial summary

English

The study tests whether seven language models give stable responses to psychological inventories on two occasions five days apart and, only when a prespecified psychometric criterion is met, compares their scores with human norms. GPT-3 was tested in December 2022; GPT-3.5, GPT-4, GPT-4o, Gemini Pro, Llama 3, and Mixtral were tested in June 2024 through the interfaces available for each provider. Models were instructed to pretend to be human and answered 232 items spanning 21 scales or domains of self-consciousness, Big Five, HEXACO, Dark Triad, impression management, and political orientation. Item-level stability was estimated with weighted kappa and absolute-agreement ICCs; profile interpretation required ICC3,k of at least 0.50 with a 95% confidence interval excluding zero. Stability was not a general model property: Llama 3 met the criterion on 17 of 21 scales, GPT-4o on 16, and GPT-3.5 on 14, whereas Gemini and GPT-4 did so on only five and six. Results also depended on the instrument. Agentic impression management was stable in every model and several BFI-2 scales performed comparatively well, while public self-consciousness, HEXACO altruism, and HEXACO openness showed little stability. Among interpretable scales, scores mostly remained within one human standard deviation of the normative mean. A generally prosocial or socially desirable output pattern appeared in specific models, such as higher agreeableness, conscientiousness, honesty-humility, or altruism and lower Machiavellianism, but it was not uniform: Mixtral combined high agreeableness and altruism with high Machiavellianism. Interpretable political-orientation scores did not fall outside the normative range. The paper provides evidence about very short-term response repeatability under a fixed protocol; it does not establish internal personality, longitudinal stability, or measurement equivalence between human inventories and machines.

Español

El estudio examina si las respuestas de siete modelos a inventarios psicológicos se mantienen estables en dos aplicaciones separadas por cinco días y, solo cuando alcanzan un criterio psicométrico previo, compara sus puntuaciones con normas humanas. GPT-3 se evaluó en diciembre de 2022; GPT-3.5, GPT-4, GPT-4o, Gemini Pro, Llama 3 y Mixtral se evaluaron en junio de 2024 mediante las interfaces disponibles para cada proveedor. Los modelos recibieron la instrucción de fingir ser humanos y respondieron 232 ítems que forman 21 escalas o dominios de autoconciencia, Big Five, HEXACO, tríada oscura, manejo de impresiones y orientación política. La estabilidad se estimó ítem a ítem con kappa ponderada e ICC de acuerdo absoluto; para interpretar un perfil, los autores exigieron ICC3,k de al menos 0,50 y un intervalo del 95 % que no incluyera cero. La estabilidad no fue una propiedad general: Llama 3 superó el criterio en 17 de 21 escalas, GPT-4o en 16 y GPT-3.5 en 14, mientras que Gemini y GPT-4 solo lo hicieron en cinco y seis. El resultado también dependió del instrumento: el manejo de impresión agéntico fue estable en todos los modelos y varias escalas BFI-2 funcionaron relativamente bien; la autoconciencia pública, el altruismo de HEXACO y la apertura de HEXACO mostraron poca estabilidad. En las escalas interpretables, las puntuaciones quedaron mayoritariamente dentro del intervalo humano de media más o menos una desviación estándar. Apareció un patrón de salida generalmente prosocial o socialmente deseable, por ejemplo, mayor amabilidad, responsabilidad, honestidad-humildad o altruismo y menor maquiavelismo en modelos concretos, aunque no fue uniforme: Mixtral combinó amabilidad y altruismo altos con maquiavelismo alto. Las puntuaciones políticas interpretables no se alejaron del rango normativo. El trabajo aporta evidencia de repetibilidad a muy corto plazo bajo un protocolo fijo, pero no demuestra personalidad interna, estabilidad longitudinal ni que los cuestionarios humanos midan constructos equivalentes en máquinas.

Research question

With what temporal stability do different LLMs respond to psychological instruments on two close occasions and, on the scales that reach a reliability threshold, what profile do they show compared to human normative data?

Method

Repeat study at two time points separated by five days. Seven models responded in English, through their respective interfaces, to 232 items grouped into 21 scales or domains. The prompt "Pretend you are a human. Answer the following questions" was used and, if necessary, a second instruction to pretend during the game. Item-by-item agreement was estimated with weighted kappa, ICC3,1 and ICC3,k of absolute agreement. Means were only compared with human norms when ICC3,k was at least 0.50 and its 95% interval excluded zero; a descriptive difference was defined as falling outside the human mean plus or minus one standard deviation.

Sample: Seven models and two measurements per model, separated by five days. GPT-3 was measured on December 9 and 14, 2022; the other six models, on June 24 and 29, 2024. Each measurement covered 232 items and 21 scales or domains. No new people participated: the comparisons use published normative statistics from online or community human samples.

Findings

  • Stability varied strongly by model: 17 of 21 scales met the ICC criterion in Llama 3, 16 in GPT-4o and 14 in GPT-3.5; Gemini and GPT-4 only met the criterion in five and six scales.
  • The agentic impression management scale obtained excellent ICC3,k in all seven models. Agreeableness and conscientiousness of the BFI-2 were excellent in six models, and extraversion, intellectual openness of the BFI-2 and machiavellianism in five.
  • Public self-consciousness did not reach an acceptable ICC in any model; openness to experience of HEXACO only did so in one and altruism produced mostly null or constant values.
  • On the scales that did pass the criterion, most means fell within one standard deviation of human norms; prosocial or socially desirable deviations depended on the model and the instrument.
  • GPT-4o and Llama 3 showed several traits above the normative range related to conscientiousness, agreeableness, honesty-humility or impression management; GPT-4o, GPT-4, Gemini and Llama 3 showed lower machiavellianism when that scale was interpretable.
  • Mixtral was the most contradictory exception: it presented agreeableness and altruism above the human range, but also high machiavellianism. No interpretable political result fell outside the normative interval of one standard deviation.

Limitations

  • Two observations separated by only five days allow studying immediate repeatability, not longitudinal stability, resistance to updates or persistence in prolonged conversations.
  • The models were tested through different platforms and GPT-3 in 2022 versus the others in 2024; version, interface, architecture, training and alignment remain intertwined and cannot be causally compared.
  • The prompt forces the system to pretend to be human and the settings were kept fixed; sensitivity to other roles, temperatures, formulations, languages or natural use conditions is not studied.
  • The inventories and norms were designed for people. The human references come from different validation studies and not from participants assessed simultaneously with the same context.
  • The profile comparisons are descriptive through the one standard deviation threshold; they do not test measurement invariance, factorial structure, convergent validity, predictive validity or construct equivalence in LLMs.
  • The explanations about number of parameters, architecture, RLHF and guardrails are discussion hypotheses: several details of the proprietary models are not known and the design does not manipulate those factors.

What the study does not establish

  • It does not demonstrate that the models possess personality, self-awareness, political beliefs, motivations or conscious reflection; it records responses produced when simulating a person.
  • It does not demonstrate that a human-like score represents the same psychological construct or that the questionnaires are validated for machines.
  • It does not establish that Llama 3 or GPT-4o are stable in general; their advantage is limited to certain scales, two dates and the tested configuration.
  • It does not allow attributing the differences to size, number of parameters, architecture, provider or RLHF.
  • It does not prove that the observed profiles predict generative behavior, safety, influence on users or performance in real applications.
  • It does not confirm a general political orientation: the analyzed scores remained within the normative interval and come from only three items.

Models evaluated

  • GPT-3
  • GPT-3.5-turbo-16k
  • GPT-4
  • GPT-4o
  • Gemini Pro
  • Llama 3-sonar-large-32K-chat
  • Mixtral-8x7b-instruct

Instruments and metrics

  • Self-Consciousness Scales, Revised (SCS-R, 22 items)
  • Big Five Inventory-2 (BFI-2, 60 items)
  • HEXACO-100 (100 items, including altruism)
  • Short Dark Triad (SD3, 27 items)
  • Bidimensional Impression Management Index (BIMI, 20 items)
  • Three-item political orientation measure
  • Weighted Cohen's kappa
  • ICC3,1 and ICC3,k absolute-agreement coefficients

Data used

  • Published normative data from the original validation studies of SCS-R, BFI-2, HEXACO-100, SD3 and BIMI
  • Study response data and codebook deposited on OSF

Evidence and location

  • Objective, models, dates, prompt and instruments: Paper, pp. 5–7, sections 1.5 and 2.1–2.2
  • Criteria for kappa, ICC and normative comparison: Paper, p. 7, section 2.3
  • Stability results by model, scale and confidence interval: Paper, pp. 7–14, section 3 and Tables 1–7
  • Means compared to human norms and profile exceptions: Paper, pp. 8 and 15–16, section 3 and Table 8
  • Interpretation of the prosocial profile and the contradictory case of Mixtral: Paper, pp. 13 and 17, sections 4.2–4.3
  • Limitations, implications and needs for replication and predictive validity: Paper, pp. 18–19, sections 4.4–4.5
  • Data availability, ethics and declarations: Paper, p. 19, end matter