This EMNLP 2024 paper examines the stability of Big Five Inventory (BFI) scores when administered to LLMs under prompt changes. It constructs a 5 instruction templates × 5 item versions × 10 languages × 5 choice-label formats × 2 choice orders factorial, yielding 2,500 configurations per model. The BFI has 44 five-point items; items are shuffled and presented in random blocks of 17–27. GPT-4-Turbo creates four paraphrases per item; Google Translate and DeepL produce nine translations from English, with only a sample manually checked. At temperature zero, the study evaluates GPT-3.5-Turbo-1106, GPT-4-Turbo-1106, Gemini-1.0-Pro, and LLaMA-3.1-8B. For GPT-3.5, DBSCAN labels 77 configurations, 3.08%, as outliers, concentrated in numerical labels, descending order, and Arabic or Chinese. Only 7 of 135 one-level-versus-rest comparisons exceed an absolute difference of .15, although the table contains many small p-values because .15 is a descriptive threshold rather than a validated psychometric criterion. GPT-3.5 mean ± SD scores are 4.31±.44 for Openness, 4.15±.39 for Conscientiousness, 3.89±.43 for Extraversion, 4.13±.38 for Agreeableness, and 2.35±.42 for Neuroticism. GPT-4, Gemini, and LLaMA have distinct distributions and outlier rates of 5.6%, 4.2%, and 4.4%; LLaMA is more dispersed. Comparing GPT-3.5 dispersion with more heterogeneous human norms finds lower model variability, but this does not prove reliability: it is also compatible with temperature zero, response bias, or determinism. The test–retest study queries GPT-3.5 biweekly from September 2023 through January 2024 across snapshots 0613 and 1106. Its conclusion of no variation conflicts with Table 2: maximum Agreeableness differs from the mean with p displayed as .00 and is marked “No” for equal means. The analysis also selects extrema post hoc, treats failure to reject as acceptance of equality, and reports no test–retest coefficient. A second part tests three ways to alter GPT-3.5 self-ratings: emotional narratives, explicit assignment of ten extreme profiles, and embodiment of eight heroes and eight villains. Narratives barely move the distribution; among question answering, biography, and portray methods, only direct portray instructions clearly shift it. Extreme instructions separate every targeted dimension from default (p < .001): the largest change is minimum Extraversion at −1.71, followed by maximum Neuroticism at +1.03. Heroes stay near the default assistant profile, while villains spread more broadly. This demonstrates that self-report outputs respond to descriptions containing the target trait, not accurate representation of people or human populations: there is no target human data, behavioral validation, external criterion, or character-fidelity evaluation. The main overreach is calling aggregate robustness to perturbations “internal consistency reliability.” The paper itself acknowledges that its transformations preclude Cronbach's alpha and that reliability does not imply validity. The defensible contribution is therefore a broad map of BFI sensitivity to formats, languages, and instructed personas; it neither validates the BFI as a measure of LLM personality nor supports replacing human participants.
Research question
To what extent do BFI scores of various LLMs remain stable under changes in instruction, paraphrase, language, format, and order, and how much can GPT-3.5 shift those scores through context, profiles, or instructed personas?