This study tests whether language-model responses to a Big Five questionnaire shift toward socially desirable profiles when the model can infer that it is being evaluated. The authors administer the 100 IPIP items in fresh contexts while varying the number of questions shown together, and compare eight models from OpenAI, Anthropic, Google, and Meta. They also test explicit disclosure of the personality assessment, manually paraphrased items, three ordering schemes, alternative coding schemes, and temperatures from 0.0 to 1.2.
For GPT-4, increasing the batch from one to twenty questions raises the four positively valued dimensions by 0.73–0.96 points and lowers Neuroticism from 2.87 to 2.02. The mean absolute change across the five traits is 0.82 points, reported as 1.20 human standard deviations under the selected external norms. The directional shift appears across the compared model families and is descriptively larger in some newer and larger models; the design does not identify scale, recency, or preference tuning as a cause. Given five random items, GPT-4, Claude 3, and Llama 3 identify the personality-test context with over 90% accuracy, compared with 55% for PaLM 2 and 45% for GPT-3.5. Explicitly naming the Big Five produces a shift comparable to presenting five questions together. Fully reverse-coding the items reduces the mean difference from 0.81 to 0.38 points but does not remove it; paraphrasing, ordering changes, and temperature changes preserve the overall pattern.
The reported internal consistency, α>0.8 for each subscale, α>0.93 across all items, and corrected split-half reliability of 0.79, does not protect scores from protocol sensitivity: an instrument can be internally consistent while changing with evaluation context. It also does not turn model outputs into human traits. The proposed “awareness” mechanism is behavioral. Recognizing test-related language and shifting under an explicit instruction are consistent with contextual inference, but they do not demonstrate self-awareness or a subjective intention to look good.
The OSF deposit supplies 2,418 raw-conversation files, 3,263 preprocessed files, code, and six figures, so the main temperature-zero GPT-4 means can be checked. It does not provide a reliable end-to-end reproduction. The generation script fails to compile because API-key placeholders are unquoted, contains undefined `throw` statements, imports packages absent from requirements, and ultimately hard-codes Groq's llama3-8b-8192 instead of honoring its model argument. Preprocessing replaces unparseable non-PaLM answers with the neutral value 3 and labels survey recognition through substring matching that includes broad terms such as “assessment,” “affect,” and “inventory.” Figure 1A hard-codes its values and uncertainty bands. The released aggregate combines 120 rows for Q1/Q5, four temperatures, three orderings, and ten ordering seeds, with 30 rows for Q10/Q20, which are present only at temperature zero, although the caption states N=30. No code computes the reported alpha or Spearman–Brown estimates, and the p-value function is not connected to figure generation.
The defensible contribution is a measurement warning: under this protocol, grouping more items and making the test context more evident systematically changes Big Five scores. This constrains psychometric profiling of LLMs and their use as participant substitutes, and supports triangulation with independent measures and tasks. It does not show that models possess personality, intend to present themselves favorably, reproduce human psychological distributions, or exhibit the effect across every culture and instrument.