Shu et al. ask a question that should precede any personality test applied to a language model: before interpreting a score, does the LLM understand the response format and preserve its answer when content is unchanged or reverse it when the proposition is reversed? They build MODEL-PERSONAS, 693 statements drawn from 39 instruments and organized into 115 axes. The collection is broader than personality: it combines traits, values, political and moral beliefs, social attitudes, descriptors, and situational responses. Items are standardized into binary Yes/No or True/False questions and tested through punctuation, whitespace, separator, answer-label, option-order, direct-negation, and paraphrastic-opposite variants.
The paper separates three properties. Comprehensibility is the proportion of responses whose first token is one of the allowed options; sensitivity is the fraction unchanged by spurious format variations; consistency is the fraction preserving an answer under equivalent labels or order and reversing it when meaning is reversed. This last quantity is within-item consistency, not alpha, omega, test–retest reliability, or cross-item internal consistency, a distinction the authors explicitly make. Seventeen models are tested at temperature 0. GPT-2 and RedPajama are removed from the main consistency analysis because of low valid-answer rates, so Figure 1 compares 15 models: 13 open models plus GPT-3.5 and GPT-4.
The results reveal substantial and practically important brittleness. BLOOMZ and FLAN-T5 usually emit valid option tokens, whereas Falcon, RedPajama, and several Llama 2 variants can move from valid answers to almost none after a space, line break, or terminal punctuation change. FLAN-T5 Base/Large/XL is the most robust family under spurious formatting. Option order and the True/False–Yes/No switch preserve many responses, although the paper correctly notes that an always-positive response policy can also score highly. Negation is the central failure: 10 of 15 models are near or below random behavior in at least these comparisons, and every model performs worse on paraphrastic reversal than on an explicit negation word.
The study manually selects 0.6 as a simultaneous threshold on four metrics and states that only FLAN-T5-XL, GPT-3.5, and GPT-4 pass, with FLAN-T5-Large close. This cutoff is neither calibrated nor preregistered and is not a psychometric criterion. Auditing the official artifact at commit fdfdf513a88de2af50294d10def9ca9ccfac87e8 reproduces the published processing logic but exposes boundary sensitivity: the released processed CSV yields about 0.605 paraphrastic-negation consistency for FLAN-T5-Large, whereas the released GPT-3.5 output yields about 0.592. Under a strict unrounded rule, membership differs from the prose. This does not overturn the general brittleness finding, but it does mean that the reported group of three is not a stable classification.
A second experiment prepends “You are a normal person,” one of six specific profiles, or a 35-attribute description. On average, every intervention lowers negation consistency; a few related axes improve as outliers, but the highly personified prompt is among the least consistent. The result supports a restrained conclusion: adding more personality adjectives does not itself create stable measurement. The experiments measure the robustness of prompt-conditioned binary answers, however; they do not show that a model lacks all behavioral tendencies, nor do they establish the presence or absence of a latent personality. The paper's reference contribution is therefore a set of format and polarity controls that should precede questionnaire interpretation, not an ontological validation of model personality.