This preprint, described by its repository as an AIES 2026 submission, asks whether Big Five questionnaires support personality attributions to LLMs that are comparable to human traits. It examines three requirements: whether items appropriately describe chatbots, whether scores differentiate models, and whether responses reproduce the expected five-factor latent structure. In phase one, three experts, two of them authors, rated five inventories. The LLM-adapted BFI-LLM and LMLPA passed the chatbot-suitability threshold, whereas BFI-44, IPIP-NEO-120, and MPI did not. A pilot with nine models, two finalist inventories, seven prompt templates, and three repetitions selected BFI-LLM and a template that explicitly frames the task as a psychological evaluation. In phase two, 44 items were sent separately to 244 model entries from 49 families, using two visual orders of the same 1–5 mapping and five repetitions per order. The released repository contains 107,352 deduplicated BFI rows, and its preprocessing reconstructs the reported 106,058 valid responses. Means cluster toward socially desirable endpoints: high Agreeableness, Conscientiousness, Extraversion, and Openness and low Neuroticism. A mixed model assigns 3% of variance to a general model intercept, 37% to item, 32% to model-by-item interaction, and 28% to residual variation. Robust CFA on the 244 × 44 matrix fits very poorly; Openness, Conscientiousness, Extraversion, and Agreeableness correlate at .92–.99, while reverse-keyed items load much more weakly than positive items. A two-factor EFA also fits inadequately. This is the strongest finding: under this self-report protocol, the human Big Five structure is not recovered and trait labels lack demonstrated structural validity. The conclusion must be narrower than some of the paper's prose. The 3% term represents each model's general response level and does not erase the 32% model-item interaction, so it does not prove that models lack stable differences. Human reference values come from the original BFI while model values use adapted stems, with no measurement-invariance or scale-linking study; they are directional context, not directly equivalent norms. Nineteen base/instruction pairs shift in a socially desirable direction on four traits, but the comparison mixes inference routes, providers, templates, and defaults; it suggests an association with instruction tuning rather than causally identifying alignment as the primary driver. The repository is unusually substantial, with raw responses and analysis code, but it is not one-command reproducible: preprocessing writes `final_df` while R reads `final_dfs`, derived files, an R environment, tests, CI, a license, and run instructions are absent. The parser also takes the first standalone 1–5 digit from many explanatory responses. A sensitivity rule prioritizing a later explicit final answer changes 2,369 BFI rows across 92 models. Global means move little, but an individual model-trait score changes by as much as 1.17 points. The aggregate pattern is therefore informative, while model rankings and family, country, or subgroup contrasts should not be treated as validated personality measurements.
Research question
Do Big Five applications to LLMs meet three basic psychometric requirements: adequate content to describe chatbots, interpretable differences between models, and an internal structure coherent with the five human factors?