This paper raises a foundational psychometric objection to administering human questionnaires to language models and interpreting their scores as equivalent traits. It evaluates GPT-3.5, GPT-4, Llama 2 70B Chat, and Llama 3.1 70B Instruct in two studies. The first compares acquiescence bias on the 50-item IPIP Big Five Markers with a distribution of more than one million human responses. The second generates BFI-2 questionnaires under persona, cumulative-context, and seeded-first-answer conditions, explores their components with varimax-rotated PCA, and tests factor models with CFA. Its most important contribution is the separation of internal consistency from validity: a high Cronbach's alpha or omega cannot support interpretation when the structural model that gives those coefficients meaning fits poorly.
All four models agree with both positively and negatively keyed items more than the human mean, at approximately 0.6 for GPT-4, 0.8 for Llama 3.1, and 1.5 for GPT-3.5 and Llama 2. Only the latter two lie outside the empirical human distribution at p < .005; GPT-4 and Llama 3.1 are not statistically significant, although GPT-4 exceeds 89% of human agree-bias values. The comparison treats each model as one response profile rather than a sample of people or model runs, so it measures how unusual that output is relative to humans without estimating between-run model uncertainty.
For the BFI-2, Llama 3.1 under cumulative context and persona prompts produces the most human-looking exploratory pattern: five clearer blocks with opposite signs for direct and reverse-keyed items. That appearance is not confirmed. Every model and prompt setting remains far from the stated CFI/TLI ≥ .95 and RMSEA ≤ .06 thresholds. In the BFI-2 model, for example, GPT-4 reaches CFI .88/.78, TLI .81/.65, and RMSEA .15/.17; Llama 3.1, despite alpha and omega of .98/.93, reaches CFI .74/.66, TLI .65/.57, and RMSEA .38/.41. The solutions converge with lavaan warnings, including negative or zero estimated variances. The evidence strongly rejects a human-equivalent interpretation of BFI-2 scores for these models and protocols, although it is not a full formal multigroup invariance sequence: failure occurs at the prior requirement that the human configural model fit each response set adequately.
The induced population also constrains the inference. The 100 PersonaChat descriptions were not designed to span the Big Five space in a balanced way, and the recovered covariance structure depends on the range and correlations of traits induced by those prompts. Poor fit may reflect invalid instrument transfer, persona distribution, deterministic response mechanics, or a combination; it does not prove the absence of every form of personality or behavioral regularity in an LLM. Repository auditing finds only partial, manual reproducibility: there is no README, license, locked environment, or raw human dataset; the R scripts hard-code a local working directory and repeatedly overwrite their input objects; and the artifact predates the EAAMO version and contains no Llama 3.1 results. OpenAI outputs are also converted to one-hot answers from a returned token, whereas Llama uses an expected score over normalized probabilities for five option tokens, an asymmetry that can affect variance and estimated factor structure. The defensible conclusion is not that LLMs “have no personality,” but that these BFI-2 scores should not be treated as human traits until the construct, administration protocol, and relationship to external behavior have been validated.