Löhn, Kiehne, Ljapunov, and Balke do not run a new personality test on models. They construct a normative framework for deciding whether human psychological tests can legitimately be transferred to LLMs. Drawing on the Standards for Educational and Psychological Testing and the International Guidelines for Test Use, they formulate seven requirements: reliability for the intended use (R1), validity for that use (R2), suitability for the test taker (R3), non-disclosure or contamination control for test materials (R4), validity for every model being compared (R5a), validity of translations (R5b), and transparent, reproducible test use (R5c). The central contribution is to separate repeatable responding from justified score interpretation: a system may always return the same answer while still failing to measure the human construct attributed to it.
To illustrate the framework, the authors review 25 studies found through keyword searches in Google Scholar, Scopus, and DBLP up to October 2023, followed by citation-network tracing. The sample covers 12 constructs and 34 tests or assessments. Twenty studies include GPT-3 or a later version, and 17 of those 20 examine no other model family. All four authors jointly rate each paper on an ordinal scale: not applicable, not addressed, discussed, an appropriate effort or study without supporting evidence, and any evidence of fulfillment. The rule is intentionally lenient: the top category recognizes some evidence for one form of reliability or validity rather than complete validation, and evidence sufficiency is not judged.
The table shows a substantial evidence gap. For R1, only 6 of 25 studies provide any reliability evidence; for R2, only 3 provide any validity evidence. R3 appears better covered, but its 13 top-rated cases benefit from a weak operational rule that treats a compatible text format as suitable. Only one study reaches the top category for R4. Among the ten comparative studies to which R5a applies, only Serapio-García et al. provides evidence. R5b applies to two studies, and only Pellert et al. uses already validated translations. Just 2 of 25 fully meet R5c. Nine studies modify or rephrase tests to reduce contamination, but only Coda-Forno et al. supplies evidence that the modification preserves results. No study provides evidence for every applicable requirement.
The diagnosis is useful as a checklist and as a warning against anthropomorphizing scores without a validation chain. It is not, however, a reproducible systematic review or a validated standard. The paper releases no search strings, selection flow, exclusion criteria, coding sheet, evidence excerpts, coder agreement, or artifact; ratings are decided in joint meetings. It also supplies no operational threshold for how much evidence is enough. Some requirements are overly absolute or narrow: proving absence of contamination is often impossible for proprietary models, suitability is not equivalent to accepting text, and the fairness section omits subgroup bias, accessibility, and harm. The paper establishes that the reviewed early literature incompletely documented its inferences under this framework. It does not establish that every substantive conclusion was false or that human constructs can never be adapted through model-specific validation.