AIPsychoBench is a 2026 Topics in Cognitive Science article developed from a CogSci 2025 paper. The definitive version has eight authors and differs from the arXiv preprint in both author order and composition. This review uses the complete XML of the definitive publication, visually inspects all eight pages of the preprint and the four definitive figures, and audits the official repository at commit d4759bff.
The study assembles 21 human Likert questionnaires: 777 items grouped into 112 subcategories and six domains. It tests six LLMs, GPT-4, GPT-4o-2024-11-20, GLM-4-plus, Gemini-2.0-flash-exp, DeepSeek-R1, and Claude-3-5-sonnet-20240620, in English, Chinese, French, Russian, German, Spanish, Arabic, and Japanese. Microsoft Azure AI Translator produces the translations, after which two researchers sample and cross-check them. Each condition is run five times at temperature 0. GPT-4o also serves as the audit model that decides whether an explanation agrees with its Likert number and removes responses judged invalid.
The main intervention is a lightweight prompt instructing the model to act as a respondent and answer from its “authentic emotions and thoughts.” Against direct questions, Base64, Caesar cipher, and the STAN jailbreak, the prompt raises mean valid-response rate from 70.12% to 90.40%; STAN reaches 81.49%. The Table 1 averages are arithmetically correct. The paper then compares prompt scores with baseline only on items valid in both conditions. It reports mean bias of +3.3% and −2.1% for the lightweight prompt versus +9.8% and −6.9% for STAN, with 86 of 112 subcategories below 4%. For language comparisons, it reports deviations of at least 5% and up to 20.2% in 43 subcategories relative to English.
The defensible result is narrower than the paper's framing: under this protocol, an anthropomorphic instruction increases the proportion of outputs that a GPT-4o judge can turn into Likert numbers, and changing prompt language through machine translation changes those numbers. This does not validate psychological properties of a model. Asking a system for “authentic emotions” when the paper itself acknowledges that such terms lack rigorous definitions changes the task from measurement to compliance or simulation. Objective or neutral answers are not necessarily accidental missing data; they can be truthful responses from an assistant without biography, race, religion, relationships, employment, or embodied experience.
The bias calculation is also intersection-selected: it compares only items valid under both baseline and method, so it does not evaluate the additional cases responsible for coverage increasing from 70.12% to 90.40%. The 4% threshold is called reasonable without a human benchmark, interval, or test. “Significant” denotes thresholded descriptive differences, not statistical inference. There are no human participants, factor-structure tests, reliability estimates, convergent validity, measurement invariance, intervals, significance tests, seeds, or judge-error study. Language effects confound translation, language proficiency, tokenization, training corpora, alignment, and judge behavior.
The repository does not reproduce the paper. It releases the 777 English items plus an extra MBTI questionnaire, but not all eight complete translations, six-model by five-repeat outputs, tables, derived figures, or analysis scripts. Only one German BFI result is committed. The program imports a missing ResultManager module; ignores all 117 reverse-keyed items across 13 scales; and configures EPQ-R so that 0 is accepted while 1 is rejected. The released BFI result averages reverse items without recoding them; correcting this changes Agreeableness, Extraversion, and Neuroticism means. Documentation misattributes Resistance to Change to Mael (1988) rather than Oreg (2003), and labels Zi's Negative Perfectionism Questionnaire as the Zuckerman–Kuhlman Personality Questionnaire even though its items and dimensions are negative-perfectionism constructs. Without raw outputs and executable analysis, the paper's numerical claims cannot be independently verified from the public artifacts.