The study compares how four conversational products available in April 2024 respond to two human personality questionnaires: ChatGPT-3.5, Gemini Advanced, Claude 3 Opus, and Grok in Regular mode. Each product completes the 32 Open Extended Jungian Type Scales (OEJTS) pairs and the 50 IPIP Big-Five items fifteen times in fresh chats. The authors remove the midpoint, force four response options, and map 1, 2, 3, and 4 onto the original 1, 2, 4, and 5 scoring scale. Prompts are not neutral: they tell the model to “play the role of yourself,” be “100% honest,” and return a mandatory number. Sixty administrations per instrument are analyzed with MANOVA, ANOVA, and frequencies; no clinical conversations or patients are evaluated.
Large product differences appear under this protocol. For OEJTS, the MANOVA reports Wilks' lambda 0.130, F(12, 140.52)=13.63, p<0.001. Claude returns INTJ in 15/15 administrations; ChatGPT's modal type is ENTJ in 7/15, Gemini's INFJ in 9/15, and Grok's INFJ in 8/15. For Big Five, the MANOVA reports lambda 0.115, F(15, 143.95)=11.44, p<0.001, and partial eta squared 0.514. Gemini scores 68.7±12.5 on Agreeableness and 82.6±9.0 on Conscientiousness, below the other three; Claude reaches 97.1±1.0 on Conscientiousness; Grok has the highest Openness, 95.0±1.4, but also the greatest dispersion in Emotional Stability, 81.1±20.4. The defensible result is that scored outputs from these four configurations differ and that some recur with substantial regularity.
Fifteen sessions do not make fifteen people or independent samples from a population of models. They are stochastic replicates of the same product sharing weights, alignment, and interface. The power calculation and inferential tests treat each administration as an independent observation, inflating the effective sample size for model-level inference. Exact snapshot identifiers, temperature, top-p, seed, provider system prompts, account configuration, and execution days are not recorded. Methods announce one nine-variable MANOVA, 95% confidence intervals, and Bonferroni comparisons; Results present two separate MANOVAs, show no intervals, omit the post-hoc table, and do not report F, degrees of freedom, p, and effect size for each ANOVA. Levene's test is also significant for all five Big Five traits, with no robust alternative or complete assumption assessment.
Construct validity is the main limitation. OEJTS and Big Five were developed for human self-report, and the paper acknowledges that they have not been validated in artificial systems. It tests no factor structure, measurement invariance, internal reliability, convergence between instruments, or correspondence with external behavior. Removing neutrality to “increase discrimination” and then concluding that models are non-neutral is circular; no neutral score is defined or tested. Variation across chats does not rule out memorization or prior exposure to questionnaire content, while the instruction to answer “as yourself” induces the very self-referential performance being interpreted. “Personality” should therefore be restricted to a prompt- and interface-conditioned response profile.
The integrity audit finds two clear numerical contradictions. Table 1 says Gemini selected Sensing in 60% of fifteen trials, nine trials, while Table 2 says it produced INFJ, which requires Intuition, in nine of those same fifteen; both cannot be true. The abstract says Claude had the highest Emotional Stability, but Table 3 gives Gemini 94.1 and Claude 93.2. The abstract also uses the 0.115 lambda from the Big Five analysis to summarize typological and dimensional traits together, whereas OEJTS has lambda 0.130. A paragraph on interdisciplinary oversight is duplicated verbatim in the discussion. The linked Zenodo record exists and lists XLSX and SAV files, but the service returned 503 during the audit and prevented inspection; the record lists no separate response transcripts despite the paper promising “raw outputs.” With no clinical task, sustained interaction, therapeutic outcome, or safety test, recommendations about model selection, patient matching, and formal clinical personality evaluation are governance proposals rather than empirical conclusions of this experiment.