PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents

Research question

Can a GPT-4-based multiagent system convert self-report scales of different constructs into interactive fiction games that retain consistency and correlation with the original scores and that, furthermore, are more coherent, interactive, interesting, immersive, and satisfying than scales or interviews generated with LLM?

Method

PsychoGAT employs a designer, a controller, and a critic based on GPT-4. The designer produces a title, outline, and binary items adapted to a plot; the controller instantiates each item as a paragraph and two instructions; the critic iterates up to three times to correct coherence, omissions, and bias; and a deterministic sum converts ten choices into a score. For automatic evaluation, GPT-4 simulates 20 extreme participants per task and method, 10 with and 10 without the construct. Cronbach's alpha, Guttman's lambda-6, and Pearson correlations with the reference scale and with a visual scale are calculated. Traditional scale, generated scale, psychological interview, Diagnosis-of-Thought, and PsychoGAT are compared. Thirty-three people rate 15 simulated samples; 12 real participants complete scale and extraversion game; and three experts assess agent ablations and redesign.

Sample: The main psychometric evaluation does not use patients or a population sample: it uses 20 characters simulated by GPT-4 for each task and method, deliberately divided into ten positive cases and ten negative cases through explicit instructions. The comparative evaluation recruits 33 people with basic knowledge of psychological assessment to judge 15 contents produced with simulators. The real-use study includes 12 English-speaking participants aged 20 to 30 years, all subjected to the traditional scale and to ten rounds of PsychoGAT for extraversion. The ethics section places the range of human evaluators at 20-45 years, excludes current mental illness and self-harm or suicidal risk, declares consent, compensation of 20 USD/h, and approval THU-03-2024-0001.

Findings

• PsychoGAT implements a chain of design, control, critique, and scoring to convert the binary choices of a scale into narrative continuations.
• The person does not respond directly to the original text of the item, but each of their two continuations remains explicitly linked to an option and score of the scale.
• The controller maintains summarized memory and the critic attempts to prevent the narration from inducing a choice based on previous choices.
• All generative components and all LLM-based baselines use GPT-4 without fine-tuning, at temperature 0.5.
• Each task and method is evaluated with 20 GPT-4 simulations, deliberately distributed between ten positive and ten negative profiles.
• In simulated personality, Table 1 reports alpha 0.97, lambda-6 0.98, convergence 0.97, and discrimination -0.59.
• In simulated depression, Table 1 reports alpha 0.77, lambda-6 0.84, convergence 0.85, and discrimination -0.07.
• In dichotomous thinking, Table 1 reports alpha 0.92, lambda-6 0.93, convergence 0.97, and discrimination -0.44.
• In mind reading, Table 1 reports alpha 0.92, lambda-6 0.95, convergence 0.97, and discrimination 0.25.
• In should statements, Table 1 reports alpha 0.88, lambda-6 0.91, convergence 0.93, and discrimination -0.18.
• The article considers a discriminant correlation with an absolute value below 0.60 acceptable, so the -0.59 of personality exceeds the threshold by only 0.01.
• All five compared methods meet the psychometric thresholds defined by the authors in the simulated scenario.
• Among 33 evaluators, 66.7% judge PsychoGAT superior to all other methods in coherence.
• The superiority percentages of PsychoGAT are 84.8% in interactivity, 87.9% in interest, 78.8% in immersion, and 84.8% in satisfaction.
• When excluding the two interviews, the superiority agreement in coherence rises from 66.7% to 75.8% and the other percentages barely change.
• The scene experiments obtain alpha between 0.98 and 0.99 and convergence 0.99 in everyday life, science fiction, and horror, again over 20 extreme simulations.
• The ablation suggests that removing agents affects simulated psychometric metrics little but reduces experience ratings.
• The published legend of Figure 7 contains two conditions called 'w/o Critic', which makes one of the ablations ambiguous.
• Twelve real participants complete an extraversion scale and ten rounds of the game in approximately 30 minutes.
• The text states that the human study reproduces reliability and convergence and that the majority prefers PsychoGAT, but Figure 6 provides no numerical values or a table.
• The abstract and the conclusion describe the results as statistically significant, although no p-values, intervals, difference tests, or effect sizes are presented.
• The ethics section declares that PsychoGAT does not replace professional assessment nor constitutes a diagnosis.
• The study received approval from Tsinghua University under protocol THU-03-2024-0001.
• Participants were remunerated at 20 USD per hour and the authors declare consent and absence of identifying data.
• The article includes no code or data link and the targeted search did not identify an official reproducible publication.

Limitations

• The main design uses GPT-4 as game generator, participant simulator, and component of the baselines, creating common-method dependence.
• The simulated profiles receive by prompt the presence or absence of the construct, so the high correlations partly reflect compliance with extreme instructions.
• Twenty samples per task are insufficient for stable estimates of reliability and correlation, especially without confidence intervals.
• Artificially dividing the sample into ten positive and ten negative cases widens the variance and may inflate correlations and internal consistency.
• Realistic distributions, subclinical cases, mixed profiles, and population prevalences are not evaluated.
• Convergent validity correlates the game with the same scale that guides its design, so it provides no independent evidence of the construct.
• Discriminant validity uses only visual learning preference as a comparator, without empirically justifying its independence in this simulated sample.
• The absolute criterion of 0.60 for discrimination allows approving a correlation of -0.59, which remains moderate and nearly fails the threshold.
• There is no factorial analysis, invariance, dimensional structure, differential item functioning, test-retest, or predictive validity.
• No correction for multiple comparisons, hypothesis tests, intervals, or effect sizes are reported.
• The expression 'statistically significant excellence' does not correspond to an inferential test described in the article.
• PHQ-9 is reduced from four frequencies to two options, altering administration, range, and scoring rules of the validated instrument.
• The binary version of PHQ-9 includes the death or self-harm item, but the system does not present a response protocol or risk scaling.
• Extraversion is represented with ten dichotomous MBTI items and not with a modern dimensional instrument or one validated for this use.
• For distortions, the standardized scale provides a single item per construct and the authors construct additional situations, so the obtained reliability does not validate the original instrument.
• Redesigning the items changes content, context, and cognitive load without a formal study of semantic or psychometric equivalence.
• The two continuations remain linked to binary values, simplifying behaviors and eliminating ambiguous, mixed, or open responses.
• The order of the options and the 1/0 association are fixed in the published prompts; position biases are not studied.
• Previous selections change the story and may condition subsequent ones; the critic attempts to mitigate this, but does not measure whether it succeeds.
• The demonstration contains socially valued choices and narrative stereotypes, including a scene with 'tribal scouts', which may introduce desirability and cultural bias.
• Only five constructs are tested in English; this does not support the claim of transforming any standardized scale.
• The GPT-4 used has no snapshot, API date, or exact identifier, preventing reproduction of the outputs.
• No seeds, raw responses, games, item scores, evaluator data, or analysis scripts are published.
• The article provides no code or official repository; the extensive prompts in the appendix are insufficient to reconstruct the pipeline and its parsing.
• The 33 evaluators judge only 15 simulated contents of dichotomous thinking and receive the instruction to imagine they are participants.
• Content evaluation is not equivalent to interactively using each method and may favor the narrative format in interest and immersion.
• The evaluators only need basic psychological knowledge; training, calibration, or inter-evaluator agreement are not detailed.
• The 1-5 means are normalized to 0.1-0.9 without justifying the effect of this transformation on the interpretation of the graphs.
• The real study has 12 participants, only extraversion, and a single session of about 30 minutes.
• Figure 6 of the real study offers no exact figures, dispersion, intervals, paired comparisons, or per-participant results.
• No order or counterbalancing between scale and game is reported; the appendix indicates that the scale is completed first, which may produce memory and carryover.
• The real participants are between 20 and 30 years old with English proficiency, with no additional diversity or representativeness data.
• People with current mental illness or self-harm risk are excluded, precisely the populations relevant to the claims about depression and diagnosis.
• There is no trial with patients, professionals using the system, clinical consequences, or comparison with human interview.
• The ablations use 20 simulations and only three expert evaluators, and their legend contains a duplicate label that prevents identifying one condition with certainty.
• The cost, latency, API stability, and data footprint of multiple GPT-4 calls are not evaluated.
• Privacy is declared for the study, but the handling of sensitive data in a future external API is not analyzed.
• Resistance to prompt injection, adversarial content, incoherent responses, or parsing failures is not tested.
• The game may hide that each choice is a psychological item, but transparency, specific consent, or manipulative use of covert assessment are not discussed.
• The article itself acknowledges the need for longitudinal studies, diverse patients, linguistic localization, and professional supervision.

What the study does not establish

• It does not demonstrate that PsychoGAT measures an internal, stable, or complete personality.
• It does not demonstrate clinical validity for depression or cognitive distortions.
• It does not demonstrate equivalence with PHQ-9 after binarizing its responses.
• It does not demonstrate equivalence with MBTI or general validity of MBTI as a personality measure.
• It does not validate the distortion scales constructed by the authors as independent instruments.
• It does not demonstrate that any standardized scale can be converted without altering the construct.
• It does not demonstrate that a high correlation in instructed simulators transfers to people.
• It does not demonstrate that GPT-4 faithfully simulates human psychological distributions.
• It does not demonstrate broad discriminant validity with a single comparison construct.
• It does not demonstrate test-retest reliability, longitudinal stability, or invariance between groups.
• It does not demonstrate statistical superiority over traditional scales or human interviews.
• It does not demonstrate that 12 participants represent general or clinical populations.
• It does not demonstrate efficacy for people with depression, self-harm, or suicidal risk, who were excluded.
• It does not demonstrate that greater interest or immersion produces more accurate diagnoses or better outcomes.
• It does not demonstrate that the experience is preferable in other languages, ages, cultures, or accessibility conditions.
• It does not demonstrate that the critic eliminates narrative biases, social desirability, or dependence between items.
• It does not demonstrate safety against harmful content, data leaks, or manipulation.
• It does not justify autonomous use for screening, diagnosis, treatment, or high-impact decisions.
• It does not allow reproducing the published figures without data, code, and exact model version.
• It does not establish that the results remain valid with current models or APIs.

Traceability

Scope: Full text

Version: ACL 2024 final proceedings paper, pp. 14470-14505, DOI 10.18653/v1/2024.acl-long.779, 36 pages; no official code or data release identified in the paper, ACL record, arXiv record, or targeted project search

Consulted source: https://aclanthology.org/2024.acl-long.779.pdf

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, psychometric, clinical-claims, experimental-design, metric-interpretation, reproducibility, privacy and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

• GPT-4 como diseñador, controlador, crítico y simulador humano; el artículo no identifica un snapshot exacto
• GPT-4 en todos los baselines basados en LLM, temperatura 0,5
• Evaluador psicométrico determinista que suma las opciones asociadas a los ítems

Instruments and metrics

• Subescala de extraversión de Myers-Briggs Type Indicator (MBTI), reducida a 10 ítems binarios
• Patient Health Questionnaire-9 (PHQ-9), transformado en respuestas binarias
• Cognitive Distortions Questionnaire para pensamiento dicotómico, lectura de mente y enunciados de debería
• Situaciones y pensamientos alternativos construidos por los autores para ampliar cada distorsión
• Escala de preferencia de aprendizaje visual usada para validez discriminante
• Alfa de Cronbach
• Lambda-6 de Guttman
• Correlación de Pearson para validez convergente y discriminante
• Valoración 1-5 de coherencia, interactividad, interés, inmersión y satisfacción
• Cuestionario de experiencia de usuario del estudio con participantes

Data used

• 20 simulaciones GPT-4 por tarea y por método, con 10 perfiles positivos y 10 negativos
• Cinco constructos objetivo: extraversión, depresión y tres distorsiones cognitivas
• Diez combinaciones de tipo y tema narrativo: fantasía, romance, ciencia ficción, vida cotidiana y horror
• 15 muestras simuladas de pensamiento dicotómico valoradas en la comparación de contenido
• 33 evaluadores humanos con conocimientos básicos de evaluación psicológica
• 12 participantes humanos en el experimento de extraversión
• 20 simulaciones por condición en las ablations de escenas y agentes
• Demostración no seleccionada manualmente de un juego de extraversión Fantasy/Adventure

Evidence and location

• Official bibliographic record: ACL Anthology 2024.acl-long.779: 9 authors, ACL 2024, pp. 14470-14505, DOI 10.18653/v1/2024.acl-long.779
• Full audited source: .cache/editorial-sources/article-095/source.pdf; official ACL PDF; 36 pages; sha256 afe1eb1f7cc8b299d85c1ea656d093554de0b970b67292698377a29b6247fad7
• Designer, controller, critic, and evaluator architecture: Full text pp. 14471-14473, sections 2.1-2.5 and Figure 2
• History bias and critic mitigation: Full text p. 14473, section 2.4
• Tasks and instruments: Full text pp. 14473-14474, section 3.1; Appendix pp. 14484-14487
• GPT-4, temperature, and iteration limits: Full text p. 14474, Baseline Methods
• Twenty extreme simulations per task: Full text p. 14475, section 3.3
• Psychometric definitions and thresholds: Full text pp. 14474-14475, section 3.2
• Exact reliability and validity results: Full text p. 14475, Table 1
• Comparison with four baselines: Full text pp. 14474-14476, Figure 4 and Comparative Experiments; Appendix p. 14483, section A
• Thirty-three evaluators and fifteen contents: Full text p. 14475, Comparative Experiments
• Superiority percentages: Full text p. 14476, Figure 5; Appendix p. 14483, Table 3
• Robustness by scene: Full text p. 14476, Table 2
• Ablations and ambiguous legend: Full text pp. 14476-14477, Figure 7 and Ablation on Agents
• Twelve real participants: Full text p. 14476, Human Participant Experiments; Appendix pp. 14483-14484, section B.3
• Human results only as a graph: Full text p. 14476, Figure 6; no numeric companion table
• Binarization of PHQ-9 and scales used: Appendix pp. 14485-14487, section C
• Construction of distortion items: Appendix p. 14484, footnote 5; pp. 14486-14487
• Complete prompts and binary association: Appendix pp. 14488-14500, section D
• Unselected narrative demonstration: Appendix pp. 14501-14505, section E
• Acknowledged limitations: Full text p. 14478, Limitations
• Non-diagnosis and professional supervision: Full text p. 14478, Ethics Statement
• IRB, exclusions, remuneration, and privacy: Full text p. 14478, Ethics Statement; protocol THU-03-2024-0001
• Absence of reproducible artifact: Paper and ACL/arXiv metadata contain no code or data URL; targeted official project and GitHub search checked 15 Jul 2026
• Comprehensive visual inspection: All 36 PDF pages rendered and visually inspected, including Figures 1-9, Tables 1-3, prompts, scales, ethics, limitations and full demonstration; checked 15 Jul 2026