Synthetic Personality Review

LLM-Based Educational Simulation: Evaluating Temporal Student Persona Stability Across ADHD Profiles

Applications, bias, and safety2026arXivApproved editorial review

Authors: Jana Gonnermann-Müller, Jennifer Haase, Nicolas Leins, Thomas Kosch, Sebastian Pokutta

Keywords: Student persona stability, ADHD-like persona prompting, Educational simulation, Within-conversation drift, Between-run repeatability, Scripted interaction, Self-report observer dissociation, CAARS-derived assessment, Control contamination, Prompt demand characteristics, Reproducibility audit

Source: Open primary source (opens in a new tab)

5
Authors
8
Findings
25
Limitations
3
Evidence

Editorial summary

English

The preprint studies whether five LLMs maintain student profiles with ADHD-like characteristics across runs and over nine-turn conversations. Across 4,968 independent narratives and 3,952 dialogues, self-reports remain nearly constant, whereas ratings from three LLM observers decline in unscripted conversations for high and moderate profiles. With scripted, symptom-relevant questions, their means are nearly flat; the “97%” is only the descriptive reduction from 4.0 to 0.1 points for the high education profile. This does not prove drift disappeared: the inferential model pools profiles, omits interactions and excludes the default control, whose scripted score rises by 2.49 points. The control is also contaminated: even without a persona description, the task tells the model to show how ADHD influences the day, and the script reminds it to answer consistently with “your ADHD symptoms.” Stability can reflect instruction following, scale boundaries and overlap with CAARS content rather than an internal persona or clinical realism. There are no participants with ADHD, educational validation, public data or code for recalculating the results.

Español

El preprint estudia si cinco LLM mantienen perfiles estudiantiles con rasgos de ADHD entre ejecuciones y durante conversaciones de nueve turnos. En 4.968 narrativas independientes y 3.952 diálogos, los autoinformes permanecen casi constantes, mientras las valoraciones de tres LLM observadores bajan en conversaciones no guionizadas para perfiles altos y moderados. Con preguntas guionizadas y relevantes para los síntomas, sus medias quedan casi planas; el «97%» es sólo la reducción descriptiva de 4,0 a 0,1 puntos en el perfil alto de educación. Esto no demuestra que la deriva desaparezca: el modelo inferencial agrupa perfiles, omite interacciones y excluye el control default, que bajo guion sube 2,49 puntos. Además, el control no está limpio: aun sin descripción de persona, la tarea ordena mostrar cómo influye el ADHD, y el guion recuerda responder conforme a «tus síntomas». La estabilidad puede reflejar seguimiento de instrucciones, límites de escala y solapamiento con CAARS, no una personalidad interna ni realismo clínico. No hay participantes con ADHD, validación educativa, datos ni código públicos para recalcular los resultados.

Research question

With what stability do different LLMs reproduce profiles with high, moderate, low, or default intensity of ADHD-like traits across independent conversations and throughout dialogues, and does that stability change when the interlocutor's questions are structured?

Method

Experiment I generates school or work narratives and a performance-based self-report; three LLMs score each narrative. Experiment II confronts the agent-persona with an unscripted interlocutor or with three standardized situations, and repeats self-report and observation at turns 3, 6, and 9. It uses three prompt formats, five proprietary families, and provider defaults. Stability across executions is summarized by SD; intra-conversation drift by mixed linear models separated by scenario, structure, and source. This review inspected the 28 pages, complete prompts and items, missingness and model tables, TeX v1/v2, arithmetic of effects, and artifact availability.

Sample: There are 4,968 of 5,000 executions of experiment I, 14,904 LLM observations, and 4,968 self-reports; and 3,952 of 4,000 conversations of experiment II, 35,430 LLM observations, and 11,796 self-reports. There are 50 repetitions per non-default cell in experiment I and 20 in II. The overall loss is 0.64% and 1.20%, but it is concentrated: Grok default loses 27 of 80 conversations of experiment II (33.75%). Five MSc psychologists score only 20 narratives; there are no students or people diagnosed with ADHD.

Findings

  • In education, experiment I reports self-report M/SD 30.2/1.79 for high, 18.0/5.04 moderate, 2.08/3.35 low, and 27.3/1.64 default.
  • The observer means/SD in those conditions are 21.1/2.57, 16.6/4.78, 1.71/5.07, and 20.1/3.01.
  • In unscripted education conversation, the observer changes -4.0 points in high and -2.86 in moderate between turns 3 and 9; the pooled turn effect is -1.13, SE 0.05, p<.001.
  • Under script, high and moderate change -0.1 points; the pooled effect is -0.02, SE 0.05, p=.722, but default increases 2.49 points.
  • The 97% comes from 1-|0.1|/|4.0|=97.5%, rounded, in a single descriptive comparison without a confidence interval.
  • The unscripted self-report shows no detectable trend, beta=0.03, p=.366; the scripted one does have a small positive trend, beta=0.14, p=.001.
  • The human comparison of 20 texts reports human ICC .92, LLM .87, and human-LLM .95, but the last CI is wide [.57,.99].
  • The results support that repeated and symptom-relevant questions sustain observable scores, not that a psychological person remains intact.

Limitations

  • The default is not a control without ADHD: the experiment I task explicitly asks to show how ADHD influences the day.
  • In the scripted condition, responding according to "your ADHD symptoms" is added, even alongside the default row.
  • The scripted questions are direct tests of attention, group work, and organization; structure and symptom re-induction are confounded.
  • The same model receives the symptom description and then answers semantically overlapping items; the stable self-report may be memory of the prompt.
  • Low SD across independent samples measures repeatability of the prompt, not test-retest stability of a persistent identity.
  • The extremes remain close to floor and ceiling, with less possible variance than the moderate level; neither limits nor heteroscedasticity are modeled.
  • There is no human target or diagnosis against which to measure fidelity; stability and validity are not equivalent.
  • CAARS is for adults, while the student persona has no age and the implications include school students.
  • The appendix administers 26 self items and 30 observer items, but does not document which 12/10 form the indices nor provide scoring code.
  • The comparison with human SD uses ASRS, another scale and 43 heterogeneous people; it does not comparably estimate measurement error.
  • The mixed turn model has no interactions with person, model, or prompt, so a null average may hide opposite trajectories.
  • The inferential models exclude default, despite its scripted observation increasing 2.49 points.
  • p=.722 does not prove equivalence or zero drift; equivalence margins are not set.
  • There is no direct contrast with uncertainty for scripted vs unscripted or for the 97% reduction.
  • The Bonferroni correction is only described for pairs of turns, not for the complete family of models and exploratory analyses.
  • The loss of Grok default in experiment II is 33.75% and may not be random.
  • The counts of observations and self-reports per checkpoint are not reconciled nor is the treatment of partial missingness explained.
  • DeepSeek and Gemini change version between pretest and full; work and education do not use exactly the same set.
  • Provider defaults, aliases, and APIs are mutable; seeds, parameters, revisions, and raw responses are missing.
  • The text calls the evaluator Claude Opus 4.5, but a results table labels it Claude Sonnet 4.5.
  • The work prompt extension mentions school and schoolday; without code it cannot be known whether it is a typo or actual execution.
  • Five psychologists rate only 20 texts; no sampling, protocol, annotations, or discrepancy resolution are published.
  • No ethical review, consent, or compensation of the human evaluators is reported.
  • No people with ADHD participate in the design; stigma, stereotypes, or representational harm are not evaluated.
  • There are no data, code, outputs, environment, preregistration, or official artifact that would allow recalculating the figures.

What the study does not establish

  • That LLMs maintain an internal personality or stable psychological representation.
  • That the profiles are clinically realistic or represent people diagnosed with ADHD.
  • That drift is completely eliminated in every person, model, prompt, or scenario.
  • That the script is causal due to its structure and not due to repeatedly recalling and eliciting symptoms.
  • That there is an inherent bias of the default student toward ADHD; the control contains explicit instructions about the construct.
  • That a stable response to questionnaires predicts stable or valid behavior.
  • That the moderate level is architecturally unstable or reflects a gap in the training data.
  • That the 97% generalizes beyond the high education profile and that descriptive difference.
  • That p>.05 demonstrates absence or practical equivalence.
  • That the comparison of 20 narratives provides ecological validity for classrooms, tutoring, or intervention.
  • That teachers or students benefit from these simulations.
  • That the figures can be reproduced from a clean environment.
  • Revised peer acceptance.

Traceability

Scope: Full text

Version: arXiv:2605.06307v2; 28-page PDF and complete v1-v2 TeX comparison; no official code or data artifact found

Consulted source: https://arxiv.org/abs/2605.06307v2

Review: Codex 28-page visual full-text, v1-v2 TeX, prompt/control, construct, measurement, statistics, missingness, ethics and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Claude Opus 4.5 as persona model; manuscript conflict with Claude Sonnet 4.5 in evaluator table
  • DeepSeek v3.2 Thinking in pretest and v4 Flash in full collection
  • GPT 5.1
  • Gemini 3 Pro in pretest and 3.1 Pro in full collection
  • Grok 4.1 Fast
  • Three LLM observers from Anthropic, OpenAI and Google

Instruments and metrics

  • CAARS-derived self-report prompt with 26 administered items
  • CAARS-derived observer prompt with 30 administered items
  • Reported 12-item self ADHD Index, range 0-36
  • Reported observer ADHD Index, range 0-30, with scored subset undocumented
  • ICC(2,1) inter-rater reliability
  • Within-condition standard deviation
  • Linear mixed-effects models with conversation random intercept
  • Scripted and unscripted conversation prompts

Data used

  • Experiment I generated narratives and ratings; not publicly released
  • Experiment II generated conversations and checkpoint ratings; not publicly released
  • Twenty-narrative human rater validation sample; not publicly released

Evidence and location

  • Text, figures, prompts, questionnaires, samples, descriptives, and models: arXiv:2605.06307v2; PDF sha256 a6ca396b472a3b7a9facab1728a3386b92747293d60ced346b44b6b0fbb121b4; TeX sha256 6b8e27df2938ab2c4c602045815d51020bebf470cc58b04600887144788683b4
  • Stability of central results across versions: arXiv source v1 sha256 799ce389ae42ef02138e78b5a0c6029f8d6172f78fcf83d5f338ca154075008e; v2 sha256 dbb38603357b05b7957ebcbb3316a7e8001e4dd573d09321efd086da967937fb
  • Contaminated control, calculation of 97%, pooled model, missingness, measurement, and reproducibility: reports/verification/article-347-adhd-student-persona-stability-control-contamination-measurement-statistics-and-reproducibility-audit.json