Synthetic Personality Review

How Frontier LLMs Adapt to Neurodivergence Context: A Measurement Framework for Surface vs. Structural Change in System-Prompted Responses

Applications, bias, and safety2026arXivApproved editorial review

Authors: Ishan Gupta, Pavlo Buryi

Keywords: Persona conditioning, Neurodivergence, Prompt adaptation, Safety evaluation, LLM-as-judge, Measurement reliability, Human-Computer Interaction, Reproducibility audit

Source: Open primary source (opens in a new tab)

2
Authors
8
Findings
18
Limitations
3
Evidence

Editorial summary

English

This preprint introduces NDBench, an audit of how GPT-5 Chat and Claude Sonnet 4.6 answer 24 English queries when the system prompt contains no context, one of four synthetic neurodivergent profiles, or that profile plus adaptation directives. The repository reconciles to 576 successful responses and 1,152 successful LLM judgments. In the released analysis, the directive condition averages 83.8 more tokens, 2.24 more headings and 12.59 more words per step than control. The two judges achieve the study's reliability threshold only for masking reinforcement (alpha 0.808) and validation quality (0.700). For masking, profile context without the safeguard has no pooled effect, while an explicit prohibition on advising users to seem normal, act neurotypical or mask lowers means by 44% for Claude and 36% for GPT on six adversarial prompts. The defensible interpretation is narrow: the two models follow detailed preferences and explicit directives, including an anti-masking safeguard. The study does not show that they independently infer appropriate neurodivergence adaptation. The condition labeled persona only already states communication, format, detail and coaching preferences plus free text requesting features such as numbered lists, headings, bullets and whitespace; structural changes therefore cannot be attributed to declaring ADHD, autism, dyslexia or AuDHD. C2 additionally orders headings, lists, granular steps, validation and anti-masking language, making the metrics primarily instruction-following manipulation checks. No neurodivergent participants evaluated the profiles or answers, and the study measures no usability, accessibility, preference, task success or real-world outcome. Stability is also overstated: the four C0 profile cells send identical requests, yet all 48 model-query groups produce multiple distinct texts at temperature zero; C1 and C2 have only one draw per cell. The mixed model treats 576 outputs as rows with query_id as the sole random intercept, omits profile and interactions, and cannot justify class-level claims from two mutable endpoints. Released code does not implement the promised C2-C1 contrast, Cohen's d, bootstrap intervals or variance check. Four harm dimensions fail reliability, and manuscript prose retains agreement and masking numbers that conflict with the CSV tables. Despite these limits, releasing prompts, responses, judgments and analysis makes NDBench a useful artifact for auditing prompt compliance and safeguards, not clinical or population evidence about neurodivergent people.

Español

Este preprint presenta NDBench, un audit de cómo GPT-5 Chat y Claude Sonnet 4.6 responden a 24 consultas en inglés cuando el system prompt no contiene contexto, incorpora uno de cuatro perfiles neurodivergentes sintéticos o añade además directivas de adaptación. El repositorio permite reconciliar 576 respuestas correctas y 1.152 juicios LLM. En el análisis publicado, la condición con directivas produce en promedio 83,8 tokens, 2,24 encabezados y 12,59 palabras por paso más que el control. Los dos jueces coinciden suficientemente sólo en refuerzo del enmascaramiento (alpha 0,808) y calidad de validación (0,700). En la primera métrica, declarar el perfil sin la salvaguarda no cambia el resultado agregado, mientras la prohibición explícita de aconsejar parecer normal, actuar de forma neurotípica o enmascararse reduce la media 44% en Claude y 36% en GPT sobre seis prompts adversariales. La interpretación correcta es estrecha: los dos modelos obedecen preferencias detalladas y directivas explícitas, incluida una salvaguarda anti-enmascaramiento. No se demuestra que infieran por sí solos una adaptación adecuada a la neurodivergencia. La condición llamada sólo persona ya especifica preferencias de comunicación, formato, detalle, tono y texto libre, como listas numeradas, encabezados, viñetas o espacios; por eso el aumento estructural no puede atribuirse a declarar ADHD, autismo, dislexia o AuDHD. En C2, además, el propio prompt ordena usar encabezados, listas, pasos, validación y lenguaje anti-enmascaramiento, de modo que las métricas son principalmente controles de seguimiento de instrucciones. No participaron personas neurodivergentes y no se midieron utilidad, accesibilidad, preferencia, éxito de tarea ni resultados reales. El diseño también sobrestima la estabilidad: las cuatro celdas de perfil de C0 envían exactamente la misma solicitud y generan textos distintos en los 48 grupos modelo-consulta, pese a usar temperatura cero; C1 y C2 sólo tienen una generación por celda. El modelo estadístico usa 576 filas con sólo query_id como intercepto aleatorio, omite perfil e interacciones y no justifica generalización desde dos endpoints. El código no implementa el contraste C2-C1, Cohen d, intervalos bootstrap ni la comprobación de varianza prometidos. Cuatro métricas de daño fallan el umbral de fiabilidad, y el texto conserva cifras de acuerdo y enmascaramiento que contradicen sus CSV. Aun con estos límites, la publicación de prompts, respuestas, juicios y análisis convierte NDBench en un artefacto útil para auditar cumplimiento de prompts y salvaguardas, no en evidencia clínica o poblacional sobre personas neurodivergentes.

Research question

Do two frontier conversational models change structure, surface style, and certain potentially harmful patterns when the system prompt includes a synthetic neurodivergent profile and explicit adaptation directives?

Method

Design labeled as factorial of two models, three conditions, four profiles, and 24 queries. C0 sends no system prompt; C1 includes neurotype, traits, and detailed preferences for communication, format, detail, and tone; C2 adds four directives for structure, decomposition, nonconformity, and validation followed by action. Responses are generated once at temperature zero. Deterministic metrics count tokens, lines, lists, headings, sentences, readability, softeners, emoji, and VADER. GPT-5 Chat and Claude Sonnet 4.6 score six harm/validation dimensions. The analysis uses mixed models with condition and model as fixed effects and query_id as the sole random intercept. The audit visually read the 15 pages, inspected TeX, configuration, 661 attempt records, 1,159 judge records, and all code, and reran metrics and analysis.

Sample: The correct artifact has 576 responses: 2 models x 3 conditions x 4 profiles x 24 queries, one output per labeled cell. C0 contains 192 calls but only 48 unique model-query combinations because the profile is not sent; its four nominal calls per profile are stochastic replicates. The judges provide 1,152 correct scores, two per response. There are no human participants or sample of neurodivergent people.

Findings

  • The repository reconciles exactly 576 correct cells and 1,152 correct response-judge pairs; it also retains 85 and seven rate limit failures, respectively.
  • C2 increases in the aggregated model by 83.8 tokens, 2.24 headings, and 12.59 words per step relative to C0.
  • List density barely changes in aggregate, while headings and depth per step increase; these measures describe textual form, not semantic utility.
  • Masking reinforcement does not change between C1 and C0, but drops 0.263 points in C2; the means decrease from 0.557 to 0.312 in Claude and from 0.776 to 0.495 in GPT.
  • Judged validation quality increases from 1.880 to 2.760 in Claude and from 1.703 to 3.312 in GPT between C0 and C2.
  • Only masking (alpha 0.808) and validation (0.700) exceed the 0.67 threshold; the other four alphas range from -0.704 to 0.010.
  • In none of the 48 C0 model-query groups are the four responses identical: 43 have four distinct texts and five have three.
  • Local rerun preserves means, alphas, and coefficients, although unfixed dependencies change standard errors, p-values, serialization, and PDFs.

Limitations

  • C1 does not isolate a neurotype declaration: it already conveys explicit preferences for format, detail, communication, and tone.
  • C2 directly orders the properties that are then counted; the structural and anti-masking results are primarily prompt obedience controls.
  • The four profiles are synthetic composites without community co-design, clinical validation, individual heterogeneity, or population representativeness.
  • There is no human evaluation or measures of utility, accessibility, preference, task success, health, or real harm.
  • Only 24 queries in English and six masking probes are tested.
  • C1 and C2 have a single output per cell; temperature zero does not eliminate variation and the check at temperature 0.7 promised in spec.md does not exist.
  • C0 crosses a profile that is not sent and treats four stochastic replicates as profile levels.
  • The mixed model uses only query_id as a random intercept, omits profile, interactions, and repetition structure, and uses asymptotic z with 24 queries.
  • Two proprietary and mutable models do not allow inferring behavior of frontier LLMs as a class.
  • The code does not compute the advertised C2-C1 contrast, Cohen d, or bootstrap intervals.
  • Holm is applied only to two contrasts within each metric, not to the family of 17 metrics.
  • Infantilization, stereotypes, rejection, and pathologization fail the agreement threshold and do not admit primary interpretation.
  • The prose gives old alphas 0.835/0.735 versus 0.808/0.700 from the table and C2 masking figures 0.40/0.29 versus 0.50/0.31 from the data.
  • The adjusted p for per-step granularity is 1.080034e-8, slightly higher than the less than 1e-8 claimed in the abstract.
  • Dependencies only have lower bounds; lockfile, Python version, container, tests, and CI are missing.
  • gpt-5-chat-latest is a moving alias and no provider fingerprint is retained.
  • Half of the judgments are self-judgments by the same audited model, not a clean separation between generator and evaluator.
  • The README contains a nonexistent command and the declared MIT/CC-BY licenses have no LICENSE file.

What the study does not establish

  • That the models know how to autonomously adapt their responses to a neurodivergent person from the neurotype.
  • That more headings, tokens, or steps imply better content, greater accessibility, or user preference.
  • That the profiles represent the diversity of people with ADHD, autism, dyslexia, or AuDHD.
  • That the responses are clinically appropriate, therapeutic, or beneficial.
  • That the four dimensions with low agreement improve or worsen between conditions.
  • That the safeguard reduces harms other than masking advice in the constructed probes.
  • That the effects are stable across runs, languages, queries, versions, or providers.
  • That two endpoints constitute a sufficient sample to generalize to frontier LLMs.
  • That NDBench estimates experiences, preferences, risks, or outcomes of real neurodivergent people.

Traceability

Scope: Full text

Version: arXiv:2605.00113v1; repository commit a0f64fd08ab3b70593470c66399bd069a8ef0ffb

Consulted source: https://arxiv.org/abs/2605.00113

Review: Codex 15-page visual full-text, prompt, response-cache, dual-judge, statistical, code and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • gpt-5-chat-latest as audited response model
  • claude-sonnet-4-6 as audited response model
  • gpt-5-chat-latest as LLM judge
  • claude-sonnet-4-6 as LLM judge

Instruments and metrics

  • Four synthetic neurodivergence profile prompts
  • Twenty-four English benchmark queries
  • Six masking-bait adversarial prompts
  • Deterministic structural and surface text metrics
  • VADER sentiment
  • TextStat readability
  • Six-dimension dual-LLM harm rubric
  • Krippendorff alpha
  • Linear mixed-effects regression with query random intercept
  • Holm correction across two contrasts within each metric

Data used

  • NDBench 576 successful response cells
  • Response cache with 661 attempts, including 85 rate-limit failures
  • 1,152 successful dual-judge scores
  • Judgment cache with 1,159 attempts, including seven rate-limit failures
  • Released structural, surface and harm metric tables
  • Released prompts, code, figures and LaTeX source

Evidence and location

  • Full text, tables, figures, limitations, and appendix prompts: arXiv:2605.00113v1; PDF sha256 c710a73af7c0fff46fb32ba1469a8c23e0d3921d5f29a4b658bac57a58da6a75
  • Prompts, profiles, responses, judgments, metrics, and audited analysis: GitHub ishansgupta/ndbench commit a0f64fd08ab3b70593470c66399bd069a8ef0ffb; archive sha256 a03405d9f4b2a49bbd9977f10861a414601e8a6ae886458abdd556d0b5f9d5fa
  • Audit of persona-directive confounding, control, replicates, statistics, judges, and reproducibility: reports/verification/article-352-ndbench-persona-directive-confound-control-replication-statistics-judge-reliability-and-reproducibility-audit.json