Math Education Digital Shadows for facilitating learning with LLMs: Math performance, anxiety and confidence in simulated students and AIs

Applications, bias, and safety2026arXivApproved editorial review

Authors: Naomi Esposito, Anthony Tricarico, Luisa Porzio, Ali Aghazadeh Ardebili, Massimo Stella

Keywords: Math Education Digital Shadows, Mathematics education, Persona conditioning, Synthetic students, Math anxiety, Mathematics self-efficacy, Psychometric scales, Forma mentis networks, Confidence calibration, Overconfidence, Dataset integrity, Reproducibility

Source: Open primary source (opens in a new tab)

5
Authors
11
Findings
20
Limitations
3
Evidence

Editorial summary

English

This preprint introduces MEDS, a corpus for studying how 14 nominal LLM folders respond about mathematics under two conditions: as a persona-free AI assistant or while role-playing a synthetic person with sociodemographics, favourite and disliked subjects, and OCEAN traits. Each run is intended to preserve five JSON artifacts covering four tasks: a seven-question interview; three self-efficacy and anxiety scales with justifications; two semantic-association batches; and 18 multiple-choice problems with explanations and confidence. The genuine contribution is the open corpus: it exposes prompts, answers, ratings, associations and metadata for descriptive comparisons across families and modes. The paper reports more varied score distributions under personas, negative mathematics associations in some models, classifier-labelled fallacies and accuracy-confidence gaps. These findings characterize generated text; they do not validate the profiles as realistic students. There are no human participants or population comparisons, and no reliability, factor structure, measurement invariance or criterion validity is reported for treating LLM scale responses as psychometric measurements. EmoAtlas measures emotional vocabulary and DistilBERT labels text; neither establishes felt emotion, anxiety, cognition or human-adjudicated fallacies. Full audit of all 34 pages, TeX source and the public commit finds 139,948 valid JSON files rather than 140,000. There are 28,000 started runs but only 27,987 complete runs: thirteen human-mode Granite runs preserve only the interview. Of 55,974 association outputs, 194 violate the required 25 cues; nine mathematical records replace expected question identifiers with values such as 20, 45, 80, 200 or 360; 23 answers have no reasoning and 59 final summaries are empty. The scale CSV exactly mirrors the 27,987 complete records and 1,287,402 items. In contrast, the demographics CSV has 27,543 personas, only 21,004 of which belong to complete human-mode runs; 6,539 are outside the selection and include two models absent from the final corpus. Ten folders mix model aliases or endpoints without checkpoint digests. The only released code is an unexecuted generation notebook: ROLE_MODES contains human twice, it requests 1,800 sessions with no seed and uses a placeholder model, so it cannot produce the 6,983 baseline LLM runs, the exact 14 environments, selection, cleaning or figures. Code for BERT correction, EmoAtlas, fallacy classification, networks, statistics and calibration is absent. Figure 8 has a further lineage break: public accuracy reproduces 13 of 14 bars, but none of the confidence bars equals the stated mean transformation `(score-1)/4`; Grok is 0.942 in released JSON versus 0.884 in the SVG, and Mistral Small 4 is 0.755 versus 0.508. Ministral 3B accuracy is 0.350 in the figure but 0.486 in the current corpus. Mathematical item 18 also uses BD in its proportion while supplying BC, so the published answer assumes an unwritten correction. MEDS is useful as a synthetic collection for exploratory audits and tool design, but it does not establish human realism, educational effectiveness, tutor safety or real group differences.

Español

Este preprint presenta MEDS, un corpus para estudiar cómo 14 carpetas nominales de LLM responden sobre matemáticas en dos condiciones: como asistente de IA sin persona o representando una persona sintética con sociodemografía, materias favoritas y rechazadas y rasgos OCEAN. Cada ejecución pretende conservar cinco JSON que cubren cuatro tareas: una entrevista de siete preguntas; tres escalas de autoeficacia y ansiedad con justificaciones; dos lotes de asociaciones semánticas; y 18 problemas de opción múltiple con explicación y confianza. El valor real del trabajo es el corpus abierto: permite inspeccionar prompts, respuestas, puntuaciones, asociaciones y metadatos, y comparar descriptivamente familias y modos. La validación del artículo encuentra distribuciones más variadas al usar personas, asociaciones negativas con matemáticas en algunos modelos, salidas clasificadas como falacias y brechas entre exactitud y confianza. Estas observaciones describen texto generado, no validan que los perfiles se comporten como estudiantes reales. No hay participantes humanos ni contraste con una población, y tampoco se presentan fiabilidad, estructura factorial, invariancia o validez criterial de las escalas cuando las responde un LLM. EmoAtlas mide léxico emocional y el clasificador DistilBERT etiqueta texto; ninguno demuestra emoción sentida, ansiedad, cognición o falacias adjudicadas humanamente. La auditoría integral de las 34 páginas, la fuente TeX y el commit público encuentra 139.948 JSON válidos, no 140.000. Hay 28.000 ejecuciones iniciadas, pero sólo 27.987 completas: trece ejecuciones humanas de Granite conservan únicamente la entrevista. De las 55.974 respuestas de asociaciones, 194 incumplen los 25 términos exigidos; nueve registros matemáticos sustituyen identificadores de preguntas por valores como 20, 45, 80, 200 o 360; 23 respuestas carecen de razonamiento y 59 resúmenes finales están vacíos. El CSV de escalas sí reproduce exactamente los 27.987 registros completos y 1.287.402 ítems. En cambio, el CSV demográfico tiene 27.543 personas, de las que sólo 21.004 pertenecen a ejecuciones humanas completas; 6.539 son restos fuera de la selección e incluyen dos modelos ausentes del corpus final. Diez carpetas mezclan alias de modelo o endpoints, sin digest que fije el checkpoint. El único código liberado es un notebook de generación sin ejecutar: fija `ROLE_MODES` a dos valores `human`, genera 1.800 sesiones con semilla nula y un modelo placeholder, por lo que no puede producir las 6.983 ejecuciones base LLM, los 14 entornos exactos, la selección, la limpieza ni las figuras. Tampoco se publica el código de corrección BERT, EmoAtlas, clasificación de falacias, redes, estadística o calibración. La Figura 8 presenta otra ruptura de trazabilidad: la exactitud pública reproduce 13 de 14 barras, pero ninguna barra de confianza coincide con el promedio de `(puntuación-1)/4` definido en el apéndice; Grok da 0,942 en los JSON frente a 0,884 en el SVG y Mistral Small 4 da 0,755 frente a 0,508. Ministral 3B muestra exactitud 0,350 en la figura, pero 0,486 en el corpus actual. Además, el problema 18 usa BD en la proporción y entrega BC como dato, de modo que la respuesta publicada presupone una corrección no escrita. MEDS es útil como colección sintética para auditorías exploratorias y diseño de herramientas, pero no demuestra realismo humano, eficacia educativa, seguridad de tutores ni diferencias reales entre grupos.

Research question

How to build and describe a multimodal corpus of synthetic responses that allows comparing mathematical performance, declared confidence, anxiety, self-efficacy, explanations, and semantic associations of different LLMs under assistant mode and simulated human person mode?

Method

For each of 14 nominal model folders, 2,000 run identifiers are selected, approximately 1,500 in human person mode and 500 in LLM mode. In human mode a profile is generated with age, gender, sexual orientation, city, employment, education, family, migration, religion, hobbies, subjects, and OCEAN; in LLM mode the person is null. Four sequential calls produce an interview of seven questions, the MAES/MSES, AMAS, and MSEAQ scales, 50 associations divided into two batches, and 18 MSES-R problems. The manuscript describes JSON repair, structural controls, BERT similarity, EmoAtlas, forma mentis networks, a DistilBERT fallacy classifier, and descriptive comparison of accuracy and confidence. The audit visually read the 34 pages, inspected TeX, commit, and notebook, censused the 139,948 JSON, reconciled the processed CSVs and the ID manifest, reproduced accuracies, and recalculated confidence with the published formula.

Sample: There are no students or human participants. The public corpus starts with 28,000 runs: 21,017 in human person mode and 6,983 in LLM assistant mode. Thirteen Granite human runs remain incomplete, so the analyzable sample for the four tasks is 27,987: 21,004 simulated human and 6,983 LLM. All folders use 1,500/500 except Mistral Small 4, with 1,517/483. The profiles are random combinations from the generator, not observed persons or a representative sample.

Findings

  • The 139,948 public JSON parse; they represent 28,000 interviews, but only 27,987 complete runs.
  • Thirteen Granite human runs have only call1 and are missing their other 52 files.
  • The scales CSV contains exactly the 1,287,402 items expected for 27,987 complete runs, with no duplicate keys or ratings outside 1-5.
  • One hundred and ninety-four association batches contain 20 or 24 terms instead of 25.
  • Nine call4 substitute ten expected IDs; there are 23 empty reasonings and 59 empty reasoning_summary.
  • The demographic CSV mixes 21,004 final persons with 6,539 persons outside the selection.
  • Ten folders combine aliases or endpoints, so a folder does not always equate to an immutable checkpoint.
  • The public notebook only generates human mode and does not contain the cleaning, validation, or analysis pipeline.
  • Thirteen accuracy bars in Figure 8 reproduce exactly; Ministral 3B changes from 0.350 to 0.486 in the current corpus.
  • None of the 14 confidence bars matches the rescaled mean defined by the article.
  • Persons induce variability in responses, but it is not demonstrated that this variability is realistic or human.

Limitations

  • It is an arXiv v1 preprint and no peer review or acceptance is recorded.
  • There are no human participants, real students, educational outcomes, or population comparison.
  • Psychometric scales are administered to LLMs and their reliability, structure, invariance, or human equivalence is not validated.
  • The paper alternates MSES and MAES for the nine-item scale.
  • EmoAtlas and DistilBERT measure or classify text, not verified emotion, anxiety, cognition, or fallacies.
  • The fallacy cutoff at the 85th percentile is selected by visual inspection and is not validated with human annotation of the corpus.
  • Failed attempts are not published nor is a regeneration manifest that allows reconstructing discards.
  • The corpus retains 13 incomplete runs and 194 semantic batches that the described cleaning should exclude.
  • Nine mathematical sets have substituted IDs; 23 responses have no reasoning and 59 summaries are empty.
  • The demographic CSV contains 6,539 persons outside the final set and two models not included.
  • Ten folders mix identifiers or endpoints and several latest aliases are mutable.
  • The only notebook fixes human/human, 1,800 runs, seed None, and a placeholder model.
  • No code is released for BERT, cleaning, reverse scoring, EmoAtlas, fallacies, networks, statistics, or figures.
  • There is no global LICENSE, requirements, lockfile, tests, CI, release tag, or numerical regressions.
  • The 14 confidence means in Figure 8 are not reproducible with the published formula.
  • The accuracy of Ministral 3B reflects a snapshot different from the current public data.
  • Item 18 uses a variable not provided and requires assuming that BD should have been BC.
  • Ordinal confidence is linearly transformed without calibration curves, Brier score, or ECE.
  • Associations and demographic traits may amplify stereotypes from training data.
  • The article does not demonstrate learning improvement, tutor safety, or fitness for deployment with students.

What the study does not establish

  • That MEDS persons represent students, demographic groups, or real populations.
  • That a run is a digital twin, a persistent individual, or a validated student model.
  • That prompt-induced variability equates to human psychological diversity.
  • That LLM ratings measure anxiety, self-efficacy, confidence, or personality with psychometric validity.
  • That emotional vocabulary implies emotions felt by the model.
  • That automatic labels are real fallacies without human validation.
  • That reasoning or reasoning_summary reveal faithful internal reasoning.
  • That persona attributes cause the observed results.
  • That folders correspond to 14 homogeneous and immutable checkpoints.
  • That the published code regenerates the corpus, the cleaning, or the figures.
  • That Figure 8 is reproducible from the public snapshot with the described formula.
  • That MEDS improves learning, reduces harm, or validates a tutor for real use.

Traceability

Scope: Full text

Version: arXiv:2604.27618v1; repository commit 78259c2c7643f9438af5ad39cc69a9ffd07b06b7

Consulted source: https://arxiv.org/abs/2604.27618

Review: Codex 34-page visual full-text, TeX, 139,948-JSON census, CSV lineage, notebook, model-identity, psychometric, calibration and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • DeepSeek Chat
  • Grok 4.1 Fast Reasoning
  • IBM Granite 4 H Tiny aliases
  • Microsoft Phi-4 Reasoning Plus aliases
  • Qwen3 4B Instruct 2507 aliases
  • Qwen3 4B Thinking 2507 aliases
  • Qwen3 4B Uncensored Unslop v2 aliases
  • Qwen3.5 9B aliases
  • Ministral 3B local aliases
  • Ministral 14B API and local reasoning aliases
  • Mistral Small 3.2 across API, LM Studio and Ollama aliases
  • Mistral Small latest, labelled Mistral Small 4
  • Magistral Small API and local aliases
  • ANITA-NEXT 24B Dolphin Mistral uncensored Italian

Instruments and metrics

  • Seven-question open mathematics interview
  • Nine-item Mathematics Self-Efficacy Scale, named MSES in the paper and MAES in code/data
  • Nine-item Abbreviated Math Anxiety Scale
  • Twenty-eight-item Mathematics Self-Efficacy and Anxiety Questionnaire
  • Fifty free-association cues with 1-5 valence
  • Eighteen-item MSES-R multiple-choice problem set
  • Synthetic persona generator with OCEAN scores
  • BERT semantic-similarity correction described at threshold 0.85
  • EmoAtlas emotional profile extraction
  • DistilBERT Base Fallacy Classification model with visually selected 85th-percentile cutoff
  • Behavioral forma mentis networks
  • Linear confidence rescaling and answer-key accuracy

Data used

  • 139,948 released original JSON files across 14 model folders
  • 28,000 started run IDs and 27,987 complete five-file runs
  • 21,004 complete human-mode and 6,983 complete LLM-mode runs
  • persona_dataset.csv with 27,543 rows, including 6,539 outside the selected corpus
  • filtered_run_ids.json with 2,000 unique IDs per folder, including 13 incomplete Granite runs
  • call2_dataset.csv with 1,287,402 scale-item rows
  • 55,974 call3 association batches and 27,987 call4 problem sets
  • Per-run task-3 edge-list CSV files

Evidence and location

  • Full text, tasks, validation, limits, instruments, schema, and availability: arXiv:2604.27618v1; PDF sha256 5d8b4ae6ea7896bcfc6650008c3ac33dace9400b5b1df82d7fd01cf1f4f87f90
  • Corpus, CSV, manifest, notebook, model identities, and figures: GitHub MassimoStel/MEDS commit 78259c2c7643f9438af5ad39cc69a9ffd07b06b7; tree 470ed5b6908902d00df5d566a54dd5dcff28eb1a
  • Census of 139,948 JSON, task contracts, metrics, and reconciliation of processed files: reports/verification/article-356-meds-corpus-schema-psychometric-calibration-code-and-reproducibility-audit.json