Synthetic Personality Review

Elder-Sim: A Psychometrically Validated Platform for Personality-Stable Elderly Digital Twins

Personas, identity, and agents2026arXivApproved editorial review

Authors: Jiaqing Wang, Zhongfang Yang, Xingyuan Zhu, Zong'an Huang, Hao Wang, Li Tian, Ying Cao, Xiaomin Qu, Xiang Qi, Bei Wu, Zheng Zhu

Keywords: Personality, Persona conditioning, Psychometrics, Human simulation

Source: Open primary source (opens in a new tab)

11
Authors
7
Findings
30
Limitations
5
Evidence

Editorial summary

English

ELDER-SIM describes an elderly-care agent architecture combining explicit OCEAN profiles, short- and long-term memory, a Beck-inspired cognitive conceptualization diagram, and LoRA adaptation. The study reports 1,200 synthetic responses from six profiles, ten scenarios, five repetitions and four cumulative conditions; its tables show little alpha change from memory, an increase from 0.702 to 0.892 with CCD, and 0.940 with LoRA, alongside rising ICC and role-discrimination values. However, responses, prompts, rubric, scorer identity, score matrices and code are not public: the availability statement literally retains '[repository URL]'. The LoRA condition also starts from Qwen2.5-7B while the general experiment specifies Qwen2.5-14B, and the statistical table is not internally reproducible: displayed alphas do not yield the reported t values, those t statistics with four degrees of freedom do not yield the stated Bonferroni p values, and the ICC/F pairs imply two measurements despite five declared repetitions. The paper is best read as an unreviewed proposal for structuring and evaluating synthetic personas, not a psychometric validation of individual digital twins or evidence of clinical readiness.

Español

ELDER-SIM describe una arquitectura de agentes para cuidados de personas mayores que combina perfiles OCEAN explícitos, memoria corta y persistente, un diagrama de conceptualización cognitiva inspirado en Beck y adaptación LoRA. El estudio informa 1.200 respuestas sintéticas de seis perfiles, diez escenarios, cinco repeticiones y cuatro condiciones acumulativas; en sus tablas, la memoria apenas cambia alfa, CCD eleva la consistencia media de 0,702 a 0,892 y LoRA alcanza 0,940, con ICC y discriminación de roles también crecientes. Sin embargo, no se publican respuestas, prompts, rúbrica, identidad del evaluador, matrices de puntuación ni código: la declaración de disponibilidad conserva literalmente '[repository URL]'. Además, la condición LoRA parte de Qwen2.5-7B mientras el resto se configura con Qwen2.5-14B, y la tabla estadística no es internamente reproducible: los alfa mostrados no producen los t publicados, sus t con 4 grados de libertad no producen las p Bonferroni indicadas y los pares ICC/F implican dos mediciones pese a declararse cinco repeticiones. Debe leerse como una propuesta no revisada por pares para estructurar y evaluar personas sintéticas, no como validación psicométrica de gemelos digitales individuales ni evidencia de preparación clínica.

Research question

Can persistent memory, a structured cognitive model, and domain adaptation reduce the apparent personality drift in synthetic eldercare agents, measured through repeated OCEAN scores and role separation?

Method

The platform uses n8n with local inference through Ollama/vLLM, JSON profiles with OCEAN targets, short-term memory, MySQL, and a CCD that links history, beliefs, coping strategies, triggers, thoughts, emotions, and behaviors. It compares Baseline, +Memory, +CCD, and +LoRA across six synthetic profiles, ten scenarios, and five generations per scenario. An unpublished rubric assigns OCEAN scores from 1 to 5; Cronbach's alpha, ICC of absolute agreement, role classification metrics, and a coherence score not defined in the methods are reported. The general configuration names Qwen2.5-14B Q8_0, but LoRA is trained on Qwen2.5-7B-Instruct at 4 bits with 19,717 pairs derived from CHARLS.

Sample: Six synthetic profiles, three of older adults and three of healthcare professionals, receive ten scenarios five times in each of four cumulative conditions: 300 responses per condition and 1,200 in total. No older adults, patients, or professionals participate as evaluators. The exact unit of the alpha, ICC, and classification calculations is not documented.

Findings

  • The tables report mean alpha 0.702 in Baseline, 0.705/0.706 with memory, 0.892 with CCD, and 0.940 with LoRA.
  • The published ICCs grow from 0.856 to 0.958 and role discrimination from 83.3% to 97.2% across cumulative conditions.
  • Memory alone produces a minimal change in internal consistency, while CCD concentrates the largest reported increase.
  • LoRA achieves the highest published values, but also changes the base model from 14B to 7B and does not constitute an isolated ablation.
  • Response coherence appears in results as 3.2/5, 3.5/5, 4.6/5, and 4.8/5 without a measurement protocol.
  • The exposed values do not allow reproducing the t tests or the corrected p values of Table 13.
  • The work provides a detailed conceptual design to separate memory and cognitive structure, but not an auditable empirical package.

Limitations

  • The work is a preprint not certified by peer review; medRxiv warns that it should not guide clinical practice.
  • Code availability is a '[repository URL]' marker and not a functional link.
  • The 1,200 responses, OCEAN scores, role predictions, confusion matrices, and analysis data are not published.
  • The exact prompts, behavioral rubric, and the identity or nature of the evaluator are missing.
  • There is no blind human coding, multiple evaluators, inter-rater reliability, or external personality criterion.
  • The general condition uses Qwen2.5-14B, but LoRA is trained on Qwen2.5-7B; the contrast changes more than the adapter.
  • The conditions are cumulative and do not include CCD alone, LoRA alone, or a matched 7B control.
  • Roles, demographics, occupation, health, OCEAN, and behavioral restrictions vary together across only six profiles.
  • OCEAN targets and behavioral rules are included in the prompt, so discriminating them from the responses is not independent validation.
  • It is not defined which items and observations enter each alpha, nor is unidimensionality of the ten heterogeneous scenarios demonstrated.
  • The ICC does not specify model, unit, occasions, aggregation, degrees of freedom, or confidence interval method.
  • The published ICC/F pairs imply approximately two measurements under the standard relation, not the five declared repetitions.
  • Five repeated generations without a temporal interval do not equate to longitudinal stability of a person.
  • It is not reported whether memory is reset between repetitions, scenarios, profiles, or conditions; retention of previous responses could inflate stability.
  • Order, randomization, counterbalancing, seeds, failures, retries, and execution dates are not declared.
  • The +Memory mean is 0.705 in one table and 0.706 in another; the five visible values average 0.706.
  • The t tests recalculated on the five visible alphas do not match any of the six published t values.
  • The published p values also do not correspond to the t and df=4 shown after a Bonferroni correction of six comparisons.
  • The df=4 suggests treating the five OCEAN dimensions as replicates, although they are not independent experimental units.
  • The classifier, train/test split, multiclass averaging, and denominator of accuracy, precision, recall, F1, or AUC are not defined.
  • The accuracies fit almost exactly with 30/36, 32/36, 34/36, and 35/36, but the article describes 300 responses per condition and never defines 36 cases.
  • Coherence is introduced only in results, without a rubric, evaluator, uncertainty, or analysis.
  • The 19,717 CHARLS pairs do not include transformation, examples, validation split, leakage control, or checkpoint selection criterion.
  • Only a final training loss near 0.05 is reported, without held-out evaluation.
  • Exact reviews of models, n8n/Ollama/vLLM/Unsloth versions, hardware, environment, workflow, and database snapshot are missing.
  • There is no factor analysis, invariance, convergent, discriminant, predictive, criterion, or ecological validity.
  • The agents are invented profiles and not calibrated digital twins with longitudinal trajectories of real individuals.
  • No older adults, caregivers, or clinicians participate to evaluate realism, utility, acceptability, or harm.
  • Crisis, harmful advice, hallucination, bias, ageism, privacy, or cognitive decline are not evaluated.
  • Ethical review, consent, deidentification, or governance for converting data derived from CHARLS into training conversations are not documented.

What the study does not establish

  • It does not demonstrate complete psychometric validation; it provides reliability values that are not verifiable without external validity.
  • It does not prove that CCD causes the improvement because the conditions are cumulative and LoRA changes the base model.
  • It does not demonstrate longitudinal stability of real older adults or fidelity of individual digital twins.
  • It does not validate role discrimination beyond labels and restrictions explicitly inserted into the prompts.
  • It does not support the p values of the paired comparisons as internally coherent Bonferroni tests.
  • It does not allow reproducing the platform or the results with the current public materials.
  • It does not establish clinical realism, safety, educational efficacy, or utility for testing interventions.
  • It does not generalize beyond six synthetic profiles, ten scenarios, one Qwen family, and a Chinese eldercare context.
  • It does not demonstrate that a stable response is correct, non-stereotyped, or representative of an older adult.
  • It does not prove that CHARLS adaptation improves human fidelity because there is no human criterion or matched 7B base control.

Traceability

Scope: Full text

Version: arXiv:2604.16343v1; medRxiv DOI 10.64898/2026.03.25.26349036

Consulted source: https://arxiv.org/abs/2604.16343

Review: Codex 19-page arXiv plus 19-page medRxiv visual full-text, psychometric-validity, model-ablation, statistical, data/code, clinical and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Qwen2.5-14B Q8_0 como modelo general declarado
  • Qwen2.5-7B-Instruct 4-bit como base distinta de la condición LoRA
  • LoRA r=16, alpha=16, exportada a GGUF Q8_0
  • Ollama
  • vLLM

Instruments and metrics

  • Big Five/OCEAN programado en escala 1-5
  • Cognitive Conceptualization Diagram basado en Beck
  • Rúbrica conductual no publicada para puntuar respuestas
  • Alfa de Cronbach
  • ICC de acuerdo absoluto sin especificación completa
  • Accuracy, precision, recall, F1 y AUC de discriminación de roles
  • Coherencia de respuesta 1-5 sin método publicado

Data used

  • 19.717 pares de instrucciones derivados de CHARLS, no publicados y disponibles solo bajo solicitud
  • 1.200 respuestas sintéticas del experimento, no publicadas
  • Seis configuraciones de agentes y diez escenarios estandarizados, no publicados como artefactos ejecutables

Evidence and location

  • Architecture, design, tables, discussion, limitations, availability, and references: arXiv:2604.16343v1, all 19/19 PDF pages rendered and individually inspected
  • Preprint status, DOI, date, clinical warning, and persistence of the repository marker: medRxiv v1 DOI 10.64898/2026.03.25.26349036, all 19/19 PDF pages rendered and individually inspected; Crossref metadata
  • arXiv version, authors, subjects, date, and license: Official arXiv abstract and Atom metadata inspected 2026-07-17
  • Absence of identifiable repository: Authenticated GitHub repository and code search for ELDER-SIM and the exact title, inspected 2026-07-17
  • Arithmetic reproduction, model conflict, psychometric, clinical, and reproducibility audit: reports/verification/article-392-elder-sim-psychometric-validity-model-ablation-statistics-data-code-clinical-and-reproducibility-audit.json