SPASM: Stable Persona-driven Agent Simulation for Multi-turn Dialogue Generation

Personas, identity, and agents2026arXivApproved editorial review

Authors: Han Luo, Guy Laban

Keywords: Persona conditioning, Human simulation

Source: Open primary source (opens in a new tab)

2
Authors
9
Findings
30
Limitations
7
Evidence

Editorial summary

English

Luo and Laban present SPASM, a pipeline for producing synthetic dialogue between a persona-enacting Client and a Responder. The system samples a structured profile, validates and verbalizes it with LLMs, generates the conversation, and uses another detector to decide when it ends. Its main technical contribution, Egocentric Context Projection (ECP), keeps a history with absolute speakers and rewrites it for each agent as SELF and PARTNER. This prevents the same turn from receiving contradictory relative role labels for the two participants.

The reported corpus crosses GPT-4o-mini, DeepSeek-V3.2, and Qwen-Plus in nine pairings. Each pairing uses 500 independently sampled personas and ten conversations per persona: 4,500 profiles and 45,000 conversations. The authors find more compact persona clusters for same-backbone interactions and especially favorable geometry when GPT-4o-mini is the Responder. Recovering persona identity from embeddings of Client turns reaches Acc@1 from .50 to .99 and Acc@10 from .82 to 1.00.

The most direct ECP evidence comes from a separate experiment: 50 personas, three conversations per persona, deterministic decoding, and a 20-utterance cap. Three questions about concerns, emotion, and motivation are answered before and during dialogue; embedding distance from the initial response is called drift. ECP lowers this distance in eight of nine model-dimension comparisons, with deltas from -.006 to -.042 and Cohen d from -.05 to -.75. For echoing, a Qwen-max judge screens conversations and two humans look for adoption of the partner's role. CONCAT human rates range from 9% to 82%; no human-positive ECP case is observed, although the judge flags 3%-24%.

This is promising, but claiming that ECP eliminates echoing exceeds a finite observation. The paper omits per-cell denominators, intervals, and an unambiguous account of work allocation between annotators. It evaluates all ECP conversations but only 50 CONCAT conversations per pairing, and the checklist confirms missing recruitment, compensation, consent, and ethics-review information. It also does not measure factuality, safety, usefulness, realism, or preference: better role separation is insufficient to call dialogues about mental health, law, or finance high quality.

There are reproducible quantitative problems. With 500 personas and ten conversations, exact chance Acc@1/3/5/10 is .0018/.0054/.0090/.0179. The table reports .02/.05/.09/.17, values corresponding to 50 personas; no such subset is disclosed. Methods retain 50 PCA components, but results attribute 68%-77% variance to the first two. Table 1 defines pairwise distances, while the appendix computes distances to centroids. That calculation includes the query in its own centroid and then applies ANOVA to paired within and between distances as if they were independent.

Backbone comparisons are also unidentified: each of the nine conditions has different personas, so model, occupations, domains, emotions, Crafter wording, and generations change together. Claimed Responder dominance is a descriptive pattern across nine unreplicated corpora, not a causal effect. Drift relies on three generated self-reports; semantic distance can reflect paraphrase or contextual adaptation rather than identity loss. The appendix proves geometric properties of cosine distance, not psychological construct validity.

The current repository does not reproduce the paper. Created on July 12, 2026 in two commits, it contains no code, GUI, environment, original corpus, annotations, or results. It publishes a later, different DeepSeek-V4-Flash/DeepSeek-V4-Flash dataset of 5,000 conversations. That JSONL is internally consistent, 500 personas with ten conversations each, 44,950 utterances, no exact duplicates, and a correct checksum, but belongs to none of the nine studied configurations. SPASM contributes a simple and plausible perspective-normalization technique; available evidence supports observed drift reductions and zero human-positive cases in the inspected samples, not a universally stable, echoing-free, publicly reproducible system.

Español

Luo y Laban presentan SPASM, un pipeline para producir diálogos sintéticos entre un Client que representa una persona y un Responder. El sistema muestrea un perfil estructurado, lo valida y redacta con LLM, genera la conversación y usa otro detector para decidir cuándo termina. Su aportación técnica principal, Egocentric Context Projection (ECP), conserva un historial con hablantes absolutos y lo reescribe para cada agente como SELF y PARTNER. La idea evita que un mismo turno aparezca con etiquetas de rol contradictorias para los dos participantes.

El corpus declarado cruza GPT-4o-mini, DeepSeek-V3.2 y Qwen-Plus en nueve pares. Cada par usa 500 personas muestreadas independientemente y diez conversaciones por persona: 4.500 perfiles y 45.000 conversaciones en total. Los autores encuentran clusters de persona más compactos cuando los backbones coinciden y una geometría especialmente favorable cuando GPT-4o-mini actúa como Responder. La recuperación del identificador de persona desde embeddings de los turnos del Client alcanza Acc@1 entre .50 y .99 y Acc@10 entre .82 y 1.00.

La evidencia más directa sobre ECP procede de un experimento separado: 50 personas, tres conversaciones por persona, decodificación determinista y máximo de 20 turnos. Tres preguntas sobre preocupaciones, emoción y motivación se responden antes y durante el diálogo; la distancia de embeddings respecto a la respuesta inicial se llama drift. ECP reduce esa distancia en ocho de nueve comparaciones modelo-dimensión, con delta de -.006 a -.042 y Cohen d de -.05 a -.75. En echoing, un juez Qwen-max filtra conversaciones y dos anotadores humanos buscan adopción del rol del compañero. CONCAT presenta tasas humanas del 9% al 82%; en las muestras ECP no se observa ningún positivo humano, aunque el juez marca 3%-24%.

Ese resultado es prometedor, pero «elimina echoing» excede una observación finita. El paper no da denominadores por celda, intervalos ni una descripción inequívoca del reparto entre anotadores. Evalúa todo ECP pero solo 50 conversaciones CONCAT por par, y el checklist confirma que no informa reclutamiento, compensación, consentimiento ni revisión ética. Tampoco mide factualidad, seguridad, utilidad, realismo o preferencia: separar mejor los roles no basta para llamar de alta calidad a diálogos sobre salud mental, derecho o finanzas.

Hay además problemas cuantitativos verificables. Con 500 personas y diez conversaciones, el azar exacto para Acc@1/3/5/10 es .0018/.0054/.0090/.0179. La tabla publica .02/.05/.09/.17, valores que corresponden a 50 personas; el texto no declara ese subconjunto. Métodos dicen retener 50 componentes PCA, pero resultados atribuyen el 68%-77% a los dos primeros. La nota de la Tabla 1 define distancias por pares, mientras el apéndice las calcula contra centroides. Ese cálculo incluye el propio punto en su centroide y después aplica ANOVA a distancias within y between emparejadas como si fueran independientes.

La comparación entre backbones tampoco está identificada: cada una de las nueve condiciones tiene personas diferentes, por lo que modelo, ocupaciones, dominios, emociones, redacción del Crafter y generaciones cambian a la vez. La supuesta dominancia del Responder es un patrón descriptivo de nueve corpus sin réplica, no un efecto causal. Y el drift se basa en tres autorreportes generados; una distancia semántica puede reflejar paráfrasis o adaptación al contexto, no pérdida de identidad. El apéndice demuestra propiedades geométricas del coseno, no validez psicológica del constructo.

El repositorio actual tampoco reproduce el paper. Fue creado el 12 de julio de 2026 con dos commits y no contiene código, GUI, entorno, corpus original, anotaciones ni resultados. Publica un dataset posterior y distinto: 5.000 conversaciones DeepSeek-V4-Flash/DeepSeek-V4-Flash. Ese JSONL sí es internamente consistente, 500 personas por diez conversaciones, 44.950 turnos, sin duplicados exactos y checksum correcto, pero no pertenece a ninguna de las nueve configuraciones estudiadas. SPASM aporta una técnica sencilla y plausible para normalizar perspectiva; la evidencia disponible apoya una reducción observada de drift y cero casos humanos en las muestras, no un sistema universalmente estable, libre de echoing o reproducible desde sus artefactos públicos.

Research question

Can an egocentric projection that relabels the history as SELF/PARTNER reduce persona drift and adoption of the interlocutor's role in LLM-LLM dialogues, and how do corpus representations vary across backbone pairs?

Method

Synthetic pipeline with nine Client-Responder pairs formed by three APIs. 500 personas and ten conversations per pair are generated. Client turns are embedded and analyzed with PCA, clustering, ANOVA, and nearest-neighbor retrieval. An ECP-CONCAT ablation uses 50 personas, three conversations, zero temperature, and periodic semantic probes. Qwen-max filters echoing and two humans annotate full ECP, 50 CONCAT samples per pair, and a double-annotated subset of 200.

Sample: The main study declares 500 independent personas per each of nine pairs and ten conversations per persona: 45,000 conversations, cap of 25 turns per agent. The ablation uses 50 personas and three conversations per condition, maximum 20 utterances. For echoing, all ECP receives human annotation and CONCAT only 50 conversations per pair; 200 conversations are double-annotated, but the exact split and ECP denominators are not published.

Findings

  • ECP reduces embedding drift in eight of nine model-dimension rows; delta -.006 to -.042 and Cohen d -.05 to -.75.
  • DeepSeek motivation is the only non-significant comparison, p=.460; the paper does not identify the test or correct nine comparisons.
  • CONCAT has 9%-82% human echoing depending on the pair; ECP registers 0% observed human, although the judge marks 3%-24%.
  • Human agreement on 200 conversations is .920 with kappa .729.
  • The judge against averaged human references reports agreement .860, precision .974, recall .861, and F1 .914.
  • Persona Acc@1 ranges from .50 to .99 and Acc@10 from .82 to 1.00 across pairs.
  • The published geometry is more compact in several same-backbone pairs and when GPT-4o-mini responds, but persona composition changes across conditions.
  • The random retrieval baseline matches 50 personas and not the 500 declared.
  • The subsequent public dataset contains exactly 5,000 records, 500 personas, 44,950 utterances, ten conversations per persona, and no exact duplicate.

Limitations

  • Each backbone pair uses 500 independently sampled personas; model differences are confounded with persona composition and wording.
  • There is only one corpus per cell, with no seed replicates or factorial model that identifies Client and Responder effects.
  • The geometry uses external text embeddings from the Client, not latent states of the conversational backbones.
  • Ten conversations share the same persona description and scenario; retrieval may exploit repeated prompt content.
  • The random baseline .02/.05/.09/.17 corresponds exactly to 50 personas, not 500; the subset or error is not declared.
  • The curve at K=50 also matches chance for 50 personas, reinforcing the scope discrepancy.
  • Methods retain 50 PCA components, but results call 68%-77% the variance of the first two.
  • Table 1 defines pairwise distances and Appendix D distances to centroids; these are distinct metrics.
  • The own centroid includes the query and mechanically reduces within distance; there is no leave-one-out.
  • ANOVA ignores that within and between are pairs of the same text and depend on estimated centroids.
  • p<10^-20 do not provide causal effect size nor validate that persona explains a substantive fraction.
  • The three probes are generated self-reports, not internal access or a validated instrument of personality stability.
  • Cosine distance may measure paraphrase or contextual adaptation; there is no human criterion, reliability, or calibration of drift.
  • The theoretical justification demonstrates cosine geometry, not psychological or behavioral validity.
  • No exact serialization of ECP or CONCAT is published and a comparison with strong chat template or per-agent history baselines is missing.
  • Drift tests do not specify test, unit, pairing, intervals, multiple correction, or exact standardized effect.
  • Observations are repeated by persona, conversation, turn, and dimension, with risk of pseudoreplication.
  • The bands in Figure 2 are not defined as SD, SE, CI, or bootstrap.
  • Echoing uses asymmetric protocols: full coverage for ECP and only 50 examples per pair for CONCAT.
  • Zero observed positives does not prove elimination and no per-cell denominators or binomial intervals are given.
  • It is unclear whether both annotators covered all ECP or split the coverage, nor how disagreements were resolved.
  • Binary references are averaged, but it is not explained how a .5 disagreement becomes ground truth for precision and recall.
  • Humans are blind to the judge, not necessarily to condition, backbone, or hypothesis.
  • Recruitment, compensation, consent, or ethical review of the annotators is not reported.
  • Factuality, safety, helpfulness, realism, diversity, preference, or task success are not evaluated.
  • The termination detector may change exposure to late turns; lengths and censoring by condition are not reported.
  • Synthetic personas and details added by the Crafter do not represent real populations or maintain a clean factorial control.
  • Dated versions of APIs, seeds, retries, environment, calls, costs, and an immutable manifest are missing.
  • The repository contains no code nor any of the nine corpora, embeddings, annotations, or outputs from the paper.
  • The only public dataset is subsequent, uses DeepSeek-V4-Flash, and does not reproduce the experiments.

What the study does not establish

  • It does not demonstrate that ECP eliminates echoing outside the observed samples, models, prompts, and runs.
  • It does not causally identify the Responder as the determinant of geometry because personas change across pairs.
  • It does not demonstrate alignment of latent spaces; it only compares external embeddings of textual outputs.
  • It does not validate the drift score as a psychological measure, internal identity, or human personality.
  • It does not test that retrieval of a persona implies behavioral stability in new contexts.
  • It does not resolve whether retrieval uses 50 or 500 personas nor validate its published random baseline.
  • It does not justify ANOVA inference with paired observations and dependent centroids.
  • It does not demonstrate global dialogue quality or safety for mental health, law, or finance.
  • It does not prove representativeness of profiles or similarity with human conversations.
  • It does not allow reproduction of generation, ablation, annotation, or results with the public artifacts.
  • It does not convert the subsequent DeepSeek-V4-Flash dataset into evidence for the nine configurations of the paper.

Traceability

Scope: Full text

Version: Findings of ACL 2026, Anthology 2026.findings-acl.412, DOI 10.18653/v1/2026.findings-acl.412; arXiv:2604.09212v1, source package and official checklist also audited

Consulted source: https://arxiv.org/abs/2604.09212

Review: Codex 21-page official Findings visual full-text, two-page checklist, TeX/source, persona-sampling, retrieval-baseline, drift-construct, echoing-human-protocol, statistics, GitHub, released-JSONL data-quality and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o-mini
  • DeepSeek-V3.2
  • Qwen-Plus
  • Qwen-max echoing judge
  • OpenAI text-embedding-3-large
  • DeepSeek-V4-Flash in the later public dataset only

Instruments and metrics

  • Egocentric Context Projection
  • CONCAT history baseline
  • Persona Validator and Persona Crafter
  • LLM Termination Detector
  • Three concerns, emotion and motivation drift probes
  • Cosine embedding distance and drift AUC
  • PCA with 50 retained components
  • Silhouette score
  • Davies-Bouldin index
  • One-way ANOVA on within/between distances
  • Top-K nearest-neighbor persona retrieval
  • Qwen-max binary echoing screen
  • Two-annotator binary echoing protocol
  • Observed agreement and Cohen kappa
  • Judge precision, recall and F1

Data used

  • Nine reported 5,000-conversation model-pairing corpora
  • 45,000 total synthetic conversations
  • 4,500 independently sampled persona profiles
  • ECP-CONCAT drift ablation with 50 personas and three conversations per model condition
  • Human echoing validation corpus with incompletely reported denominators
  • Later public SPASM v1.0.0 DeepSeek-V4-Flash 500 x 10 JSONL

Evidence and location

  • Publication, design, results, limitations, and appendices: ACL Anthology 2026.findings-acl.412, DOI 10.18653/v1/2026.findings-acl.412; 21 pages rendered and inspected
  • Editable source, prompts, formulas, tables, and internal discrepancies: arXiv:2604.09212v1 source package SHA-256 7a99917853a0f8ce611c8608b36d4a42a42f6d9eaa2fbb56417e59753f76fc2a; main TeX SHA-256 fc7a7abb6903614e5ed159ebd77021af1f89898d5310dda526b80caa424f0adcd
  • Annotators, human reporting, and absence of ethical review: Official Responsible NLP Checklist for 2026.findings-acl.412, both pages rendered and inspected
  • Real state of the repository and absence of original code/corpus: https://github.com/lhannnn/SPASM commit ee49b9288498688225405f2720b511261b306f50
  • Integrity of the subsequent public dataset: SPASM DeepSeek-V4-Flash v1.0.0 JSONL SHA-256 5e3021afd27779a540cb5f5e22e89dd8ab78d58302b0f28b0ac84bc2dd40bd5c; all 5,000 records independently parsed
  • Exact retrieval baseline: Independent hypergeometric calculation for nine same-persona neighbors among 4,999 versus 499 candidates
  • Comprehensive audit of sampling, retrieval, drift, echoing, statistics, artifacts, and reproducibility: reports/verification/article-370-spasm-persona-sampling-retrieval-baseline-drift-echoing-statistics-code-data-release-and-reproducibility-audit.json