Test-Time-Matching: Decouple Personality, Memory, and Linguistic Style in LLM-based Role-Playing Language Agent

Personas, identity, and agents2025arXivApproved editorial review

Authors: Xiaoyu Zhan, Xinyu Fu, Hao Sun, Yuanqi Li, Jie Guo, Yanwen Guo

Keywords: Test-time scaling, Role-playing agents, Retrieval-augmented generation, Personality-memory-style decoupling, Text style transfer

Source: Open primary source (opens in a new tab)

6
Authors
7
Findings
21
Limitations
5
Evidence

Editorial summary

English

Test-Time-Matching (TTM) constructs a role-playing agent from a novel without updating model weights. Offline, it splits the book into 512-token chunks with 64-token overlap, identifies dialogue and characters, extracts a personality description, structured background and linguistic preferences, and builds one memory database per book plus a historical-utterance index per character. During conversation it uses Qwen3-32B in three stages: generate a styleless reply from personality and background; rewrite the query, retrieve book passages and correct or enrich that reply; then retrieve similar utterances and progressively rewrite each segment in the target style. It is training-free only in the no-parameter-update sense: the paper reports two to three hours on four RTX 3090 GPUs for a roughly 2 MB book, plus up to three hours for background extraction, while each dialogue turn uses four or five LLM calls plus retrieval. The main evaluation selects six literary characters, three Chinese and three English. It generates only one 4-6 turn conversation per available character-method pair, 37 dialogues in total. Twelve general participants and five linguistics experts rate persona consistency, knowledge accuracy and conversation quality from 0 to 10. Relative to direct Qwen3-32B, TTM rises from 6.50/6.57/6.43 to 7.26/7.49/7.07 for general participants and from 5.43/7.37/6.00 to 6.43/7.60/6.30 for experts. In the latter group Gemini-2.5-pro scores 6.40 for conversation quality, above TTM's 6.30. GPT-4.1 also rates TTM above the same base model: 9.21/9.56/9.27 versus 8.03/8.47/8.21. Those are not 62 independent outputs, however: the script repeatedly scores the same six TTM conversations against each opponent in both orders and averages their 62 appearances. TTM is also far more verbose. Across the three English conversations its files average roughly 1,346 words, versus 946 for Gemini, 896 for Qwen3-235B, 863 for Qwen3-32B, 432 for GPT-4o and 424 for CoSER. The paper acknowledges that GPT-4.1 favors longer responses and that humans experienced reading fatigue; there is no length-matched test. A second study asks 21 people to identify a generated sentence inserted among three genuine sentences. Across 504 repeated selections they choose TTM 78 times, 15.48%, and Qwen2.5 125 times, 24.80%. Lower selection is compatible with closer style matching, but raw data, a repeated-measures model, significance test and position analysis are absent. The paper also provides no intervals, significance tests, inter-rater agreement, participant demographics or ethics protocol. The faithful conclusion is therefore narrow: on one sample per character, the complete profile-memory-rewriting stack improves Qwen3-32B's mean ratings. It does not demonstrate actual disentanglement of personality, memory and style: there are no leakage metrics, interventions, swaps or evaluations of recombinations. The MIT repository provides substantial implementation code, prompts, outputs and judge records, but no tests or CI, and 41 of 46 dependencies are unpinned. The public 6.42 GB cache has no valid provenance card and distributes chunks and dialogue derived from works including Harry Potter and modern Chinese novels without documenting source editions, legal basis or licenses for the underlying text.

Español

Test-Time-Matching (TTM) construye un agente de rol a partir del texto de una novela sin actualizar los pesos del modelo. En una fase offline divide el libro en chunks de 512 tokens con solapamiento 64, identifica diálogos y personajes, extrae una descripción de personalidad, un background estructurado y preferencias lingüísticas, y crea una base de memoria por libro junto con un índice de enunciados históricos por personaje. En conversación usa Qwen3-32B en tres etapas: primero genera una respuesta sin estilo desde personalidad y background; luego reescribe la consulta, recupera pasajes del libro y corrige o amplía la respuesta; por último recupera enunciados similares y reescribe progresivamente cada segmento para imitar el estilo. Es training-free solo porque no ajusta parámetros: procesar un libro de unos 2 MB requiere según el paper entre dos y tres horas en cuatro RTX 3090, más hasta tres horas para extraer background, y cada turno usa cuatro o cinco llamadas LLM más retrieval. La evaluación principal escoge seis personajes literarios, tres chinos y tres ingleses. Genera una única conversación de 4-6 turnos por combinación disponible de personaje y método: 37 diálogos en total. Doce participantes generales y cinco especialistas en lingüística puntúan consistencia del personaje, precisión del conocimiento y calidad conversacional de 0 a 10. Frente al Qwen3-32B directo, TTM sube de 6,50/6,57/6,43 a 7,26/7,49/7,07 entre participantes generales y de 5,43/7,37/6,00 a 6,43/7,60/6,30 entre especialistas. En este último grupo Gemini-2.5-pro obtiene 6,40 en calidad conversacional y supera el 6,30 de TTM. GPT-4.1 también puntúa TTM por encima del mismo modelo base: 9,21/9,56/9,27 frente a 8,03/8,47/8,21. Sin embargo, esos no son 62 outputs independientes: el script vuelve a puntuar las mismas seis conversaciones contra cada rival y en ambos órdenes, y promedia sus 62 apariciones. TTM además es mucho más verboso. En las tres conversaciones inglesas, sus ficheros promedian unas 1.346 palabras, frente a 946 de Gemini, 896 de Qwen3-235B, 863 de Qwen3-32B, 432 de GPT-4o y 424 de CoSER. El propio paper reconoce que GPT-4.1 favorece respuestas largas y que los humanos sufrieron fatiga; no hay control de longitud. Un segundo estudio pide a 21 personas identificar una frase generada insertada entre tres frases reales. En 504 selecciones repetidas eligen TTM 78 veces, 15,48%, y Qwen2.5 125, 24,80%; una selección menor es compatible con mayor parecido estilístico, pero no hay datos crudos, modelo de medidas repetidas, prueba estadística ni análisis de posición. El paper tampoco publica intervalos, significación, acuerdo entre jueces, datos demográficos o protocolo ético. Por ello la conclusión fiel es estrecha: en una sola muestra por personaje, la pila completa de perfil, memoria y reescritura mejora las medias de Qwen3-32B. No demuestra que personalidad, memoria y estilo estén realmente desacoplados: no hay métricas de fuga, intervenciones, swaps ni evaluación de combinaciones. El repositorio MIT contiene una implementación sustantiva, prompts, outputs y registros del juez, pero no tests o CI; 41 de 46 dependencias quedan sin versión. El dataset público de 6,42 GB carece de una ficha válida de procedencia y distribuye chunks y diálogos derivados de obras, incluidas Harry Potter y novelas modernas chinas, sin documentar ediciones, base jurídica o licencias del texto subyacente.

Research question

Can a pipeline without weight tuning extract personality, memory, and style from novels, apply them in stages at inference, and improve the fidelity of a role-playing agent compared to direct prompting and other systems?

Method

TTM extracts profiles and memory with Qwen2.5-32B-Instruct, builds RAG and uses Qwen3-32B for style-free response, memory correction, and progressive stylistic rewriting. It compares 37 unique conversations from six characters with two human groups and GPT-4.1; it adds a stylistic detection task with 21 responses and 504 repeated decisions.

Sample: Twelve general participants and five linguistics specialists rated 37 dialogues; 21 responses participated in a 24-item task, 504 selections. The paper does not report recruitment, demographics, native language, exact qualification, exclusions, overlap, compensation, consent, or ethical review.

Findings

  • TTM improves over direct Qwen3-32B in the nine published means: three dimensions by GPT-4.1, general participants, and specialists.
  • Among general participants, TTM achieves 7.26 in character, 7.49 in knowledge, and 7.07 in conversation, compared to 6.50/6.57/6.43 for the same base model.
  • Among specialists, TTM achieves 6.43/7.60/6.30 compared to 5.43/7.37/6.00; Gemini obtains the best conversational quality, 6.40.
  • GPT-4.1 gives TTM 9.21/9.56/9.27 compared to 8.03/8.47/8.21, reusing each dialogue in multiple comparisons.
  • Phrase detection selects TTM in 78/504 cases, 15.48%, and Qwen2.5 in 125/504, 24.80%.
  • TTM averages about 1,346 words in the three English outputs, approximately 1.5 times the next method and more than three times GPT-4o or CoSER.
  • The current repository includes an additional automated ablation where later stages improve GPT-4.1 means, but not a human ablation.

Limitations

  • Only one dialogue is generated per character and method; there is no repetition by seed or estimation of generation variance.
  • No intervals, standard errors, hypothesis tests, multiple correction, or repeated-measures model are published despite significance language.
  • Inter-judge agreement and reliability of the three scales are not reported.
  • Ratings are crossed by participant, character, and method, but are reduced to simple means.
  • The GPT judge reuses the same dialogues against several rivals and in both orders: 62 scores do not equal 62 independent samples.
  • The judge code truncates Harry_Potter to Harry and Pride_and_Prejudice to Pride when building the prompt from the directory name.
  • TTM is much longer; the paper itself acknowledges judge length bias and human fatigue, without length-matched evaluation.
  • Universal models do not receive the same profile, complete book, RAG, and utterance history, so the most interpretable contrast is TTM versus Qwen3-32B.
  • The separation of personality, memory, and style is an architecture, not a measured decoupling with leakage, intervention, swaps, or orthogonality.
  • Free combinations of personality, memory, and style promised in the abstract are not evaluated.
  • The stylistic task treats 504 repeated decisions from only 21 people as raw proportions; uncertainty, per-participant analysis, and position data are missing.
  • A 15.48% below the 25% chance level may reflect similarity, but also item or position biases that are not ruled out.
  • Six literary roles do not represent arbitrary characters, public figures, demographic identities, or other domains.
  • The common memory base per book may mix narrator or other characters' knowledge and does not model chronology, filtering future facts.
  • LLM extraction of personality/background is not validated against gold profiles nor does it preserve uncertainty.
  • Training-free does not mean cheap: construction uses hours of four GPUs and the turn uses several calls; latency, energy, cost, and throughput are missing.
  • Participants, consent, compensation, IRB, or ethical protocol are not documented.
  • Safety, impersonation, emotional dependence, manipulation, copyright, or privacy are not evaluated.
  • The code has no tests, CI, lockfile, or container; 41 of 46 dependencies remain unpinned.
  • Exact scripts/prompts and seeds to generate all baselines are missing, and human ratings or questionnaires are not released.
  • The public cache lacks a valid provenance record and does not separate the MIT license from the copyright of derived literary texts.

What the study does not establish

  • It does not demonstrate statistical or causal decoupling of personality, memory, and style.
  • It does not demonstrate fluid combinations between unobserved traits, memory, and style.
  • It does not demonstrate statistical significance or robustness across seeds.
  • It is not the best system on every metric: Gemini wins conversational quality for specialists.
  • It does not prove that advantages survive matching length and retrieved context.
  • It does not generalize beyond six Chinese and English literary characters.
  • It does not demonstrate reliable human agreement.
  • It does not guarantee chronological memory, absence of leakage, or character-exclusive knowledge.
  • It does not demonstrate efficiency from being training-free.
  • It does not establish safety for companionship, public figures, or real deployment.
  • It does not allow reproducing human ratings or all baselines end to end.
  • It does not certify that the public distribution of book derivatives has complete provenance and license.

Traceability

Scope: Full text

Version: arXiv:2507.16799v2, submitted 2025-07-22 and revised 2025-07-23; code, saved evaluations and cache dataset audited separately

Consulted source: https://arxiv.org/abs/2507.16799

Review: Codex 13-page visual, official-arXiv-v2, full-method, single-sample, repeated-measures, GPT-judge-pseudoreplication, length-bias, style-detection, decoupling, code, dependency, cached-data, copyright, reproducibility and claim-boundary audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • Qwen3-32B, TTM dialogue base
  • Qwen2.5-32B-Instruct, extraction and analysis
  • Qwen3-Embedding-0.6B
  • bge-large-zh-v1.5
  • Qwen3-235B-A22B
  • GPT-4o
  • Gemini-2.5-pro
  • ChatHaruhi with Qwen3-32B
  • CoSER with Llama-3.1-70B
  • GPT-4.1 as judge

Instruments and metrics

  • Three-stage styleless, memory-checked and stylized generation pipeline
  • Graph-based RAG and hybrid BM25/embedding retrieval
  • Three 0-10 rating dimensions: persona consistency, knowledge accuracy and conversation quality
  • GPT-4.1 pairwise scoring in both presentation orders
  • Generated-sentence detection among three historical utterances
  • Repository output-length and score-arithmetic audit

Data used

  • Six-character evaluation: Lin Daiyu, Duan Yu, Xuzhu, Dumbledore, Hermione and Elizabeth Bennet
  • Thirty-seven single-sample multi-turn dialogues across seven methods
  • Public asinmhk/TTM_cache, 441 files and about 6.42 GB
  • Saved GPT-4.1 pairwise evaluation records in ZhanxyR/TTM
  • Unreleased participant-level human ratings and style-detection responses

Evidence and location

  • Metadata, version, authors, license, and abstract: arXiv:2507.16799v2
  • Method, models, evaluation, table, limitations, resources, and appendix: arXiv v2 PDF, 13 pages, sha256 e76340999c05c4bf436493ac97d98bd3eb2b63647c7ecef81bf2a192b228f0bb
  • Implementation, prompts, outputs, judge logs, dependencies, and title bug: ZhanxyR/TTM commit 32823bb27675a36caa65dde817d35d8e2420b749
  • Size, access, content, and cache metadata: asinmhk/TTM_cache commit 4284156233adb7255f5049d86414bdc7d379da07
  • Decoupling audit, human sample, length, code, data, and boundary audit: reports/verification/article-248-arxiv-ttm-decoupling-human-evaluation-length-bias-code-data-and-claim-audit.json