PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

Personas, identity, and agents2026arXivApproved editorial review

Authors: Minseo Kim, Sujeong Im, Junseong Choi, Junhee Lee, Chaeeun Shim, Hwajung Hong, Edward Choi

Keywords: Persona conditioning

Source: Open primary source (opens in a new tab)

7
Authors
8
Findings
23
Limitations
7
Evidence

Editorial summary

English

PICon subjects persona agents to 10 demographic questions, 40 logically chained follow-ups, web verification of entities, and repetition of the initial questions. It evaluates eight groups of 10 agents and 63 humans. No synthetic group exceeds the human descriptive area combining internal, external and retest consistency, although Character.ai exceeds humans on the external axis and Twin 2K 500 and Li et al. on retest. The central result must be decomposed: internal consistency mixes non-contradiction with cooperativeness, so OpenCharacter and Consistent LLM score poorly mainly because of evasive or irrelevant responses. External consistency rewards producing searchable names and explicitly excludes claims about the speaker's own biography; it does not validate whether the agent represents the person. The paper also says NEI labels are excluded, while the public code counts them in the denominator as non-refuted, and an appendix redefines EC as a Wilson bound. Statistical support is weak: n=10 per group, B=3 purported bootstrap resamples, and Welch tests on overlapping cumulative human waves. The public package makes the framework reusable but releases no paper outputs, web evidence or table pipeline. PICon is a useful factual-coherence stress test, not validation of personality, identity, human similarity or fitness to replace participants.

Español

PICon somete agentes persona a 10 preguntas demográficas, 40 seguimientos encadenados, verificación web de entidades y repetición de las preguntas iniciales. Evalúa ocho grupos de 10 agentes y 63 personas. Ningún grupo sintético supera el área descriptiva humana que combina consistencia interna, externa y retest, aunque Character.ai sí supera a humanos en el eje externo y Twin 2K 500 y Li et al. en retest. El resultado central debe descomponerse: la consistencia interna mezcla no contradicción con cooperación, por lo que los mínimos de OpenCharacter y Consistent LLM reflejan sobre todo respuestas evasivas o irrelevantes. La consistencia externa recompensa producir nombres buscables y excluye afirmaciones sobre la propia biografía; no valida que el agente represente a la persona. Además, el artículo dice excluir NEI del cálculo, pero el código público las cuenta en el denominador como no refutadas, y el apéndice redefine EC como un límite de Wilson. La inferencia estadística es débil: n=10 por grupo, B=3 supuestos remuestreos bootstrap y tests de Welch sobre olas humanas acumuladas y solapadas. El paquete público hace el marco reutilizable, pero no libera outputs, evidencias web ni el pipeline de tablas. PICon es un stress test útil de coherencia factual, no una validación de personalidad, identidad, similitud humana o aptitud para sustituir participantes.

Research question

How to evaluate in black box whether a persona agent maintains factual claims without internal contradictions, without refutation by web evidence, and stable when repeating questions during and between conversations.

Method

PICon uses 10 WVS demographic questions in random order, 40 questions generated from previous responses, entity and claim extraction, web search, and a confirmation by the agent itself on the recovered evidence. GPT-5 asks, GPT-5.1 extracts, and Gemini-2.5-Flash evaluates. IC is the harmonic mean of cooperation and non-contradiction; EC, according to the main method, that of coverage of turns with searchable claims and non-refutation; RC semantically compares the ten repeated answers. Eight agent families are tested, resets between sessions, 25/50/75 turns, intensive interrogation by topic, and an alternative open configuration.

Sample: Eight groups of 10 synthetic persons: Character.ai, OpenCharacter, Consistent LLM, Twin 2K 500, DeepPersona, Li et al. 2025, Human Simulacra, and Nemotron. The human baseline gathers 63 participants with functional English, recruited by snowball in five accumulated waves over two weeks; ages 20-49 and Korean nationality predominate, although several countries participate. Each person completes 50 turns and receives 30 USD. The selection of auxiliary agents uses 25 evaluators for 220 pairs of questions and five annotators over four transcripts for extraction/evaluation.

Findings

  • Humans obtain IC 0.90, EC 0.66, and RC 0.94; none of the eight synthetic mean triangular areas exceeds the human one.
  • Character.ai reaches EC 0.71, above humans, but RC 0.46; Twin 2K 500 reaches RC 0.95 and Li et al. 0.98.
  • Human Simulacra presents the highest synthetic IC, 0.79; Nemotron combines 0.81 IC, 0.60 EC, and 0.93 RC.
  • OpenCharacter has IC 0.16 and Consistent LLM 0.31; their non-contradiction is 0.54/0.96, but cooperation only 0.11/0.20.
  • Character.ai obtains coverage 0.66 and non-refutation 0.79; Consistent LLM and Twin 2K 500 approach 1.00 non-refutation with coverage 0.18/0.16.
  • OpenCharacter discards 0.77 of the evidence after confirmation and Consistent LLM 0.69, showing strong selection before calculating non-refutation.
  • On session restart, Character.ai/OpenCharacter/Consistent LLM score 0.55/0.59/0.31; greedy decoding with shuffled order does not consistently stabilize.
  • Intensive interrogation of one topic reduces IC means from 0.69 to 0.59-0.64 and EC from 0.52 to 0.42-0.53 in one case per group.

Limitations

  • IC combines logical coherence and conversational utility; a refusal, irrelevance, or lack of data lowers the score even if no contradiction exists.
  • EC excludes claims about the speaker itself and verifies external entities; it does not check that the biography matches the represented person.
  • Coverage measures production of searchable names, not truth; it penalizes private lives and may reward verbosity or invention with a web footprint.
  • Not refuted is not equivalent to supported: absence of evidence, ambiguous snippets, and uneven web coverage do not prove factuality.
  • The agent confirms whether the evidence corresponds to its claim; discard rates of 0.69-0.77 allow selecting which claims reach the score.
  • The paper says it excludes NEI, but public commit 2e50635 adds supported, refuted, and NEI to the denominator; NEI increases non-refutation and EC.
  • Appendix F.2 calls EC a Wilson lower bound on supported claims, incompatible with the harmonic mean of coverage/non-refutation of the method and code.
  • The code assigns TRUE to persistent intra-session judge failures and FALSE to inter-session failures; failure rates are not published and the treatment is asymmetric.
  • Main RC repeats questions with previous answers still in context and may measure copying/context memory, not stable identity across executions.
  • The area combines three distinct constructs with equal weight and without unit, validation, or threshold; it is a descriptive index.
  • There are only 10 instances per group, standard deviations but no intervals for the average nor group-human tests for the main claim.
  • B=3 does not allow a reliable 95% percentile bootstrap; the negative intervals for RC reveal an unbounded approximation and do not demonstrate absence of selection bias.
  • Welch tests compare overlapping accumulated samples 20/30/41/53/63 as if they were independent; non-rejection does not demonstrate stabilization.
  • Human stopping depends on observed metrics without a preregistered threshold, equivalence, power, or sequential method.
  • Snowball, Korean predominance, functional English, privacy, and fatigue limit the baseline; it does not represent a universal human population.
  • The comparison with open evaluators is visual, without a numerical table, correlation, ranking agreement, equivalence, or variability between runs.
  • The current official abstract says seven groups, while PDF v4, tables, and package contain eight.
  • The repo/package includes implementation, scripts, and prompts, but not outputs, interviews, searches, annotations, resampling results, or the exact pipeline for tables.
  • There are no automated tests or lockfile; main preserves local paths and the README packaging has obsolete anonymous references.
  • There is no superior code license and GitHub/PyPI declare license null, although some data licenses are documented.
  • Gemini models do not have an immutable snapshot, web search changes, and logs are not published; reproducing exact values is temporally unstable.
  • The questions ask for religion, finances, immigration, children, names, address, and employer; retention, deletion, Azure region, search/geocoding providers, and reidentification/third-party analysis are missing.
  • The prompt assumes a fabricated identity and orders demanding details and constructing a dossier; it may pressure disclosure and has dual use in profiling or surveillance.

What the study does not establish

  • It does not demonstrate that all agents are below humans on each axis; several exceed EC or RC separately.
  • It does not demonstrate that a low IC implies contradictions: it may reflect non-cooperation.
  • It does not validate that the agent embodies the biography of its target, because it excludes speaker-centric claims.
  • It does not convert absence of web refutation into factual support or coverage into realism.
  • It does not validate personality, style, preferences, values, behavior, or downstream outcomes; those constructs are out of scope.
  • It does not establish that the triangular area has psychometric interpretation or a human equivalence threshold.
  • It does not demonstrate stability of the human baseline through tests on accumulated waves or absence of bias through three resamples.
  • It does not provide a robust inferential comparison with the human average with n=10 per group.
  • It does not guarantee that the published EC follow the described method while NEI is implemented contradictorily.
  • It does not allow exactly reconstructing tables and figures without outputs, web evidence, analysis, and historical snapshots.

Traceability

Scope: Full text

Version: arXiv:2603.25620v4

Consulted source: https://arxiv.org/abs/2603.25620v4

Review: Codex 29-page visual full-text, official arXiv/project, GitHub two-branch, PyPI, construct, code-metric, statistical, human-baseline, privacy, ethics and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • GPT-5-2025-08-07
  • GPT-5.1-2025-11-13
  • Gemini-2.5-Flash
  • gemini-3-flash-preview
  • Qwen3-235B-A22B-Thinking
  • Qwen3-Next-80B-A3B-Thinking
  • Qwen3-Next-80B-A3B-Instruct
  • Character.ai
  • OpenCharacter
  • Consistent LLM
  • Twin 2K 500
  • DeepPersona
  • Human Simulacra
  • Nemotron Personas

Instruments and metrics

  • 10 preguntas demográficas de World Values Survey
  • 40 preguntas de interrogación encadenada
  • Extracción incremental de entidades y claims
  • Búsqueda web y confirmación de evidencia
  • Cooperativeness y non-contradiction rate
  • Coverage y non-refutation rate
  • Retest intra e inter-sesión
  • Media armónica IC/EC
  • Área normalizada del radar triangular
  • Gwet AC1 y evaluación humana de selección de modelos

Data used

  • Character.ai public-figure personas
  • xywang1/OpenCharacter
  • Consistent LLM persona configurations
  • LLM-Digital-Twin/Twin-2K-500
  • DeepPersona profiles
  • Tianyi-Lab/Personas descriptive tier
  • Human Simulacra characters
  • Seven country-specific NVIDIA Nemotron Personas datasets
  • 63 PICon human interviews, not publicly released

Evidence and location

  • Framework, scope, IC/EC/RC equations and speaker-centric exclusion: arXiv v4, pp. 1-6; all 29 PDF pages visually inspected
  • Main results and decomposition by cooperation, coverage, and discard: arXiv v4, pp. 6-9 and p. 13, Tables 3-5
  • Bootstrap B=3, length, intensity, and human evaluation: arXiv v4, pp. 13-17, Tables 6-12
  • Privacy, sample, ethics, sensitive questions, and full prompts: arXiv v4, pp. 17-23
  • Availability, versions, and seven/eight groups discrepancy: Official arXiv Atom/HTML and KAIST project page inspected 2026-07-17
  • NEI implementation, judge defaults, artifacts, tests, and licenses: Official GitHub main aa660d8 and packaging 2e50635 branches plus PyPI picon-eval 0.1.8 inspected 2026-07-17
  • Comprehensive audit of construct, statistics, human baseline, code, reproducibility, and ethics: reports/verification/article-383-picon-consistency-construct-external-metric-nei-code-statistics-human-baseline-artifacts-and-ethics-audit.json