From Personas to Talks: Revisiting the Impact of Personas on LLM-Synthesized Emotional Support Conversations

Personas, identity, and agents2025ACL AnthologyApproved editorial review

Authors: Shenghan Wu, Yimo Zhu, Wynne Hsu, Mong-Li Lee, Yang Deng

Keywords: Emotional Support Conversations, Persona Conditioning, Persona Extraction, Synthetic Dialogue, HEXACO-60, Communication Styles Inventory, GPT-4o mini, Claude 3.5 Haiku, Llama 3.1 8B Instruct, PersonaHub, ESConv, CAMS, Dreaddit, Human Evaluation, Strategy Distribution, Mental Health AI

Source: Open primary source (opens in a new tab)

5
Authors
12
Findings
21
Limitations
12
Evidence

Editorial summary

English

This study examines how persona information changes LLM-synthesized emotional-support conversations. The pipeline uses GPT-4o mini to extract age, gender, occupation, socio-demographic description, and emotional problem from ESConv, CAMS, and Dreaddit. GPT-4o mini, Claude 3.5 Haiku, and Llama 3.1 8B then augment cards with descriptions aligned to HEXACO and the Communication Styles Inventory, answer both questionnaires, and correlate their six dimensions. In a second experiment, 1,000 PersonaHub descriptions are enriched with demographics and trait-indicative statements, scored, used to generate seeker-supporter dialogues, re-extracted, and scored again. Aggregate distributions remain similar for Honesty-Humility, Agreeableness, Conscientiousness, and Openness, while dialogue-derived personas show higher Emotionality and lower Extraversion. Finally, ESConv histories are continued with and without the seeker's twelve HEXACO/CSI scores. With persona information, GPT-4o mini and Claude generate more questions and affirmation/reassurance and less self-disclosure; Llama changes less. Ten English-speaking undergraduates compare 50 pairs: persona-conditioned versions win 37-39% and lose 33-35% on suggestion, comforting, identification, and overall quality, while consistency ties in 54% of judgments. The defensible contribution is that explicit persona information measurably changes synthetic dialogue text, length, and self-labelled strategy use, with a weak but favorable human preference signal. It does not validate stable psychological traits or therapeutic improvement. The same LLM writes trait descriptions, completes both inventories, generates both speakers, re-extracts the persona, and labels its own strategies, creating circularity and common-method bias. Claimed stability rests on violin plots of group distributions rather than person-level retention: distributions can match even if every individual's scores change. The no-persona control is not matched for information or length, and an omniscient model changes both seeker and supporter. Human evaluation reports no intervals, tests, agreement, blinding, or raw ratings, so 3-5-point margins do not establish superiority. The GPT-4o-mini with-persona distribution also sums to 101.10%, which rounding cannot explain, while the text calls strategy differences significant without reporting tests. No code, generated data, or judgments are publicly linked. The checklist acknowledges that PII/offensive-content checks and artifact documentation were not performed, despite deriving detailed persona cards from sensitive mental-health data. The faithful conclusion is that personas are an influential design variable for conversation synthesis; evidence is not yet sufficient for valid psychometric inference, efficacy, safety, privacy, or real-world readiness.

Español

Este trabajo analiza cómo la información de persona modifica conversaciones de apoyo emocional sintetizadas por LLM. El pipeline usa GPT-4o mini para extraer edad, género, ocupación, descripción sociodemográfica y problema emocional de ESConv, CAMS y Dreaddit; después GPT-4o mini, Claude 3.5 Haiku y Llama 3.1 8B amplían las tarjetas con descripciones alineadas a HEXACO y al Communication Styles Inventory, responden ambos cuestionarios y correlacionan sus seis dimensiones. En un segundo experimento, 1.000 descripciones de PersonaHub se enriquecen con datos y frases indicativas de rasgos, se puntúan, se usan para generar diálogos buscador-soporte, se vuelven a extraer y se puntúan de nuevo. Las distribuciones agregadas permanecen parecidas en Honesty-Humility, Agreeableness, Conscientiousness y Openness, pero las personas derivadas del diálogo muestran mayor Emotionality y menor Extraversion. Finalmente, se continúan historias ESConv con y sin las doce puntuaciones HEXACO/CSI del buscador. Con persona, GPT-4o mini y Claude generan más preguntas y affirmation/reassurance y menos self-disclosure; Llama cambia menos. Diez estudiantes anglófonos comparan 50 pares: la versión con persona gana 37–39 % y pierde 33–35 % en suggestion, comforting, identification y overall, mientras consistency empata en 54 %. La aportación defendible es que añadir información explícita de persona cambia de forma medible el texto, la longitud y las estrategias autoetiquetadas del diálogo sintético, y ofrece una señal humana débil pero favorable. Sin embargo, no valida rasgos psicológicos estables ni una mejora terapéutica. El mismo LLM redacta descripciones de rasgo, contesta los dos inventarios, genera ambos interlocutores, vuelve a extraer la persona y etiqueta sus propias estrategias; esto introduce circularidad y common-method bias. La supuesta estabilidad se apoya en violin plots de distribuciones, no en retención persona-a-persona: una distribución puede coincidir aunque todos los individuos hayan cambiado. El control sin persona tampoco iguala información o longitud, y el modelo omnisciente cambia tanto al buscador como al soporte. La evaluación humana no publica intervalos, tests, acuerdo, cegamiento ni ratings crudos, por lo que márgenes de 3–5 puntos no demuestran superioridad. Además, la distribución GPT-4o mini con persona suma 101,10 %, imposible por redondeo, y el texto habla de diferencias significativas sin reportar tests. No hay código, datos generados ni juicios enlazados públicamente. El checklist reconoce que no se comprobaron PII/contenido ofensivo ni se documentaron los artefactos, pese a derivar tarjetas personales detalladas de datos sensibles de salud mental. La conclusión fiel es que las personas son una variable de diseño influyente para síntesis de conversaciones; aún no hay evidencia suficiente de inferencia psicométrica válida, eficacia, seguridad, privacidad o preparación para despliegue real.

Research question

Can LLMs infer coherent HEXACO traits and communication styles from persona cards, preserve those traits when synthesizing emotional support conversations, and modify through their injection the strategies and perceived quality of those dialogues?

Method

GPT-4o mini extracts cards from 1,155 ESConv cases, 1,140 CAMS and 730 Dreaddit. Three models, at temperature 0, generate descriptions for six HEXACO dimensions and six CSI, complete HEXACO-60 and CSI from those cards, and calculate Pearson matrices between scales. For consistency, 1,000 PersonaHub individuals are expanded with sociodemographic data and trait phrases, scored, generate synthetic dialogues, re-extracted, and re-scored; distributions before/after are compared. For impact, each model continues ESConv stories generating both roles under two prompts: one adds HEXACO/CSI scores and definitions and another omits them. The output itself annotates the strategy of each support turn. Ten native English students compare 50 pairs across five criteria. The audit read and visually reviewed the 15 pages and the two checklist pages, recalculated sums of percentages and p-values of control, evaluated constructs, comparators, statistics, privacy and security, and searched for public artifacts by title and arXiv ID.

Sample: RQ1 uses 1,155 ESConv cards, 1,140 CAMS and 730 Dreaddit; each is processed by three LLMs across two inventories. RQ2 randomly selects 1,000 PersonaHub descriptions, without publishing seed or sample. RQ3 continues ESConv stories; the appendix reports 10,398 turns and 218,433 words with persona versus 12,666 turns and 232,674 words without persona. The human evaluation includes 10 students and 50 pairs; it is not clarified how many judges score each pair nor are individual ratings published.

Findings

  • The HEXACO-CSI matrices usually contain the expected theoretical associations, but not in all cases are they the largest absolute correlation in the row, and there is no external human ground truth.
  • GPT-4o mini produces more coherent relationships in several tables, although the claim that it is the strongest across the six dimensions does not hold literally cell by cell.
  • The before/after distributions are similar for Honesty-Humility, Agreeableness, Conscientiousness and Openness; Emotionality rises and Extraversion falls after the dialogue across the three models.
  • The published evidence is aggregate distribution and does not report correlation, error, or person-to-person equivalence.
  • With persona, GPT-4o mini increases questioning from 16.45% to 27.23% and affirmation/reassurance from 21.06% to 29.72%, and reduces self-disclosure from 10.64% to 2.64%.
  • Claude shows the same general direction; Llama 8B presents smaller changes and questioning remains nearly the same.
  • The GPT-4o mini categories with persona sum to 101.10% and are repeated as a CSI column in Table 8; the error far exceeds what is explicable by rounding.
  • The prompts with persona produce 18% fewer total turns, but longer turns, so content and length also change between conditions.
  • In human evaluation, persona wins 39% and loses 34% overall; the other substantive criteria have victory margins of only 3-4 points and consistency is mainly a tie.
  • The paper does not publish intervals, significance, inter-annotator agreement, judge assignment, blinding, or raw data for the human evaluation.
  • No linked public repository or dataset was located; samples, generations, inventories, counts, and ratings cannot be reproduced.
  • The authors warn that the work is academic and should not be deployed in real emotional support without safeguards.

Limitations

  • The same model generates descriptions aligned to the traits and answers both questionnaires from those descriptions, creating a circular test of semantic coherence.
  • Correlations between two scales answered by the same LLM do not validate accuracy with respect to real people's traits or temporal stability.
  • There are no human self-reports, independent judges, behavioral criteria, or test-retest to validate the inferred personas.
  • RQ2 compares marginal violin plots; it does not report individual correspondence, rank-order, paired distance, interval, or equivalence.
  • PersonaHub descriptions are explicitly enriched with indicative phrases of HEXACO/CSI before measuring and regenerating those same traits.
  • The control without persona omits twelve scores, definitions, and an instruction to use them; it is not a placebo matched in information, length, or prompt attention.
  • A single omniscient model generates seeker and support with access to all the information; the two conditions may change both sides of the conversation.
  • Strategies are self-labeled by the same generating model and are not validated with independent annotators or classifier.
  • The GPT-4o mini distribution with persona sums to 101.10%, a sign of at least one erroneous percentage or denominator.
  • Strategy differences are called significant without tests, intervals, units of analysis, or correction for multiple comparisons.
  • Table 2 claims p<0.01 for all metrics despite showing correlations of 0.01-0.04, which are not significant with n=1,155 if the statement refers to the 36 cells.
  • The human evaluation uses only 10 students and 50 pairs, with narrow margins, without uncertainty, agreement, blinding, or raw data.
  • The annotators have task experience, but no clinical or emotional support training is reported.
  • There are no real users, well-being outcomes, crisis, escalation, harmful advice, dependency, follow-up, or review by mental health professionals.
  • ESConv is crowdsourced and RQ3 generates both interlocutors; the design does not reproduce a real interaction with an independent user.
  • No seed, PersonaHub sample, total cost/compute, infrastructure, or generation parameters are documented apart from temperature 0 and API snapshots.
  • The checklist marks several items as reported, computation, error bars, complete instructions, consent, and demographics, which the paper does not develop in a verifiable manner.
  • There is no public code/data artifact to review or correct the results.
  • The checklist answers no to PII/offensive content checks and documentation, and leaves licenses and intended use as N/A.
  • Extracting demographics and detailed problems from mental health posts may invent or amplify attributes; the example converts young into 23 years without an accuracy audit.
  • The study is in English and the personas and evaluators do not support cultural, linguistic, or clinical generalization.

What the study does not establish

  • It does not demonstrate that LLMs accurately infer stable psychological traits of real people.
  • It does not validate HEXACO or CSI for psychometric measurement of cards generated and answered by the same model.
  • It does not demonstrate that each individual persona retains its traits after the dialogue.
  • It does not convert distribution similarity into test-retest reliability or equivalence.
  • It does not demonstrate statistical significance of strategy differences nor that all published percentages are correct.
  • It does not validate the model's self-labels as independent annotations of support strategies.
  • It does not causally isolate the effect of persona versus more information, explicit instructions, length, and changes in the generated seeker.
  • It does not demonstrate that margins of 3-5 points across 50 pairs represent better empathy or overall quality.
  • It does not demonstrate that rhetorical questions improve well-being, safety, or therapeutic outcomes.
  • It does not demonstrate privacy, consent, absence of PII, or compatibility of use for personas derived from mental health data.
  • It does not establish safety or readiness for real emotional support.
  • It does not allow independent reproduction from public code, data, and ratings.

Traceability

Scope: Full text

Version: EMNLP 2025 main proceedings, pages 5439-5453, DOI 10.18653/v1/2025.emnlp-main.277; 15-page paper plus 2-page Responsible NLP Checklist

Consulted source: https://aclanthology.org/2025.emnlp-main.277/

Review: Codex complete bilingual fidelity pass using the full EMNLP paper and Responsible NLP Checklist, all-17-page visual inspection, independent percentage and p-value checks, construct/common-method/control/human-evaluation/privacy/safety review, and targeted public-artifact search; summaries written from full evidence rather than abstract keywords, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o mini 2024-07-18
  • Claude 3.5 Haiku 2024-10-22
  • Meta Llama 3.1 8B Instruct

Instruments and metrics

  • HEXACO-60
  • Communication Styles Inventory
  • Pearson correlation
  • Persona-card extraction prompts
  • Eight-class emotional-support strategy taxonomy
  • Five-criterion pairwise human evaluation
  • Principal component analysis visualization

Data used

  • ESConv
  • CAMS
  • Dreaddit
  • PersonaHub
  • LLM-synthesized emotional-support conversations

Evidence and location

  • Question, pipeline, results, and declared limits: Official paper pages 1-9, Sections 1-7, Limitations and Ethical Considerations
  • Extraction and counts ESConv, CAMS, and Dreaddit: Official paper page 3, Section 3 and Table 1
  • HEXACO, CSI, snapshots, temperature, and correlation matrices: Official paper pages 4-5 and 13-14, Section 4 and Tables 2-4 and 10-15
  • PersonaHub design and stability by distributions: Official paper pages 5-6 and 13-15, Section 5, Figures 4-7 and 19-21
  • Prompts with/without persona and self-labeling of strategies: Official paper pages 7 and 14, Section 6 and Figures 16-18
  • Strategy distributions and sum error of 101.10%: Official paper page 7, Tables 5-8; independent percentage-sum recalculation on 15 July 2026
  • Human evaluation, margins, and absence of uncertainty: Official paper page 8, Section 6.2 and Table 9
  • Length and turn differences: Official paper page 15, Appendix H and Table 16
  • Privacy, licenses, documentation, computation, and annotators: Official two-page Responsible NLP Checklist, fully rendered and inspected
  • Absence of linked artifact: Paper, ACL and arXiv records plus targeted GitHub title/arXiv-ID repository and code searches on 15 July 2026
  • Comprehensive audit of measurement, evaluation, and reproducibility: reports/verification/article-199-persona-measurement-and-evaluation-audit.json
  • Complete visual inspection: All 15 paper pages and both checklist pages rendered and visually inspected on 15 July 2026