Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization

Reviews, theory, and governance2024ACL AnthologyApproved editorial review

Authors: Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, Yun-Nung Chen

Keywords: Large Language Models, Persona, Role-Playing, Personalization, LLM Personality Evaluation

Source: Open primary source (opens in a new tab)

7
Authors
9
Findings
12
Limitations
11
Evidence

Editorial summary

English

This survey offers a simple and useful distinction for organizing a literature that often mixes different meanings of “persona.” In role-playing, the persona is assigned to the model itself, a doctor, judge, programmer, or character, so that it acts inside an environment. In personalization, the persona belongs to the user, their history, preferences, context, or attributes, and the model adapts its response to that person. The paper turns this difference in ownership and purpose into a shared taxonomy, adds a section on personality evaluation, and assembles representative tasks, datasets, and papers published through 2024.

The role-playing branch is organized by environment, software development, games, medicine, and LLM-as-evaluator, by interaction schema, single agent or cooperative/adversarial multi-agent, and by behavior described as voluntary, conforming, or destructive. The personalization branch covers recommendation, search, education, healthcare, and either task-oriented or user-persona dialogue. This map helps distinguish a model performing an assigned role from a model inferring and satisfying the preferences of a real user, while recognizing that both mechanisms may coexist in one application.

The personality section surveys Big Five, MBTI, the Machine Personality Inventory, and LLM-rated interviews used to assess character alignment. Its most important point is cautionary: output that appears consistent with an assigned persona does not establish that a human psychometric questionnaire transfers validly to a model. The section is nevertheless brief and mainly descriptive. It does not appraise the psychometric quality of each study, compare instruments under common criteria, or reanalyze the evidence behind strong claims reproduced from cited work, such as MBTI types or “emergent behaviors.”

The paper should be read as a narrative survey and design map, not a systematic review or meta-analysis. It reports no searched databases, query, cutoff dates, inclusion or exclusion criteria, record counts, screening procedure, extraction process, bias assessment, or quality appraisal. It also performs no original experiment or homogeneous comparison, and explicitly acknowledges that heterogeneous metrics prevent a fair comprehensive performance table. The companion repository is a reading list, but contains only a README and images, has no license or structured data, and its latest commit is the October 2024 camera-ready update despite a promise of continuous maintenance. The defensible contribution is therefore a strong introductory taxonomy for separating model roles from user profiles, not evidence that role-playing or personalization generally makes LLMs more effective, safe, or psychometrically valid.

Español

Este survey propone una distinción sencilla y útil para ordenar literatura que suele mezclar significados diferentes de «persona». En el role-playing, la persona se asigna al propio modelo, médico, juez, programador o personaje, para que actúe dentro de un entorno. En la personalización, la persona pertenece al usuario, historial, preferencias, contexto o atributos, y el modelo adapta su respuesta a esa persona. El artículo convierte esa diferencia de propiedad y objetivo en una taxonomía común, añade una sección sobre evaluación de personalidad y reúne tareas, datasets y trabajos representativos publicados hasta 2024.

La rama de role-playing se organiza por entornos, desarrollo de software, juegos, medicina y LLM como evaluador, por esquema de interacción, agente único o multiagente cooperativo/adversarial, y por conductas descritas como voluntarias, conformistas o destructivas. La rama de personalización cubre recomendación, búsqueda, educación, salud y diálogo orientado a tareas o modelado del usuario. Esta cartografía ayuda a no confundir que un modelo interprete un rol con que infiera y satisfaga preferencias de una persona real, aunque ambos mecanismos pueden coexistir en una aplicación.

La sección de personalidad revisa Big Five, MBTI, Machine Personality Inventory y entrevistas evaluadas por LLM para comprobar alineación con personajes. Su formulación más importante es cautelar: que una salida parezca coherente con la persona asignada no demuestra que un cuestionario psicométrico humano sea transferible al modelo. Sin embargo, esta sección es breve y principalmente descriptiva; no evalúa la calidad psicométrica de cada estudio, no compara instrumentos bajo criterios comunes y en ocasiones reproduce conclusiones fuertes de los trabajos citados, por ejemplo, tipos MBTI o «conductas emergentes», sin volver a analizar su evidencia.

Debe leerse como survey narrativo y mapa de diseño, no como revisión sistemática ni metaanálisis. No declara bases consultadas, consulta de búsqueda, fechas de corte, criterios de inclusión/exclusión, número de registros, cribado, extracción, evaluación de sesgo ni valoración de calidad. Tampoco realiza experimentos propios o una comparación homogénea de resultados, y reconoce que la diversidad de métricas impide construir una tabla de eficacia justa. El repositorio asociado funciona como lista de lectura, pero solo contiene README e imágenes, carece de licencia y datos estructurados y su último commit es la versión camera-ready de octubre de 2024, pese a la promesa de mantenimiento continuo. La contribución defendible es, por tanto, una taxonomía introductoria muy útil para distinguir roles del modelo y perfiles del usuario, no evidencia de que role-playing o personalización mejoren de forma general, segura o psicométricamente válida los LLM.

Research question

How can research on personas in LLMs be organized under a common vision, distinguishing the roles assigned to the model from adaptation to user profiles, and what tasks, schemes, evaluations, and risks characterize each line?

Method

Narrative survey of literature on persona and LLMs. The authors synthesize works into a hierarchical taxonomy: role-playing by environment, agent scheme, and observed behavior; personalization by task or domain; and a cross-cutting layer of personality evaluation. The tables gather web benchmarks, systems, and datasets for recommendation, search, and dialogue. The article does not describe a reproducible strategy for search, screening, or quality evaluation, does not report the total number of studies included, and does not perform quantitative synthesis. The editorial review read the 20 pages, including tables and references, verified the ACL Anthology record, and audited the associated public collection.

Sample: There is no experimental sample or formally declared set of included studies. The text covers pre-LLM dialogue literature and work on agents, role-playing, personalization, and evaluation published mainly up to mid-2024. The tables list systems and datasets, but the article does not report how many records were identified, discarded, or analyzed, nor a reproducible cutoff date. The audited repository contains a reading list in the README with links repeated across categories, not a structured bibliographic database.

Findings

  • The central distinction assigns the persona to the LLM in role-playing, whose goal is to adapt to the environment, and to the user in personalization, whose goal is to adapt the response to individual needs.
  • Role-playing is organized by environment (software, games, medicine, and evaluation), by interaction (single-agent or multi-agent), and by behaviors described during collaboration.
  • Multi-agent systems are subdivided into cooperation and adversariality; the survey relates division of labor, debate, critique, and memory to concrete applications, but does not estimate their mean effect.
  • Personalization is deployed in recommendation, search, education, health, and dialogue, typically through prompting, retrieval, memory, profiling, or model tuning.
  • The same scenarios can combine both branches: an agent can play a professional role and, at the same time, adapt its response to the user profile.
  • The cited personality evaluations use Big Five, MBTI, inventories designed for machines, and interviews, but the survey acknowledges that the direct transferability of human tests remains open.
  • The main challenges identified are task-dependent frameworks, long personas, lack of datasets and benchmarks, biases, jailbreaks, toxicity, privacy, and personal information leakage.
  • The article acknowledges that incompatible metrics across tasks prevent a complete and fair comparison; therefore it delivers a taxonomy and directions, not a ranking of methods.
  • Social implications include potentially more affordable educational and health access, along with risks of inequality, clinical accountability, polarization, and unequal replacement of human support.

Limitations

  • No systematic review protocol is presented: missing databases, query, period, selection criteria, deduplication, double screening, flow diagram, and quality or bias assessment.
  • The number of included works per branch is not reported, nor is a verifiable boundary defined between central literature, general context, and illustrative examples.
  • The taxonomy mixes different levels: application domain, agent architecture, personalization technique, behavioral phenomenon, benchmark, and evaluation instrument.
  • Several categories overlap. LLM-as-evaluator can be role-playing or simply task instruction; user persona modeling can infer attributes without personalizing; a medical system can belong to both branches.
  • Claims of improvement, accuracy, expert knowledge, or human correlation come from the cited articles and are neither reanalyzed nor weighted by methodological quality.
  • The personality evaluation section occupies a small fraction of the survey and does not address in depth measurement invariance, construct validity, conditional reliability, acquiescence, or prompt dependence.
  • Presenting cooperation, conformity, or destructive intent as emergent behaviors may anthropomorphize regularities of generation conditioned by prompts and protocols.
  • There is no quantitative synthesis, effect sizes, intervals, common results table, or controlled comparison of prompting, fine-tuning, memory, and retrieval.
  • Coverage is frozen around 2024 and does not represent later models, benchmarks, regulatory risks, or techniques; its claim to be the first survey is temporal and difficult to prove exhaustively.
  • Privacy is treated as a general risk, but no threat framework, data classification, consent requirements, retention, or differential evaluation of memories and profiles is offered.
  • The public collection contains no bibliographic export, structured metadata, inclusion history, verifiable labels, or automation to detect broken links or update coverage.
  • The repository does not include a license and its last audited commit is from 11 October 2024; therefore, "continuously maintain" describes an intention, not a guarantee of currency.

What the study does not establish

  • It does not demonstrate that assigning roles generally improves reasoning, accuracy, or task success compared to equivalent prompts without a persona.
  • It does not demonstrate that a multi-agent set outperforms a single model with the same budget, information, and number of calls.
  • It does not demonstrate that behaviors termed voluntary, conformist, or destructive correspond to internal human motivations or social processes.
  • It does not demonstrate that coherence with a character equates to possessing a personality, nor that Big Five or MBTI are valid for LLMs.
  • It does not demonstrate that personalization is always beneficial: a more tailored response can amplify bias, polarization, manipulation, dependence, or data exposure.
  • It does not establish clinical, educational, or legal safety for the described systems, nor does it replace domain-specific evaluation and human oversight.
  • It does not offer an exhaustive, mutually exclusive, or stable classification; it is a useful lens over a changing literature.
  • It does not allow comparing efficacy across domains or methods because metrics, datasets, models, costs, and protocols are not harmonized.
  • It does not prove that the repository lists are complete or up to date as of the current date.

Traceability

Scope: Full text

Version: Findings of EMNLP 2024 proceedings, pp. 16612–16631, DOI 10.18653/v1/2024.findings-emnlp.969; official collection commit 6567151fdcc1fe7c0a7d757c1443282a87df78ff audited

Consulted source: https://aclanthology.org/2024.findings-emnlp.969.pdf

Review: Codex editorial review and methods/artifact audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • No LLM is evaluated directly (narrative survey)
  • GPT-3.5 and ChatGPT family discussed across cited studies
  • GPT-4 family discussed across cited studies
  • Llama family discussed across cited studies
  • PaLM family discussed across cited studies
  • Role-playing and personalized agent systems using heterogeneous backbones

Instruments and metrics

  • Role-playing versus personalization taxonomy
  • Role-playing environments: software, games, medicine and LLM-as-evaluator
  • Single-agent and multi-agent schemas
  • Voluntary, conformity and destructive behavior categories
  • Big Five personality inventories
  • Myers–Briggs Type Indicator (MBTI)
  • Machine Personality Inventory (MPI)
  • Personality-test interviewing and LLM rating
  • Task-specific success, accuracy, preference and dialogue metrics reported by cited work

Data used

  • Surveyed recommendation datasets: Amazon Review, MovieLens, Yelp, TripAdvisor and MIND
  • Surveyed task-oriented dialogue datasets: MultiWOZ, SGD, STAR, AirDialogue and UniDA
  • Surveyed user-persona datasets: PersonaChat, ConvAI2, Baidu PersonaChat, JPersonaChat, JEmpatheticDialogues and DailyDialog
  • Surveyed web-agent benchmarks: WebShop, Mind2Web, WebArena, VisualWebArena and VisualWebBench
  • Official PersonaLLM-Survey GitHub reading list

Evidence and location

  • Definitive publication, authors, pages, and DOI: ACL Anthology 2024.findings-emnlp.969; Findings of EMNLP 2024, pp. 16612–16631; DOI 10.18653/v1/2024.findings-emnlp.969
  • Definitions of role-playing and personalization and general taxonomy: Proceedings paper, pp. 16612–16614, Figures 1–3 and sections 1–2
  • Environments, agent schemes, and behavior categories: Proceedings paper, pp. 16614–16616, sections 2.1–2.3
  • Personalization tasks and methods: Proceedings paper, pp. 16616–16618, sections 3.1–3.5 and Figure 4
  • Personality evaluation and doubt about transfer of human tests: Proceedings paper, p. 16618, section 4
  • Challenges of generalization, long context, data, biases, safety, and privacy: Proceedings paper, pp. 16619–16620, sections 5.1–5.5
  • Educational and health implications and need for new measures: Proceedings paper, p. 16620, section 6
  • Recognized absence of complete comparison due to metric heterogeneity: Proceedings paper, p. 16620, Limitations
  • Tabulated benchmarks, systems, and datasets: Proceedings paper, pp. 16629–16631, Tables 1–6
  • Reading list, artifact structure, lack of license, and last update: GitHub MiuLab/PersonaLLM-Survey commit 6567151fdcc1fe7c0a7d757c1443282a87df78ff dated 11 Oct 2024; README.md and repository tree
  • Absence of reproducible search and selection method: Full proceedings paper and companion README: no search strategy, inclusion/exclusion criteria, screening counts or quality appraisal reported