Revealing Personality Traits: A New Benchmark Dataset for Explainable Personality Recognition on Dialogues

Evaluation and psychometric validity2024ACL AnthologyApproved editorial review

Authors: Lei Sun, Jinming Zhao, Qin Jin

Keywords: Personality recognition, Explainable AI, Dialogue analysis, Personality traits, Machine learning, Chain-of-Personality-Evidence (CoPE)

Source: Open primary source (opens in a new tab)

3
Authors
13
Findings
32
Limitations
14
Evidence

Editorial summary

English

The paper proposes CoPE, a chain linking dialogue context, momentary personality states, and long-term traits, and turns it into two tasks. EPR-S predicts a Big Five label for each dialogue and the supporting utterances; EPR-T infers a character's trait from multiple dialogues and identifies supporting dialogues. The “explanation” is a supervised textual rationale, summaries and BFI-2-derived descriptors plus evidence identifiers, not a causal model explanation or a direct measurement of a psychological construct.

PersonalityEvd is built from CPED, a Chinese television-series dialogue corpus. After excluding short dialogues, the authors select 72 characters and up to 30 dialogues per character. The final dataset contains 1,924 dialogues and 32,673 utterances. GPT-4 Turbo pre-annotates each dimension; twelve psychology undergraduates correct states, two graduate students inspect quality, and the authors perform a final review. About 30% of samples are re-annotated. Six other undergraduates produce trait annotations: three independently annotate each character, reach consensus, may alter state labels, and convert the mean of twelve BFI-2 items into labels through a median split. Dialogues with all five dimensions uncertain or with contradictory information are removed.

For EPR-S, Qwen1.5-7B-Chat has the highest mean Big Five accuracy at 66.45%, versus 64.62% for ChatGLM3-6B-32K and 62.09% for zero-shot GPT-4 Turbo. Evidence-ID F1 scores are 75.94, 75.42, and 71.20. For three-fold EPR-T, ChatGLM reaches 77.78% accuracy and 40.28 evidence-ID F1, while Qwen reaches 76.59% and 44.39. These accuracies must be read against the imbalance in the released data: among 72 characters, high labels occur 62 times for Openness, 62 for Conscientiousness, 70 for Extraversion, 49 for Agreeableness, and 45 for Neuroticism. Always selecting each dimension's majority class would average about 80%, above both EPR-T results. Raw accuracy therefore does not demonstrate effective trait recognition.

Evaluation combines BERTScore, Claude 3 Sonnet and GPT-4 Turbo ratings, plus a human evaluation of 50 samples from ten characters with five evaluators per sample. References score 4.61/5 for fluency, 4.38 for coherence, and 4.31 for plausibility; ChatGLM scores 3.82/2.51/2.59 and Qwen 3.90/2.68/2.65. No inter-rater agreement, uncertainty, or sampling strategy is reported, so the figures support high perceived quality in a small sample but do not “guarantee” the entire corpus.

The ablations qualify the claim that generating evidence improves recognition. For states, direct fine-tuning reaches 64.84% and CoT 64.62%; hybrid settings reach 66.94% and 66.55%. For traits, CoT improves from 75.83% to 77.78%, but hybrid settings remain at 75.53% and 75.55%. Adding predicted states raises trait accuracy only from 77.78% to 77.99%; reference states raise it to 83.52%, but these annotations are unavailable in real use and tied to the same process that produced trait labels.

The repository allows inspection of the characters, dialogues, splits, labels, and part of the code. It lacks the announced English translation, complete outputs, checkpoints, an end-to-end judge pipeline, a code/derivative-data license, and releases. Many scripts retain local paths or xxx placeholders; Qwen dependencies are unpinned, and seed variability is absent. The defensible contribution is a novel evidence-annotated benchmark over fictional Chinese characters and initial baselines. It does not validate human personality, psychological stability, causal explanations, cross-cultural generalization, or a system ready to profile real users.

Español

El artículo propone CoPE, una cadena que enlaza contexto de diálogo, estados de personalidad momentáneos y rasgos de largo plazo, y la convierte en dos tareas: EPR-S predice para cada diálogo una etiqueta Big Five y las intervenciones que la sustentan; EPR-T infiere el rasgo de un personaje a partir de varios diálogos y señala los diálogos usados como evidencia. La «explicación» es una justificación textual supervisada, resúmenes y descriptores derivados de BFI-2 junto con identificadores de evidencia, no una explicación causal del modelo ni una medición directa del constructo psicológico.

PersonalityEvd parte de CPED, un corpus chino de series de televisión. Tras descartar diálogos cortos, los autores seleccionan 72 personajes y hasta 30 diálogos por personaje. El conjunto final contiene 1.924 diálogos y 32.673 intervenciones. GPT-4 Turbo preanota cada dimensión; doce estudiantes de Psicología corrigen estados, dos estudiantes de posgrado inspeccionan la calidad y los autores hacen una revisión final. Aproximadamente el 30 % de las muestras se reanota. Otros seis estudiantes producen los rasgos: tres anotan independientemente cada personaje, consensúan la decisión, pueden modificar estados y convierten la media de doce ítems BFI-2 en etiquetas mediante una división por la mediana. También se eliminan diálogos con las cinco dimensiones inciertas o con información contradictoria.

En EPR-S, Qwen1.5-7B-Chat obtiene la mayor exactitud Big Five media, 66,45 %, frente al 64,62 % de ChatGLM3-6B-32K y el 62,09 % de GPT-4 Turbo en zero-shot. Los F1 de identificadores de evidencia son 75,94, 75,42 y 71,20. En EPR-T, evaluado con tres pliegues, ChatGLM alcanza 77,78 % de exactitud y 40,28 de F1 de evidencia; Qwen obtiene 76,59 % y 44,39. Estas exactitudes deben leerse junto al desequilibrio de la versión publicada: de 72 personajes, las etiquetas altas son 62 en apertura, 62 en responsabilidad, 70 en extraversión, 49 en amabilidad y 45 en neuroticismo. Elegir siempre la clase mayoritaria de cada dimensión daría una media aproximada del 80 %, superior a ambos resultados EPR-T. La exactitud bruta no prueba buen reconocimiento de rasgos.

La evaluación combina BERTScore, puntuaciones de Claude 3 Sonnet y GPT-4 Turbo, y una evaluación humana de 50 muestras de diez personajes con cinco evaluadores por muestra. La referencia recibe 4,61/5 en fluidez, 4,38 en coherencia y 4,31 en plausibilidad; ChatGLM obtiene 3,82/2,51/2,59 y Qwen 3,90/2,68/2,65. No se informa acuerdo, intervalos ni estrategia de muestreo, por lo que estas cifras apoyan calidad percibida alta en una muestra pequeña, pero no «garantizan» todo el corpus.

Las ablaciones matizan la afirmación de que generar evidencia mejora el reconocimiento. En estados, el ajuste directo obtiene 64,84 % y CoT 64,62 %; los esquemas híbridos llegan a 66,94 % y 66,55 %. En rasgos, CoT mejora de 75,83 % a 77,78 %, pero los esquemas híbridos quedan en 75,53 % y 75,55 %. Añadir estados predichos solo eleva la exactitud de rasgos de 77,78 % a 77,99 %; usar estados de referencia la sube a 83,52 %, pero son anotaciones no disponibles en uso real y ligadas al mismo proceso que produjo los rasgos.

El repositorio permite comprobar personajes, diálogos, particiones, etiquetas y parte del código. No contiene la traducción inglesa anunciada, resultados completos, checkpoints, un flujo ejecutable de extremo a extremo para los jueces, licencia del código/datos derivados ni releases. Muchos scripts conservan rutas locales o marcadores xxx; las dependencias de Qwen no están versionadas y no hay variabilidad entre semillas. La contribución defendible es un benchmark novedoso de evidencias anotadas sobre personajes ficticios chinos y baselines iniciales. No valida personalidad humana, estabilidad psicológica, explicaciones causales, generalización intercultural ni un sistema listo para perfilar usuarios reales.

Research question

Can explainable personality recognition be represented as a chain from concrete dialogues to Big Five states and from these to traits, and what capacity do several LLMs have to predict the labels and recover the annotated evidence at both levels?

Method

Benchmark construction and supervised evaluation. CPED is filtered, 72 characters from Chinese series are selected, and 1,924 dialogues are annotated in five dimensions. GPT-4 Turbo performs a pre-annotation; Psychology students and two inspectors correct states, evidence, and justifications. Another group annotates traits from about 30 dialogues and BFI-2, with three annotators per character and consensus. EPR-S uses a 51/7/14 speaker split and EPR-T uses three-fold cross-validation of 24 characters. ChatGLM3-6B-32K and Qwen1.5-7B-Chat are fine-tuned with LoRA; GPT-4 Turbo is evaluated in zero-shot only for EPR-S. Accuracy, identifier F1, BERTScore, and LLM and human ratings are measured. The editorial review read and rendered the 15 pages, audited the data and code from the official repository, and recomputed distributions and majority baselines.

Sample: PersonalityEvd contains 72 fictional characters and 1,924 dialogues, 26.72 per character on average, with 32,673 utterances. EPR-S is split by characters into 51 for training, 7 for validation, and 14 for testing, with 1,380, 181, and 363 dialogues. EPR-T contains 360 dimension-character decisions in three folds of 24 characters. Only four trait labels are uncertain; 288 of the 360 are high. The human evaluation covers 50 samples from ten characters and five evaluators per sample.

Findings

  • The published corpus matches the main figures: 72 characters, 1,924 dialogues, and EPR-S partitions of 51/7/14 characters with 1,380/181/363 dialogues.
  • Qwen achieves the highest mean EPR-S accuracy, 66.45%, followed by ChatGLM with 64.62% and GPT-4 zero-shot with 62.09%.
  • The utterance identification F1 scores in EPR-S are 75.94 for Qwen, 75.42 for ChatGLM, and 71.20 for GPT-4.
  • In EPR-T, ChatGLM reaches 77.78% accuracy and 40.28 evidence F1; Qwen reaches 76.59% and 44.39.
  • The published file contains 288 high labels, 68 low labels, and 4 uncertain labels across 360 trait decisions; extraversion has 70 high and only 2 low.
  • A baseline that chooses the majority class for each dimension averages approximately 80% EPR-T accuracy, above both models.
  • In EPR-S, the per-dimension majority baseline on test averages approximately 59.50%, below the three LLMs.
  • Human references receive 4.61/5 in fluency, 4.38 in coherence, and 4.31 in plausibility; the generations remain at 3.82–3.90, 2.51–2.68, and 2.59–2.65.
  • In states, the direct hybrid improves from 64.84% to 66.94%, while pure CoT drops slightly to 64.62%.
  • In traits, CoT improves from 75.83% to 77.78%, but the hybrid schemes remain around 75.5%.
  • Using predicted states raises trait accuracy by only 0.21 points; reference states raise it from 77.78% to 83.52%.
  • The repository preserves Chinese dialogues, annotations, and folds, but not the announced English translation.
  • The code compiles syntactically, but requires adapting paths, dependencies, models, and absent artifacts to reproduce the experiments.

Limitations

  • The characters come from television scripts: they are written and narratively oriented dialogues, not spontaneous behavior of people.
  • There is no self-report, close-observer report, or other validated psychometric measure as an external criterion.
  • The labels are annotators' impressions of fictional characters, not observed psychological traits of real individuals.
  • GPT-4 Turbo pre-annotates the entire corpus, so its associations, style, and biases may anchor human corrections.
  • Each state sample is initially reviewed by a single annotator and only about 30% is re-annotated; there is no complete double annotation.
  • No inter-annotator agreement indicator is published for states, traits, or evidence.
  • Disagreements are resolved by consensus or elimination, and trait annotators may modify states; both levels are not independent.
  • Removing contradictory dialogues and those with the five uncertain dimensions biases the corpus toward consistent cases with evidence.
  • The median split converts continuous dimensions into categories and loses magnitude differences.
  • The definition allows uncertain traits, but the procedure is described as binary; the released file contains four uncertain ones.
  • The EPR-T imbalance is extreme: 70 of 72 characters are high in extraversion and 62 of 72 in openness and conscientiousness.
  • Majority baselines, balanced accuracy, macro-F1, per-class sensitivity, or confusion matrices are not reported.
  • The two EPR-T models remain below the mean majority baseline derived from the released labels.
  • EPR-T only has 72 characters and 360 labels; each test fold contains 24 characters.
  • No intervals, dispersion across seeds, or statistical tests are published to compare models and ablations.
  • The split is by character, but no separation by series is documented; shared plots, interlocutors, and styles could leak between sets.
  • BERTScore may reward formulas similar to the reference without verifying that each claim is supported by the dialogue.
  • GPT-4 participates in pre-annotation and evaluation; Claude reduces, but does not eliminate, the dependence on LLM judges.
  • Claude scoring somewhat higher does not prove the absence of self-evaluation bias; calibration against humans is lacking.
  • The human evaluation uses 50 samples from ten characters, without stratification, agreement, variance, or reported intervals.
  • Human scores from a small sample do not guarantee the quality of the entire corpus.
  • The textual evidence is an annotated and templated justification; it does not reveal the internal process or a causal explanation.
  • Reference states are not available in real use and are tied to the same pipeline that generates the traits.
  • Predicted states improve trait accuracy by only 0.21 points, and the evidence does not help in all configurations.
  • GPT-4 is only evaluated in EPR-S; there is no equivalent zero-shot comparison in traits.
  • The corpus is limited to Chinese and characters from Chinese series; it does not evaluate other languages, cultures, or domains.
  • The announced English translation is not in the audited repository, nor is its fidelity evaluated.
  • The repository does not include an explicit license for the code or derived data, checkpoints, complete outputs, or evaluation calls.
  • Many scripts contain absolute paths or xxx placeholders; the Qwen dependencies do not fix versions.
  • The article does not report seeds or repetitions; ChatGLM uses 1234 in scripts, but robustness across runs is not shown.
  • The repository has no tests, CI, versioned release, or documentation for clean reproduction.
  • Privacy, consent, and harm from profiling remain open if the technique is transferred from fictional characters to users.

What the study does not establish

  • It does not demonstrate that characters, LLMs, or people possess the assigned Big Five traits.
  • It does not validate the labels as equivalent to BFI-2 administered to real people.
  • It does not demonstrate longitudinal stability; it aggregates impressions from fiction scenes.
  • It does not demonstrate that a justification is causal or faithful to the model's internal process.
  • It does not establish that CoPE is a validated psychological theory.
  • It does not prove that the models outperform trivial baselines in EPR-T.
  • It does not demonstrate global superiority of GPT-4 or of the fine-tuned models.
  • It does not establish a material improvement of traits through predicted states.
  • It does not demonstrate generalization to people, spontaneous conversation, other languages, or cultures.
  • It does not justify inferring personality of users or automated decisions about them.
  • It does not validate clinical, educational, occupational, credit, police, or selection uses.
  • It does not demonstrate that LLM judges measure truth, human quality, or absence of hallucination.
  • It does not offer complete reproduction of the figures with an executable environment without repair.

Traceability

Scope: Full text

Version: Proceedings of EMNLP 2024, pp. 19988–20002; DOI 10.18653/v1/2024.emnlp-main.1115; ACL Anthology ID 2024.emnlp-main.1115; CC BY 4.0

Consulted source: https://aclanthology.org/2024.emnlp-main.1115.pdf

Review: Codex full-text, visual, dataset, code and reproducibility audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • ChatGLM3-6B-32K fine-tuned with LoRA
  • Qwen1.5-7B-Chat fine-tuned with LoRA
  • GPT-4 Turbo 2024-04-09 as zero-shot EPR-S baseline
  • GPT-4 Turbo as dataset pre-annotator and evidence evaluator
  • Claude 3 Sonnet 2024-02-29 as evidence evaluator
  • Chinese BERT used through BERTScore

Instruments and metrics

  • Big Five Inventory-2 with five dimensions, fifteen facets and sixty items
  • Chain-of-Personality-Evidence annotation framework
  • Personality state labels: high, low or uncertain
  • Personality trait labels derived through BFI-2 scoring and median split
  • Evidence utterance and dialogue identifiers
  • Natural-language evidence summaries and personality descriptors
  • Accuracy and binary evidence-ID F1
  • BERTScore F1
  • Claude 3 Sonnet and GPT-4 Turbo evidence ratings from one to five
  • Human fluency, coherence and plausibility ratings

Data used

  • CPED Chinese personalized and emotional dialogue corpus
  • PersonalityEvd with 72 fictional television characters, 1,924 dialogues and 32,673 utterances
  • PersonalityEvd EPR-S speaker-disjoint train, validation and test annotations
  • PersonalityEvd EPR-T trait annotations and three speaker folds
  • Official PersonalityEvd repository at commit 1f41e8de94ae712ff3e2d0d3c75265d97494ef3e

Evidence and location

  • Metadata, abstract, DOI, pages, and license: ACL Anthology 2024.emnlp-main.1115; final paper p. 19988; CC BY 4.0
  • CoPE framework, states, traits, and tasks: Final paper, Figure 1, sections 1 and 4, pp. 19988–19993
  • CPED origin, 72 characters, and 2,160 candidates: Final paper, section 3.1, pp. 19990–19991
  • GPT-4 pre-annotator and human correction: Final paper, sections 3.3.1–3.3.2, pp. 19990–19992
  • Trait consensus, state modification, and median: Final paper, section 3.4, p. 19992
  • Filtering of uncertain cases and contradictions: Final paper, section 3.3.2, p. 19992
  • Size, statistics, and partitions: Final paper, Table 1 and section 3.5, pp. 19992–19993; released dataset commit 1f41e8d
  • Models, metrics, and main results: Final paper, sections 4–5.2 and Table 2, pp. 19993–19994
  • Human evaluation: Final paper, section 5.3 and Table 3, pp. 19994–19995
  • Ablations and states as cues: Final paper, section 5.4 and Tables 4–5, p. 19995
  • Declared limitations and ethics: Final paper, Limitations and Ethics Statements, pp. 19995–19996
  • Real distribution and majority baseline: Official repository commit 1f41e8d, released state and trait annotation JSON; recomputed 15 Jul 2026
  • Translation, license, and absent artifacts: Official PersonalityEvd repository commit 1f41e8de94ae712ff3e2d0d3c75265d97494ef3e; audited 15 Jul 2026
  • Paths, dependencies, and seeds: Official repository commit 1f41e8d, state and trait code; audited 15 Jul 2026