Synthetic Personality Review

Big-Five Backstage: A Dramatic Dataset for Characters Personality Traits & Gender Analysis

Evaluation and psychometric validity2024ACL AnthologyApproved editorial review

Authors: Marina Tiuleneva, Vadim A. Porvatov, Carlo Strapparava

Keywords: Big Five personality traits, character analysis, psycholinguistics, dramatic dialogue, dataset creation, fictional characters, computational personality analysis

Source: Open primary source (opens in a new tab)

3
Authors
20
Findings
34
Limitations
16
Evidence

Editorial summary

English

Big-Five Backstage releases a corpus of English theatre-character lines aggregated by character and labeled for binary gender and the presence or absence of five Big Five traits. The paper starts from 178 Project Gutenberg files, removes non-English and verse works, retains 400 plays by 132 authors, and excludes characters with fewer than five lines. It reports 3,265 texts, 3,419,136 words, and 1,047.2 words per character. Gender labels are assigned manually, but personality labels are produced by GPT-3.5-turbo from each character's full text. To validate those annotations, 10% of texts are sampled and distributed across two humans who are blind to GPT; each text receives only one human decision, so there is no inter-annotator agreement or consensus ground truth. Against those labels, GPT reaches accuracies of 0.872, 0.889, 0.894, 0.791, and 0.900 for Extraversion, Agreeableness, Openness, Neuroticism, and Conscientiousness; corresponding F1 scores are 0.760, 0.903, 0.829, 0.781, and 0.886. These figures support agreement with one human annotation on this sample, unevenly across traits; they do not validate that characters possess those traits or that GPT measures them psychometrically. The analysis uses 44 LIWC markers for gender and 65 LIWC/MRC markers for personality, Mann–Whitney and Wilcoxon tests, Cohen's d, and point-biserial correlations at α=0.05. For gender, 32 markers are significant: several follow directions previously reported for human texts but with larger effects, while others reverse, including you being more frequent in female fictional speech. Among authors with enough characters, correspondence between fictional and human effects varies; the Strindberg example reaches r=0.8861. For personality, 14 MRC and 48 LIWC variables associate with at least one trait; five are significant for all five traits and 24 for four. Neuroticism is especially linked to length and negative-affect vocabulary; Conscientiousness to word and phoneme length; Extraversion to leisure; Agreeableness to positive emotion; and Openness to meaningfulness, spelling, and leisure. However, hundreds of combinations are tested at α=0.05 without multiplicity correction, analyzed traits come from GPT's own labels, and complete effect sizes and uncertainty are not reported. The official-repository audit confirms a useful MIT-licensed release containing texts, labels, LIWC/MRC variables, demographics, human/GPT annotations, a notebook, and supplementary tables. It also reveals divergences: current CSV files contain 3,258 rows rather than 3,265, normalize to 133 authors rather than 132, and contain six duplicated author-play-character keys. There is no dependency manifest, exact GPT snapshot, executable prompt, seeds, or complete extraction and analysis pipeline. The resource is valuable for studying dramatic language and annotation bias, but its Big Five labels should be treated as partially validated GPT-3.5 predictions, not as true character personality or direct evidence about real people.

Español

Big-Five Backstage publica un corpus de intervenciones de personajes de teatro en inglés, agregadas por personaje y etiquetadas con género binario y presencia/ausencia de cinco rasgos Big Five. El artículo parte de 178 archivos de Project Gutenberg, elimina obras no inglesas y en verso, conserva 400 obras de 132 autores y excluye personajes con menos de cinco líneas. Declara 3.265 textos, 3.419.136 palabras y 1.047,2 palabras por personaje. Las etiquetas de género se asignan manualmente, pero las de personalidad las produce GPT-3.5-turbo a partir del texto completo de cada personaje. Para validar esa anotación, se muestrea el 10 % y se reparte entre dos humanos ciegos a GPT; cada texto recibe la decisión de un solo humano, por lo que no existe acuerdo interanotador ni un ground truth consensuado. Frente a esas etiquetas humanas, GPT obtiene exactitudes de 0,872, 0,889, 0,894, 0,791 y 0,900 para extraversión, amabilidad, apertura, neuroticismo y responsabilidad; los F1 respectivos son 0,760, 0,903, 0,829, 0,781 y 0,886. Estas cifras apoyan que el modelo coincide con una anotación humana individual en esa muestra, con rendimiento desigual por rasgo; no validan que los personajes tengan esos rasgos ni que GPT los mida psicométricamente. El análisis usa 44 marcadores LIWC para género y 65 marcadores LIWC/MRC para personalidad, pruebas Mann–Whitney y Wilcoxon, d de Cohen y correlaciones punto-biseriales con α=0,05. Para género encuentra 32 marcadores significativos: varios siguen direcciones publicadas en textos humanos, pero con efectos mayores; otros se invierten, como el pronombre you, más frecuente en personajes femeninos. En autores con suficientes personajes, la relación entre efectos ficticios y humanos varía y el ejemplo de Strindberg alcanza r=0,8861. Para personalidad, 14 variables MRC y 48 LIWC se asocian con al menos un rasgo; cinco son significativas para los cinco rasgos y 24 para cuatro. Neuroticismo se relaciona especialmente con longitud y vocabulario afectivo negativo; responsabilidad con longitud de palabra y fonemas; extraversión con ocio; amabilidad con emociones positivas; y apertura con meaningfulness, ortografía y ocio. Sin embargo, se prueban cientos de combinaciones con α=0,05 sin corrección por comparaciones múltiples, las etiquetas analizadas proceden del propio GPT y no se informan tamaños de efecto completos ni incertidumbre. La auditoría del repositorio confirma una publicación útil bajo MIT con textos, etiquetas, variables LIWC/MRC, demografía, anotaciones humanas/GPT, notebook y tablas suplementarias. También descubre divergencias: los CSV actuales contienen 3.258 filas, no 3.265; normalizan 133 autores, no 132; y seis claves autor-obra-personaje están duplicadas. No hay manifiesto de dependencias, versión exacta de GPT, prompt ejecutable, semillas ni pipeline completo de extracción y análisis. El recurso es valioso para estudiar lenguaje dramático y sesgos de anotación, pero las etiquetas Big Five deben tratarse como predicciones débiles de GPT-3.5 validadas parcialmente, no como personalidad verdadera de personajes ni como evidencia directa sobre personas reales.

Research question

To what extent does the language attributed to theatrical characters reproduce or exaggerate linguistic patterns associated with gender and Big Five in human texts, and can GPT-3.5 provide personality labels sufficiently concordant with a partial human annotation to build an exploratory corpus?

Method

The utterances of each character in 400 English plays are extracted from Project Gutenberg and concatenated by character, excluding those with fewer than five lines. Gender is labeled manually and GPT-3.5-turbo assigns five binary Big Five labels. 10% of the texts are distributed between two human annotators, one per text, to estimate accuracy, precision, recall and F1 of GPT. 44 LIWC categories are calculated for gender and 51 LIWC plus 14 MRC for personality. Mann-Whitney contrasts men and women across the whole corpus, Wilcoxon compares authors, Cohen's d quantifies differences and point-biserial correlations relate markers to binary labels at the global and author level, using α=0.05.

Sample: The paper declares 3,265 character texts from 400 plays and 132 authors, with 3,419,136 words and a mean of 1,047.2. The audited official CSV contains 3,258 rows, 400 author-play pairs and 133 author values; six author-play-character keys appear twice. Gender labels are distributed as 2,007 masculine and 1,251 feminine in the CSV. Approximately 10% of texts are validated by humans, but each text receives only one human annotation.

Findings

  • GPT-3.5 achieves accuracy of 0.872, 0.889, 0.894, 0.791 and 0.900 for extraversion, agreeableness, openness, neuroticism and conscientiousness.
  • The respective F1 values are 0.760, 0.903, 0.829, 0.781 and 0.886; extraversion has the lowest F1 and neuroticism the lowest accuracy.
  • Validation compares each prediction with an individual human label, not with consensus or a psychometric measure of the character.
  • The published distribution shows a higher proportion of positive agreeableness labels and lower openness and extraversion.
  • The gender analysis finds 32 significant linguistic markers in the complete corpus.
  • BigWords, prepositions, work vocabulary, numbers and swear words follow in male characters the direction reported in people, but with larger effects.
  • Family, home, social processes, pronouns and negations are more frequent in female characters and also appear exaggerated.
  • The pronoun you is an inverted marker: it appears more in fictional female speech, while the cited human baseline associates it more with men.
  • The Strindberg example shows correlation r=0.8861 between human and fictional d for selected markers, but it does not represent all authors.
  • In personality, 14 MRC markers and 48 LIWC have some significant correlation.
  • Five markers are significant for all five traits and 24 for four traits.
  • Neuroticism shows the strongest positive association with word count and with negative vocabulary, anger and death.
  • Emotionally stable characters use more punctuation, according to the interpretation of the binary labels.
  • Conscientiousness is associated with word length and number of phonemes; non-conscientious characters with exclamations, quotes and disfluencies.
  • Extraversion is associated with leisure; introversion with long phrases.
  • Agreeableness is associated with positive emotion and its negative pole with negative emotion and anger.
  • Openness is associated with meaningfulness, spelling, word length and leisure, and presents more inversions compared to human baselines.
  • The repository publishes the data and derived variables under the MIT license, which improves the auditability of the resource.
  • The official CSV has 3,258 records, seven fewer than the 3,265 declared by the article and README.
  • The CSV contains 133 author values and six duplicated author-play-character keys, divergences not documented in the published version.

Limitations

  • The Big Five labels are inferences from GPT-3.5, not self-reports, longitudinal observations or clinical assessments of characters.
  • Fiction offers no objective ground truth about latent traits; interpretation of text and personality can vary between readers.
  • Each validation text has a single human annotator, so inter-human agreement is not calculated.
  • Two annotators split the sample but do not label the same cases; personal differences are confounded with the assigned texts.
  • The exact size per annotator, annotator characteristics, training, adjudication and uncertainty of metrics are not reported.
  • The 10% human sample may not preserve authors, genders, lengths, periods or trait prevalences.
  • The exact snapshot of gpt-3.5-turbo, temperature, number of calls, error handling and seed are not specified.
  • The annotation prompt does not appear in executable form in the audited repository.
  • Reducing each continuous trait to presence/absence eliminates intensity, facets and ambiguity.
  • Accuracy depends on prevalence and no intervals, confusion matrices or calibration are provided.
  • Metrics are calculated against a subjective human label called ground truth, a designation that is too strong.
  • The loneliness example itself shows plausible interpretive disagreement, not necessarily an unequivocal error by the model.
  • Correlations between language and personality are calculated on labels produced by the same LLM that the study aims to validate.
  • 65 markers are tested per five traits and multiple per-author analyses with α=0.05 without correction for multiplicity.
  • The claim of many significant markers may include false positives and dependencies between LIWC variables.
  • No intervals, bootstrap stability or replication by play or author are published.
  • Texts by the same author and play are not independent, but global tests do not model their hierarchical structure.
  • Length, dramatic genre, period, author, translation and character may explain associations attributed to personality.
  • Gender labels are binary and the article does not explain ambiguous cases, non-human characters, disguises or non-binary identities.
  • Human baselines come from different studies, corpora and periods; matching in direction does not establish equivalence.
  • Comparing only signs of significant effects loses magnitude, uncertainty and context differences.
  • The per-author analysis selects those with enough characters and then highlights the top-10, introducing subsequent selection.
  • The r=0.8861 example uses a subset of significant markers and does not summarize the distribution of authors.
  • LIWC is proprietary and limits full reproduction even if derived columns are published.
  • MRC and LIWC were developed primarily for contemporary English and may behave differently with archaic theatrical language.
  • The corpus excludes verse and non-English plays, so it does not represent theatre or fiction in general.
  • Project Gutenberg overrepresents historical public-domain texts and certain cultural canons.
  • The paper declares 3,265 samples and 132 authors, while the current artifact contains 3,258 rows and 133 authors.
  • Six duplicated author-play-character keys can bias counts if analyzed as independent observations.
  • There is no changelog or dataset version explaining the differences between paper, README and CSV.
  • The repository lacks requirements, lockfile and a complete pipeline to reconstruct extraction, annotation and figures.
  • The repository's MIT license does not individually document the conditions of each source text or of LIWC/MRC.
  • GPT biases by gender, period, dialect, length or character stereotype are not evaluated.
  • No ethical evaluation of personality and gender labeling or of possible uses to profile authors or individuals is carried out.

What the study does not establish

  • It does not demonstrate that fictional characters have objectively true Big Five traits.
  • It does not demonstrate psychometric validity of GPT-3.5 as a personality test.
  • It does not demonstrate equivalence between a binary prediction from text and stable human personality.
  • It does not demonstrate human consensus on the labels, because there is no overlap between annotators.
  • It does not demonstrate that the reported metrics generalize outside the 10% subsample.
  • It does not demonstrate causality between gender or personality and language use.
  • It does not demonstrate that fiction reflects real people; it identifies coincidences and inversions under heterogeneous comparisons.
  • It does not demonstrate that the effects are universally more expressive in fiction.
  • It does not demonstrate that r=0.8861 represents authors other than Strindberg or all markers.
  • It does not demonstrate that the hundreds of results with p<0.05 survive a correction for multiplicity.
  • It does not demonstrate independence from author, play, period, length or dramatic genre.
  • It does not demonstrate that 3,265 is the size of the current public version of the corpus.
  • It does not allow fully reconstructing the dataset and figures from scratch with the published repository.
  • It does not validate applications for profiling individuals, LLM evaluation or robotics mentioned in the discussion.
  • It does not establish that the labels are safe or appropriate for decisions about real individuals.

Traceability

Scope: Full text

Version: CogALex at LREC-COLING 2024 final paper, pp. 114-119, 6 pages; official Big-Five-Backstage repository main audited at commit cab452be63919474227cc3f441355a98b8bec84a; MIT dataset release present, published CSV contains 3,258 rows versus 3,265 reported in paper

Consulted source: https://aclanthology.org/2024.cogalex-1.13.pdf

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, psychometric, LLM-annotation, human-validation, LIWC-MRC, multiple-testing, dataset-artifact, reproducibility, bias, licensing and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-3.5-turbo, instantánea no especificada, como anotador de cinco rasgos binarios
  • Stanza para normalización y tokenización

Instruments and metrics

  • Definiciones Big Five de Mairesse y Walker para anotación binaria
  • LIWC-22, 51 variables usadas en el análisis de personalidad y 44 en género
  • MRC Psycholinguistic Database, 14 variables
  • Accuracy, precision, recall y F1 contra una etiqueta humana individual
  • Mann–Whitney U y Wilcoxon signed-rank
  • d de Cohen
  • Correlación punto-biserial y correlación entre efectos por autor

Data used

  • Big-Five Backstage, corpus de personajes teatrales de Project Gutenberg
  • Big-Five_Backstage.csv con texto y etiquetas binarias
  • Big-Five_Backstage_extended.csv con variables LIWC y MRC
  • BIG_5_human_and_GPT_annotated.xlsx con la submuestra de validación
  • authors_demographics.csv
  • Baselines psicolingüísticos humanos de Newman et al. y Mairesse y Walker

Evidence and location

  • Record and official abstract: ACL Anthology 2024.cogalex-1.13; CogALex at LREC-COLING 2024, pp. 114-119
  • Full audited source: .cache/editorial-sources/article-100/source.pdf; official ACL PDF; 6 pages; sha256 9961f096dbdfb38b378dcecc4730970aeb373c93ee456d7e7b1f7d908cbc9fdc
  • Extraction and declared size: Full text pp. 114-115, sections 1 and 2.1
  • GPT annotation and human validation: Full text p. 115, section 2.2, example and Table 1
  • Label distribution: Full text p. 115, Table 2
  • Tests and markers: Full text p. 116, section 3
  • Gender results: Full text pp. 116-117, section 4.1 and Figures 1-2
  • Big Five results: Full text pp. 116-118, section 4.2 and Figure 3
  • Acknowledged limitations: Full text pp. 117-118, section 5
  • Interpretation and applications: Full text p. 118, section 6
  • Official repository: Official https://github.com/estiei/Big-Five-Backstage main, commit cab452be63919474227cc3f441355a98b8bec84a; checked 15 Jul 2026
  • Published CSV count: Official dataset/Big-Five_Backstage.csv at audited commit: 3,258 parsed data rows, 400 author-play pairs, 133 author values, 2,007 M and 1,251 F labels
  • Duplicated keys: Official CSV at audited commit: six repeated author-play-character keys across Baker, Denison and Galsworthy records
  • Open artifacts: Official repository contains raw plays, base and extended CSVs, human/GPT annotation workbook, demographics, supplementary tables, notebook, utils.py and MIT license
  • Reproduction limits: Official repository at audited commit has no dependency manifest, exact GPT snapshot, executable annotation prompt, seed or complete extraction/analysis pipeline
  • Integral visual inspection: All 6 PDF pages rendered and visually inspected, including two tables, three figures, annotation example, methods, limitations, discussion and references; checked 15 Jul 2026