Synthetic Personality Review

Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment

Society, culture, and collective behavior2026arXivApproved editorial review

Authors: Bryan Chen Zhengyu Tan, Zhengyuan Liu, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Roy Ka-Wei Lee

Keywords: Persona conditioning, Safety and bias

Source: Open primary source (opens in a new tab)

7
Authors
9
Findings
17
Limitations
5
Evidence

Editorial summary

English

Tan and colleagues study whether an LLM can predict the modal value of demographic subgroups in Singapore. They start from 2,012 WVS Wave 7 participants, retain 214 questions and build aggregate labels by sex, age, ethnicity, religion and selected intersections. Seven open models receive one epoch of LoRA training to output each subgroup's most frequent numerical option. On held-out age-religion, age-ethnicity and ethnicity-religion intersections, mean accuracy rises from 0.450 to 0.624 and NMAE falls from 0.269 to 0.173. Transfer to free text against GPT-4.1 is much smaller and heterogeneous: +2.2 win-rate points for value alignment, +1.1 overall and -0.6 for persona; several intervals include zero and Phi-4-mini worsens. Three annotators support the automatic judge more strongly for value than for persona authenticity. Fairness conclusions depend on the metric: average disparity falls under exact accuracy but rises under ordinal error. This is a holdout of demographic combinations, not questions, people or cultures: the same 214 questions, category levels and 2,012 respondents feed training and evaluation. Modal labels erase distributions and minority views, there is no simple statistical baseline or repeated training, the bootstrap ignores clustering, and the text disagrees between 20,877 and 22,837 pairs. No code, derived data, outputs or annotations are released; WVS prohibits redistribution of its files, but the execution recipe, official version/DOI and reproducibility artifacts are also missing. The study shows behavioral transfer within a fixed ontology, not individual values, cultural authenticity or an internal persona representation.

Español

Tan y colaboradores estudian si un LLM puede predecir el valor modal de subgrupos demográficos de Singapur. Parten de 2.012 participantes de WVS Wave 7, conservan 214 preguntas y forman etiquetas agregadas por sexo, edad, etnia, religión y algunas intersecciones. Siete modelos abiertos reciben un epoch de LoRA para responder con la opción numérica más frecuente de cada subgrupo. En intersecciones reservadas de edad-religión, edad-etnia y etnia-religión, la exactitud media sube de 0,450 a 0,624 y NMAE baja de 0,269 a 0,173. La transferencia a texto libre frente a GPT-4.1 es mucho menor y desigual: +2,2 puntos de win rate en alineación de valor, +1,1 global y -0,6 en persona; varios intervalos incluyen cero y Phi-4-mini empeora. Tres anotadores respaldan mejor al juez automático en valor que en autenticidad de persona. La equidad depende de la métrica: la disparidad media baja con exactitud, pero aumenta con error ordinal. Este es un holdout de combinaciones demográficas, no de preguntas, personas ni culturas: las mismas 214 preguntas, categorías y 2.012 encuestados alimentan train y evaluación. La moda borra distribución y voces minoritarias, no hay baseline estadístico simple ni repeticiones de entrenamiento, el bootstrap ignora agrupación, y el texto discrepa entre 20.877 y 22.837 pares. No se publican código, datos derivados, outputs ni anotaciones; WVS prohíbe redistribuir sus ficheros, pero faltan también receta, versión/DOI oficial y artefactos reproducibles. El trabajo demuestra transferencia conductual dentro de una ontología fija, no valores individuales, autenticidad cultural ni una representación interna de persona.

Research question

Can supervised tuning with modal responses of subgroups teach several LLMs to predict values of demographic intersections not included in training and transfer that behavior to open text without aggravating disparities?

Method

The study filters the Singapore sample of WVS Wave 7 to 214 questions and defines as the target the modal numeric option of each subgroup with at least 30 respondents. It trains seven open LLMs with LoRA for one epoch on 50 subgroups of individual axes and crosses with sex, and evaluates 48 reserved intersections of age, ethnicity, and religion. It measures accuracy and NMAE; for open text, Mistral-Small-3.1-24B compares each response with GPT-4.1 on persona, value, and overall quality. It calculates normalized range and coefficient of variation across subgroups, uses paired bootstrap by question-subgroup pair, and validates the judge with three annotators over 100 comparisons.

Sample: The primary source is 2,012 respondents from Singapore. The modeled dataset declares 10,700 pairs for 50 training subgroups and 10,177 for 48 evaluation subgroups, 20,877 in total. The rows are not independent persons: each individual response contributes to multiple overlapping subgroups and each of the 214 questions reappears across subgroups. The judge validation uses three Singaporean annotators or residents of at least five years over 100 comparisons.

Findings

  • In the intersection holdout, the mean of seven open models goes from 0.450 to 0.624 in accuracy and from 0.269 to 0.173 in NMAE; Sailor2-8B achieves the largest jump, from 0.356 to 0.720.
  • The bootstrap intervals per pair exclude zero for the seven accuracy improvements, under a resample that does not cluster by question, subgroup, or overlapping respondents.
  • In open text against GPT-4.1, Value WR increases from 0.300 to 0.322 and Overall WR from 0.236 to 0.247, while Persona WR drops from 0.240 to 0.234; the direction and significance vary by model.
  • Phi-4-mini worsens significantly in Value and Overall WR; Llama-3.2 does not change distinguishably and several overall improvements include zero.
  • The mean disparity by accuracy decreases, with range 0.240 to 0.179 and CV 0.078 to 0.054, but by NMAE it increases, with range 0.280 to 0.336 and CV 0.094 to 0.116.
  • The judge agrees with humans better on value, kappa 0.631, and overall, 0.568, than on persona, 0.318; on persona it also falls below the human-human agreement of 0.388.
  • Religious Values leads both Modal Diversity Score and Wasserstein, but the correlation between both category orderings is only 0.091, so other conclusions about divisiveness depend on the metric.
  • SFT reduces non-parseable outputs, including abstentions on sensitive questions; the study does not evaluate whether that reduction preserves safety or avoids harm.
  • No repository, derived dataset, executable configuration, outputs, judge decisions, or human annotations appear in the article, ACL, source package, or specific searches.

Limitations

  • The OOD split reserves combinations of labels, but reuses all questions, demographic levels, and respondents; it does not measure population, cultural, or new-question generalization.
  • The same individuals feed modes of overlapping subgroups in train and evaluation, so the pairs are not independent observations.
  • Each question is seen in training with other subgroups; without a split by question or baselines of global mode, margins, interpolation, or demographic regression, thematic memorization is not separated from persona composition.
  • The mode discards dispersion, ties, uncertainty, and minority opinions; balancing rows by subgroup does not balance persons or representativeness.
  • The threshold N greater than or equal to 30 is called robust without analysis of power, precision, or probability of change of the mode.
  • The nearly null correlation between N and modal margin does not prove stability: it only denies a monotonic association and does not perform label resampling or external validation.
  • The stability analysis says it uses 22,837 pairs, while the final composition declares 20,877; the difference of 1,960 is not explained.
  • The bootstrap resamples individual pairs and does not respect dependency by question, subgroup, or respondent; its intervals may be too narrow.
  • A single global seed is reported and there are no training replicas; the intervals do not capture variation across fine-tunings.
  • The disparity changes have no intervals, tests, or sensitivity to weighting strata by persons, subgroups, or precision.
  • Three annotators and 100 comparisons are a small calibration; descriptively similar kappas do not prove equivalence between human and automatic judges.
  • A single 24B judge compares against a single GPT-4.1 and sees the target value; there is no alternative judge, absolute scale, or human replica of the full experiment.
  • Performance differences by group are biases relative to this task and target, not causal evidence that pretraining data produce discriminatory social treatment.
  • The article acknowledges essentialism, silencing of minorities, targeted propaganda, and risk of using aggregate profiles as if they were individuals.
  • WVS prohibits redistributing its files, but code, acquisition instructions, version and official DOI, hashes of legal inputs, adapters, outputs, and commands are also missing.
  • The article does not cite in references the official WVS Wave 7 dataset with DOI 10.14281/18241.18 and does not document ethical review, consent, recruitment, or compensation of the annotators.
  • Reducing refusals on topics of violence, sexuality, abortion, or terrorism may weaken safety abstentions; no harm or subsequent compliance is measured.

What the study does not establish

  • It does not demonstrate values, personality, identity, cultural authenticity, mind, or individual experience in an LLM.
  • It does not allow attributing to a real person the modal response of their demographic categories.
  • It does not demonstrate generalization to new cultures, countries, periods, questions, respondents, demographic axes, or unseen categories.
  • It does not prove that the model has learned an internal representation of persona; it only observes output changes after SFT.
  • It does not establish a general improvement of open text: the average is small, one model worsens, and several changes are not significant.
  • It does not establish that the automatic judge measures persona authenticity with high reliability; that is precisely its weakest dimension.
  • It does not demonstrate that lower disparity in accuracy implies greater equity, because NMAE moves in the opposite direction.
  • It does not identify pretraining as the cause of the patterns across subgroups or demonstrate discriminatory impact on users.
  • It does not validate the stability of the modal labels or resolve the count discrepancy between 20,877 and 22,837 pairs.
  • It does not offer integral reproduction from public artifacts, even though it is a Long Paper peer-reviewed for ACL 2026.

Traceability

Scope: Full text

Version: arXiv:2604.12851v1; ACL 2026 final inspected as authoritative publication

Consulted source: https://arxiv.org/abs/2604.12851

Review: Codex dual 20-page visual full-text, TeX/source, ACL checklist, WVS provenance, split, construct, statistics, fairness, human-judge and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Llama-3.1-8B-Instruct
  • Llama-3.2-3B-Instruct
  • SEA-LION-v3-8B-Instruct
  • Phi-4-mini-instruct
  • Qwen2.5-7B-Instruct
  • Sailor2-8B-Chat
  • SeaLLMs-v3-7B-Chat
  • GPT-4.1
  • GPT-4o
  • GPT-4.1-mini
  • GPT-4o-mini
  • Mistral-Small-3.1-24B-Instruct-2503 como juez

Instruments and metrics

  • Exactitud de respuesta modal
  • Error absoluto medio normalizado, NMAE
  • Modal Diversity Score
  • Distancia de Wasserstein entre distribuciones
  • Win rate LLM-as-a-Judge con orden intercambiado
  • Rango normalizado y coeficiente de variación
  • Kappa ponderado y acuerdo humano
  • Bootstrap pareado de 2.000 remuestras

Data used

  • World Values Survey Wave 7, muestra de Singapur
  • 20.877 pares agregados pregunta-subgrupo declarados en la tabla principal
  • 214 preguntas WVS retenidas
  • 98 subgrupos demográficos con N igual o mayor que 30

Evidence and location

  • Publication, design, results, prompts, appendices, limits, and ethics: ACL 2026 Long Paper, DOI 10.18653/v1/2026.acl-long.1127, 20 pages rendered and inspected
  • Editable source and scope of artifacts, without implementation or data: arXiv:2604.12851v1 source SHA-256 24c9c731bcb89624b626f86a444d34bcb02ae60fc81f1e6a54627e6ef4bc547c
  • Reproducibility, data, and human evaluation statements: Responsible NLP Checklist SHA-256 8eb042bcccb49ed182c6df9d8d8e8e3eac36590bea0c1f2ccc40a7c3f1ea0547, 2 pages inspected
  • Official terms of use and non-redistribution of WVS: World Values Survey download license and Wave 7 documentation, inspected 2026-07-17
  • Audit of split, construct, statistics, fairness, human judge, data, and reproducibility: reports/verification/article-364-acl-subgroup-values-wvs-compositional-split-fairness-human-judge-data-and-reproducibility-audit.json