Synthetic Personality Review

Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks

Society, culture, and collective behavior2024ACL AnthologyApproved editorial review

Authors: Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent V. Frigo, Sijia Yang, Dhavan V. Shah, Junjie Hu, Timothy T. Rogers

Keywords: Human belief networks, Role-playing agents, Opinion alignment, In-context learning, Supervised fine-tuning

Source: Open primary source (opens in a new tab)

9
Authors
11
Findings
22
Limitations
5
Evidence

Editorial summary

English

The study asks whether one real opinion from a person helps an LLM representing that person approximate their other opinions better than a demographic profile does. It uses 564 US Amazon Mechanical Turk participants surveyed in 2018: each rated 64 statements on a forced six-point scale and supplied demographic attributes. A prior analysis of the 64-topic correlation matrix applies PCA with Varimax rotation, retains nine orthogonal factors explaining 72% of matrix variance, and assigns every topic to its highest-loading factor. The highest-loading topic in each factor becomes the seed opinion and the remaining 55 topics are tests. With ChatGPT, GPT-4o-mini, Mistral-7B and Llama-3.1-8B, the paper compares a generic role, demographics only, the seed opinion only, demographics plus a same-factor seed, demographics plus a different-factor seed, and an upper bound that also reveals the target answer. Demographics alone are not consistently helpful: relative to the generic role, mean MAE falls for GPT-4o-mini and Llama but rises for ChatGPT and Mistral. By contrast, demographics plus the same-factor seed produces the lowest average non-upper-bound MAE for all four models: 1.34, 1.16, 1.29 and 1.60, versus 1.67, 1.35, 1.55 and 2.16 with a different-factor seed. A reverse-framing control preserves the ordering while slightly weakening the result. Fine-tuning gpt-3.5-turbo-0125 gives the same descriptive pattern, but only for the Ghost and Partisan factors, 18 topics in total. The narrow conclusion is useful: within this sample, knowing one correlated opinion predicts the same person's other responses better than demographics alone. It is not external validation, however. The same 564 participants and their eventual test responses were used to discover the factors, assign topics and choose seeds; there are no held-out people, independent survey or later wave. The text calls effects 'significant' without intervals or tests, reports no seeds or stochastic replications, and says MAE ranges from 0 to 4 even though its published -3 to +3 coding permits differences up to 6. The data require contacting the authors and, despite a promised release, no public code, outputs or scoring implementation was found. The work therefore supports internal association and prediction, not causality, belief propagation, or faithful and generalizable digital twins.

Español

El estudio pregunta si una opinión real de una persona ayuda a un LLM que la representa a aproximar sus demás opiniones mejor que una ficha demográfica. Parte de 564 participantes estadounidenses de Amazon Mechanical Turk encuestados en 2018: cada uno valoró 64 afirmaciones con una escala forzada de seis puntos y aportó datos demográficos. Un análisis previo de las correlaciones entre los 64 temas aplica PCA con rotación Varimax, retiene nueve factores ortogonales que explican el 72% de la varianza de la matriz y asigna cada tema al factor donde más carga. El tema con mayor carga de cada factor funciona como opinión semilla y los otros 55 como pruebas. Con ChatGPT, GPT-4o-mini, Mistral-7B y Llama-3.1-8B, el paper compara un rol genérico, solo demografía, la opinión semilla sola, demografía más una semilla del mismo factor, demografía más una semilla de otro factor y un techo que revela también la respuesta objetivo. La demografía sola no mejora de forma consistente: frente al rol genérico, el MAE medio baja en GPT-4o-mini y Llama, pero sube en ChatGPT y Mistral. En cambio, demografía más semilla del mismo factor logra el menor MAE medio no supervisado en los cuatro modelos: 1,34, 1,16, 1,29 y 1,60, frente a 1,67, 1,35, 1,55 y 2,16 al usar una semilla de otro factor. Un control con enunciados invertidos conserva el orden, aunque debilita algo el resultado. El ajuste fino con gpt-3.5-turbo-0125 muestra el mismo patrón descriptivo, pero solo para los factores Ghost y Partisan, 18 temas en total. La conclusión estrecha es útil: dentro de esta muestra, conocer una opinión correlacionada permite predecir mejor otras respuestas de esa misma persona que conocer solo sus datos demográficos. No obstante, no es validación externa. Los mismos 564 participantes y también sus respuestas de prueba se usaron para descubrir los factores, asignar temas y elegir las semillas; no hay personas, encuesta ni oleada independientes. El texto habla de efectos 'significativos' sin intervalos ni tests, no reporta semillas o repeticiones y afirma que el MAE va de 0 a 4 aunque la codificación publicada de −3 a +3 permite diferencias de hasta 6. Los datos solo se obtienen contactando a los autores y, pese a prometer su liberación, no se localizó código, outputs ni scoring públicos. Por ello el trabajo respalda asociación y predicción interna, no causalidad, propagación de creencias ni 'gemelos digitales' fieles y generalizables.

Research question

Does adding to the agent the actual response of a person to a central topic of a belief network improve agreement with their responses to other related topics, compared to using only demographics or an opinion from a different network?

Method

564 complete responses to 64 topics are used. A prior PCA with Varimax rotation retains nine factors and assigns each topic to its maximum loading; the highest-loading topic of each factor is the seed and the other 55 are test. Four LLMs respond with six Likert categories under six prompting conditions and are compared by MAE. gpt-3.5-turbo-0125 is additionally fine-tuned with the 564 profiles and labels, with upsampling, but the published SFT result is limited to Ghost and Partisan.

Sample: 564 people residing in the United States recruited on Amazon Mechanical Turk in 2018. Each responded to all 64 topics with no neutral option and provided nine types of demographic information. In ICL, nine seed topics and 55 test topics are used. The reported SFT evaluates only 18 topics from two factors.

Findings

  • Demographics alone do not consistently improve mean MAE compared to a generic role: it helps GPT-4o-mini and Llama 3.1, but worsens ChatGPT and Mistral.
  • Demographics plus same-factor seed achieves the lowest unsupervised mean MAE across the four models.
  • ChatGPT goes from 1.70 with demographics and 1.67 with random seed to 1.34 with same-factor seed; mean Relative Gain, 22.54%.
  • GPT-4o-mini goes from 1.33 and 1.35 to 1.16; mean Relative Gain, 13.88%.
  • Mistral goes from 1.51 and 1.55 to 1.29; mean Relative Gain, 26.29%.
  • Llama 3.1 goes from 1.91 and 2.16 to 1.60; mean Relative Gain, 39.99%.
  • The seed opinion without demographics also helps, but the combination of both gives the best average across the four models.
  • The control with original and inverted formulations maintains the ordering, with same-factor MAE of 1.41, 1.21, 1.31, and 1.71.
  • ChatGPT sensitivity maintains the ordering at temperatures 0, 0.7, and 1, with MAE 1.37, 1.34, and 1.44 for the main condition.
  • GPT-3.5 SFT on Ghost obtains 1.29 compared to 2.58 with demographics and 2.31 with random factor; on Partisan, 1.25 compared to 1.41 and 1.35.
  • The results are descriptive and heterogeneous by factor; the main condition is not the best in each individual combination, although it is in the average of each model.

Limitations

  • The factor network and the evaluation use the same 564 people and the same 64 responses; the test outcomes contribute to constructing the factors that are then evaluated.
  • There are no held-out participants, external sample, new wave, or confirmatory validation of the factor structure.
  • SFT sees the 564 demographic profiles and then evaluates other topics from those same people; it does not test generalization to new individuals.
  • The paper uses 'significantly' and 'significantly correlated' without reporting correlations, hypothesis tests, intervals, or standard errors.
  • The scale declares values −3, −2, −1, +1, +2, and +3, but the text says the maximum MAE is 4; a transformation explaining this incompatibility is missing.
  • Relative Gain is the mean of nine ratios per factor, not the ratio computed over the mean MAEs; the aggregation should be interpreted with care.
  • The random factor does not publish selection, seed, number of repetitions, or average over draws.
  • Parsing, invalid responses, failed calls, and missingness of the LLM outputs are not explained.
  • There are no repetitions by seed; testing three temperatures does not quantify stochastic uncertainty.
  • The survey forces a direction with no neutral option and comes from a 2018 US MTurk sample.
  • The detailed combination of state, age, race, income, city, and affiliation may identify profiles in SFT; privacy or memorization is not analyzed.
  • A rotated PCA of correlations does not establish a causal network or belief propagation.
  • The ceiling reveals the target response; it measures instruction-following in addition to serving as a normalizer.
  • SFT only covers Ghost and Partisan, not the nine factors.
  • The limitations section says 18 topics and two factors, in tension with the ICL study of 64 topics and nine factors unless it refers only to SFT.
  • The conclusion speaks of distant topics 'within' the network although the control uses different factors; this is an internally inconsistent formulation.
  • The data were obtained only by contacting the dataset authors, and raw responses, reproducible loadings, and splits are not published.
  • No code, outputs, scoring, or public fine-tuning files were located despite the ethics statement saying the code will be released.
  • Reproduction depends on historical snapshots of proprietary APIs and on a Mistral model that was then subject to a non-production license.
  • One author's public page links the title to the wrong arXiv, 2305.14328, instead of 2406.17232.
  • The study measures Likert choices, not dialogue, behavior, persuasion, social dynamics, or realistic communication.
  • Harm, manipulation, targeting, equity across groups, and misuse of psychographic profiles are not evaluated.

What the study does not establish

  • It does not demonstrate that demographics never matter; two models improve descriptively compared to the generic role.
  • It does not demonstrate prediction of opinions for unseen people.
  • It does not validate the factor network outside the 564 responses that generated it.
  • It does not establish causality or that a belief propagates to others.
  • It does not turn the agent into a faithful digital twin of the person.
  • It does not demonstrate statistical significance of the differences.
  • It does not demonstrate the same effect across all factors, models, or participants.
  • It does not generalize SFT to the nine factors; it only reports two.
  • It does not allow reconstructing the metric with certainty from the described scale.
  • It does not allow reproducing end-to-end results with public artifacts.
  • It does not generalize to other populations, cultures, time periods, surveys, or ways of expressing opinion.
  • It does not establish safety or suitability for social simulation, persuasion, or real segmentation.

Traceability

Scope: Full text

Version: Findings of ACL: EMNLP 2024, pages 14010-14026; arXiv:2406.17232v2 and public-artifact availability audited separately

Consulted source: https://aclanthology.org/2024.findings-emnlp.819/

Review: Codex 17-page visual, official-ACL, arXiv-v2, full-method, prompt, table-arithmetic, data-quality, transductive-design, metric-scale, stochastic-control, privacy, artifact-availability, reproducibility and claim-boundary audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • gpt-3.5-turbo-0125 (ChatGPT and supervised fine-tuning)
  • gpt-4o-mini-2024-07-18
  • Mistral-7B-Instruct-v0.2
  • Llama-3.1-8B-Instruct

Instruments and metrics

  • Controversial Beliefs Survey, 64 statements
  • Forced six-point opinion scale: -3, -2, -1, +1, +2, +3
  • PCA with Varimax rotation and scree-plot factor retention
  • Nine highest-loading training topics and 55 test topics
  • Six in-context agent-construction conditions
  • Mean absolute error over paired human and LLM ratings
  • Factor-wise Relative Gain against target-revealing upper bound
  • Original plus reverse-framed prompt control
  • Temperature sensitivity at 0, 0.7 and 1
  • Two-factor supervised fine-tuning comparison

Data used

  • Controversial Beliefs Survey from Frigo (2022), contact-only access
  • 564 US Amazon Mechanical Turk respondents collected in 2018
  • 64 opinion topics assigned to nine rotated factors
  • Demographics: age, gender, education, race, household income, city size, urban/rural setting, state and political leaning

Evidence and location

  • Metadata, DOI, pages, authors, license, and exact abstract: ACL Anthology 2024.findings-emnlp.819
  • Method, prompts, tables, metrics, results, limitations, ethics, and appendices: Findings EMNLP 2024 PDF, 17 pages, sha256 97104f79c6d4e3bae94ab4f0b6f6628f8dc02db3d24dc4504e5b65b6061e8a1e
  • History, version, and publication reference: arXiv:2406.17232v2
  • Code availability and public repositories: ACL/arXiv artifact links, GitHub global title and ID searches, github.com/yunshiuan public repositories; checked 2026-07-16
  • Transductive audit, MAE scale, random control, statistics, privacy, reproducibility, and claim boundaries: reports/verification/article-246-emnlp-belief-network-transductive-evaluation-metric-code-and-claim-audit.json