Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approach

Society, culture, and collective behavior2026arXivApproved editorial review

Authors: Xiaoyou Qin, Zhihong Li, Xiaoxiao Cheng

Keywords: Persona conditioning, Human simulation

Source: Open primary source (opens in a new tab)

3
Authors
6
Findings
11
Limitations
8
Evidence

Editorial summary

English

This study examines whether audience segmentation can better preserve heterogeneity when LLMs simulate social opinion. It starts from 594 U.S. participants recruited through Prolific in October 2025, with quotas aligned to Census distributions for gender, age, and region, and assigns them to six SASSY climate segments: Alarmed, Concerned, Cautious, Disengaged, Doubtful, and Dismissive. This is a quota-aligned nonprobability sample, not a nationally representative sample; weights, response rate, full composition, and segment sizes are not reported. Llama 3.1-70B and Mixtral 8x22B answer three seven-point scales asking whether climate change is pleasant/unpleasant, favorable/unfavorable, and positive/negative. Six configurations are compared: five demographic variables; demographics plus 59 or 15 theory-selected identifiers; 15 gradient-boosting predictors; the 15 SASSY items; and its four segmentation items. Evaluation separates distributional, structural, and a dimension called predictive fidelity. The main result is qualified: no configuration wins everywhere, and adding variables does not improve performance monotonically. Against the demographic profile's average KL divergence of 4.65, Theory-15 falls to 1.10 and Theory-59 reaches 2.76, while F1 and MAE change little. Item-15 has the lowest mean KL, .47, and the closest within-group variation, SD .99 and CV .54 against human values of 1.19 and .53. Data-driven comes closest to between-group separation, with mean nEMD .18 against a human benchmark of .19, although that average is driven by Mixtral at .25 while Llama remains .11. Every configuration retains over-regularization or unstable geometry. The so-called predictive fidelity is averaged Cramer's V association between identifiers and only three outcomes; it is not out-of-sample prediction or causal evidence. Item-4 and Item-15 also supply climate attitudes to predict three closely related climate attitudes from the same questionnaire, so their advantage may reflect near-target proxies rather than general heterogeneity. The Data-driven selection reports no split, cross-validation, or holdout and therefore does not establish out-of-sample performance. The two models were chosen after comparing other candidates partly on persona consistency and response diversity, properties close to the later target; that benchmark is not reported, creating favorable-selection bias. Exact generation counts, seeds, checkpoints, serving stack, retries, parsing failures, and reproducible definitions of KL, nEMD, and classification metrics are absent. There are no intervals, bootstrap analyses, tests, or stochastic replications. CV is questionable for Likert data because it depends on the scale's arbitrary zero, while MDS/Procrustes on only six groups lacks stress or stability analysis. The most serious documentary problem is that the text repeatedly cites Appendix A and Tables A1-A10 for questions, identifiers, prompts, and full results, yet the 43-page PDF ends after the references and the official arXiv source ends at the bibliography with no appendix. No official repository, dataset, code, outputs, or supplement was found. The paper contributes a useful multidimensional framework and descriptive evidence that variable choice matters, but it does not establish that segmentation generally restores human heterogeneity and its results cannot be reproduced exactly from the available artifacts.

Español

Este estudio examina si la segmentación de audiencias puede conservar mejor la heterogeneidad al simular opiniones sociales con LLM. Parte de 594 participantes de Estados Unidos reclutados en Prolific en octubre de 2025, con cuotas alineadas con el censo en género, edad y región, y los agrupa en seis segmentos climáticos SASSY: Alarmed, Concerned, Cautious, Disengaged, Doubtful y Dismissive. Es una muestra no probabilística con cuotas, no una muestra nacional representativa; no se publican pesos, tasa de respuesta, composición completa ni tamaños de los segmentos. Llama 3.1-70B y Mixtral 8x22B responden tres escalas Likert de siete puntos sobre si el cambio climático resulta agradable/desagradable, favorable/desfavorable y positivo/negativo. Se comparan seis configuraciones: cinco datos demográficos; esos datos más 59 o 15 identificadores teóricos; 15 predictores elegidos por gradient boosting; los 15 ítems SASSY; y sus cuatro ítems de segmentación. La evaluación separa fidelidad distribucional, estructural y una dimensión denominada predictiva. El resultado central es matizado: ninguna configuración gana en todo y añadir más variables no mejora de forma monotónica. Frente al promedio de KL 4,65 del perfil demográfico, Theory-15 baja a 1,10 y Theory-59 queda en 2,76, pero F1 y MAE apenas cambian. Item-15 obtiene el menor KL medio, 0,47, y la variación intragrupo más próxima a la humana, SD 0,99 y CV 0,54 frente a 1,19 y 0,53. Data-driven se acerca más a la separación entre grupos: nEMD medio 0,18 frente al benchmark humano 0,19, aunque la cifra depende de Mixtral, 0,25, mientras Llama queda en 0,11. Todas las configuraciones conservan sobre-regularización o geometrías inestables. La supuesta fidelidad predictiva es el promedio de asociaciones Cramér V entre identificadores y solo tres resultados; no es predicción fuera de muestra ni evidencia causal. Además, Item-4 e Item-15 introducen actitudes climáticas para predecir otras tres actitudes climáticas muy próximas del mismo cuestionario, de modo que su ventaja puede reflejar proxies cercanos al objetivo y no heterogeneidad general. La selección Data-driven tampoco informa split, validación cruzada ni holdout, por lo que no establece rendimiento fuera de muestra. Los dos modelos se escogieron tras comparar otros candidatos parcialmente por consistencia de persona y diversidad, propiedades afines al resultado posterior; no se publica ese benchmark y existe sesgo de selección favorable. Faltan número exacto de generaciones, semillas, checkpoints, motor, reintentos, fallos de parseo y definiciones reproducibles de KL, nEMD y las métricas de clasificación. No hay intervalos, bootstrap, tests ni replicaciones estocásticas. El CV es discutible en Likert porque depende del cero arbitrario, y MDS/Procrustes sobre solo seis grupos carece de estrés o estabilidad. El problema documental más grave es que el texto remite repetidamente al Apéndice A y a las Tablas A1-A10 para preguntas, identificadores, prompts y resultados completos, pero el PDF de 43 páginas termina tras las referencias y el fuente arXiv termina en la bibliografía sin apéndice. Tampoco se localizó repositorio, dataset, código, outputs o suplemento oficial. El trabajo aporta un marco multidimensional útil y evidencia descriptiva de que la elección de variables importa, pero no demuestra que la segmentación restaure en general la heterogeneidad humana ni permite reproducir exactamente sus resultados.

Research question

Can audience segmentation reduce the homogenization of social simulations with LLMs, and how do distributional, structural, and association fidelity change when varying the granularity, parsimony, and selection logic of the identifiers?

Method

594 Prolific participants are surveyed and assigned to six SASSY climate segments. Two open-weight LLMs receive six persona configurations and respond to three Likert climate items in zero-shot, temperature 0.8, and top-p 1.0. Means, classification metrics, and KL are compared; SD/CV, nEMD, and MDS/Procrustes; and Cramer's V associations between identifiers and outcomes.

Sample: 594 U.S. Prolific participants, with gender, age, and region quotas aligned to the census, collected in October 2025 and filtered by IP, attention, response time, and manual inspection. No weights, response rate, exclusions, full composition, or n per segment are published. The exact number of synthetic responses and repetitions is not specified.

Findings

  • No configuration dominates the three dimensions, and more identifiers do not imply more fidelity: Theory-15 outperforms Demo and Theory-59 on several metrics, but the differences depend on the indicator.
  • Mean KL drops from 4.65 in Demo to 1.10 in Theory-15 and 2.76 in Theory-59; mean F1 is 0.425, 0.455, and 0.445, and MAE 0.64, 0.63, and 0.69.
  • Item-15 achieves the lowest mean KL, 0.47, and the intragroup variation closest to human, SD 0.99/CV 0.54 versus 1.19/0.53.
  • Data-driven reaches mean nEMD 0.18 versus 0.19 human and mean Procrustes 0.48, but nEMD depends on Mixtral 0.25; Llama remains at 0.11.
  • Item-4 achieves the highest Cramer's V associations, but uses the four climate items that define the segment to predict three proximate climate outcomes.
  • All designs still compress or restructure part of the human variation; the study acknowledges that segmentation is not a total solution.

Limitations

  • The Prolific sample with quotas on three variables is neither probabilistic nor representative of the United States; weights, recruitment denominator, composition, and subgroup sizes are missing.
  • Only three affective climate items are predicted with two models chosen after unpublished screening; there is no basis to generalize to other domains, behaviors, populations, or LLMs.
  • The Item-4/15 identifiers are climate proxies proximate to the outcomes, and the missing appendix prevents measuring the exact overlap.
  • The Data-driven selection does not document split, cross-validation, holdout, or nested selection, so it may be evaluated on the same participants used to choose variables.
  • The models are partially selected for diversity before evaluating heterogeneity, without a table or independent protocol, which introduces favorable selection bias.
  • No synthetic n, repetitions, seeds, checkpoints, serving, chat template, parser, failures, retries, or exclusions are reported.
  • KL lacks direction/smoothing/base; nEMD does not define normalization; accuracy/precision/recall/F1 do not explain how they convert two distributions into comparable classes.
  • CV is not suitable without reservations for a Likert with an arbitrary zero, and MDS/Procrustes of six groups does not report stress, seed, bootstrap, or stability.
  • There are no intervals, tests, bootstrap, stochastic replication, or multiplicity control; the differences are descriptive.
  • Appendix A and Tables A1-A10 cited do not exist in the PDF or in the official TeX package; questions, identifiers, prompts, and complete tables are missing.
  • No code, data, outputs, supplement, or reproducible analysis are located, and IRB/ethics, consent, compensation, or preregistration are not documented either.

What the study does not establish

  • It does not demonstrate that the sample represents U.S. public opinion, nor does it allow estimating national or per-segment prevalences.
  • It does not demonstrate that segmentation restores human heterogeneity in general; it only describes three climate items in two preselected models.
  • It does not establish uniform superiority of Item-15, Data-driven, or any other configuration: each favors different metrics.
  • It does not test out-of-sample predictive capacity or causal relationships; Cramer's V only compares associations within this design.
  • It does not fully separate fidelity from target-proximate information in Item-4/15 or the risk of in-sample selection in Data-driven.
  • It does not support that more identifiers are always worse; it shows non-monotonic performance within six specific configurations.
  • It does not allow exact reproduction of treatments, prompts, metrics, or results without the missing appendix and artifacts.
  • It does not validate that small differences are stable across seeds, samples, metric definitions, or alternative models.

Traceability

Scope: Full text

Version: arXiv:2604.06663v1 preprint

Consulted source: https://arxiv.org/pdf/2604.06663v1

Review: Codex 43-page visual full-text, TeX/source, appendix completeness, sampling, target-proxy, model-selection, metric, statistical, human-subjects, artifact and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Llama 3.1-70B
  • Mixtral 8x22B
  • Qwen, Gemma 3 and BLOOM in an undocumented preliminary model screen
  • Gradient-boosting model for Data-driven identifier selection

Instruments and metrics

  • SASSY six-segment climate audience framework
  • Three seven-point climate-affect items Q25-Q27
  • Mean absolute error
  • Accuracy, weighted precision, weighted recall and F1
  • KL divergence
  • Within-group standard deviation and coefficient of variation
  • Median normalized Earth Mover's Distance
  • Multidimensional scaling and Procrustes distance
  • Cramer's V identifier-outcome associations

Data used

  • October 2025 U.S. Prolific climate-opinion survey, n=594
  • Six SASSY audience segments
  • LLM-generated responses under six segmentation configurations

Evidence and location

  • Objective, heterogeneity framework, and six segments: arXiv v1, pp. 1-6, Abstract, Introduction and theoretical framework
  • Prolific sample, screening, six configurations, and models: arXiv v1, pp. 7-11, Methods; official TeX lines 224-290
  • Three outcomes, decoding and three-dimensional evaluation: arXiv v1, pp. 11-16, Methods and Figure 1
  • Granularity, distribution, SD/CV, nEMD and Procrustes: arXiv v1, pp. 16-24, Results and Figures 2-4
  • Parsimony, identifier-selection logic and Cramer's V: arXiv v1, pp. 24-34, Results and Figures 2-5
  • Conclusions, residual over-regularization and declared limitations: arXiv v1, pp. 34-38, Discussion
  • Absence of the repeatedly cited Appendix A and Tables A1-A10: All 43 PDF pages; official arXiv source package SHA-256 9390c8c4cf3250b088585425fd97ecd84ef0e40bd0bc6fe9cc952885dddd9c7a; main.tex ends after bibliography
  • Integrated sample, model-selection, target-proxy, metric, statistics and reproducibility audit: reports/verification/article-375-audience-segmentation-missing-appendix-model-selection-target-proxy-metric-validity-statistics-human-sample-and-reproducibility-audit.json