Mapping how LLMs debate societal issues when shadowing human personality traits, sociodemographics and social media behavior

Society, culture, and collective behavior2026arXivApproved editorial review

Authors: Ali Aghazadeh Ardebili, Massimo Stella

Keywords: Cognitive Digital Shadows, Synthetic personas, LLM-generated discourse, Sociodemographic prompting, OCEAN, Textual forma mentis networks, Topic anchoring, Corpus integrity, Pooling dashboard, Reproducibility audit

Source: Open primary source (opens in a new tab)

2
Authors
11
Findings
29
Limitations
4
Evidence

Editorial summary

English

This preprint introduces Cognitive Digital Shadows (CDS), a corpus of LLM responses on four social topics: vaccines and health, fake social-media content, the gender gap in science and stereotype threat in STEM. Records are generated either in AI-assistant mode or under a synthetic persona built from age, gender, sexuality, work, education, city, migration, religion, social-media use, psychological labels and OCEAN scores. Its useful contribution is infrastructural: it preserves prompts, selections and responses, releases derived textual networks and supplies a Colab notebook for filtering profiles and visualizing semantic and emotional networks. Validation through textual forma mentis networks asks only whether words taken from the prompted topic are frequent or structurally different from other words. The paper correctly states that this does not validate truth, quality, persona realism or faithfulness to any human population. Full audit of the public commit finds 226,571 valid JSON records rather than an identifiable 190,000-record subset: 133,219 are human mode and 93,352 AI mode, with the four topics nearly balanced. However, no human persona contains the stated 17 attributes because biological_sex is absent from all 133,219. Only 128,676 opinions, 56.79%, satisfy the promised 250-500 word range; 94,836 are shorter and 3,059 longer. In addition, 2,415 GPT-oss records use temperature 0.0 although the text says every simulation uses 0.7. The 19 analysis CSVs contain 226,924 names: 560 have no raw JSON and 207 released JSONs are omitted, so raw-to-analysis lineage does not close. Several directories mix versions or families: Claude Sonnet 4.5 contains two Claude 3.5 outputs; Mistral Small combines mistral-small-latest and mistral-large-2512; and Phi-4-mini-instruct actually contains Phi-4-mini-reasoning. Released code also cannot regenerate the corpus: both ROLE_MODES entries are fixed to llm, biological sex is omitted, current_marital_status is written where the parser reads marital_status, only one Qwen configuration is covered, no random seed is preserved, and the reasoning-summary description is contradicted by a prompt requesting full step-by-step derivations. Topic-anchoring tests produce tiny p-values from flattened, dependent observations; significance does not imply higher centrality, and the executed LiquidAI notebook has topic-keyword medians equal to or below other words for all four reasoning-summary tests. The dashboard is a notebook rather than a hosted application: it requires Colab, unpinned dependencies, a mounted Drive and a mutable 773 MB parquet. Zenodo preserves only the 76 kB v3 notebook; the repository has no global license, tests, CI or locked environment. CDS remains a large resource for studying conditioned outputs and designing future audits, but it should be cited as a synthetic collection with material schema and reproducibility drift, never as a survey, population simulator or evidence of real-world group beliefs.

Español

Este preprint presenta Cognitive Digital Shadows (CDS), un corpus de respuestas de LLM sobre cuatro temas sociales: vacunas y salud, contenido falso en redes, brecha de género en ciencia y amenaza de estereotipo en STEM. Los registros se generan en modo asistente de IA o con una persona sintética construida a partir de edad, género, sexualidad, trabajo, educación, ciudad, migración, religión, uso de redes, rasgos psicológicos y puntuaciones OCEAN. La contribución útil es infraestructural: conserva prompts, selecciones y respuestas, publica redes textuales derivadas y ofrece un notebook de Colab para filtrar perfiles y visualizar redes semánticas y emocionales. La validación mediante textual forma mentis networks sólo pregunta si palabras tomadas del tema del prompt son frecuentes o estructuralmente distintas del resto. El propio artículo delimita correctamente que esto no valida verdad, calidad, realismo de persona ni fidelidad a una población humana. La auditoría integral del commit público encuentra 226.571 JSON válidos, no un subconjunto identificable de 190.000: 133.219 son modo humano y 93.352 modo IA, con los cuatro temas casi equilibrados. Sin embargo, ninguna persona humana contiene los 17 atributos declarados porque `biological_sex` falta en las 133.219. Sólo 128.676 opiniones, 56,79%, cumplen las 250–500 palabras prometidas; 94.836 son más cortas y 3.059 más largas. Además, 2.415 registros GPT-oss usan temperatura 0,0 aunque el texto afirma 0,7 para todas las simulaciones. Los 19 CSV de análisis suman 226.924 nombres: incluyen 560 sin JSON fuente y omiten 207 JSON publicados, de modo que la línea raw-análisis no cierra. Varios directorios mezclan versiones o familias: el de Claude Sonnet 4.5 contiene dos salidas de Claude 3.5; Mistral Small mezcla `mistral-small-latest` y `mistral-large-2512`; y Phi-4-mini-instruct contiene en realidad Phi-4-mini-reasoning. El código liberado tampoco regenera el corpus: fija los dos valores de `ROLE_MODES` a `llm`, omite sexo biológico, escribe `current_marital_status` donde el parser busca `marital_status`, cubre una sola configuración Qwen, no conserva semilla y contradice la descripción del reasoning summary al pedir derivaciones completas paso a paso. Los tests de anclaje producen p muy pequeños con observaciones aplanadas y dependientes; además, significación no implica mayor centralidad, y en el notebook ejecutado de LiquidAI las cuatro medianas de palabras temáticas de los resúmenes son iguales o inferiores al resto. El dashboard es un notebook, no una aplicación alojada: exige Colab, dependencias sin fijar, montar Drive y cargar un parquet mutable de 773 MB. Zenodo preserva sólo el notebook v3 de 76 kB; el repositorio carece de licencia global, tests, CI y entorno bloqueado. CDS sigue siendo un recurso grande para estudiar respuestas condicionadas y diseñar auditorías futuras, pero debe citarse como colección sintética con importantes derivas de esquema y reproducibilidad, nunca como encuesta, población simulada o evidencia de opiniones reales.

Research question

How can a traceable corpus be constructed and made available to compare the discourse generated by different LLMs under assistant roles and synthetic configurations of personality and sociodemographics on sensitive social topics?

Method

JSON documents are generated with 19 nominal model folders, a target temperature of 0.7, and four social prompts. Each execution selects topic and mode; in human mode a synthetic combination of attributes and OCEAN is attached, and in AI mode there is no persona. The model returns opinion, reasoning summary, tone, and source names. After structural controls, the texts are converted with EmoAtlas into textual forma mentis networks. For each model, topic, and layer, degrees of thematic words and of the rest are flattened and Kruskal-Wallis is applied; frequency is also examined. A Panel/Colab notebook loads a unified parquet, filters by metadata, and draws networks, emotional flowers, ego networks, and mindset streams. The audit visually read the 18 pages, inspected TeX, commit, the 226,571 JSON, 19 CSV, code, notebooks, pickles, Zenodo, and Drive; reconstructed schema, modes, topics, lengths, temperatures, model identities, and file lineage, and performed syntactic validation of the code.

Sample: The article calls the corpus 190,000 records and states a minimum of 10,000 validated for each of 19 models. The audited commit contains 226,571 JSON: 133,219 in human mode and 93,352 in AI mode. The selected topics are vaccines/health 57,015, gender gap 56,660, STEM stereotype 56,496, and false content 56,400. The files span from October 6, 2025 to April 28, 2026. There are no human participants, population samples, or observed opinions; each persona is a random combination of prompt attributes.

Findings

  • The 226,571 public JSON are parsed and retain non-empty opinion and reasoning summary.
  • The four topics are nearly balanced, between 56,400 and 57,015 records.
  • The public snapshot exceeds the described size of 190,000 by 36,571 and does not identify which subset corresponds exactly to the manuscript.
  • None of the 133,219 human personas has the 17 fields: biological_sex is missing in all of them.
  • 56.79% of the opinions respects 250-500 words; 41.86% falls below and 1.35% above.
  • 2,415 GPT-oss records report temperature 0.0 and the other 224,156, 0.7.
  • The processed CSV include 560 file_name without source JSON and omit 207 published JSON.
  • Some folders mix identifiers or model families, so folder does not always equate to an immutable checkpoint.
  • The TFMN results allow checking the presence and organization of thematic vocabulary, but not realism, factuality, or human likeness.
  • The LiquidAI notebook shows p<0.001 in eight tests although the four thematic medians of reasoning summary are equal to or lower than the rest, illustrating that significance does not imply the expected direction.
  • The pooling notebook archived in Zenodo matches byte for byte the v3 version of the repository.

Limitations

  • The work is a Data Descriptor and does not estimate effects of attributes on stance, bias, tone, accuracy, or safety.
  • There is no comparison with real people, surveys, interviews, or observed behavior.
  • The corpus itself is not representative of Italy, the United States, or any other population.
  • Combining sexuality, religion, migration, psychological health, and other attributes may induce stereotypes from training.
  • The public number of JSON is 226,571 and there is no release manifest that delimits the 190,000 of the text.
  • Biological sex is missing in all raw personas despite the requirement of 17 complete attributes.
  • Nearly 43.21% of opinions violates the declared length interval.
  • Temperature is not uniform: 2,415 records are at 0.0.
  • No failed attempts or generation manifest are published, so discard rates of 0-15.3% cannot be verified.
  • The raw-CSV table does not close: 560 rows have no public source and 207 sources do not enter CSV.
  • The Claude folder mixes Claude 3.5 and 4.5; Mistral Small includes Mistral Large; Phi-mini is mislabeled.
  • Alias latest, lack of digest, and absence of fingerprint prevent fixing several models.
  • The released generator disables human mode by using llm twice in ROLE_MODES.
  • The public Persona class omits biological_sex and only offers man/woman, without reproducing the schema or the historical data.
  • current_marital_status of the generator does not match marital_status read by the parser.
  • Only one Qwen miner is released and not the exact variants used for 19 models and providers.
  • No seed or RNG state is preserved per record.
  • Categorical distributions are not uniform: religion and sexual orientation are weighted by repetition and there are more dependencies than the two described.
  • Supplementary Table S1, cited for probabilities, is not in the arXiv source or repository.
  • The prompt requests complete step-by-step derivations, incompatible with presenting it without nuance as a concise summary or internal reasoning.
  • Response validation requires only four fields and does not enforce length, tone, citations, complete persona, or equality with the selection.
  • The tests flatten dependent degrees of words and networks, generating overly precise p-values.
  • There is no adjustment for the up to 152 model-topic-layer comparisons.
  • The keywords come from the prompt itself; high frequency is largely expected lexical obedience.
  • The dashboard requires Colab, unfixed installations, mounted Drive, manual path, and loading a 773 MB parquet.
  • Zenodo archives the notebook, not the unified parquet or the complete corpus.
  • There is no global LICENSE, requirements, lockfile, tests, CI, release tag, or numerical regressions.
  • The notebooks contain Windows and Google Drive paths of the authorial team.
  • Pickles should be opened only from a trusted source due to the inherent risk of their deserialization.

What the study does not establish

  • That CDS represents beliefs, attitudes, or opinions of real people.
  • That a synthetic persona is a digital twin, a persistent individual, or an autonomous agent.
  • That prompt attributes cause differences in discourse.
  • That synthetic groups reproduce distributions or relationships of a population.
  • That generated OCEAN scores correspond to a psychometrically valid personality.
  • That an output attributed to age, gender, religion, or sexuality is typical of the named human group.
  • That the reasoning summary reveals internal reasoning, cognition, or intention of the model.
  • That the citations named by the models are correct or support the claims.
  • That thematic anchoring implies truth, coherence, quality, neutrality, safety, or absence of bias.
  • That a significant p-value demonstrates greater centrality of thematic words.
  • That the 19 folders are 19 homogeneous and immutable checkpoints.
  • That the public code allows regenerating the corpus or exactly reproducing discard and selection rates.
  • That the dashboard is a hosted service ready to use without technical preparation.
  • That CDS can replace surveys, interviews, experiments, or human behavioral measures.

Traceability

Scope: Full text

Version: arXiv:2604.27624v1; repository commit e39570e8cd8d188b2dd019fc3aa138e81770ffb9; Zenodo 10.5281/zenodo.19816544

Consulted source: https://arxiv.org/abs/2604.27624

Review: Codex 18-page visual full-text, TeX, 226,571-JSON census, CSV lineage, model-identity, prompt, TFMN, code, Zenodo, Drive dashboard and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Claude Sonnet folder, mixing Claude 3.5 latest and Claude Sonnet 4.5 identifiers
  • DeepSeek-R1-0528-Qwen3-8B
  • DeepSeek-R1-Distill-Llama-70B
  • DeepSeek V3.2 through deepseek-chat
  • ERNIE-4.5-21B-A3B
  • ERNIE-4.5-21B-A3B-Thinking identifiers
  • GPT-oss-20b identifiers
  • Google Gemma 4 e4b and e4b-it identifiers
  • IBM Granite 4 H Tiny
  • LiquidAI LFM2 1.2B
  • Llama 2 7B Chat
  • Llama 3.3 70B aliases
  • Magistral Small 2509
  • Mistral Small latest mixed with Mistral Large 2512
  • Microsoft Phi-4-mini-reasoning in a folder labelled Phi-4-mini-instruct
  • Microsoft Phi-4-reasoning-plus
  • Qwen3 30B A3B 2507
  • Qwen3 4B Instruct 2507 identifiers
  • Qwen3 4B Thinking 2507

Instruments and metrics

  • Four fixed societal-topic prompts
  • Synthetic persona randomization pool
  • Seventeen-attribute claimed persona schema
  • OCEAN scores from 0 to 100 with low, moderate and high buckets
  • JSON structured-output prompt and parser
  • 250-500 word requested opinion
  • Generated reasoning summary, tone and named citations
  • EmoAtlas textual forma mentis networks
  • Vertex degree and occurrence frequency
  • Kruskal-Wallis topic-keyword comparisons
  • Panel and ipywidgets Colab pooling interface
  • Emotional flower, semantic network, ego network and mindset stream visualizations

Data used

  • 226,571 released raw JSON records across 19 model directories
  • 133,219 human-mode records with 16 complete persona fields and missing biological_sex
  • 93,352 AI-assistant-mode records with null selected persona
  • Nineteen processed Data_visualization CSV files with 226,924 rows
  • TFMN edge-list parquet chunks
  • 152 topic-layer hypothesis-testing pickle slots plus generated plots
  • TFMN and EmoAtlas aggregate statistics
  • Zenodo-archived TFMN_Pooling_from_CDS_v3.ipynb
  • Mutable Google Drive unified edge-list parquet and v5 notebook

Evidence and location

  • Full text, design, schema, validation, dashboard, limits, and availability: arXiv:2604.27624v1; PDF sha256 4269a50973b038e21fd49677f3c1932055dd3341ab741637cec96e03bdd65dfe
  • Code, 226,571 JSON, CSV, networks, pickles, notebooks, and figures: GitHub NaviDATA-Repos/PENSO_Data_WP-ConvinceMe_FIS2_UniTrento commit e39570e8cd8d188b2dd019fc3aa138e81770ffb9; tree 549360e699132439e98fe44f683b4e7f0051775b
  • Stable snapshot of the pooling notebook v3: Zenodo 10.5281/zenodo.19816544; sha256 fe5481db43d79654017b42910d744bc3745fa3419e221f65002057141e807a53
  • Audit of schema, corpus, model identities, anchoring, dashboard, and reproducibility: reports/verification/article-355-cognitive-digital-shadows-corpus-schema-model-identity-topic-validation-dashboard-and-reproducibility-audit.json