Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need

Personas, identity, and agents2025DOIApproved editorial review

Authors: Yuqi Bai, Tianyu Huang, Kun Sun, Yuting Chen

Keywords: Computers and Society, Artificial Intelligence, Computation and Language, Social experiments, Persona role-playing

Source: Open primary source (opens in a new tab)

4
Authors
18
Findings
40
Limitations
18
Evidence

Editorial summary

English

Bai and coauthors ask whether adding detail and apparent realism to a textual persona makes DeepSeek-chat produce Big Five profiles that are more repeatable, distinguishable, and similar to human age curves. The work remains a preprint. It proposes a SQLite pipeline that samples demographic skeletons from the 1994 Adult Income Dataset, expands them with the same LLM, administers the 120 IPIP-NEO items in six blocks, and computes OCEAN scores. Only one service is tested: DeepSeek-chat version 3.0 before 3.1. It is invoked through a mutable API alias, and the released code neither freezes the model snapshot nor sets the temperature reported as 0.7 in the paper.

At the individual level, five personas are selected and severely shortened versions are compared with their full profiles across 300 repeated questionnaire administrations per condition. After IQR outlier removal, 275–298 observations remain. The coefficient of variation of Mahalanobis distances decreases in all five cases: 0.2806→0.2384, 0.2818→0.2168, 0.3380→0.2451, 0.2787→0.2523, and 0.2811→0.2369. K-means separation also improves: pooled ARI rises from 0.7349 to 0.9835 and pair 4–5 from 0.0795 to 0.9151. Yet three pairs were already between 0.9531 and 1 with poor profiles. Distances are computed from the sample-derived mean, not an external ideal prototype; K-means is given the correct number of personas and evaluated on the same observations. The experiment therefore supports the narrower claim that additional text can constrain and differentiate questionnaire outputs under this protocol. It does not demonstrate an internal personality or an identity that persists through interaction.

At the population level, the paper compares four conditions, each reported as n=600: enriched census profiles; the same profiles with an anti-idealization instruction during testing; at-least-2,000-word narrative personas; and literary characters sourced through Wikidata. A cubic polynomial is fitted for each trait against age and evaluated at six points from ages 20 to 70 against human curves. Joint distance decreases from 70.25 to 63.45, 51.21, and 23.75. Appendix metrics reproduce the same ordering. This is an interesting descriptive pattern, not a scaling law: there is no measured quantity of detail, exponent, replicated levels, held-out prediction, or extrapolative validation. Each step simultaneously changes population source, prompt, content, and construction method. The final group is not a more detailed version of the census population but a different population of fictional characters, and the LLM itself fills gaps in their descriptions.

The artifact audit further limits the conclusion. GitHub contains code, prompts, census.csv, and some CFA outputs, but the SQLite files with responses and results are hosted behind a Baidu captcha and the figures cannot be recomputed from the repository. No run manifests bind each figure to a commit, configuration, and model snapshot. The released human-data importer marks every item as not reverse-scored, whereas the LLM scoring path applies the 55-item reverse key. Without human.db, whether the published reference was affected cannot be established, but the available pipeline does not demonstrate scoring equivalence. Curve comparison uses unvalidated cubic fits without uncertainty; Sliced Wasserstein uses unseeded random projections; and Average Marginal Wasserstein duplicates the calculation labeled Wasserstein Mean.

The defensible contribution is an exploratory workbench for studying how persona context constrains synthetic self-reports. The findings suggest that richer profiles induce greater response self-consistency and that some generated populations more closely match five human marginal age means in this setup. They do not establish a general law, psychological fidelity, psychometric validity, a causal effect of detail, or that synthetic personas can replace human participants. The design can instead confound realism with prompt obedience, circularity between the model writing the persona and the model answering the questionnaire, and reproduction of demographic stereotypes. Claims that the LLM “possesses” personality, that CFA and construct validity are irrelevant, or that more detail is all that is needed go beyond the presented evidence.

Español

Bai y coautores preguntan si añadir detalle y aparente realismo a una persona textual hace que DeepSeek-chat produzca perfiles Big Five más repetibles, distinguibles y parecidos a curvas humanas por edad. El trabajo, que sigue siendo un preprint, propone un pipeline de SQLite: muestrea esqueletos demográficos del Adult Income Dataset de 1994, los amplía con el propio LLM, administra los 120 ítems IPIP-NEO en seis bloques y calcula OCEAN. Solo se usa un servicio llamado DeepSeek-chat versión 3.0 anterior a 3.1; la API se invoca mediante un alias mutable y el código liberado no fija el snapshot ni la temperatura que el texto sitúa en 0,7.

A escala individual, se eligen cinco personas y se comparan versiones muy recortadas con sus perfiles completos mediante 300 cuestionarios repetidos por condición. Tras descartar outliers por IQR quedan entre 275 y 298 respuestas. El coeficiente de variación de las distancias de Mahalanobis baja en los cinco casos: 0,2806→0,2384; 0,2818→0,2168; 0,3380→0,2451; 0,2787→0,2523; y 0,2811→0,2369. La separación por K-means también aumenta: el ARI conjunto pasa de 0,7349 a 0,9835 y el par 4–5 de 0,0795 a 0,9151. Sin embargo, tres pares ya estaban entre 0,9531 y 1 con perfiles pobres. Las distancias se calculan respecto de la media de la propia muestra, no de un prototipo ideal externo; K-means recibe de antemano el número correcto de personas y se evalúa sobre los mismos datos. Esto apoya que más texto puede restringir y diferenciar respuestas en este protocolo, pero no prueba una personalidad interna ni una identidad estable a lo largo de una interacción.

A escala de población se comparan cuatro condiciones declaradas de 600 casos: perfiles censales ampliados; los mismos perfiles con una instrucción anti-idealización durante el test; personas narrativas de al menos 2.000 palabras; y personajes literarios obtenidos a partir de Wikidata. Se ajusta un polinomio cúbico para cada rasgo en función de la edad y se miden seis puntos, de 20 a 70 años, frente a curvas humanas. La distancia conjunta desciende de 70,25 a 63,45, 51,21 y 23,75. Los apéndices muestran la misma ordenación con varias métricas distribucionales. Es un patrón descriptivo interesante, no una ley de escalado: no existe una variable cuantitativa de detalle, un exponente, niveles repetidos, validación fuera de muestra ni predicción extrapolable. Además, cada escalón cambia simultáneamente la fuente de población, el prompt, el contenido y el modo de construcción. El último grupo no es una versión más detallada de la misma población censal, sino una población distinta de personajes ficticios, y el propio LLM completa sus descripciones.

La auditoría del artefacto reduce aún más la fuerza de la conclusión. El repositorio publica código, prompts, census.csv y algunos resultados de CFA, pero los SQLite con las respuestas y resultados están en un enlace Baidu bloqueado por captcha y no pueden recomputarse desde GitHub. No hay manifiestos que unan cada figura con un commit, configuración y modelo. La importación humana liberada marca todos los ítems como no invertidos, mientras el scoring de respuestas LLM aplica la clave inversa de 55 ítems; sin el human.db no puede comprobarse si la referencia publicada quedó afectada, pero el pipeline disponible no demuestra equivalencia de scoring. La comparación de curvas usa polinomios cúbicos sin validación ni incertidumbre; el Sliced Wasserstein usa proyecciones aleatorias sin seed; y Average Marginal Wasserstein duplica exactamente el cálculo llamado Wasserstein Mean.

La aportación defendible es un banco de trabajo exploratorio para estudiar cómo el contexto de persona constriñe autoinformes sintéticos. Los resultados sugieren que perfiles ricos inducen mayor autoconsistencia y que ciertas poblaciones generadas se acercan más a cinco medias marginales humanas por edad bajo este setup. No demuestran una ley general, fidelidad psicológica, validez psicométrica, causalidad del detalle, ni que las personas sintéticas permitan sustituir participantes humanos. De hecho, el diseño puede confundir realismo con obediencia al prompt, circularidad entre el modelo que escribe la persona y el que responde, y reproducción de estereotipos demográficos. Las afirmaciones de que el LLM «posee» personalidad, que CFA y validez de constructo son irrelevantes o que más detalle es todo lo necesario exceden la evidencia presentada.

Research question

How do the repeatability, identifiability, and similarity to human curves of DeepSeek-chat IPIP-NEO responses change when textual personas move from demographic skeletons or truncated versions to expanded, narrative, or literary-character-based profiles?

Method

Exploratory experimental preprint with a single LLM endpoint. The pipeline samples the Adult Income Dataset, generates personas, splits IPIP-NEO-120 into six blocks of 20 items, and computes OCEAN. At the individual level it compares five poor and complete profiles through 300 administrations, Mahalanobis distances, KDE, coefficient of variation, kurtosis, PCA, K-means, and ARI. At the population level it compares four declared conditions of 600 personas, fits cubic trait-age polynomials, and calculates distances to a human reference at ages 20, 30, 40, 50, 60, and 70, plus eight distributional metrics. The editorial audit read the 31 pages of Research Square and the 21 of arXiv, rendered and inspected figures and appendices, and reviewed tag v1.0.0, the current commit, prompts, generation, parsing, scoring, and statistical implementation.

Sample: The individual analysis declares 300 questionnaires per combination of five personas and two detail levels; after the IQR filter, 275–298 observations remain per cell. The identification analysis combines adjacent pairs and the five personas. The population analysis declares 600 cases in each of four conditions and a human reference, but the repository does not include the SQLite files nor document the exact selection of the 600 humans. A single model alias is evaluated, in English, without provider or snapshot replications.

Findings

  • The CV of the Mahalanobis distance decreases with the complete profile across the five personas: 0.2806→0.2384; 0.2818→0.2168; 0.3380→0.2451; 0.2787→0.2523; and 0.2811→0.2369.
  • After IQR removal, 275–298 of the 300 declared administrations per condition remain; the effect of the exclusions is not tested with a sensitivity analysis.
  • The joint ARI of five personas rises from 0.7349 to 0.9835 and pair 4–5 from 0.0795 to 0.9151.
  • Pairs 1–2, 2–3, 3–4, and 5–1 already obtain ARI 0.9538, 1, 0.9531, and 0.9863 with poor profiles; the benefit of detail is not uniform and shows a ceiling.
  • The joint distance of the human and synthetic OCEAN curves decreases in the order standard 70.25, antialign 63.45, narrative 51.21, and wiki-fiction 23.75.
  • By trait, the wiki-fiction condition obtains distances 4.85 in Neuroticism, 13.77 in Extraversion, 11.07 in Openness, 13.23 in Agreeableness, and 7.28 in Conscientiousness.
  • The appendix metrics preserve the ordering: cosine similarity increases from 0.8300 to 0.8422, 0.9260, and 0.9802; MMD decreases from 0.6578 to 0.6141, 0.4864, and 0.2156.
  • Wasserstein Mean and Average Marginal Wasserstein have identical values across all rows because the code implements the same marginal average; they are not two independent confirmations.
  • The anti-alignment condition produces a smaller improvement than narrative or wiki-fiction, but does not isolate alignment: it modifies response instructions without controlling content, generation, or population.
  • The article acknowledges that the sequence was heuristic and that the literary-character experiment was designed after observing the narrative improvement.
  • The repository runs K-means with known k, standardizes the combined data, and evaluates ARI on the same sample, without holdout or uncertainty interval.
  • The curves are fitted with cubic np.polyfit and compared at only six ages; there is no cross-validation, bootstrap, confidence bands, or weighting by the age distribution.
  • The data link points to Baidu and requires captcha; GitHub does not contain the response SQLite files, intermediate results, or sufficient artifacts to regenerate tables and figures.
  • The generation code uses a DeepSeek-chat alias and does not set temperature, although the method declares 0.7; the effectively served model is not frozen.
  • The parser accepts any object between braces and does not validate uniqueness, completeness, range, or exactly 120 responses before saving; the pipeline may drop failed cases later.
  • The LLM path applies 55 IPIP reverse keys, but the released human importer sets reverse_scored=0 for all items; without the human baseline it cannot be verified that both sides were scored identically.
  • The appendix example attributes empathy, reliability, and resilience to a Black woman in a service job; it shows that apparent coherence may stem from stereotypical inferences.
  • The Research Square cover lists Kun Sun first, while the body, arXiv, and contributions list Yuqi Bai first; the editorial discrepancy is not explained.

Limitations

  • The study is a non-peer-reviewed preprint; Research Square explicitly warns of this on the cover.
  • It tests only one DeepSeek endpoint and one service epoch; it does not compare models, sizes, families, or snapshots.
  • The API label is mutable and the repository stores no provider identification response, model hash, or per-run manifest.
  • The purported causal variable, detail/realism, is neither quantified nor manipulated in isolation.
  • The four population conditions change population, data source, prompt, length, content, and generative process all at once.
  • No scaling function, exponent, interval, saturation point, or prediction outside the four nominal levels is estimated.
  • The standard, antialign, narrative, and wiki-fiction levels are post hoc ordered categories, not equidistant measurements of a single variable.
  • Wiki-fiction represents another population of fictional characters, not the same census population with more detail; its age, selection, and composition are not documented.
  • The prompt completes sparse literary profiles with generated content, so they are not purely human-authored as the text claims.
  • The same type of model generates the personas and answers the questionnaire, creating circularity and possible lexical self-consistency.
  • The narrative prompt explicitly asks for psychological states and traits; the test then measures traits, which introduces target leakage.
  • More detail may impose more deterministic responses without increasing truth, construct validity, or human fidelity.
  • IPIP-NEO is split into six conversations/blocks with mandatory explanations; order, context, and reasoning may induce artificial consistency.
  • No factorial structure, invariance, test–retest reliability, convergent, discriminant, or criterion validity is demonstrated in the LLM responses.
  • The individual Mahalanobis distance uses the sample mean and covariance as reference, not an ideal prototype of the persona.
  • The CV of self-normalized distances is not a direct measure of temporal persistence or psychological convergence.
  • The 300 administrations are independent repeated calls, not a longitudinal conversational trajectory of the same agent.
  • Outliers are removed via 1.5 IQR without preregistration, failure-mechanism justification, or results with and without exclusion.
  • K-means knows the true number of personas and is evaluated in-sample; there is no new-identity classification, holdout, or replication.
  • Five personas selected without a seed or published criterion are too small a sample to generalize identifiability.
  • The interpretation of kurtosis is incorrect or inconsistent: zero corresponds to normality, not uniformity, and the complete profile does not always move closer to zero.
  • The 1994 Adult Income Dataset is old, selective, and US-based; it does not represent contemporary or global populations.
  • No demographic tables or comparable age distributions are published for the four groups of 600.
  • The human reference is cited as [22], but that reference concerns invariance across cultural groups; the data source appears to correspond to other references and is not traced.
  • The repository importer processes the first 10,000 rows of an absent IPIP120.dat, without documenting the sampling of 600 or its seed.
  • The released human scoring path appears to omit the item reversal that the LLM path applies; without human.db the comparability cannot be verified.
  • Fitting cubics to individual points can produce curves sensitive to composition and extremes; goodness of fit or validation is not reported.
  • Comparing only the marginal means of five traits by age does not validate the joint distribution, subfacets, behavior, or individual coherence.
  • Sliced Wasserstein uses 100 Gaussian projections without a seed and MMD uses a V-statistic without a permutation test or interval.
  • The eight metrics are presented without uncertainty or correction and two of them are mathematically duplicated in the implementation.
  • The exploratory sequence was adapted after seeing results; there is no preregistration, independent confirmatory hypothesis, or validation set.
  • There are no human participants or judges evaluating whether each persona is realistic, faithful, or recognizable.
  • The experiment SQLite files are not on GitHub; the external link requires captcha and blocks an independent audit of raw data.
  • No automated test suite exists; several files called test are manual scripts that require API access.
  • The current repository and the tag use different configurations and there are no per-experiment snapshots; the current configuration even selects a prompt that omits the persona.
  • The exported notebook from learn contains shell syntax and avoids compiling the full tree, although the core modules do compile.
  • The explanations about alignment, pretraining, world models, or Bayesian probabilities are speculative and lack ablations.
  • Inferring traits from race, sex, occupation, or marital status may reinforce stereotypes and cause harm if used for real decisions.
  • No ethical review, consent, or governance of the human reference is described beyond citing prior sources.
  • The Research Square cover and the manuscript disagree on author order, which requires bibliographic caution.

What the study does not establish

  • It does not demonstrate a mathematical or generalizable scaling law.
  • It does not demonstrate that more detail is sufficient or necessary for faithful human simulation.
  • It does not demonstrate that DeepSeek possesses personality, a self, or internal psychological states.
  • It does not demonstrate that repeatable IPIP responses are true, human, or psychometrically valid.
  • It does not demonstrate identity persistence across conversations or over time.
  • It does not demonstrate that the generated characters faithfully correspond to specific real or literary persons.
  • It does not demonstrate that the improvement causally stems from detail rather than prompt, population, length, or target leakage.
  • It does not demonstrate that alignment causes the observed positive bias or that anti-alignment instructions eliminate it.
  • It does not demonstrate equivalence between human and LLM scores or that they share measurement structure.
  • It does not demonstrate that similar marginal curves imply similar distributions, behaviors, or decisions.
  • It does not validate substituting human participants with synthetic personas in social sciences.
  • It does not demonstrate that CFA, invariance, or construct validity are irrelevant; the study offers no equivalent alternative validation.
  • It does not generalize to other models, providers, languages, cultures, questionnaires, or populations.
  • It does not allow independent reproduction of the figures with the artifacts accessible on GitHub.
  • It does not establish safety, absence of bias, or fitness for surveillance, marketing, politics, or decisions about people.

Traceability

Scope: Full text

Version: arXiv v1, 21 pages, submitted 10 October 2025; cross-checked in full against Research Square v1, 31-page covered manuscript posted 10 October 2025, DOI 10.21203/rs.3.rs-7777787/v1

Consulted source: https://arxiv.org/pdf/2510.11734

Review: Codex editorial review and methods/artifact audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • DeepSeek-chat version 3.0 before the 3.1 upgrade, accessed through an OpenAI-compatible API alias

Instruments and metrics

  • IPIP-NEO-120, administered in six 20-item blocks
  • Big Five/OCEAN trait scoring
  • Mahalanobis-distance KDE, coefficient of variation and Fisher kurtosis
  • PCA, K-means and Adjusted Rand Index
  • Cubic personality-by-age curves and six-point Euclidean distance
  • Wasserstein, Sliced Wasserstein, Fréchet, Euclidean, cosine, MMD and marginal-distance metrics

Data used

  • 1994 UCI Adult Income Dataset for demographic skeletons
  • Five randomly selected generated personas and author-created poor variants
  • Four reported 600-person conditions: standard, anti-alignment, narrative and wiki-fiction
  • Human IPIP-NEO reference expected by the repository as IPIP120.dat
  • Wikidata-derived literary-character profiles expanded through prompts
  • SQLite experiment databases distributed through a Baidu link but absent from GitHub

Evidence and location

  • Preprint status, DOI, date, and authorship discrepancy: Research Square v1 cover and manuscript p. 1, posted 10 Oct 2025, DOI 10.21203/rs.3.rs-7777787/v1
  • Question, pipeline, and methodological positioning: Research Square v1 abstract and sections 1–3, manuscript pp. 1–9
  • Model, census data, and setup: Research Square v1 sections 4.1–4.4, manuscript pp. 9–10
  • CV, kurtosis, and counts after outliers: Research Square v1 section 5.2.1, Figure 2 and Table 1, manuscript pp. 10–12
  • Pairwise and joint ARI: Research Square v1 section 5.2.2, Tables 2–3 and Figures 3–4, manuscript pp. 12–15
  • Four population experiments and n=600: Research Square v1 sections 3 and 5.3, manuscript pp. 9 and 16–19
  • Curve distances by condition and trait: Research Square v1 Figures 6–9 and Table 4, manuscript pp. 16–19
  • Scaling law claim and post hoc exploration: Research Square v1 sections 6–7, manuscript pp. 20–21
  • Risks, data, code, and conclusions: Research Square v1 Ethical Risks, Conclusion, Data availability and Code availability, manuscript pp. 21–22
  • Complete distributional metrics: Research Square v1 Appendix 4, Table 2 and Figure 9, manuscript pp. 29–30
  • Code, configurations, and absence of databases: Official GitHub repository baiyuqi/agentic-society-neo, tag v1.0.0 commit d7ceecc6957fdccfc8e876ef9d4fcd791151d129 and current commit 25b92f69cefebee9aba859c3f3e1e7da2b7b2e1d, audited 15 Jul 2026
  • Model alias and unfixed temperature: Official repository asociety/generator/llm_engine.py and paper section 4.2, audited 15 Jul 2026
  • Parsing, completeness, and LLM scoring: Official repository quiz service, quiz extractor and IPIP scoring modules, audited 15 Jul 2026
  • Possible human/LLM reverse-scoring difference: Official repository tools/importers/import_human_data.py and IPIP-NEO-120 scoring path, audited 15 Jul 2026; human.db absent
  • Mahalanobis, K-means, and ARI: Official repository single_density_panel.py and clustering analysis scripts, audited 15 Jul 2026
  • Cubics and distributional metrics: Official repository personality-curve and ocean_density_panel.py analysis scripts, audited 15 Jul 2026
  • Actual data availability: Paper Baidu data link checked 15 Jul 2026: extraction page reachable but download verification blocked by captcha; raw SQLite databases absent from GitHub
  • Visual integrity of full text: arXiv v1 PDF SHA-256 50036c94617485db1cf0c3eb49cfc10cbe4eaaed80727809269c6096cba320df and Research Square v1 PDF SHA-256 57216787d8f05a4186876125cde7839349d1fbf7244f0a7adc4998bc89ccbec4; all relevant pages, figures, tables and appendices rendered and inspected 15 Jul 2026