Synthetic Personality Review

From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilers

Evaluation and psychometric validity2026OpenReviewApproved editorial review

Authors: Yi-Fei Liu, Yi-Long Lu, Di He, Hang Zhang

Keywords: Large Language Models, Psychological Profiling, Big Five, Psychometrics, Structural Correlation, LLM Evaluation, Reasoning Traces, Privacy and Reproducibility

Source: Open primary source (opens in a new tab)

4
Authors
10
Findings
15
Limitations
12
Evidence

Editorial summary

English

This ICLR 2026 poster asks whether LLMs can reconstruct the correlation structure among psychological traits from minimal inputs. For 816 Chinese participants from a pandemic-era study, the authors gave twelve model conditions the 20 Big Five items and asked them to role-play each participant's answers to nine additional instruments. The primary result is not individual accuracy: it compares 105 model-generated Big-Five-to-target correlations with the corresponding human correlations. For Gemini 2.5, the reported regression is k = 1.42 and R² = 0.92; our reproduction from the released outputs gives k = 1.415 and R² = 0.917. All twelve conditions have k > 1, which the paper calls structural amplification. This aggregate effect is well supported by the released outputs, but it does not mean that a model explains more than 88% of an individual's psychological traits. The individual-prediction table reports mean correlations from 0.345 to 0.445. Reasoning-trace analysis shows strong factor-level agreement with a Bayesian-ridge baseline (r = 0.981) but weak item-level agreement (r = 0.207), so it does not validate the model's internal reasoning. Adding a narrative summary to the scores yields modest improvements, but the summary comparisons use variable subsets of 193 to 711 participants and are not cleanly matched to the 816-person ScoreOnly baseline. The repository permits a partial check of the structural effect but releases no code, linked human-scale records, 20-item Big Five prompt inputs, executable baselines, tests, dependencies, or license. It also retains highly specific quasi-identifiers in the Big Five file: 680 of 816 rows are unique on gender, province, exact age, job, education, and survey date, which conflicts with a strong reading of fully anonymized. The defensible conclusion is that these LLMs reproduce and amplify an aggregate psychological covariance pattern in this sample and prompt setting; the paper does not establish precise, valid, interpretable, or safe individual profiling.

Español

Este trabajo aceptado como póster en ICLR 2026 estudia si varios LLM pueden reconstruir la estructura de correlaciones entre rasgos psicológicos a partir de información mínima. Para 816 participantes chinos de un estudio realizado durante la pandemia, los autores proporcionaron a doce condiciones de modelo los 20 ítems del Big Five y les pidieron representar a cada persona al contestar nueve escalas adicionales. El resultado principal no es una medida de precisión individual: compara 105 correlaciones Big-Five–escala-objetivo generadas por los modelos con las correlaciones humanas. En Gemini 2.5, la regresión entre ambos vectores obtiene k = 1,42 y R² = 0,92; nuestra reproducción con los archivos publicados da k = 1,415 y R² = 0,917. Los doce modelos muestran k > 1, lo que el artículo denomina amplificación estructural. Este hallazgo agregado es sólido en los outputs liberados, pero no significa que el modelo explique más del 88 % de los rasgos de una persona. La tabla de predicción individual presenta correlaciones medias de 0,345 a 0,445. El análisis de trazas encuentra gran acuerdo entre modelos a nivel de factores (r = 0,981 frente a un baseline Bayesian Ridge), pero poco a nivel de ítems (r = 0,207), por lo que no valida el razonamiento interno. Los resúmenes narrativos mejoran modestamente el resultado al añadirse a los scores, aunque las comparaciones usan subconjuntos variables de 193 a 711 participantes y no están emparejadas limpiamente con el baseline de 816. El repositorio permite comprobar parcialmente el efecto estructural, pero carece de código, enlaces individuales entre escalas humanas, los 20 ítems Big Five usados como input, baselines ejecutables, tests, dependencias y licencia. Además, el archivo demográfico y Big Five conserva cuasi-identificadores muy específicos: 680 de 816 filas son únicas al combinar sexo, provincia, edad exacta, ocupación, educación y fecha, lo que contradice una interpretación fuerte de «totalmente anonimizado». La lectura defendible es que estos LLM reproducen y amplifican una estructura de covarianza psicológica en esta muestra y bajo estos prompts; no que realicen perfiles individuales precisos, válidos, explicables o seguros.

Research question

Can LLMs reconstruct, from a person's 20 Big Five items, the aggregated correlation structure that connects those traits with nine psychological instruments, and what do their traces and summaries indicate about the mechanism of that simulation?

Method

Data from 816 participants and twelve LLM conditions via OpenRouter were used. Each model received each participant's 20 Big Five items and generated responses to the items of nine target instruments. 21 target subscales were scored, 26-variable correlation matrices were constructed, and the model's 105 Big-Five–target coefficients were regressed onto the humans'; the slope k measures amplification and R² measures the fit between coefficients. A 1,000-iteration permutation test, order and individual-question variants for Gemini, ML and semantic similarity baselines, internal consistency and attenuation analyses, and a second experiment that had four LLMs annotate traces from five reasoning models and compared ScoreOnly, SummaryOnly, and Summary+Score were added. The audit independently reproduced the structural effect with the public outputs and examined coverage, missingness, privacy, and repository completeness.

Sample: 816 Chinese participants from an online longitudinal study on perceptions of the pandemic. The article provides no external validation in another cohort, culture, language, or period. A subgroup of 309 participants is defined as more attentive via response times.

Findings

  • Gemini 2.5 obtains k = 1.42 and R² = 0.92 when comparing 105 Big-Five–target correlations; the independent reproduction obtains k = 1.415 and R² = 0.917.
  • The twelve published LLM conditions show k > 1 and reproduce an amplified version of the human correlation structure; the semantic baseline reports k = 0.99 and R² = 0.52.
  • The effect persists for Gemini with standard order, random order, and one question at a time, with k = 1.42, 1.41, and 1.42; the 1,000-iteration permutation test reports p < .001.
  • Individual accuracy is much lower than the structural R²: the mean correlations in Table 2 range from 0.345 to 0.445; the best value is Claude Summary+Score with r = 0.445.
  • Attribution profiles converge across models (mean r = 0.934; KL = 0.0475), but align weakly with the human baseline at the item level (r = 0.207) and strongly at the factor level (r = 0.981).
  • Summary+Score improves modestly over ScoreOnly in the five models of Table 2, while SummaryOnly performs worse than ScoreOnly in all five.
  • Mean internal consistency is higher in LLM responses (alpha = 0.87) than in human responses (0.75), but this may reflect homogenization or stereotyped responses and does not equate to validity.
  • The one-sided correction for attenuation brings k closer to 1, the attentive subgroup of 309 participants produces k = 1.08, and adding Gaussian noise to a regression reduces k from 1.55 to 1.12; these are results compatible with attenuation, not a causal demonstration of the LLM mechanism.
  • The repository allows reproducing approximately the structural amplification, but not individual prediction, the trained baselines, or the full pipeline.
  • The public file basic_info_BF.csv has 680 of 816 unique rows when combining six quasi-identifiers, a high risk of singling out for sensitive psychological data.

Limitations

  • The main R² operates on aggregated correlation coefficients and should not be presented as R² of individual profiles.
  • A single Chinese sample in a COVID-19 context limits external validity; there is no independent cohort, culture, language, or period.
  • No MAE, RMSE, calibration, prediction intervals, or useful person-level thresholds are published.
  • Reasoning traces are not validated as faithful explanations; LLMs annotate texts produced by other LLMs and may share biases.
  • The SummaryOnly and Summary+Score subsets vary between 193 and 711 cases; Gemini averages only 35.8% coverage, while ScoreOnly uses 816.
  • The fifteen points used to relate k to r include three dependent measures per model and both metrics share the same covariance structure.
  • Higher alpha may be due to more homogeneous or redundant responses, not to greater psychological validity.
  • The attenuation correction depends on classical assumptions, the selection of attentive participants is observational, and adding noise to a linear model attenuates correlations in an expected manner.
  • The BGE baseline does not rule out learned psychological priors, trait stereotypes, or prompt-induced consistency.
  • The repository does not include code, notebooks, dependencies, tests, executable baselines, table outputs, or a license.
  • The human files were shuffled independently and do not allow reproducing individual links; the Big Five file publishes five aggregate scores, not the 20 items used in prompts.
  • An Experiment 1 output for DeepSeek Chat is missing; the 144 item maps contain 77,435 missing cells and some complete rows drop to 304.
  • OpenRouter, proprietary models, and generation parameters are not frozen; temperature, top-p, seeds, repetitions, dates, and request traces are missing.
  • The claim of total anonymization is not supported by a disclosure risk analysis; quasi-identifiers allow singling out the majority of rows.
  • Redistribution permissions for the six questionnaire PDFs or a repository license are not documented.

What the study does not establish

  • It does not demonstrate that LLMs explain more than 88% of a person's psychological traits.
  • It does not validate individual use for diagnosis, selection, intervention, recommendation, or high-impact decisions.
  • It does not demonstrate construct validity, calibration, or psychometric equivalence between human and simulated responses.
  • It does not prove that chain-of-thought traces reveal the model's internal causal mechanism.
  • It does not fully separate psychological reasoning from learned priors, stereotypes, or prompt-induced consistency.
  • It does not demonstrate that a summary contains new observed information about the participant or a sufficient statistic in the formal sense.
  • It does not prove that higher internal consistency implies greater accuracy or validity.
  • It does not causally prove that LLMs filter human noise to recover true scores.
  • It does not establish generalization to other populations, cultures, languages, instruments, providers, or model versions.
  • It does not demonstrate that the public dataset is fully anonymized or that publication is low-risk.
  • It does not allow end-to-end reproduction of all tables, baselines, individual analyses, and mechanistic conclusions.

Traceability

Scope: Full text

Version: ICLR 2026 Poster; accepted 29-page paper mirrored as arXiv:2511.03235v2, revised 23 March 2026

Consulted source: https://openreview.net/forum?id=JXFnCpXcnY

Review: Codex complete bilingual full-text fidelity pass using the accepted ICLR 2026 paper, all-page visual inspection, official repository and data audit, independent structural-regression reproduction, aggregate-versus-individual metric reconciliation, reasoning-trace and summary-condition assessment, missingness and privacy measurement, and reproducibility and licensing review; summaries written from paper and artifact evidence rather than abstract keywords, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • DeepSeek V3.1 Chat
  • DeepSeek V3.1 Thinking
  • GPT-5 Chat
  • Claude 3.7 Sonnet Chat
  • Claude 3.7 Sonnet Thinking
  • Gemini 2.5 Flash Chat
  • Gemini 2.5 Flash Thinking
  • GLM-4.5 Chat
  • GLM-4.5 Thinking
  • Kimi K2 Chat
  • Qwen3-235B Chat
  • Qwen3-235B Thinking
  • BAAI bge-reranker-large semantic baseline
  • KNN, SVM, Linear Regression and Bayesian Ridge baselines

Instruments and metrics

  • Short IPIP-BFM-20 Big Five inventory
  • Perceived Stress Scale
  • Simplified Coping Style Questionnaire
  • State-Trait Anxiety Inventory, trait subscale
  • Self-Compassion Scale
  • Connor-Davidson Resilience Scale
  • Intolerance of Uncertainty Scale
  • Emotion Regulation Questionnaire
  • Risk Perception and Behavior Questionnaire
  • Future Time Perspective Scale

Data used

  • Psychological responses from an 816-participant Chinese longitudinal diary study during COVID-19
  • Official From-Five-Dimensions-to-Many repository outputs at commit 16058c7f85c01a3dd42c3ee73fae0b2b4eebe03d
  • Released 26-by-26 true human correlation matrix
  • Experiment 1 LLM item-response outputs
  • Experiment 2 reasoning traces, extracted summaries, item mappings and summary-condition outputs

Evidence and location

  • Question, sample, procedure, and structural metric: Accepted PDF pages 1-5, Abstract and Sections 1-3.3
  • Amplification, robustness, reliability, attenuation, and noise: Accepted PDF pages 5-7 and 23-28, Sections 3.2-3.4 and Appendix C.1/C.4
  • Traces, item-level and factor-level attribution: Accepted PDF pages 7-8 and 18-24, Sections 4.1, Appendix A.3 and C.2
  • Summary conditions and individual results: Accepted PDF pages 9 and 24-25, Section 4.2, Figure 5 and Table 2
  • Declared limitations, ethics, and reproducibility: Accepted PDF pages 10-11, Discussion, Ethics Statement and Reproducibility Statement
  • Prompts and OpenRouter models: Accepted PDF pages 16-19, Appendix A and Table 1
  • Independent reproduction of k and R²: Official repository commit 16058c7 with released Experiment 1 outputs and True_correlation_matrix.csv, audited 15 July 2026
  • Missingness, summary coverage, and missing file: Official repository commit 16058c7, 107 Experiment 1 CSVs, 90 summary-condition CSVs and 144 item-mapping CSVs audited 15 July 2026
  • Human linkage destruction and absence of inputs/code: Official repository README, data tree and complete 449-file inventory at commit 16058c7
  • Singling-out risk via quasi-identifiers: Official repository data/basic_info_BF.csv: 680/816 singleton rows on gender+province+age+job+education+date
  • Comprehensive audit of structure versus individual, privacy, and artifact: reports/verification/article-195-structural-vs-individual-and-artifact-audit.json
  • Complete visual inspection: All 29 accepted ICLR 2026 PDF pages rendered and visually inspected on 15 July 2026