PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation

Personas, identity, and agents2025ACMApproved editorial review

Authors: Chenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, Jeffrey Chan

Keywords: Information Retrieval, Recommender Systems, User Behavior Simulation, Personality Traits, Large Language Models

Source: Open primary source (opens in a new tab)

5
Authors
13
Findings
41
Limitations
15
Evidence

Editorial summary

English

PUB is an interaction simulator for recommender-system evaluation. Its architecture aggregates each user's history, computes signals such as purchase frequency and rhythm, category preferences, prices, review length, and sentiment, adds product metadata, and asks an unidentified LLM to return five Big Five scores. The mapping is fixed through heuristics: openness is tied to category entropy and metaphor density; conscientiousness to regularity, review-length consistency, and rating deviation; extraversion to social references; agreeableness to positive sentiment and politeness; and neuroticism to negative-emotion volatility. These scores condition an LLM agent that generates choices or synthetic feedback. The paper calls this a psychometric mapping, but it administers no psychometric instrument and does not validate scores against observed human personality.

Experiments use Amazon Reviews 2023, described as 571.54 million interactions in 30 categories from 1996 to 2023. Categories are pooled, users and items with fewer than 20 interactions are removed, and a chronological split is made, but retained user, item, and interaction counts are not reported. To test sequence similarity, at every iteration the agent receives ten candidates deliberately containing the true next test item and nine random negatives. It selects one, and the resulting sequence is compared with the real sequence using Jaccard similarity. PUB averages 0.31 and visually outperforms Random, RecSim, NEST, and RecAgent; similarity increases for groups with longer histories. This design measures recovery of positives exposed to the agent, not free-running trajectory generation, and 0.31 is partial overlap rather than equivalence to real behavior.

A second evaluation trains or tests seven recommenders, Pop, MF, BPR, NeuMF, LightGCN, GRU4Rec, and SASRec, on real and synthetic data using nDCG@20. The figure preserves some broad tendencies but also shows systematic deviations: MF, BPR, NeuMF, and LightGCN perform worse on synthetic data, whereas Pop, GRU4Rec, and SASRec perform better. No numerical table, repeated runs, error bars, rank correlation, or equivalence test is reported. The plot therefore suggests partial reproduction of some algorithms' ordering or scale, but does not support the stronger claim that PUB is a reliable substitute for real-data evaluation.

The personality analysis describes the distribution of the five inferred scores and compares the top and bottom 10% of GRU4Rec users by nDCG@20. The authors associate high agreeableness and conscientiousness with better recommendations and high openness with lower accuracy. Yet the traits derive from the same behaviors and rules that feed the simulator, not from independent measurements; no coefficients, confound controls, tests, or effect sizes are reported. The paper also interprets lower neuroticism as lower emotional stability, reversing the conventional Big Five relationship. Phrases such as most prevalent trait, significant correlations, or algorithms favoring personalities are unsupported by the presented analysis and risk turning design-induced associations into stereotypes about users.

The paper links a repository as source code and refers readers to it for omitted definitions. Auditing current commit 52651aec from June 2025 finds only three files: .gitignore, a GPL-3.0 license, and a README saying the code will be released soon. There is no implementation, prompt, configuration, processed data, or result artifact. The LLM, provider, version, temperature, cost, seeds, and privacy policy are also unspecified. The defensible contribution is a conceptual design and preliminary five-page evaluation of simulation conditioned on signals labeled as Big Five. It does not establish psychometric validity, a causal link between personality and recommendation, high-resolution human fidelity, or that PUB can replace user studies or online evaluation.

Español

PUB es un simulador de interacciones para evaluar sistemas de recomendación. Su arquitectura agrega el historial de cada usuario, calcula señales como frecuencia y ritmo de compra, preferencias de categoría, precios, longitud y sentimiento de reseñas, añade metadatos de productos y solicita a un LLM no identificado que devuelva cinco puntuaciones Big Five. La correspondencia se fija mediante heurísticas: apertura se vincula con entropía de categorías y densidad de metáforas; responsabilidad con regularidad, longitud de reseñas y desviación de puntuaciones; extraversión con referencias sociales; amabilidad con sentimiento positivo y cortesía; y neuroticismo con volatilidad emocional negativa. Esas puntuaciones condicionan a un agente LLM que produce elecciones o feedback sintético. El artículo llama «psychometric mapping» a este procedimiento, pero no administra un instrumento psicométrico ni valida las puntuaciones contra personalidad humana observada.

Los experimentos utilizan Amazon Reviews 2023, descrito con 571,54 millones de interacciones, 30 categorías y fechas de 1996 a 2023. Se agregan categorías, se eliminan usuarios e ítems con menos de 20 interacciones y se hace una división cronológica, pero no se informa cuántos usuarios, ítems o interacciones quedan. Para evaluar semejanza de secuencias, en cada iteración el agente recibe una lista de diez candidatos que incluye deliberadamente el siguiente ítem positivo del test y nueve negativos aleatorios. Selecciona uno y la secuencia resultante se compara con la real mediante Jaccard. PUB alcanza una media de 0,31 y supera visualmente a Random, RecSim, NEST y RecAgent; la similitud aumenta en grupos con más historial. Este diseño mide capacidad de reidentificar positivos presentados al agente, no generación libre de una trayectoria, y 0,31 representa solapamiento parcial, no equivalencia con el comportamiento real.

Una segunda evaluación entrena o prueba siete recomendadores, Pop, MF, BPR, NeuMF, LightGCN, GRU4Rec y SASRec, con datos reales y sintéticos, usando nDCG@20. La figura conserva algunas tendencias generales, pero también muestra desviaciones sistemáticas: MF, BPR, NeuMF y LightGCN rinden peor con los datos sintéticos, mientras Pop, GRU4Rec y SASRec rinden mejor. No se publican valores tabulares, múltiples ejecuciones, barras de error, correlación de rankings o pruebas de equivalencia. Por tanto, la figura sugiere que el simulador reproduce parcialmente el orden o la escala de ciertos algoritmos, pero no sustenta que sea una plataforma fiable y sustituible por evaluación real.

El análisis de personalidad describe la distribución de las cinco puntuaciones inferidas y compara usuarios del 10 % superior e inferior de nDCG@20 para GRU4Rec. Los autores asocian amabilidad y responsabilidad altas con mejores recomendaciones y apertura alta con menor precisión. Sin embargo, los rasgos proceden de las mismas conductas y reglas que alimentan el simulador, no de mediciones independientes; no hay coeficientes, controles de confusión, tests o tamaños de efecto. Además, el texto interpreta un neuroticismo más bajo como menor estabilidad emocional, cuando en el Big Five la relación es inversa. Expresiones como «rasgo más prevalente», «correlaciones significativas» o algoritmos que «favorecen» personalidades no están respaldadas por el análisis presentado y pueden convertir asociaciones inducidas por el diseño en estereotipos sobre usuarios.

El paper enlaza un repositorio como supuesto código fuente y remite a él para definiciones omitidas. La auditoría del commit actual 52651aec, de junio de 2025, encuentra solo tres archivos: .gitignore, una licencia GPL-3.0 y un README que anuncia que el código se publicará próximamente. No hay implementación, prompts, configuración, datos procesados ni resultados. Tampoco se identifica el LLM, proveedor, versión, temperatura, coste, semillas o política de privacidad. La aportación defendible es un diseño conceptual y una evaluación preliminar de cinco páginas sobre simulación condicionada por señales etiquetadas como Big Five. No demuestra validez psicométrica, causalidad entre personalidad y recomendación, fidelidad humana de alta resolución ni que PUB pueda reemplazar estudios de usuarios o evaluaciones online.

Research question

Can an LLM agent conditioned on Big Five traits inferred from histories and metadata generate synthetic sequences that resemble real interactions, reproduce relative outcomes of various recommenders, and allow for the exploration of associations between inferred personality scores and recommendation performance?

Method

Proposal and preliminary experimental evaluation of a hybrid simulator. An aggregator extracts statistics from Amazon histories; a module incorporates metadata; psycholinguistic rules and an unidentified LLM produce Big Five scores; and an agent conditioned by those scores selects synthetic interactions. The evaluation compares Jaccard of sequences against four simulators, nDCG@20 of seven recommenders on real and synthetic data, score distributions, and profiles of the top and bottom 10% in GRU4Rec. The editorial audit read and rendered the five published pages, verified DOI, version and license, and froze and reviewed the linked official repository.

Sample: The raw source is described with 571.54 million interactions across 30 categories between 1996 and 2023. After aggregating categories and filtering users and items with fewer than 20 interactions, the article does not report retained sizes, distribution by domain, nor the number of prompts or LLM calls. For RQ1, lists of ten candidates are formed with one real positive and nine negatives; for RQ2, the synthetic test size is equated to the real one. RQ4 uses the top and bottom 10% percentiles of nDCG@20 from GRU4Rec, without indicating their counts.

Findings

  • PUB combines statistical profiles, metadata, inferred Big Five scores, and an LLM agent to generate synthetic interactions.
  • Trait inference uses proxies designed by the authors, not an inventory administered to users.
  • In the candidates task, PUB achieves a mean Jaccard similarity of 0.31 with the observed sequence.
  • PUB visually outperforms Random, RecSim, NEST, and RecAgent in the Jaccard figure.
  • Similarity increases in groups with higher interaction frequency and, therefore, more available history.
  • The sequence evaluation includes the following true item among ten candidates at each step.
  • Synthetic data approximately reproduce some of the nDCG@20 patterns of seven recommenders.
  • MF, BPR, NeuMF, and LightGCN perform worse on the synthetic test than on the real one.
  • Pop, GRU4Rec, and SASRec perform better on the synthetic test than on the real one.
  • The inferred scores show higher mean extraversion and lower mean neuroticism in the figure.
  • For GRU4Rec, the top 10% profiles show higher agreeableness and conscientiousness than the bottom 10%.
  • Inferred openness appears slightly higher in the worst nDCG@20 group.
  • The official repository does not contain the code that the article announces, so none of these results can be reproduced from the public artifact.

Limitations

  • The LLM used is not identified by provider, family, checkpoint, version, or date.
  • Prompts, system messages, output schemas, or handling of invalid responses are not published.
  • Temperature, top-p, maximum length, seeds, number of samples, or retry strategy are not reported.
  • Hardware, cost, latency, dependencies, or execution environment are not detailed.
  • The article refers to the code for feature definitions, but the official repository does not contain an implementation.
  • The current repository contains only .gitignore, LICENSE, and a README stating that the code will be published soon.
  • The GPL-3.0 license does not provide reproducibility when no program is distributed.
  • Processed data, splits, IDs, retained statistics, or model outputs are not published.
  • It is not reported how many users, items, and interactions remain after the 20 interaction filter.
  • Exact proportions or dates of the chronological split are not specified.
  • It is not explained how test signals are prevented from entering profiles, normalization, metadata, or prompts.
  • Traits are inferred from behaviors via proxies and are not validated against self-reports, informants, or experts.
  • The correspondence between traits and characteristics mixes normative decisions with correlational evidence from other domains.
  • The same history contributes to defining traits and evaluating associated behaviors, creating circularity.
  • Associations may reflect rules encoded in the prompts rather than psychological differences among users.
  • The agent receives the true positive from the test along with nine negatives, so the task is not free simulation.
  • It is not clarified if negatives are sampled by popularity, category, time, or difficulty.
  • Jaccard ignores order and time, although the stated goal is behavioral sequences.
  • A Jaccard similarity of 0.31 implies partial overlap and is not sufficient on its own for high fidelity.
  • Complete distributions by baseline, confidence intervals, or paired tests are not reported.
  • The ten frequency groups do not have published sizes and the figure does not separate the effect of history quantity.
  • RQ2 does not publish numerical nDCG@20 values, tables, deviations, or multiple runs.
  • Ranking correlation, absolute error, calibration, or equivalence tests between real and synthetic data are not calculated.
  • Synthetic results overestimate some algorithms and underestimate others, which may change evaluation conclusions.
  • Pop is aligned by construction with popularity signals incorporated into the profile, favoring benchmark circularity.
  • Recommendations generated by PUB are not compared with an ablation without personality.
  • There are no ablations for metadata, psycholinguistic proxies, history, temporal sampling, or LLM.
  • It is not evaluated whether a simulator without Big Five labels achieves the same Jaccard or nDCG.
  • Claims of significant correlations lack coefficients, p-values, intervals, or correction for comparisons.
  • The word prevalence is applied to inferred continuous scores, not to diagnostic categories or observed proportions.
  • The text interprets lower neuroticism as lower emotional stability, reversing the usual definition of the construct.
  • RQ4 uses only GRU4Rec and 10% extremes, without controlling for activity, history length, category, time, or other confounders.
  • Explanations about openness, agreeableness, and conscientiousness are post hoc and do not demonstrate causal mechanisms.
  • Demographic differences, domains, countries, languages, or generalization outside of Amazon are not evaluated.
  • The method is declared dataset-agnostic, but is only tested on one collection and one recommendation task.
  • There is no user study, online experiment, A/B test, or validation of utility for real decisions.
  • Privacy is not analyzed when inferring psychological attributes from reviews and purchase histories.
  • Fairness, stereotypes, discrimination, or consequences of personalizing based on non-consented traits are not evaluated.
  • No limitations, ethics, or social impact section is included despite the automated psychological profiling.
  • Five pages limit the methodological detail and leave essential elements delegated to an empty artifact.
  • The conclusion describes the platform as scalable and reliable without direct measurements of scalability or reliability.

What the study does not establish

  • It does not demonstrate that inferred scores measure the actual personality of users.
  • It does not psychometrically validate the mapping to Big Five.
  • It does not prove that traits cause purchase decisions or recommendation outcomes.
  • It does not demonstrate that PUB generates free trajectories without access to real positives.
  • It does not establish statistical equivalence between synthetic and human sequences.
  • It does not demonstrate that a Jaccard of 0.31 is sufficient to replace real data.
  • It does not guarantee that the ranking of recommenders is preserved stably.
  • It does not demonstrate that personality provides incremental value over history and metadata.
  • It does not prove generalization to other datasets, domains, languages, or platforms.
  • It does not demonstrate stability against another LLM, prompt, seed, or configuration.
  • It does not allow inferred distributions to be interpreted as psychological prevalence in Amazon users.
  • It does not establish that recommenders causally favor certain personalities.
  • It does not validate the explanatory stereotypes about openness, agreeableness, conscientiousness, or neuroticism.
  • It does not evaluate the security, privacy, consent, or fairness of the profiling.
  • It does not currently offer a reproducible system despite linking to a code repository.

Traceability

Scope: Full text

Version: arXiv:2506.04551v1 (5 Jun 2025); published at SIGIR '25; DOI 10.1145/3726302.3730238; CC BY 4.0

Consulted source: https://arxiv.org/pdf/2506.04551v1

Review: Codex full-text, visual, experimental-design, psychometric and repository audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • Unspecified large language model used for metadata fusion, Big Five score generation and user-agent simulation
  • Pop
  • Matrix Factorization
  • Bayesian Personalized Ranking
  • NeuMF
  • LightGCN
  • GRU4Rec
  • SASRec

Instruments and metrics

  • Handcrafted Big Five proxy mapping from behavioral and linguistic features
  • LIWC-22 social-reference markers
  • VADER positive sentiment ratio
  • Circular statistics for purchase rhythm
  • Shannon entropy for purchase intervals and categories
  • Jaccard similarity between synthetic and observed item sequences
  • nDCG@20 for recommender evaluation
  • Top-versus-bottom 10% GRU4Rec performance profiles

Data used

  • Amazon Reviews 2023: reported 571.54 million interactions across 30 categories from 1996 to 2023
  • Chronologically split interactions after removing users and items with fewer than 20 interactions
  • Official PUB repository at commit 52651aec47d79feb04d54fc57e8d4054e8972a3e; no source code present

Evidence and location

  • Problem, contributions and declared code: SIGIR '25 paper, abstract, section 1 and footnote 1, pp. 1–2
  • Four module architecture: SIGIR '25 paper, Figure 1 and section 2, p. 2
  • Profile extraction and temporal sampling: SIGIR '25 paper, section 2.1 and equations 1–2, p. 2
  • Metadata fusion via prompt: SIGIR '25 paper, section 2.2, p. 3
  • Proxies of the five traits: SIGIR '25 paper, section 2.3, p. 3
  • Simulator and task scope: SIGIR '25 paper, section 2.4, p. 3
  • Amazon source, period, categories and filter: SIGIR '25 paper, section 3.1, p. 3
  • List with one positive and nine negatives and Jaccard metric: SIGIR '25 paper, section 3.2.1, p. 3
  • Mean Jaccard 0.31 and simulator comparison: SIGIR '25 paper, Figure 2 and section 3.2.1, p. 3
  • Seven recommenders and real versus synthetic nDCG@20: SIGIR '25 paper, Figure 3a and section 3.2.2, p. 4
  • Inferred distributions and interpretation of neuroticism: SIGIR '25 paper, Figure 3b and section 3.2.3, p. 4
  • Extreme profiles of GRU4Rec: SIGIR '25 paper, Figure 3c and section 3.2.4, p. 4
  • Conclusion and only declared future work: SIGIR '25 paper, section 4, p. 4
  • Published version, DOI and license: arXiv:2506.04551v1 and SIGIR '25 metadata; DOI 10.1145/3726302.3730238; CC BY 4.0
  • Repository without code: Official PUB repository commit 52651aec47d79feb04d54fc57e8d4054e8972a3e; three tracked files; audited 15 Jul 2026