Synthetic Personality Review

SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators

Applications, bias, and safety2026arXivApproved editorial review

Authors: Yada Pruksachatkun, Elaine Wan, Lyanna Chen, Kai-Wei Chang, Chien-Sheng Wu

Keywords: Retail user simulation, Multimodal agents, Synthetic shopper personas, Persona adherence, Decision alignment, Tool-augmented dialogue, UserGRPO, Trajectory-level reinforcement learning, Conversational fidelity, Product constraint grounding

Source: Open primary source (opens in a new tab)

5
Authors
8
Findings
18
Limitations
3
Evidence

Editorial summary

English

SalesSim evaluates whether multimodal models acting as shoppers follow explicit profiles, preferences and constraints in tool-augmented sales conversations. It contains 674 synthetic personas and 274 products across six categories; six backbones score from 0.324 to 0.786 on Decision Alignment, a purchase-or-reject metric against a predefined acceptable-product set. UserGRPO is trained once on Qwen3-VL-8B and female clothing; on five unseen categories it reaches 0.655 versus a recalculated 0.490 base mean on those same categories (+16.5 points). The abstract’s 13.8% mixes the six-category base overall with the five-category UserGRPO overall and is actually an absolute-point difference. The evidence supports improved compliance with synthetic constraints and tool use, not realistic human simulation: no human purchasing decisions validate the benchmark, conversational metrics use an unmatched corpus, statistical tests and central reward details are missing, denominators/data versions conflict, and no code, data, outputs or checkpoints are released. Its strongest contribution is therefore a controlled synthetic benchmark result whose external validity remains an open empirical question.

Español

SalesSim evalúa si modelos multimodales que actúan como compradores respetan perfiles, preferencias y restricciones explícitas en conversaciones de venta con herramientas. Reúne 674 personas sintéticas y 274 productos de seis categorías; seis backbones obtienen entre 0,324 y 0,786 de Decision Alignment, una métrica de compra o rechazo respecto a un conjunto predefinido de productos aceptables. UserGRPO se entrena una sola vez sobre Qwen3-VL-8B y ropa femenina; en cinco categorías no vistas alcanza 0,655 frente a 0,490 del baseline recalculado sobre esas mismas categorías (+16,5 puntos). El 13,8% del abstract mezcla el overall base de seis categorías con el de UserGRPO de cinco y en realidad expresa puntos absolutos. El resultado apoya una mejora en cumplimiento de restricciones sintéticas y uso de herramientas, no una simulación humana realista: no hay validación con decisiones humanas, las métricas conversacionales usan un corpus no emparejado, faltan tests estadísticos y detalles centrales de recompensa, existen denominadores/versiones inconsistentes y no se publican código, datos, outputs ni checkpoints.

Research question

To what extent can MLLMs simulate shoppers that maintain explicit persona constraints during a multimodal, multi-turn interaction, and can a multi-objective trajectory reward improve their final decision, tool protocol, and conversational similarity with a human corpus?

Method

SalesSim pairs an MLLM shopper with a fixed GPT-5.4 seller that retrieves guides and products. Synthetic personas include background, preferences, dealbreakers, and a set of acceptable products. Decision Alignment scores acceptable purchases or rejection when no recommendation is acceptable; tool errors and three conversational proxies are added, compared with 515 RecQuest dialogues. Two closed models and four open models are tested with distinct decoding. UserGRPO fine-tunes via LoRA a Qwen3-VL-8B for 200 steps on womens clothing with a six-component reward and is compared with human prompting and SFT. This review inspected the 24 pages, all TeX, tables, prompts, transcripts, arithmetic, data sources, and artifact availability.

Sample: The manuscript reports 674 persona-category pairs: 73 womens clothing, 77 mens clothing, 147 rings, 83 smart watches, 147 cars, and 147 games, along with 274 products. Six backbones are evaluated. UserGRPO and SFT are trained only on womens clothing and are reported on five unseen categories; the text mentions eight rollouts per training item, but does not report the exact number of evaluated trajectories, exclusions, or unit of independence.

Findings

  • Base Decision Alignment: GLM-Thinking 0.324; GLM-Reasoning 0.443; Qwen 0.517; Gemini 0.539; Gemma3 0.592; ChatGPT-5.4 0.743; Gemma4 0.786.
  • UserGRPO obtains 0.632, 0.513, 0.578, 0.803, and 0.748 on mens clothing, rings, smart watch, cars, and games.
  • Over those five common categories, base Qwen averages 0.4898 and UserGRPO 0.6548: +16.5 points.
  • The +13.8 in the abstract is 0.655-0.517 and mixes five trained categories with six from the baseline; furthermore these are points, not a relative 13.8%.
  • SFT improves four of the five common categories and the macro average from 0.4898 to 0.5272; it is inferior to UserGRPO, but does not fail to generalize in absolute terms.
  • UserGRPO reports 0% early termination and 1.64% format error across the five unseen categories.
  • Gemma4 is the best baseline on the benchmark with 0.786; UserGRPO does not surpass that overall result.
  • The environment evidences compliance with explicit constraints, not predictive fidelity with respect to human shoppers.

Limitations

  • Synthetic ground truth without complete protocol, double annotation, agreement, or exhaustive QA of 674 personas and 274 products.
  • Decision Alignment also depends on which products the GPT-5.4 seller retrieves and recommends.
  • No controlled persuasion metric despite claiming susceptibility to suggestions.
  • No text-only ablation; the contribution of images or multimodality is not isolated.
  • Temperature 0.7 and penalty 1.1 in open models versus temperature 1.0 in closed models; ICL only explicit for open models.
  • Checkpoints/API revisions not fixed and GLM/Gemma/Qwen names inconsistent between text and tables.
  • No intervals, tests, repeated seeds, or hierarchical analysis; "significantly" has no statistical test.
  • Overall base of six categories compared with trained overall of five.
  • Smart Watch rates reveal denominators 89 and 83 although the persona table fixes 83, a sign of unexplained snapshots/exclusions.
  • The appendix says seven categories, while the final benchmark contains six and retains commented laptops.
  • The RecQuest human baseline is not matched with catalogs, constraints, images, or the SalesSim protocol.
  • The human mean of initial criteria is not published, nor how criteria and grammatical completeness are extracted.
  • The UserGRPO completeness delta is -0.089 in commented TeX, but the published table removes the sign and shows 0.09.
  • Reward and evaluation reuse the same decision ground truth and proxies related to RecQuest.
  • Weights, normalization, judge prompt/scale, n-gram classifier, length targets, and GRPO parameters absent; the TeX retains a TODO.
  • A single backbone, training category, and run; no real ablation of the six rewards.
  • Commercial sources and images without URLs, dates, licenses, or permissions; infringement is not presumed, but redistribution cannot be assessed.
  • No repository, dataset, environment, transcripts, executable metrics, adapter, or public checkpoint.

What the study does not establish

  • Realism of shoppers or predictive validity of human decisions.
  • Representativeness of United States consumers or prevention of distributional biases.
  • Economic, sociotechnical, or market conclusions from these personas.
  • That the models are causally persuaded by a seller.
  • That images improve the simulation or are used in the decision.
  • Controlled comparison between closed and open models.
  • Valid relative improvement of 13.8% over the same set.
  • Necessity of GRPO versus the specific rewards, data, and proxies used.
  • Generalization of the training to other families, scales, sellers, or tasks.
  • Human fluency or naturalness from reducing grammatical completeness and TF-IDF.
  • Redistributable license of the inventory and images.
  • Independent reproduction.
  • Acceptance at NeurIPS, COLM, or another conference.

Traceability

Scope: Full text

Version: arXiv:2605.08334v1; 24-page PDF and complete TeX package

Consulted source: https://arxiv.org/abs/2605.08334v1

Review: Codex 24-page visual full-text, complete TeX, persona-ground-truth, metric, denominator, human-baseline, GRPO, statistics, data-license and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • ChatGPT-5.4 shopper and fixed sales agent
  • Gemini-3-Flash
  • Gemma3-12B-Instruct
  • Gemma4-31B-Instruct
  • Qwen3-VL-8B
  • GLM-4.6V-Flash (9B)
  • GPT-4o for persona rewriting/validation

Instruments and metrics

  • SalesSim dual-agent retail environment
  • Precomputed acceptable-product sets
  • Decision Alignment
  • Premature termination rate
  • Tool/action format error rate
  • First-turn criteria count
  • Sentence-completeness gap
  • Within-category TF-IDF redundancy gap
  • GPT-5.4 reasoning-quality judge
  • N-gram human-vs-model linguistic discriminator
  • UserGRPO trajectory reward

Data used

  • SalesSim (not publicly released at audit time)
  • Nemotron-Personas-USA
  • Amazon Product Review Dataset
  • RecQuest/CRS: 515 human shopper-AI recommendation dialogues
  • Unspecified retailer, boutique, manufacturer and automaker listings

Evidence and location

  • Text, tables, prompts, transcripts, sources, and limitations: arXiv:2605.08334v1; PDF sha256 60f99da3379eef7e847652512bffce23b838ac300dd4f57333dd44c18c758b65; main TeX sha256 a203cd4e5e3fb7cef571d5a614c01ca86787e7cee012fec24ec55ae74e923ea9
  • Code, dataset, checkpoint, and official page not found: arXiv source package plus exact-title, Salesforce Research, GitHub and Hugging Face searches on 2026-07-17
  • Recalculations, denominators, human metrics, reward overlap, governance, and reproducibility: reports/verification/article-345-salessim-persona-ground-truth-metric-denominator-human-baseline-grpo-statistics-data-license-and-reproducibility-audit.json