Synthetic Personality Review

Towards Automated Crowdsourced Testing via Personified-LLM

Personas, identity, and agents2026arXivApproved editorial review

Authors: Shengcheng Yu, Yuchen Ling, Chunrong Fang, Zhenyu Chen, Chunyang Chen

Keywords: Persona conditioning, Human simulation

Source: Open primary source (opens in a new tab)

5
Authors
8
Findings
27
Limitations
6
Evidence

Editorial summary

English

PersonaTester turns nine interaction patterns derived from 1,500 crowdtesting traces into instructions for agents testing 15 Android apps. Compared with the same system without persona conditioning, profiles produce more repeatable and distinct paths, more events judged effective, and a larger union of failures: the personified ensemble reports 132 crashes versus 36 across nine baseline repetitions and 11 functional bugs confirmed by three authors, while the baseline triggers three. This is useful evidence that structured diversity can broaden exploration. It does not, however, demonstrate faithful crowdworker reproduction: categories are assigned to traces rather than repeated behavior of each worker; prompts instruct agents to click, enter text or follow workflows, and the metrics measure those same actions again. The 20-person study rates fit to written profiles, not similarity to humans. Nor are the three dimensions shown to be orthogonal. Inference lacks intervals or hierarchical models, and effectiveness/crash oracles depend on GPT-4o without published human validation for crashes. MIT-licensed code and auditable cost sheets exist, but the public entrypoint fails because of incorrect unpacking and IDs, omits scenario wiring, and releases no raw results, BiLSTM, or figure scripts. The paper supports persona prompting as a testing heuristic, not an equivalent automated human crowd.

Español

PersonaTester convierte nueve patrones de interacción extraídos de 1.500 trazas de crowdtesting en instrucciones para agentes que prueban 15 apps Android. Frente al mismo sistema sin persona, los perfiles producen rutas más repetibles y distintas, más eventos considerados efectivos y una unión mayor de fallos: el conjunto personificado reporta 132 crashes frente a 36 en nueve repeticiones del baseline y 11 bugs funcionales confirmados por tres autores, mientras el baseline activa tres. Es evidencia útil de que la diversidad estructurada puede ampliar la exploración. No demuestra, sin embargo, reproducción fiel de crowdworkers: las categorías se asignan a trazas, no a conducta repetida de cada trabajador; los prompts ordenan hacer clic, introducir texto o seguir flujos, y las métricas vuelven a medir esas mismas acciones. El estudio de 20 personas juzga encaje con descripciones de perfil, no similitud con humanos. Tampoco se prueba que las tres dimensiones sean ortogonales. La inferencia carece de intervalos o modelos jerárquicos, y el oracle de efectividad/crash depende de GPT-4o sin validación humana publicada para crashes. Hay código MIT y costes auditables, pero el entrypoint público falla por desempaquetado e ID incorrectos, omite el escenario y no libera datos, BiLSTM, resultados ni scripts de figuras. El trabajo respalda persona prompting como heurística de testing, no una multitud humana automatizada equivalente.

Research question

Whether injecting mindset profiles, exploration strategy, and entry habit into LLM agents can produce controlled diversity of GUI tests and discover more failures than repeating a non-personified agent.

Method

Three authors label by consensus 1,500 traces from a public crowdtesting corpus and select nine combinations that exceed 1% and cover 95.40% of the sample. PersonaTester combines CV/OCR, GPT-4o to understand and validate screens, o4-mini to decide actions, and ADB to execute them. One task is tested in each of 15 apps, five runs per configuration, and 20 minutes per run. RQ1 uses verb-object phrases, SBERT, BiLSTM, and cosine; RQ2 calculates events considered effective; RQ3 compares crashes and functional bugs. Twenty participants score videos against written profiles, and five students only fix the time limit.

Sample: Nine profiles derived from 1,500 labeled traces; nine combinations out of 18 possible exceed 1%, range from 2.27%-21.80%, and cover 95.40%. Experiments on 15 Android apps (four open-source and eleven commercial) with one manually chosen task per app, five runs per configuration, and 20 minutes. Twenty graduate students/QA professionals evaluate videos; five graduate students perform the tasks to estimate a human time of ten minutes. No distribution of videos, subgroups, payments, or exclusions is published.

Findings

  • Intra-persona cohesion usually sits at 0.88-0.99 versus 0.31-0.70 for the baseline; similarities between personas are usually lower, approximately 0.27-0.67.
  • The article summarizes relative consistency improvements of 117.86%-126.23%, with no intervals or inferential test.
  • The 20 evaluators score 8.35 (SD 0.67) for mindset, 8.70 (SD 1.08) for strategy, and 8.80 (SD 0.77) for entry habit.
  • Overall event effectiveness improves 33%-47% in most profiles; input-oriented ones report 683.32%-697.50% relative to a low baseline.
  • The personified union contains 132 distinct crashes (113 exclusive and 19 shared) versus 36 in nine repetitions of the baseline (17 exclusive and 19 shared).
  • Each persona reports 29-38 crashes versus 22 for the baseline in individual comparisons.
  • 11 functional bugs are confirmed by consensus; Persona H triggers six, B and G four, and the baseline three, all also covered by persona agents.
  • The public sheets reproduce approximately 0.404 USD of GUI analysis and 0.279 USD of decision, approximately 0.684 USD per task and agent.

Limitations

  • It does not compare agents and real crowdworkers on the same apps, tasks, and metrics; prompt-controlled diversity does not equal human fidelity.
  • It labels isolated traces, not repeated behavior per worker; it does not demonstrate that the profile is stable within a person.
  • The prompts prescribe click/input/core, sequence, and length/validity, and RQ1/RQ2 measure those same signals; there is construct circularity.
  • The study of 20 people shows recognizability against written profiles, not human-likeness, population distribution, or equivalence with crowdworkers.
  • It calls the three dimensions orthogonal without an association table, independence test, factor analysis, or per-worker evidence.
  • It does not report pre-consensus agreement for the 1,500 labels, nor a codebook, adjudications, or uncertainty.
  • The BiLSTM lacks a training corpus, objective, split, loss, checkpoint, held-out validation, and comparison with simple distances; its tuning was assisted by GPT.
  • The 10/25 similarity pairs reuse the same five runs; without cluster treatment they may produce pseudoreplication.
  • Runs are nested in agent, app, and task, but there are no random effects, cluster bootstrap, intervals, or RQ1-RQ3 tests.
  • There is only one manually chosen task per app, with no task sampling, preregistration, or sensitivity to alternatives.
  • The baseline only removes persona; it does not control prompt length/detail, other diversity strategies, or compare with previous GUI LLM systems.
  • The relationship between five runs per configuration and nine accumulated repetitions of X is not explained with a total table of runs, failures, timeouts, and exclusions.
  • Effectiveness is a semantic judgment by GPT-4o with no human sample, accuracy, agreement, blinding, or error analysis.
  • The input percentages near 700% stem from a low baseline and instructions that order searching for inputs; they are not a general quality improvement.
  • Only functional bugs are described as manually confirmed. The public crash oracle uses two screenshots, not logcat, process exit, or stack trace, and the deduplication of 132 crashes is not released.
  • Missing screenshots or GPT-4o errors pass assertions by default; failure rates are not published.
  • Temperature 0 does not guarantee determinism, and o4-mini does not even receive temperature in the code; there are no snapshots, seeds, dates, or archived responses.
  • The public entrypoint does not work: get_app_info returns five values and main unpacks four; it uses app-1 although the CSV uses 001, 015, etc.
  • The entrypoint omits the scenario returned by CSV, even though the scenario is a central part of the experimental design.
  • There are /home/lyc/Projects paths, empty credentials, cwd-dependent access, requirements without scikit-image, no example .env, container, tests, or lockfile.
  • Labels, worker IDs, frequencies, raw runs, BiLSTM, event labels, bugs, statistical scripts, and the table/figure pipeline are not released.
  • Random and DeepSeek only appear as supplementary images without numerical tables, exact models, raw data, runs, or uncertainty.
  • The ADB commands use shell=True with model-generated text and incomplete escaping, a risk if the input is not reliable.
  • There is no ethics/IRB/consent/compensation for secondary traces, the video study, or the five students; nor details of privacy and retention.
  • Eleven apps are commercial, but versions, accounts, dates, network, terms, sandbox, and responsible disclosure procedure are missing.
  • Screenshots, OCR, and history are sent to hosted APIs with no discussion of PII, redaction, region, logging, or retention.
  • The cost comparison does not include hardware, engineering, retries, and human validation, nor does it publish crowdworker payment under equivalent conditions.

What the study does not establish

  • It does not demonstrate faithful reproduction of human patterns or equivalence with a crowd of crowdworkers.
  • It does not demonstrate that a trace corresponds to a stable personality of the worker.
  • It does not validate that the three dimensions are orthogonal, psychometric, or independent.
  • It does not fully separate the persona effect from obedience to action instructions that are then measured again.
  • It does not show that observers confuse agents with humans; only that they recognize the written profile.
  • It does not prove generalizable statistical superiority beyond one task for each of 15 apps.
  • It does not causally attribute bugs to specific dimensions, nor does it rule out that other structured instructions produce similar diversity.
  • It does not validate 132 crashes with system evidence or human review equivalent to the functional bugs.
  • It does not demonstrate that it is cheaper than human crowdtesting with the same budget, quality, time, and total cost.
  • It does not allow reproducing the numbers of the paper with the current repository and supplements.

Traceability

Scope: Full text

Version: arXiv:2603.24160v2; accepted FSE 2026 paper, DOI 10.1145/3808173

Consulted source: https://arxiv.org/abs/2603.24160v2

Review: Codex 23-page visual full-text, official arXiv/project, GitHub code/runtime, spreadsheet, construct, metric, statistical, human-fidelity, privacy and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o
  • o4-mini
  • DeepSeek-R1 (supplementary replacement, exact snapshot unspecified)
  • Sentence-BERT
  • BiLSTM

Instruments and metrics

  • Taxonomía de tres dimensiones y nueve perfiles
  • CV/OCR y GUI state textualization
  • Prompt ReAct de intención y operación
  • ADB Android GUI execution
  • SBERT 384d + BiLSTM 256d + similitud coseno
  • Intra-cluster cohesion e inter-cluster separation
  • General/input event generation effectiveness
  • Venn de crash bugs y consenso de functional bugs
  • Estudio de vídeo con Likert 1-10
  • Dos hojas Excel de tokens y costes

Data used

  • Public crowdsourced GUI testing corpus: approximately 23,000 reports, 50+ systems and 1,100+ workers
  • 1,500 manually annotated exploration traces, not released
  • 15 Android app tasks in app.csv
  • PersonaTester experiment runs and bug evidence, not released

Evidence and location

  • Profile design, system, apps, runs, and metrics: Accepted FSE 2026 PDF, pp. 1-15; all 23 pages visually inspected
  • Effectiveness, 132/36 crashes, 11 functional bugs, and declared limits: Accepted FSE 2026 PDF, pp. 15-20, Figures 6-7 and Table 2
  • Publication, version, DOI, license, and acceptance: Official arXiv v2 Atom/HTML and accepted PDF inspected 2026-07-17
  • Code, entrypoint, scenarios, crash oracle, fail-open, dependencies, and ADB security: Official GitHub commit 6bae4143f27f217a644c24fa1384571dcc0921cb inspected and byte-compiled 2026-07-17
  • Random/DeepSeek figures and costs: Official project page and both embedded Excel workbooks inspected and visually verified 2026-07-17
  • Comprehensive audit of human fidelity, circularity, metrics, statistics, artifacts, privacy, and reproducibility: reports/verification/article-384-personatester-human-fidelity-circularity-bug-oracle-statistics-code-and-reproducibility-audit.json