Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception

Society, culture, and collective behavior2026arXivApproved editorial review

Authors: Neemias B da Silva, Rodrigo Minetto, Daniel Silver, Thiago H Silva

Keywords: Persona prompting, Multimodal LLMs, Synthetic annotators, Urban sentiment, Human-AI agreement, Extremity bias, Persona validity, Reproducibility audit

Source: Open primary source (opens in a new tab)

4
Authors
10
Findings
21
Limitations
3
Evidence

Editorial summary

English

This preprint asks whether simple persona labels produce urban-sentiment judgments that are stable, distinct across profiles and valid as proxies for human perception. Using Qwen3-VL:8B through Ollama, it crosses gender, economic status, political orientation and personality style into 24 profiles. It makes 50 calls per profile on the same 50 PerceptSent images: 60,000 attempts, of which the artifact releases 59,708 valid responses and 292 failures. The same model is also run without a persona, with and without thinking, five times per image. Repetitions under an identical profile converge strongly: across 1,200 profile-image groups, agreement with the modal class averages 0.871 and has a median of 0.98. Stability, however, is not the same as valid diversity. The global profile contrast is statistically significant under the very large row count but explains less than 1% of variation; gender has no effect, while economic status, politics and personality produce small differences. Persona-conditioned outputs also place 77.34% of predictions in the extreme Negative or Positive classes, versus 65.2-66.0% without a persona. Human agreement deteriorates from coarse polarity to five classes, and no-persona controls match or exceed persona-conditioned predictions in every published task variant. The defensible conclusion is that these four labels induce highly repeatable and somewhat more extreme outputs but add little value as synthetic human annotators in this setting. Audit identifies several material reporting issues. Figure 5 compares model predictions on 50 images against the human distribution over all 5,000 images: the cited 35.3% human extreme share is not the experimental sample, where extremes are 44.0%. Extremity bias remains, but the matched gap is about 33.3 points rather than 42.5. The paper describes five-fold cross-validation, whereas the code draws one seeded 60% subsample per image and then bootstraps images. For no-persona controls the point estimate uses three of five runs, not the all-five modal class stated in the paper. Recalculation with all five preserves the conclusion but narrows several control advantages. Agreement CSVs linked by the README also have filenames and columns incompatible with the loader, so that analysis is not reproducible without reconstruction. Finally, the 1,200 entities called agents are stateless calls over only 24 label combinations, not persistent identities or independent people. There are no humans whose attributes are matched to the labels, no hierarchical model for images and repetitions, no tests or CI, and the Ollama tag does not preserve a model-weight digest. The work demonstrates consistency of prompt compliance and a useful limitation of simple personas, not demographic realism, psychological personality or valid individual simulation.

Español

Este preprint pregunta si etiquetas simples de persona producen juicios de sentimiento urbano estables, diferentes entre perfiles y válidos como aproximación a la percepción humana. Con Qwen3-VL:8B servido en Ollama, cruza género, nivel económico, orientación política y estilo de personalidad en 24 perfiles. Para cada perfil realiza 50 llamadas sobre las mismas 50 imágenes de PerceptSent: 60.000 intentos, de los que el artefacto publica 59.708 respuestas válidas y 292 fallos. También ejecuta el mismo modelo sin persona, con y sin razonamiento, cinco veces por imagen. Las repeticiones bajo un perfil idéntico convergen mucho: en 1.200 grupos perfil-imagen el acuerdo con la clase modal promedia 0,871 y la mediana es 0,98. Sin embargo, esa estabilidad no equivale a diversidad válida. El contraste global entre perfiles es significativo por el gran número de filas, pero explica menos del 1% de la variación; género no tiene efecto, y nivel económico, política y personalidad muestran diferencias pequeñas. Además, el 77,34% de las predicciones con persona cae en las clases extremas negativa o positiva, frente a 65,2–66,0% sin persona. La concordancia con humanos empeora al pasar de polaridad gruesa a cinco clases, y los controles sin persona igualan o superan a la condición con persona en todas las variantes publicadas. La conclusión defendible es que estas cuatro etiquetas inducen respuestas muy repetibles y algo más extremas, pero aportan poco valor como anotadores humanos sintéticos en este escenario. La auditoría corrige varios detalles materiales. La Figura 5 compara predicciones del modelo sobre 50 imágenes con la distribución humana de las 5.000 imágenes: el 35,3% humano citado no corresponde a la muestra experimental, donde los extremos son 44,0%. El sesgo de extremidad continúa, pero la brecha comparable es unos 33,3 puntos, no 42,5. El texto describe validación cruzada de cinco pliegues, aunque el código sólo toma una submuestra sembrada del 60% por imagen y luego bootstrappea imágenes. Para los controles sin persona usa tres de cinco ejecuciones en el estimador puntual, no la moda de las cinco que declara el artículo. Al recalcular con las cinco, la conclusión no cambia, pero se reducen varias ventajas del control. Los CSV de acuerdo enlazados en README tampoco coinciden en nombres ni columnas con el cargador, por lo que esa parte no se reproduce sin reconstrucción. Finalmente, los 1.200 llamados agentes son ejecuciones sin memoria con sólo 24 combinaciones de etiquetas, no identidades persistentes ni personas independientes. No hay humanos con atributos emparejados a esas etiquetas, modelo jerárquico para imágenes y repeticiones, tests o CI; el tag Ollama no conserva digest de pesos. El trabajo muestra consistencia de obediencia a un prompt y una limitación útil de las personas simples, no realismo demográfico, personalidad psicológica ni simulación individual válida.

Research question

Do four demographic and style labels produce multimodal personas whose judgments about urban scenes are reproducible within profile, differentiable between profiles, and concordant with human perception, or does the same model without persona work equally or better?

Method

The study uses Qwen3-VL:8B locally through Ollama with temperature 0.1, seed 42 and reasoning activated. It crosses male/female gender, low/high economic level, progressive/conservative politics and analytical/empathetic/pragmatic style to form 24 profiles. Each profile is repeated 50 times over 50 images from PerceptSent selected by human modal class and indoor/outdoor context. The prompt presents the four labels, instructs the model to be that person and not an AI, and requests sentiment in five classes, perceptions, description and justification. Modal agreement and variance within profile are measured, Kruskal-Wallis between profiles and Mann-Whitney or Kruskal-Wallis per factor. Two controls without persona, with and without reasoning, produce five annotations per image. Concordance with humans is evaluated under four label mappings and three agreement thresholds. The audit visually read the 8 pages, inspected TeX, commit, code, notebooks, data and documentation, reconstructed the 60,000 keys and distributions, reproduced tests and sensitivity per image, reconstructed the 12 agreement subsets and recalculated the control using the five declared runs.

Sample: There are 24 combinations of labels and 50 calls per combination, called agents by the article, over the same 50 images: 60,000 identifier-image pairs planned. 59,708 successes and 292 failures are published, whose union covers exactly the 60,000 keys without duplicates; each profile has 50 calls. For each control without persona there are five runs per image, 250 responses. The original human sample contains 5,000 images with five annotations each, but the experiment uses 50: 11 Negative, 11 Positive, 11 Neutral, 9 Slightly Negative and 8 Slightly Positive according to the human mode. No people from the simulated demographic groups or styles are recruited.

Findings

  • The modal agreement within 1,200 profile-image groups is 0.8707 on average and 0.98 median; the mean sentiment variance is 0.2114 and its median 0.02.
  • The contrast of the 24 profiles reproduces H(23)=489.52, p=5.33e-89 and epsilon squared=0.0078; the difference between extreme means is only 0.552 on a scale from -2 to +2.
  • Gender shows no effect; economic level, politics and personality style show small differences that also retain direction when blocking by image.
  • Predictions with persona are 42.88% negative, 34.46% positive, 11.48% neutral, 6.08% slightly negative and 5.09% slightly positive: 77.34% in extremes.
  • Without persona, extremes are 65.2% with reasoning and 66.0% without reasoning.
  • The experimental human sample has 44.0% extreme classes; 35.3% of Figure 5 belongs to the complete corpus of 5,000 images.
  • Concordance with humans worsens when increasing the resolution from two to five classes.
  • Under the code procedure, the controls without persona equal or surpass the persona condition in all mappings and thresholds.
  • Using the mode of the five runs without persona, as the article states, reduces several macro-F1 of the control but does not reverse any main comparison.
  • The data allow verifying integrity, distributions, convergence and main tests, but the documented agreement download does not feed the loader without repair.

Limitations

  • Only one mutable tag of Qwen3-VL:8B, one prompt in English and 50 images are tested.
  • The 1,200 units called agents are calls without memory with only 24 profiles, not persistent agents or independent subjects.
  • The four labels simplify identities and may activate model stereotypes.
  • There are no humans whose demographic and personality attributes are matched to the synthetic profiles.
  • Stability under identical labels measures repeatability, not realism or construct validity.
  • Row-wise tests treat 59,708 observations as independent despite sharing 50 images and 1,200 call identifiers.
  • No mixed or blocked model is used that jointly represents image, profile and repetition.
  • Multiple tests of profiles and factors have no declared family-wise correction.
  • Extreme significance coexists with small effect sizes, including epsilon squared 0.0078.
  • Figure 5 mixes 50 model images with 5,000 human images and overestimates the extremity gap.
  • Table IV gives median variance 0.000 versus 0.020 in data and documentation, and inconsistently rounds group size.
  • The article says five-fold cross-validation, but the code does a single seeded subsample of 60% and bootstrap of images.
  • The comparison without persona uses three of five runs per image in the published estimator, not the five declared.
  • The legend of Figure 8 calls out-of-fold matrices that the code computes with the full baseline mode.
  • The bootstrap fixes a single modal subsample beforehand and does not incorporate generation uncertainty, weights, profiles or image selection.
  • The downloadable agreement files have alpha names and text/sentiment columns, while the loader requires sigma and image_id/ground_truth.
  • JPEG images are not versioned and depend on an external download.
  • The model is identified by qwen3-vl:8b without digest, weight checksum, Ollama version or hardware.
  • The default image directory contains an absolute path /mnt/raid5 from the authors' machine.
  • There are no tests, CI or numerical regressions, although there is a CC BY 4.0 license, pyproject and uv.lock.
  • Generated perceptions, captions and justifications do not receive independent validation to support persona validity.

What the study does not establish

  • That the labels create a stable psychological personality.
  • That the profiles represent real people of low or high economic level, progressive, conservative, analytical, empathetic or pragmatic.
  • That a difference provoked by a label reproduces a human difference of the corresponding group.
  • That 50 repetitions of a prompt constitute 50 independent agents.
  • That high internal concordance demonstrates identity, memory, autonomy or agent continuity.
  • That the model is a valid substitute for individual human annotators or urban populations.
  • That row-wise significance implies large or useful effects in practice.
  • That the human comparison of 35.3% is valid for the 50 experimental images.
  • That five-fold cross-validation was executed or out-of-fold predictions were obtained.
  • That the intervals include model, generation, prompt, profile or corpus selection variation.
  • That the result generalizes to other VLMs, weights, prompts, languages, cities, tasks or sentiment scales.
  • That persona prompting improves annotation: in this design the control without persona equals or surpasses its results.

Traceability

Scope: Full text

Version: arXiv:2604.28048v2; repository commit 71fb693244686b0c1b6642d31008a1a39edd06c3

Consulted source: https://arxiv.org/abs/2604.28048

Review: Codex 8-page visual full-text, TeX, repository, 60,000-key integrity, denominator, agreement reconstruction, statistical, code and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Qwen3-VL:8B through a local Ollama tag
  • The same Qwen3-VL:8B setup without persona labels, with thinking enabled
  • The same Qwen3-VL:8B setup without persona labels, with thinking disabled

Instruments and metrics

  • Four-factor 24-profile persona prompt
  • Five-class urban sentiment classification
  • 593-term urban perception vocabulary
  • Within-profile modal agreement and sentiment variance
  • Kruskal-Wallis profile and personality tests
  • Mann-Whitney gender, economic-status and political-orientation tests
  • Image-paired Wilcoxon and Friedman sensitivity checks added in audit
  • Four human-agreement label mappings at sigma 3, 4 and 5
  • Macro-F1 and Cohen kappa
  • Image bootstrap confidence intervals

Data used

  • PerceptSent metadata for 5,000 urban images with five human annotations per image
  • Fifty selected PerceptSent image IDs; raw JPEGs require external download
  • 59,708 successful persona-conditioned annotations
  • 292 failed persona-conditioned annotation records
  • 250 no-persona annotations with thinking
  • 250 no-persona annotations without thinking
  • Twelve human-agreement subsets reconstructed during audit because documented downloads do not match the loader

Evidence and location

  • Full text, tables, figures, design, discussion and limitations: arXiv:2604.28048v2; PDF sha256 de46c343df3278e520b2cdf597e4b376b05d36fd515ad199fed7321fc0d63d42
  • Code, data, results, documentation, notebooks and figures: GitHub neemiasbsilva/mllm-persona-evaluation commit 71fb693244686b0c1b6642d31008a1a39edd06c3; archive sha256 5f43c900ad9aa3eef172abeee5dc9004a1a0b01a073cfe530e9b8ff4353a9a28
  • Audit of units, denominators, human comparison, cross-validation, statistics, construct and reproducibility: reports/verification/article-354-urban-sentiment-persona-validity-denominator-cross-validation-statistics-and-reproducibility-audit.json