FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agents

Applications, bias, and safety2026arXivApproved editorial review

Authors: Muhammad Usman Safder, Ayesha Gull, Rania Elbadry, Fan Zhang, Yankai Chen, Xueqing Peng, Xue (Steve) Liu, Preslav Nakov, Zhuohan Xie

Keywords: Agent behavioral consistency, Financial agents, Mandate re-grounding, Synthetic market simulation, Long-horizon evaluation, Persona prompting, MBTI personas, Big Five personas, Prompt-position effects, Reproducibility

Source: Open primary source (opens in a new tab)

9
Authors
8
Findings
12
Limitations
6
Evidence

Editorial summary

English

This preprint introduces FinPersona-Bench to study whether LLM-based financial agents maintain behavioral mandates over a simulation. The paper crosses 18 models, three financial prompts associated with ENTJ, ISFJ, and INTJ, three synthetic markets, two prompt architectures, and five seeds over 200 steps. The flat market measures deviation from an author-assigned cash fraction, the crash uses maximum drawdown, and the bubble uses a binary rule for buying below and selling above a hidden fundamental value. Aggregated results report that repeating the mandate reduces flat-market cash deviation by 12.7% and crash drawdown by 12.6%, but worsens the bubble value-rule score by 8.8%. Effects depend strongly on prompt and scenario. In a flat market, repetition helps the conservative ISFJ prompt in 17 of 18 models and hurts the aggressive ENTJ prompt in 16 of 18. This interaction is the most useful descriptive result: reinforcing an instruction can improve or degrade policy depending on whether the mandate fits the environment. However, the implementation does not measure the mechanism called Mandate Salience Decay through accumulating context. The static agent is a stateless predictor. Each day makes a fresh call, re-sends the complete persona as a system message, and includes only the current observation and two portfolio values. It retains no conversation, rationales, or prior tokens. The memory agent receives the same system prompt and additionally gets a short imperative mandate in the user message. The contrast therefore identifies effects of duplication, position, wording, and immediate prompt emphasis together with portfolio trajectory; it does not demonstrate long-context forgetting. The reported 4.4x widening of the crash gap is also not a direct salience measure because it uses cumulative drawdown, market phases that change over time, and path-dependent portfolios. Nor is this a psychometric validation. MAS is distance from author-chosen cash targets .2, .5, and 1.0. The Big Five appendix tests two prompt archetypes and a numerical control without an inventory, reliability, factor structure, or criterion validation. The repository audit finds major drift: its public runner uses three models, 100 days, and temperature .2, whereas the paper reports 18, 200, and temperature 0. It lacks the stated Hugging Face/vLLM route and publishes no raw results or classifier. The market generator also uses NumPy's global RNG inside ten threads. Forty concurrent constructions labeled seed 42 produced 40 distinct markets, while five sequential constructions were identical. Static-memory pairing by seed is therefore not guaranteed by the released orchestrator, threatening the paired Wilcoxon design. Tests do not pass from a clean clone. The defensible conclusion is narrow: repeating a mandate in the current prompt heterogeneously changes a synthetic financial policy; longitudinal psychometric decay and instruction loss from accumulating context have not been demonstrated.

Español

Este preprint introduce FinPersona-Bench para estudiar si agentes financieros basados en LLM mantienen mandatos conductuales a lo largo de una simulación. El artículo cruza 18 modelos, tres prompts financieros asociados a ENTJ, ISFJ e INTJ, tres mercados sintéticos, dos arquitecturas de prompt y cinco semillas durante 200 pasos. El mercado plano evalúa desviación respecto a una fracción de caja fijada por los autores; el crash usa máximo drawdown; y la burbuja usa una regla binaria de compra por debajo y venta por encima de un valor fundamental oculto. La comparación agregada informa que repetir el mandato reduce 12,7% la desviación de caja en el mercado plano y 12,6% el drawdown en el crash, pero empeora 8,8% la regla de valor en la burbuja. El efecto depende fuertemente del prompt y el escenario: en plano, la repetición ayuda al perfil conservador ISFJ en 17 de 18 modelos y perjudica al agresivo ENTJ en 16 de 18. Esta interacción es el resultado descriptivo más útil: reforzar una instrucción puede mejorar o degradar la política según si el mandato es apropiado para el entorno. Sin embargo, la implementación no mide el mecanismo que el trabajo denomina Mandate Salience Decay por acumulación de contexto. El agente static es un predictor sin estado: en cada día hace una llamada nueva, vuelve a enviar la persona completa como mensaje de sistema y añade solo la observación actual y dos valores de cartera. No conserva conversación, racionales ni tokens previos. El agente memory recibe el mismo sistema y además una versión corta e imperativa del mandato en el mensaje de usuario. Por tanto, el contraste identifica efectos de duplicación, posición, redacción y saliencia inmediata del prompt, junto con la trayectoria de la cartera; no demuestra olvido por contexto largo. El aumento 4,4x de la brecha del crash tampoco es una medida directa de saliencia: usa drawdown acumulativo, mercados que cambian de fase y carteras dependientes del camino. Tampoco es una validación psicométrica. MAS mide distancia a objetivos de caja .2, .5 y 1.0 asignados por los autores; el apéndice Big Five prueba dos arquetipos de prompt y un control numérico, sin cuestionario, fiabilidad, estructura factorial ni validez de criterio. La auditoría del repositorio encuentra una deriva grave: el runner público usa tres modelos, 100 días y temperatura .2, frente a 18, 200 y temperatura 0 en el artículo; no soporta las rutas Hugging Face/vLLM declaradas y no publica resultados ni clasificadores. Además, el mercado usa el RNG global de NumPy dentro de diez hilos. Cuarenta construcciones concurrentes con la misma semilla 42 produjeron 40 mercados distintos, mientras cinco secuenciales fueron idénticos. El emparejamiento static/memory por semilla no está garantizado en el orquestador liberado, lo que amenaza los Wilcoxon pareados. Los tests tampoco pasan desde un clon limpio. La conclusión defendible es estrecha: repetir un mandato en el prompt actual cambia de forma heterogénea una política financiera sintética; no se ha demostrado decaimiento psicométrico longitudinal ni pérdida de instrucciones por acumulación de contexto.

Research question

Do LLM-based financial agents maintain a policy compatible with risk mandates defined by prompts over 200 steps, and how does their behavior change when repeating a short mandate in each user message?

Method

Factorial simulation benchmark. Eighteen models receive one of three MBTI prompts with financial extensions and cash objectives, and operate in flat, crash, and bubble synthetic markets. Compares a stateless agent that resends the full persona in the system on each call with another that additionally repeats a short mandate in the user. Runs five market seeds, 200 steps, and analyzes distance to target cash, drawdown, and a binary value rule via paired Wilcoxon. Includes placebo, two Big Five prompts, numeric-only control, reinjection frequency, temporal calibration, crash sensitivity, and linguistic analysis.

Sample: Declared main grid: 18 models, three persona prompts, three scenarios, two conditions, and five seeds; 1,620 simulations of 200 steps, equivalent to 324,000 decisions. Diagnostics use subsets of one to three models and between three and five seeds.

Findings

  • Repeating the mandate reduces aggregate MAS deviation from .391 to .342 in the flat market and improves drawdown from -26.18% to -22.89% in the crash.
  • In the bubble, the same intervention reduces the value rule score from 85.9 to 78.4; reinforcing a mandate misaligned with the scenario can harm behavior defined as rational.
  • In flat, reinjection helps the conservative ISFJ prompt in 17/18 models and harms the aggressive ENTJ in 16/18, showing a strong interaction between mandate content and environment.
  • Effects by model are heterogeneous; several small or open models reverse the crash pattern or incur greater costs in the bubble.
  • The placebo is descriptively close to static, but the pooled placebo-memory and static-memory comparisons are p=.720 and p=.890; a formal interaction test by persona is missing.
  • The two Big Five archetypes reproduce the conservative/aggressive direction in three models and the numeric control reduces the effect, consistent with prompt vocabulary influence.
  • The public implementation does not accumulate context: each call receives the full persona again and only the current state; the memory condition adds redundancy and immediate imperativeness.
  • The concurrent orchestrator does not preserve market paths per seed due to global RNG; the statistical pairing of the released code is not reproducible.

Limitations

  • There is no conversational history or token growth across days; therefore the design does not observe degradation due to long context.
  • Static and memory differ simultaneously in duplication, position, length, imperative tone, and user content; the placebo does not match vocabulary or instructional force.
  • MAS uses cash objectives defined by the authors and not psychometric scores; MBTI and Big Five function as prompt templates, not validated instruments.
  • Maximum drawdown does not directly measure panic or caricature, and the binary value rule imposes a coarse and normative definition of rationality.
  • Temporal curves use means and cumulative extremes, path-dependent portfolios, and scenarios that change phase, conflating time with regime and mathematical accumulation.
  • The Wilcoxon tests aggregate related models, personas, and seeds without modeling clustering or applying a global correction for the multiple tests.
  • T=200 is chosen after a calibration with many horizons; the appendix itself acknowledges implicit multiple testing and there is no held-out temporal evaluation.
  • The classifier splits rows randomly instead of grouping by model, run, or seed, allowing leakage of near-duplicate rationales between train and validation.
  • The code converts the decision to HOLD after three provider/parser errors without a specific flag; failures may appear as conservative behavior.
  • There is only one asset, one agent, and synthetic markets; there are no transaction costs, slippage, multi-asset portfolio, other agents, or validation in real deployment.
  • API aliases are mutable, there are no model sampling seeds, and results/outputs are not published.
  • The repository diverges from the article, lacks a license, versioned lock, and CI, and its tests fail.

What the study does not establish

  • It does not demonstrate that the mandate loses salience because context accumulates; the LLM input context does not grow in the implementation.
  • It does not validate Mandate Salience Decay as a psychometric construct nor measure stable personality, MBTI, or the Big Five structure.
  • It does not demonstrate that the 4.4x gap is longitudinal forgetting; it may arise from cumulative drawdown, market phases, and portfolio dependence.
  • It does not isolate semantic content as the mechanism of reinjection, because the pooled comparisons with memory are non-significant and a formal interaction is missing.
  • It does not convert drawdown into a validated measure of panic selling nor the P/V rule into a universal measure of financial rationality.
  • It does not guarantee that the static/memory pairs share the same market in the released concurrent code.
  • It does not allow reproducing the figures of 18 models and 200 days with the public runner of three models and 100 days.
  • It does not support trading recommendations, fiduciary management, or real financial deployment.

Traceability

Scope: Full text

Version: arXiv:2606.31522v2; preprint in COLM 2026 style without an identified acceptance

Consulted source: https://arxiv.org/pdf/2606.31522

Review: Codex 29-page full-text visual, TeX, publication, construct, causal, metric, statistical, repository, executable test, RNG and reproducibility audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o
  • GPT-4o-mini
  • GPT-4.1
  • GPT-4.1-mini
  • GPT-5-mini
  • GPT-5.4
  • GPT-5.4-mini
  • Claude Haiku 4.5
  • Claude Sonnet 4.6
  • Claude Opus 4.6
  • Gemini 2.5 Flash
  • Gemini 2.5 Pro
  • Gemini 3.1 Pro Preview
  • DeepSeek Chat
  • Llama-3.1-8B-Instruct
  • Gemma-2-9B-it
  • Qwen2.5-7B-Instruct
  • Gemma-3-4B-it
  • DistilBERT auxiliary persona-rationale classifier

Instruments and metrics

  • Mandate Adherence Score: absolute deviation from author-assigned cash target
  • Caricature Index: maximum portfolio drawdown
  • Rationality Gap: binary action alignment with hidden fundamental value
  • MBTI-derived persona prompts with financial extensions
  • Two Big Five prompt archetypes and numerical-only ablation
  • Static, placebo and mandate re-injection prompt conditions
  • DistilBERT intended-persona probability
  • Lexical Conflict Rate

Data used

  • FinPersona-Bench synthetic single-asset market
  • Flat low-signal scenario
  • Crash scenarios with price/value discounts .85, .92 and .95
  • Bull-trap bubble scenario
  • Unreleased model decision and rationale logs

Evidence and location

  • v2 status, corrected authors, date, and declared scope: Official arXiv record 2606.31522v2, TeX source and 29-page PDF, checked 2026-07-16
  • Architectures, markets, prompts, metrics, and grid of 18 models: arXiv v2, Sections 3-4 and Appendices A-D
  • Aggregate, temporal, per-persona, and per-model results: arXiv v2, Tables 1 and 9-11, Figures 3-5
  • Placebo, Big Five, frequency, calibration, and classifier: arXiv v2, Appendices E-H and L
  • Absence of accumulated context, runner drift, concurrent RNG, tests, and artifacts: Official repository usmansafdarktk/FinPersona-Bench at commit af2bd7a3ef3a65a12895bb87458b80c753ff12f0
  • Consolidated audit of construct, causality, code, data, and reproducibility: reports/verification/article-286-arxiv-finpersona-stateless-prompt-reinjection-rng-pairing-code-data-and-claim-audit.json