PTCBENCH studies how Big Five scores from LLM-based systems change when a NEO-FFI assessment is situated in different contexts. It includes four foundation models, Gemini 2.0 Flash, GPT-4o-mini, Claude Sonnet 4, and GPT-OSS-120b, and two agents, AutoGen and CAMEL, both using GPT-4o-mini as their backbone. The design combines six locations with six life events. Although the paper describes “12 external conditions,” every situated prompt contains both a location and an event: the released averaged results have 37 rows, one baseline plus 36 factorial pairs. The reported 39,240 records reconstruct exactly as 6 systems × (36 pairs × 3 repetitions + 1 baseline) × 60 items. These are therefore not twelve isolated interventions, and location or event effects are estimated within constructed combinations.
Each system answers the 60 NEO-FFI items using options 1–5. The code sums twelve items per trait and reverse-scores negative-keyed items, producing a 12–60 range; the paper's equation instead defines a 1–5 average while its tables report sums up to 60. Native profiles generally show low Neuroticism and high Openness/Conscientiousness. For locations, Table 1 reports human B coefficients of 0.319–0.795 versus 0.016–0.082 for foundation models, meaning smaller model changes on that scale. For events, Table 2 gives much larger ranges for Gemini, 0.904–6.271, than GPT-4o-mini, 0.256–1.462, Claude, 0.071–0.114, or GPT-OSS, 0.086–0.137. These metrics are not directly interchangeable with raw-difference heatmaps or appendix standardized effects.
High/medium/low preset profiles demonstrate instruction control in GPT-4o-mini. Openness, for example, averages 50.17/35.83/29.09; Conscientiousness 55.05/38.62/26.49; Extraversion 52.52/34.91/20.85; Agreeableness 51.05/42.03/25.67; and Neuroticism 55.13/33.97/13.53. This shows that prompt labels and descriptions shift questionnaire answers, but ceilings, floors, and regression toward the mean constrain how much each profile can subsequently change. The repository generator creates 3^5=243 combinations; the paper's “244th” None control is not in that generated list and can only be substituted manually.
Across-run stability is calculated through ICC(3,1) and ICC(3,k). For baseline/situated conditions, GPT-4o-mini reports ICC(3,1) 0.67/0.64 and ICC(3,k) 0.91/0.85; Claude 0.87/0.79 and 0.97/0.94; Gemini 0.96/0.86 and 0.99/0.95; AutoGen 0.93/0.91 and 0.99/0.98; and CAMEL 0.75/0.65 and 0.94/0.90. GPT-OSS is omitted. The code treats the five traits as targets and runs as raters, so the statistic captures preservation of the rank ordering of a five-point profile, not instrument internal consistency or item-level stability. The appendix also says any ICC(3,k)<0.8 was rerun automatically, yet released outputs retain GPT-4o-mini bar+divorce at ICC(3,k)=0.164 and bar+unemployment at 0.572.
The most important interpretive issue is that the situated condition does not change context alone. The public prompt includes the baseline Big Five score, all 60 baseline questions and answers, and, as the questionnaire progresses, all prior situated conversation. The paper calls this a “compressed” history or a small subset, but the code accumulates the full 60-answer record. The comparison therefore mixes location, event, explicit anchoring on prior scores, verbatim answer recall, item order, and growing context length. Scenarios are synthetic one-line instructions, not realistic evolving environments. Changes cannot be cleanly attributed to situational context.
Standardized effects do not support strong psychological comparisons either. Table 6 reaches values such as dE=-18.528 for Divorce on Gemini Neuroticism, CI [-33.385,-3.671], driven by tiny baseline standard deviations and few runs. Most intervals for GPT-4o-mini, Claude, and GPT-OSS cross zero. The prose is internally inconsistent: it first says foundation-model location changes are smaller than human changes, then concludes LLMs, “especially foundation models,” change substantially more. Raw sums, regression coefficients, and standardized effects are alternated without adequate reconciliation.
The AGIEval experiment does not show that altered personality changes reasoning. It runs only on GPT-OSS-120b and retrospectively selects, for each trait, the three location-event pairs producing the largest trait deviation. The script adds only the location-event text to each AGIEval question; it does not add or intervene on a personality score, profile, NEO-FFI answer history, or mediating state. Accuracy differences are therefore direct effects of selected context text, not causal estimates of personality effects. The same pair is assigned multiple trait labels and yields identical rows: Vehicle+Unemployment appears for Conscientiousness and Extraversion; Home+Unemployment for Conscientiousness, Extraversion, and Neuroticism; and Vehicle+Divorce for Conscientiousness, Agreeableness, and Neuroticism.
Table 8 reports a positive average improvement in all fifteen selected conditions, from +0.0065 to +0.0439 absolute accuracy. This conflicts with the narrative that decreases in Conscientiousness/Extraversion or increases in Neuroticism degrade reasoning; the boxed Finding 5 even says stress tends to improve reasoning, while the discussion reverts to degradation. The “up to +20% overall AGIEval” claim is not supported by the table: the largest mean change is +4.39 percentage points; roughly 20% refers to trait change, not overall accuracy. There is no independent replication, uncertainty, significance test, correction for selecting maxima, or mediation identification.
A public repository, FishAnonymous/PTCBench, was audited at commit 6545f85f1cb13e16816d28e674bd7b2ff6352725. It was released after arXiv v1 and is not linked by the paper or official record. It contains roughly 397 MB and 86,932 tracked files, mainly raw outputs, but has no license, tests, CI, requirements, pyproject, lockfile, container, or installation and execution commands. The README names missing camel/ and figures/ directories; the Dockerfile claimed by the paper is also absent. A reproduction attempt under Python 3.10+, the only declared requirement, fails immediately with ModuleNotFoundError: statsmodels. Installing current dependencies would not be faithful reproduction because no versions are pinned.
The code audit found additional material divergences: big5.py creates 50 ad hoc questions while naming its output NEO_FFI_60; the actual pipeline uses a different English 60-item CSV; credentials and model are selected by editing globals; output directories are not created; an answer lacking 1–5 silently becomes 3 and an exception becomes 0; a run is discarded when all five traits sum to 36 even though a neutral profile may be valid; repairing a zero situated response mistakenly invokes the baseline prompt; and the task list accumulates between repetitions. Released outputs mix three and five runs by model, conflicting with paper descriptions, and the ICC code requires five indices even though Gemini results have three.
PTCBENCH is useful as a contextual stress-test corpus and as evidence that LLM-generated self-report answers are sensitive to prompts, history, and provider. It does not establish authentic personality, psychometric equivalence to humans, longitudinal personality change, or a causal personality effect on reasoning. Its contribution should be described as an exploratory benchmark with partially auditable artifacts, not proof that life events produce real psychological change or that such change causes capability loss.