The study converts psychological questionnaire items about empathy, emotion regulation, assertiveness, and impulsiveness into situational judgment tests for assistants. It starts from 332 statements, manually reduces them to 260, uses Gemini 3 Pro to filter and reframe them, and retains 161. Gemini 3 generates 16 scenarios per statement, 2,576 candidates, conditioned on a provisional AGREE, OPPOSE, or AMBIGUOUS class. Three annotators must unanimously confirm that each scenario contains a dilemma, that its actions oppose each other, and that the agree action reflects the statement; after excluding 8%, the paper reports 2,357 SJTs. Each then receives human preferences nominally from 10 people drawn from a pool of 550, with neutral counted as half a vote and N/A filtered.
For 25 LLMs, the authors sample 20 responses per scenario at temperature 1.0. The prompt requires exactly one of the two actions, forbids neutrality or recommending both, and limits the response to two sentences; Gemini 3 Flash classifies each text as agree, oppose, or neither. They compare the frequency of the trait-oriented action with the human distribution through Trait-Positive Rate, absolute difference, majority-choice consistency, and directional alignment around 0.5. They report that models remain above 90% consistency when human preference is near 50/50; that alignment improves with consensus and capability, although some frontier models disagree in 15%-20% of non-unanimous high-consensus cases; and that smaller models are often near chance. They also show gaps between direct 1-7 ratings and SJT behavior, especially for impulsiveness.
The interpretation should remain bounded to behavior elicited by this instrument. Forcing a single action removes the mixed and neutral responses at issue, so consistency across 20 samples is not calibrated epistemic confidence and the reported overconfidence is partly prompt-induced. Ten votes per scenario provide a coarse, predominantly US/UK preference reference rather than a universal social norm. The 2,357 SJTs are nested within 161 source statements and the same raters and models contribute repeated measurements, yet the paper provides no hierarchical analysis, intervals, cluster bootstrap, or inferential tables. Judge validation covers only 100 responses reported all correct, without class balance, agreement, blinding, or error analysis. The self-report/SJT comparison shows weak cross-format prediction under this design, not internal traits or general invalidity of every self-report protocol.
Official artifacts released after the preprint improve transparency but support only partial reproduction. Kaggle v3 contains 2,262 scenarios, 95 fewer than the paper's 2,357, and omits source statements, individual judgments, validation records, original outputs, and 25-model results; it does exactly reproduce the 1,348 high-consensus Figure 4 denominators. Moreover, 109 aggregate scores do not advance in half-point increments despite documentation claiming 10 annotators and neutral=0.5. The official notebook defaults to one Gemini model, 500 rows, and 6 replications rather than 25 models, 1,348 rows, and 20 replications; it is invalid strict JSON because of a trailing comma, and the code treats model TPR exactly equal to 0.5 asymmetrically relative to the paper's strict formula. The defensible contribution is a useful, partially reproducible SJT benchmark showing prompt-, sample-, and format-specific gaps, not latent dispositions or universal human misalignment.