This preprint introduces TRAIT, an English-language benchmark of 8,000 multiple-choice questions for studying response patterns associated with the Big Five and Dark Triad. It starts from 44 BFI and 27 SD-3 items: GPT-4 expands them into 1,600 descriptions, selects five of twenty ATOMIC10X situations for each description, and constructs a situation, question, and four recommendations, two labeled high and two low. The result is about 112 times the 71 seeds; the 225× claim in the pipeline figure does not match the item count. Two graduate students with psychology training label only 200 items and reach 97.5% high/low accuracy. This supports the intended polarity in a sample, but does not validate all 8,000 items, trait specificity, factor structure, invariance, or behavioral relevance. The paper calls GPT-4-assigned facet diversity and evenness “content validity,” score change after high/low instructions “internal validity,” and sensitivity to three prompts, option order, and paraphrases “reliability.” These are useful audits of coverage, instruction following, and robustness, but are not by themselves equivalent to those psychometric constructs. Averaged across eight models, TRAIT scores 71.9 for diversity and 77.5 for score difference on the Big Five, and 51.0/83.3 on the Dark Triad; mean sensitivity is 29.8% and 24.4%, respectively. It improves on the baselines, yet changing roughly one quarter to one third of answers shows relative rather than strong consistency. Refusal rates are near zero in multiple choice and 3.1–3.3% in generation, much lower than self-report tests; however, the detector only checks predefined opening phrases and may conflate response format with refusal. TRAIT scoring averages A–D token probabilities under two option layouts and reports the proportion of high choices. It is not a factor score, and two permutations do not exhaust the 24 possible orders. Across nine models, GPT-4 and Claude-Sonnet score 86–87 on Agreeableness and 0–11 on Dark Triad traits. Grouping aligned and base models yields higher Agreeableness (78.3 vs 66.7) and Conscientiousness (91.0 vs 81.7), and lower Openness (56.3 vs 67.8), Extraversion (32.8 vs 46.9), and Dark Triad scores (9.3 vs 27.0). Because this comparison mixes model families and sizes, it does not identify a causal alignment effect. The more controlled Llama2-7B→Tulu2-SFT→Tulu2-DPO sequence suggests that SFT strongly shifts choices, for example +22.9 Agreeableness, −22.9 Extraversion, and −49.8 Psychopathy, while DPO changes little. Yet Table 7's caption describes the subtraction opposite to the interpretation, and T-EVALUATOR, used to label the training data, is itself trained on synthetic TRAIT. The reported .7893 correlation between corpus balance and trait change excludes Openness post hoc, uses only seven traits, and does not establish causality. Three persona prompts yield an average directional score of 85.2, but fine-grained results vary sharply by template and model: GPT-4 high-Psychopathy scores, for example, range from 37.3 to 99.7. Induced correlations among Dark Triad traits reach .90–.97, far above human values; they reflect at least partly shared prompt and generator semantics rather than a natural psychological structure. Five scenarios derived from the same description also alter choices, which is compatible with context sensitivity but also with scenario differences or measurement noise. Finally, correlations between TRAIT and general benchmarks use eight models without controlling model family, size, or alignment and do not establish predictive power of personality. The defensible contribution is a large scenario bank and multifactor response audit, not evidence of internal personality, human-equivalent psychometrics, or behavioral validity.
Research question
Can a benchmark of scenarios built specifically for LLMs measure differentiated and relatively consistent response patterns, and what do those patterns show about prompting, alignment, and training data composition?