This v2 preprint proposes assigning Big Five scores to text from four corpora and continuations from six models: GPT-2, GPT-3, GPT-3.5-Turbo, a 4-bit-quantized LLaMA 7B, Transformer-XL, and XLNet. It does not ask models to select Likert options. Instead, it uses the 50 questionnaire statements as prompts, generates 30 independent continuations per statement and model, and passes every text to a natural-language-inference zero-shot classifier. For each trait, the preferred “Approach 3” compares two opposite labels, such as extraversion/introversion, and maps entailment probability onto a 1–5 scale. The paper does not identify the NLI checkpoint, validate its scores against human judgments or a psychometric instrument, or empirically show that Approach 3 outperforms the other two approaches. The scores therefore describe an auxiliary classifier’s lexical associations with questionnaire-conditioned text, not internal psychological traits. Corpora are processed as sentences or short paragraphs using subsamples: 10% of BookCorpus, 2% of an English Wikipedia snapshot from May 2020, 20% of the WebText Test Set, and all of WikiText-103. These substitutes are not necessarily the exact training data. In the model comparison, the main table reports medians with an undefined dispersion: GPT-3.5-Turbo is highest on agreeableness, 4.41, and extraversion, 4.06; LLaMA on conscientiousness, 4.01, and emotional stability, 3.76; Transformer-XL on openness, 4.02. GPT-2 remains close to 3 on all five traits. Comparisons are confounded by scale, training data, alignment, API version, and decoding. GPT-2 scores also change substantially depending on whether the classifier sees the whole response, first sentence, or median across sentences; Wasserstein distances reach .610 for openness. Comparing the WebText Test Set with GPT-2 produces small distances without prompts.025 to .102, and much larger distances when Big Five statements are included, up to .630 for extraversion. This mainly shows prompt-induced changes in what the classifier receives; it does not validate the questionnaire as a measure of inherent personality or show that GPT-2 psychologically “inherits” a corpus personality. The paper also fine-tunes GPT-2 on labeled SIOP responses. Method 1 keeps texts with scores above 4 and trains for 20 epochs; many traits move together and the target trait does not always increase. Method 2 turns each trait into binary classification at five thresholds and trains for 10 epochs. For extraversion, a 3.07 baseline becomes 2.89, 3.19, 3.24, 2.90, and 3.25: it decreases in two of five settings even though the prose calls the result an improvement. There are no seeds, fine-tuning replications, statistical tests, confidence intervals, language-quality evaluation, behavioral tests, persistence tests, or human judges. The recommendation to use LLaMA in mental-health care because of its “emotional stability” median is unsupported by any clinical, empathy, or safety evaluation. The defensible evidence is narrower: different models, prompts, text segments, and fine-tuning runs yield different distributions under one unspecified NLI classifier.
Research question
Can a zero-shot NLI-based classifier convert corpus text and continuations elicited by a Big Five questionnaire into profiles comparable across models, and can those profiles be modified by fine-tuning GPT-2 with labeled data?