This Findings of EMNLP 2023 paper studies the perceived personality of BERT-base and 124M-parameter GPT-2 through an adaptation of the 50-item IPIP Big Five questionnaire. It replaces agreement choices with the adverbs never, rarely, sometimes, often, and always and turns each item into sentence completion. BERT selects the most likely masked token, while GPT-2 selects the most likely completed sentence. Deterministic scores are mapped to percentiles from an external dataset of 1,015,000 human questionnaires. The fact that all ten base-model results fall within 26 percentile points of the human median does not show that pretraining reproduces a population: the paper does not report TOST equivalence bounds or justify treating one model output as a human-comparable observation. The central experiment prefixes each target item with one of ten same-trait statements combined with five intensity modifiers, yielding 500 responses and 50 scores per trait and model. Expected direction r_cm is constructed from the same scoring key and lexical intensity, then correlated with observed change. Aggregate correlations are .40 for BERT and .54 for GPT-2; the median of fifty per-item correlations, each computed from only five points, is .84 and .81. However, every statement is also used as context for itself, allowing the model to copy the modifier. Replacing that response with baseline lowers mean correlations to .25 and .40. Double negations create clusters near −1, and an alternative framing excludes neutral and reports only six of ten trait-model combinations. The defensible result is thus predictable sensitivity to quantifiers and lexical content, not validated personality control. An exploratory analysis uses 1,119 Reddit personality descriptions as context. Bag-of-words and n-gram regressions are fitted only to cases with absolute change ≥1 and interpreted through extreme weights; no validation split or predictive performance is reported, and several associations are noisy or spurious. An IRB-approved Mechanical Turk study retains 404 people: 199 see Big Five definitions before writing and 205 do not; roughly one quarter miss the 75-word minimum. After outlier or length filtering, correlations between human scores and scores obtained by conditioning on their self-description reach .44 for BERT and .48 for GPT-2 in directed responses; for undirected responses they are .40 and .48 after outlier removal. The paper gives no intervals, p-values, remaining sample sizes per filter, external validation, or text-classifier comparison. This moderately supports that explicit self-descriptions contain signals associated with their authors' self-reports, not that a model adopts their personality or can replace human assessment. Finally, GPT-2 generates 50 tokens from approximately 1,500 combinations of trait statements and six neutral prompts. The generated text is then fed back into the same GPT-2 as questionnaire context, and its score correlates .49 with the initial statement score. This is a circular design without human judges and can reflect lexical continuation rather than personality perceived by users. The paper is a useful early study of contextual priming and contributes two description datasets, but it does not establish internal traits, persistence, behavioral transfer, robustness in modern models, or clinical safety. An audit of the PDF-linked repository at commit 3388f194a6410162e4c2a614e12846f02cf5a939 confirms two valid CSV files with exactly 199 and 205 rows, but finds no code even though the README advertises “dataset and code.” The purported `reddit-data.json` contains 1,119 strings but cannot be parsed as JSON because it is enclosed in braces rather than brackets. The repository also has no license, derived outputs, execution parameters, or scripts for regenerating tables and figures. The human data are partially inspectable, but the complete experiments are not reproducible from the published artifact.
Research question
Can Big Five scores obtained from BERT and GPT-2 be predictably modified with different contexts, reflect personality signals in human self-descriptions, and transfer to generated text?