This preprint evaluates binary automatic prediction of the five Big Five traits from text with GPT-4 and four local models, Phi-3, Gemma 7B, Llama 3.1 8B, and Mistral 7B, on Essays, MyPersonality, and Pandora. It runs 135 configurations: 75 with a simple prompt across all five models and 60 with an enriched prompt for the four open models only. Each model classifies each trait separately as 0 or 1 at temperature 0 and a 20-token maximum. The enriched prompt supplies Costa and McCrae definitions, positive and negative IPIP phrases, and long Goldberg adjective lists. The paper reports accuracy, macro-F1, and class-specific precision, recall, and F1, together with invalid outputs. The most useful result is diagnostic rather than strong performance. Only 13 of 135 configurations, 9.6%, reach F1 of at least 0.5 for both classes. The strongest balanced row is Mistral on Agreeableness in MyPersonality with the complex prompt: accuracy 0.608, negative-class F1 0.600, positive-class F1 0.620, and macro-F1 0.610, only 7.2 accuracy points above the majority class. GPT-4 on Agreeableness in Essays reaches accuracy 0.593 and macro-F1 0.585; Mistral on Openness in Essays reaches 0.584 and 0.575. Many rows with superficially acceptable accuracy predict the positive class almost exclusively. Enriched prompts usually remove formatting failures and raise positive recall, but often reduce negative recall; they do not uniformly improve accuracy or balance. Extraversion and Neuroticism are the weakest dimensions under these conditions. Interpretation needs additional caution. Invalid outputs are discarded before metrics are calculated, so performance is conditional on model compliance. Phi-3 loses roughly 80–85% of Essays under the simple prompt; its remaining metrics describe a highly selected subset. The F1 ≥ 0.5 cutoff is repeatedly called a significance threshold even though there is no test, null distribution, interval, or multiplicity correction, and the authors acknowledge that 0.5 is not a chance threshold under imbalance. GPT-4 receives a different prompt, is tested only in the simple condition, and has no identified snapshot; claims that complex-prompt open models match it confound model and prompt. GPT-4 is also told that 1 means high or moderate trait level, while the other models classify trait presence or expression. Big Five dimensions are continuous rather than present-or-absent properties, and author-level labels do not guarantee that one text expresses a trait. Pandora adds an unvalidated transformation: its 1–100 scores are interpolated to 1–5 and thresholded using the highest MyPersonality score still labeled 0; the five thresholds are not released. Its 14,221 texts come from 1,608 users, but metrics treat texts as units without author-clustered uncertainty. Internal inconsistencies also matter. For GPT-4/Openness/Pandora, Table 2 reports 679 invalids, while the Table 3 diagonal and Table 4 supports of 7,532+6,209 imply exactly 480; an invalid union cannot be smaller than 679 invalids for one trait. MyPersonality reports 255 texts, but each trait's classes total 250 without explaining five missing labels. Table 1 labels median and SD while the prose says mean and SD. No code, predictions, processed labels, or logs were located. Ollama tags are mutable and lack digests, dates, quantization, and runtime version; GPT-4 lacks a snapshot and date. The defensible conclusion is therefore that zero-shot classification is usually weak or asymmetric in these prompts and datasets, and prompt enrichment changes compliance and class bias. The paper usefully warns about class-specific metrics and abstentions, but it does not establish reliable prediction, statistical significance, a controlled parity comparison with GPT-4, or validity for profiling people.
Research question
With what reliability do five LLMs classify binary presence or absence of Big Five traits from texts in three domains, and how do their compliance, class balance, and performance change when enriching the prompt with psychological descriptors?