The paper evaluates whether two early ChatGPT versions can rank Facebook users by Big Five traits from text that was not labeled for that task. It randomly selects 1,000 adult MyPersonality participants who completed the 100-item IPIP and posted at least 200 status updates; mean age is 24.2 and 63.1% are women. Each person's 200 most recent updates are concatenated into ten chunks of 20. GPT-3.5 Turbo 0301 and GPT-4 0314 receive the prompt “Rate the text on the Big Five personality dimensions,” return five scores from 1 to 5, and repeat every inference three times. The authors average repetitions and chunks, then compare the result with self-reports. GPT-3.5 correlations range from r = 0.223 to 0.298 and GPT-4 correlations from 0.264 to 0.327; their means are 0.271 and 0.307, respectively, and 0.289 across all ten results. This corresponds to roughly 5–11% shared variance per trait: there is a modest ranking signal, not precise recovery of an individual's personality. GPT-4 is numerically higher on all five traits, but none of its differences from GPT-3.5 is statistically significant. The score scales are also poorly calibrated. GPT-3.5 underestimates Conscientiousness by 1.254 points and Agreeableness by 1.032 on a 1–5 scale; GPT-4 underestimates Conscientiousness by 0.690 and overestimates Extraversion by 0.881. Correlation therefore does not make model outputs interchangeable with IPIP scores. Twenty status updates already yield correlations of 0.176–0.257; using 200 raises them to 0.222–0.327, with different gains by trait. The demographic analyses compare group means and absolute residuals. For both models, absolute errors are lower for women on Conscientiousness, Agreeableness, and Neuroticism; the Openness difference appears only for GPT-3.5, while GPT-4 makes larger Extraversion errors for women. Within-gender correlations do not differ significantly. By age, GPT-3.5 has larger errors for older users on Openness and Conscientiousness but smaller errors on Agreeableness; GPT-4 shows no significant absolute-error differences. The headline of greater accuracy for women and younger people therefore applies to selected traits and metrics, not a general pattern. Among 68 people with friend ratings, observer–self-report correlations average 0.304 and observer–LLM correlations 0.269–0.276, but equivalence is never formally tested. Table S6.1 has a verifiable reporting error: its confidence intervals and p-values are identical to the GPT-4 rows in S6.2 and do not correspond to the observer–self-report coefficients. It cannot support a strong claim of human parity or shared cue use. The study shows that these snapshots capture textual regularities associated with Big Five self-reports in a highly selected sample. It does not validate a personality test, clinical inference, or a tool suitable for individual decisions. The clearest implication is a privacy risk: even a modest signal can be exploited at scale without users' knowledge. Current reproducibility is inadequate. The paper points to OSF hq2ra, but that project now returns HTTP 401 without authentication; an archived page shows that it was public and stored 6.4 MB in November 2024, while the arXiv source contains only the manuscript and figures, not data or analytical code.
Research question
Can GPT-3.5 and GPT-4 infer in zero-shot the relative positions of users on the five Big Five traits from their Facebook statuses, how does the association change with text volume, and how do errors vary by gender and age?