Schoenegger and coauthors study a narrow but useful task: estimating the empirical correlation between two personality items without collecting new questionnaire responses. The benchmark contains 249 pairs drawn from 103 SAPA items. It is not a natural sample of correlations: one third was selected below −0.2, one third between −0.2 and 0.2, and one third above 0.2. The study compares 254 laypeople, 272 psychology or behavioral-science academics, three temperature-zero GPT-4o runs, three Claude 3 Opus runs, and one deterministic prediction from PersonalityMap, a proprietary network trained specifically on item pairs. Each human judged 30 randomly assigned pairs, apart from some incomplete lay responses, whereas every AI system covered all 249.
In individual comparisons, the seeded randomly selected GPT-4o run had mean absolute error 0.14, Claude 0.11, and PersonalityMap 0.07, compared with participant-level means of 0.29 for laypeople and 0.20 for experts. GPT-4o ranked at the 95.67th percentile against laypeople and 70.22nd against experts; Claude ranked at 100 and 95.22; PersonalityMap ranked at 100 against both. Win rates over the items actually answered by each person were 90.94%/69.85% for GPT-4o, 97.64%/86.40% for Claude, and 100%/99.26% for PersonalityMap against laypeople/experts. This supports the claim that one query to these systems is usually more accurate than one individual on this task. It does not establish broad personality understanding: the target is an item-level population parameter, and the individual comparison gives each AI 249 pairs while each human sees only 30.
Median aggregation by item changes the comparison. Mean error is 0.164 for laypeople, 0.086 for experts, 0.140 for GPT-4o, 0.104 for Claude, and 0.072 for PersonalityMap. Correlations between aggregate predictions and empirical values are 0.88, 0.90, 0.78, 0.80, and 0.91, respectively. The three-bucket test (<−0.1, −0.1 to 0.1, >0.1) reports no group difference, χ²(4)=6.42, p=.170. The published error tests, however, treat five measurements on the same 249 pairs as independent: they use Kruskal–Wallis and label Paired=False Mann–Whitney comparisons as “Dunn’s test.” An editorial paired robustness check yields Friedman χ²(4)=96.59, p<.001 and Holm-adjusted Wilcoxon tests. It preserves the broad ranking but changes two borderline conclusions, GPT-4o versus laypeople becomes p=.047 and PersonalityMap versus experts p=.047, while experts versus Claude becomes p=.053. A paired Cochran Q test still finds no bucket difference, Q(4)=8.70, p=.069. These checks are not a substitute for a formal statistical reanalysis, but they show that some inferential detail depends on ignoring the paired design.
Analyses added after first-round peer review test 30 unseeded GPT-4o queries at temperature 1 and SurveyBot3000. Aggregating 30 higher-temperature runs raises GPT-4o’s prediction–truth correlation to 0.88 without uniformly improving the other metrics. SurveyBot3000 appears competitive, but it was trained on SAPA and therefore on the benchmark used here; the authors correctly state that its result cannot distinguish generalization from memorization. The transparent review record also shows that reviewers pressed the authors to narrow the title from “understanding personality” to predicting item correlations and to curb speculative applications.
The OSF artifact reproduces the headline numbers but not the study end to end. It contains two processed Excel files, two monolithic scripts, and two truth JSON files, with no README, requirements file, lockfile, CLI, or code for collecting the LLM outputs or running PersonalityMap. Paths remain as XXX/...; after reconstructing them and using the package versions listed in the reporting summary, the main script completes and reproduces the published tables. The additional script also completes, but sets num_bootstraps = 10 although the paper reports 10,000, produces intervals different from the supplement, and returns nan CIs for two inapplicable conditions. Both scripts print the Shapiro p-value as both W and p, and the Levene p-value as both F and p. PersonalityMap remains proprietary: two authors created it, and one also founded Positly and GuidedTrack, all disclosed conflicts; the public artifact cannot independently verify that none of the evaluated items entered its training data.
The defensible contribution is that synthetic item-correlation forecasts can be cheap and surprisingly accurate, while aggregated human expertise remains competitive with general LLMs. The proper scope is linear, cross-sectional self-report correlations from one English inventory under an artificially stratified test distribution. The evidence does not validate hiring, diagnosis, marketing, automated psychological assessment, or wholesale replacement of human data. Nor does a predicted correlation establish causation: the authors themselves state that scientific uses must ultimately be confirmed in real humans and that PersonalityMap still lacks prediction-level uncertainty estimates.