The article examines whether fixed representations of Reddit text produced by BERT, RoBERTa, and OpenAI text-embedding-3-small can predict human Big Five labels and whether those representations can be considered psychometrically reliable and valid. It starts from PANDORA, retains 1,568 users, and reports 935,102 posts after language, length, and duplicate filtering. It compares regression with prior work and GPT-4o zero-shot, and binary classification with five algorithms, linguistic variables, and feature combinations. As a computational contribution, the comparison is broad, and the paper supplies trait-level tables, a settings appendix, and a code repository that permits a partial audit of the workflow.
Descriptive predictive results favor OpenAI representations: mean MSE is 526.90 versus 531.11 for the prior BERT result, 618.44 for RoBERTa, and 765.647 for GPT-4o zero-shot; in classification, the BiLSTM using OpenAI embeddings reaches AUROC 0.82–0.83, RoBERTa 0.80–0.82, and BERT 0.73–0.75. Linguistic variables alone reach 0.63–0.66, and adding them to embeddings does not improve results consistently. However, the claim of “45% lower error” reverses the denominator: 526.90 is approximately 31.2% lower than 765.647; 45.3% is how much larger the zero-shot value is relative to the embedding model. OpenAI improves on the prior BERT result by only 4.21 MSE points, with no interval, repetition, or statistical test. The zero-shot prompt also outputs scores on a 1–5 scale while labels are described on a 0–100 scale, without a reproducible transformation making the published comparison commensurate.
The central psychometric conclusion is not supported by the public artifact. Table 4 attributes Cronbach alpha values of 0.574–0.664 to “LLM embeddings,” but the released notebook reproduces those exact five values by running cronbach_alpha on 134 standardized LIWC/NRC/VADER linguistic variables; the line that would calculate alpha on embeddings is commented out. A different exploratory notebook reports alpha −0.1725 on selected embedding dimensions and is not discussed in the paper. Even if alpha had been calculated on vector coordinates, treating arbitrary embedding dimensions or PCA components as interchangeable scale items would not establish Big Five internal consistency. Correlations between selected LIWC variables and PCA components show that embeddings contain lexical information, not convergence with an independent validated trait measure; comparing the intertrait correlation matrices of labels and predictions is likewise insufficient evidence of convergent or discriminant validity.
The predictive design also has identity leakage and pseudoreplication. Labels belong to users, but every post is treated as an independent sample; the code drops the author identifier and randomly splits rows, allowing posts from the same person to occur in training, validation, and test sets. Performance can therefore reflect a known writer’s style or identity rather than generalization to unseen people. Although the paper says that five-fold validation was used, the main code path explicitly passes kFold=False, and the included checkpoint does not document such validation. The repository lacks the data, embeddings, environment, license, and sufficient saved results to reconstruct the paper’s tables and includes absolute paths and settings inconsistent with the manuscript. The defensible result is therefore that text embeddings carry signals useful for predicting PANDORA labels under this protocol; the study does not establish psychometric validity, unseen-user generalization, clinical utility, or reliable personality measurement.