Psychometric Evaluation of Large Language Model Embeddings for Personality Trait Prediction

Evaluation and psychometric validity2025JMIRApproved editorial review

Authors: Julina Maharjan, Ruoming Jin, Jianfeng Zhu, Deric Kenne

Keywords: Large Language Models, Embeddings, Personality prediction, Psychometric validation, Big Five, LIWC, Emotional markers

Source: Open primary source (opens in a new tab)

4
Authors
14
Findings
27
Limitations
14
Evidence

Editorial summary

English

The article examines whether fixed representations of Reddit text produced by BERT, RoBERTa, and OpenAI text-embedding-3-small can predict human Big Five labels and whether those representations can be considered psychometrically reliable and valid. It starts from PANDORA, retains 1,568 users, and reports 935,102 posts after language, length, and duplicate filtering. It compares regression with prior work and GPT-4o zero-shot, and binary classification with five algorithms, linguistic variables, and feature combinations. As a computational contribution, the comparison is broad, and the paper supplies trait-level tables, a settings appendix, and a code repository that permits a partial audit of the workflow.

Descriptive predictive results favor OpenAI representations: mean MSE is 526.90 versus 531.11 for the prior BERT result, 618.44 for RoBERTa, and 765.647 for GPT-4o zero-shot; in classification, the BiLSTM using OpenAI embeddings reaches AUROC 0.82–0.83, RoBERTa 0.80–0.82, and BERT 0.73–0.75. Linguistic variables alone reach 0.63–0.66, and adding them to embeddings does not improve results consistently. However, the claim of “45% lower error” reverses the denominator: 526.90 is approximately 31.2% lower than 765.647; 45.3% is how much larger the zero-shot value is relative to the embedding model. OpenAI improves on the prior BERT result by only 4.21 MSE points, with no interval, repetition, or statistical test. The zero-shot prompt also outputs scores on a 1–5 scale while labels are described on a 0–100 scale, without a reproducible transformation making the published comparison commensurate.

The central psychometric conclusion is not supported by the public artifact. Table 4 attributes Cronbach alpha values of 0.574–0.664 to “LLM embeddings,” but the released notebook reproduces those exact five values by running cronbach_alpha on 134 standardized LIWC/NRC/VADER linguistic variables; the line that would calculate alpha on embeddings is commented out. A different exploratory notebook reports alpha −0.1725 on selected embedding dimensions and is not discussed in the paper. Even if alpha had been calculated on vector coordinates, treating arbitrary embedding dimensions or PCA components as interchangeable scale items would not establish Big Five internal consistency. Correlations between selected LIWC variables and PCA components show that embeddings contain lexical information, not convergence with an independent validated trait measure; comparing the intertrait correlation matrices of labels and predictions is likewise insufficient evidence of convergent or discriminant validity.

The predictive design also has identity leakage and pseudoreplication. Labels belong to users, but every post is treated as an independent sample; the code drops the author identifier and randomly splits rows, allowing posts from the same person to occur in training, validation, and test sets. Performance can therefore reflect a known writer’s style or identity rather than generalization to unseen people. Although the paper says that five-fold validation was used, the main code path explicitly passes kFold=False, and the included checkpoint does not document such validation. The repository lacks the data, embeddings, environment, license, and sufficient saved results to reconstruct the paper’s tables and includes absolute paths and settings inconsistent with the manuscript. The defensible result is therefore that text embeddings carry signals useful for predicting PANDORA labels under this protocol; the study does not establish psychometric validity, unseen-user generalization, clinical utility, or reliable personality measurement.

Español

El artículo estudia si representaciones fijas de textos de Reddit obtenidas con BERT, RoBERTa y OpenAI text-embedding-3-small permiten predecir etiquetas Big Five humanas y si esas representaciones pueden considerarse psicométricamente fiables y válidas. Parte de PANDORA, conserva 1.568 usuarios y declara 935.102 publicaciones después de filtrar idioma, longitud y duplicados. Compara regresión frente a un trabajo anterior y GPT-4o zero-shot, y clasificación binaria con cinco algoritmos, variables lingüísticas y combinaciones de variables. Como contribución computacional, la comparación es amplia y el artículo publica tablas por rasgo, un anexo de configuraciones y un repositorio de código que permite auditar parte del flujo.

Los resultados predictivos descriptivos favorecen a las representaciones de OpenAI: MSE medio 526,90 frente a 531,11 del BERT previo, 618,44 de RoBERTa y 765,647 de GPT-4o zero-shot; en clasificación, el BiLSTM con embeddings de OpenAI alcanza AUROC 0,82–0,83, RoBERTa 0,80–0,82 y BERT 0,73–0,75. Las variables lingüísticas solas quedan en 0,63–0,66 y añadirlas a embeddings no mejora de forma estable. Sin embargo, la afirmación de «45% menos error» invierte el denominador: 526,90 supone aproximadamente un 31,2% menos MSE que 765,647; 45,3% es cuánto mayor resulta el zero-shot respecto al modelo con embeddings. La ventaja de OpenAI frente al BERT previo es de solo 4,21 puntos de MSE y no se acompaña de intervalos, repetición o contraste estadístico. Además, el prompt zero-shot produce puntuaciones 1–5 mientras las etiquetas se describen en escala 0–100, sin una transformación reproducible que haga comparable esa tabla.

La principal conclusión psicométrica no está respaldada por el artefacto público. La Tabla 4 atribuye a «LLM embeddings» alfas de Cronbach entre 0,574 y 0,664, pero el cuaderno publicado reproduce exactamente esos cinco valores ejecutando cronbach_alpha sobre 134 variables lingüísticas estandarizadas LIWC/NRC/VADER; la línea que calcularía alfa sobre embeddings está comentada. Otro cuaderno exploratorio obtiene alfa −0,1725 sobre dimensiones seleccionadas del embedding y tampoco aparece en el artículo. Incluso si se hubiera calculado sobre coordenadas del vector, tratar dimensiones arbitrarias o componentes PCA como ítems intercambiables de una escala no demostraría consistencia interna del Big Five. Las correlaciones entre variables LIWC seleccionadas y componentes PCA muestran que el embedding contiene información léxica, no validez convergente con una medida independiente del rasgo; y comparar la matriz de correlaciones entre etiquetas y predicciones no constituye por sí solo validez convergente o discriminante.

El problema predictivo también sufre fuga por identidad y pseudorreplicación. Las etiquetas son de usuario, pero cada publicación se trata como muestra independiente; el código elimina el identificador de autor y realiza particiones aleatorias por fila, por lo que textos del mismo usuario pueden estar simultáneamente en entrenamiento, validación y prueba. El rendimiento puede reflejar estilo o identidad conocida y no generalización a personas nuevas. Aunque el artículo afirma validación de cinco pliegues, la ruta principal del código pasa kFold=False y el checkpoint incluido no documenta esa validación. El repositorio no incluye los datos, embeddings, entorno, licencia ni resultados suficientes para reconstruir las tablas; contiene además rutas absolutas y configuraciones inconsistentes con el manuscrito. En consecuencia, el trabajo aporta evidencia descriptiva de que embeddings de texto codifican señales útiles para predecir etiquetas PANDORA bajo este protocolo, pero no establece validez psicométrica, generalización a nuevos usuarios, utilidad clínica ni una medición fiable de personalidad.

Research question

To what extent do contextual embeddings from BERT, RoBERTa, and OpenAI allow predicting Big Five labels in Reddit posts, outperform psycholinguistic variables or GPT-4o zero-shot, and retain properties that the authors interpret as reliability, convergent validity, and divergent validity?

Method

Retrospective prediction study on PANDORA. Filters the corpus to users with Big Five, English posts of at least 30 words, and non-duplicate texts. Generates 768-dimensional embeddings with bert-base-uncased and roberta-base and 1536 real dimensions with text-embedding-3-small, although two tables in the manuscript say 1568. For regression, trains a multilayer perceptron and compares MSE with a published BERT and a GPT-4o zero-shot prompt. For classification, normalizes and dichotomizes each trait by the median and compares logistic regression, XGBoost, random forest, MLP, and BiLSTM with attention using LIWC/NRC/VADER variables, embeddings, or combinations. The editorial review read the 21 pages, rendered and verified the five pages of the appendix, recalculated arithmetic comparisons, and audited the PersonaClassifier repository at commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3, including splits, models, psychometric notebooks, and saved artifacts.

Sample: The original psychological unit is the user, but the unit introduced into the models is each post. The article retains 1.568 users and declares 935.102 final posts, with a mean of 90 words. The three counts by label type sum to 934.102, exactly one thousand less than the declared total. The 0–100 labels are assigned to the user and repeated across their posts. The code removes #AUTHID before splitting rows, so it cannot guarantee that a user appears in a single split. The variable selection appendix additionally shows 908.432 rows without reconciling that figure with the 935.102 published cases. There is no clinical sample, new human evaluation, external dataset, or documented test on never-seen users.

Findings

  • OpenAI text-embedding-3-small obtains a mean MSE of 526.90, compared to 531.11 for the previous BERT, 618.44 for RoBERTa, and 765.647 for GPT-4o zero-shot.
  • The correct reduction in MSE relative to zero-shot is approximately 31.2%; the 45.3% corresponds to how much larger 765.647 is relative to 526.90, not to how much smaller the latter is relative to the former.
  • The improvement of OpenAI over the BERT result cited as state of the art is 4.21 MSE points, less than 0.8%, without statistical testing or uncertainty.
  • In the appendix, the BiLSTM with OpenAI embeddings achieves AUROC 0.82 in openness, conscientiousness, and neuroticism, and 0.83 in extraversion and agreeableness.
  • RoBERTa achieves AUROC 0.80–0.82 and BERT 0.73–0.75; no intervals or tests of difference between models are published.
  • Linguistic variables alone produce AUROC 0.63–0.66. Combining all of them with embeddings lowers the BiLSTM to 0.62–0.66, while selecting variables recovers 0.81–0.82 without consistently outperforming embeddings alone.
  • The article reports alphas between 0.574 and 0.664 and interprets them as moderate reliability of embeddings.
  • The public notebook reproduces exactly those alphas on a standardized matrix of 134 LIWC/NRC/VADER variables, not on the embeddings; the call on embeddings is commented out.
  • An exploratory CFA notebook computes alpha −0.1725, CI [−0.318, −0.036], on selected dimensions of the embedding; that result is not discussed in the article.
  • The reported correlations between selected LIWC variables and PCA components reach 0.63 in some cells, but arise from data-guided selection and multiple comparisons without independent validation.
  • The correlation matrix between predictions retains directions similar to that of labels in several pairs of traits, with a conscientiousness–neuroticism difference of −0.02 versus 0.003.
  • The manuscript declares OpenAI embeddings of 1568 dimensions on pages 4–5, but page 17 and the actual model use 1536.
  • The article declares 80:10:10 training and five-fold validation; the available code uses random row-by-row splits and the main run.py path forces kFold=False.
  • The architecture called BiLSTM with attention receives each pooled vector as a sequence of length one; the attention operates before the LSTM on that single step, so it does not select among multiple tokens or segments.

Limitations

  • The labels are per user and the observations are posts, but there is no author-level partition. This introduces dependence, identity leakage, and pseudoreplication between training, validation, and test.
  • Treating 935.102 posts as independent samples inflates the effective size: they come from only 1.568 users and repeat their same five labels.
  • There is no user-held-out evaluation, external dataset, temporal validation, replication, or comparison on another platform.
  • The counts by label type sum to 934.102 and not 935.102; the appendix uses 908.432 rows for variable selection without explaining the difference.
  • Scores, percentiles, and descriptions are normalized and dichotomized separately before being concatenated, but it is not validated that these label types are equivalent.
  • Dichotomizing by the median removes continuous information and creates classes relative to the corpus, not clinical thresholds or validated psychological categories.
  • The regression table compares a trained model with a zero-shot prompt on apparently different scales; the repository contains incompatible experiments with standardization, 0–5, MSE, and RMSE, without unambiguous reconstruction of the published table.
  • The claim of 45% less error uses an incorrect percentage comparison and the advantage over the cited BERT is small and non-inferential.
  • The published formula for TPR is TP/(TP+TN), when it should be TP/(TP+FN). The code uses sklearn utilities for the curves, so it appears to be a manuscript error, but it weakens the reproducible description of the metric.
  • No intervals, seeds, variability across runs, statistical tests, sensitivity analyses, or correction for multiple comparisons are published for the predictive metrics.
  • The claim of five folds contradicts the main code path, which passes kFold=False; optional or commented-out branches do not prove what produced the tables.
  • Table 4 does not document embedding alpha: the published values correspond exactly to linguistic variables in the audited notebook.
  • The dimensions of an embedding are not parallel items designed to measure a construct. Applying alpha to coordinates or components does not establish the reliability of a personality scale.
  • Alpha changes the number of rows per trait, 3.635, 1.961, 2.363, 2.024, and 555 in the notebook, without the article explaining this reduced sample relative to the corpus of hundreds of thousands of posts.
  • The purported convergent validity correlates selected linguistic variables with PCA components of the same texts, not predictions with an independent validated instrument.
  • Selection by Lasso, PCA, and searching for high cells is performed on the same data, with many correlations and no holdout, intervals, or correction, so the maxima are exploratory.
  • Comparing correlation matrices of labels and predictions does not prove discriminant validity, factorial structure, invariance, or measurement equivalence.
  • The BiLSTM architecture treats an aggregated embedding as a single time step; attention prior to the LSTM is trivial for a sequence of length one and the sequential justification does not match the actual tensor.
  • The main text says 512 hidden units, the appendix selects 128 or 256 per trait, and run.py fixes 512; it is unclear which configuration produced each result.
  • The appendix lists dropout [0.2, 0.3, 10.5] but selects 0.5 for openness, a value absent from the literal space; it is likely a typo, although it prevents reproducing the search as published.
  • The manuscript attributes 355 million parameters to text-embedding-3-small without a verifiable source and calls BERT/RoBERTa and the embedding model LLMs, mixing model categories.
  • The zero cost of BERT/RoBERTa omits GPU, electricity, and storage, while OpenAI includes API pricing; the economic comparison does not use homogeneous accounting.
  • The repository contains no data, OpenAI embeddings, complete checkpoints, fixed dependencies, reproducible environment, or license; the README is minimal and numerous notebooks have absolute paths from the author team.
  • Data Availability states that data and code are publicly available, but .gitignore excludes CSV, embeddings, processed results, checkpoints, and the entire analysis folder; access to PANDORA is acknowledged directly to one of its authors.
  • The text is described as anonymized, but public pseudonymous data may allow reidentification. No privacy, memorization, fairness, demographic error, or cultural invariance audit is performed.
  • The statement that no attempt is made to infer sensitive information is in tension with the explicit goal of inferring authors' personality from their texts.
  • There is no clinical, occupational, diagnostic, human, calibration, outcomes, or harm validation, even though the article discusses high-impact applications.

What the study does not establish

  • It does not demonstrate that embeddings measure Big Five constructs with psychometric reliability or validity.
  • It does not demonstrate generalization to new users, because the protocol does not keep authors separate across splits.
  • It does not demonstrate that 935.102 posts are equivalent to 935.102 independent psychological observations.
  • It does not demonstrate that OpenAI significantly outperforms the previous BERT; the published difference is small and lacks uncertainty.
  • It does not demonstrate a 45% error reduction relative to GPT-4o zero-shot; with the conventional denominator it is approximately 31.2%.
  • It does not demonstrate that zero-shot and regression are compared on the same outcome scale.
  • It does not demonstrate internal reliability of the embeddings: the published alphas come from linguistic variables according to the available code.
  • It does not demonstrate convergent validity through LIWC–PCA correlations, because there is no independent psychometric criterion or confirmatory analysis.
  • It does not demonstrate discriminant validity through similarity between label and prediction correlation matrices.
  • It does not demonstrate that a BiLSTM provides sequential modeling when the input is a single embedding vector.
  • It does not demonstrate utility for diagnosis, treatment, occupational selection, education, or other high-impact decisions.
  • It does not allow generalization to other platforms, languages, cultures, populations, personality instruments, or future versions of the models.
  • It does not provide a complete reproduction of the tables from the public repository.
  • It does not study the personality of LLMs as subjects; it studies the use of representations generated by models to predict human personality labels.

Traceability

Scope: Full text

Version: J Med Internet Res 2025;27:e75347, DOI 10.2196/75347; 21-page article, official 5-page Multimedia Appendix 1, and GitHub commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3 audited

Consulted source: https://www.jmir.org/2025/1/e75347/PDF

Review: Codex editorial review and methods/artifact audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • OpenAI text-embedding-3-small
  • OpenAI GPT-4o zero-shot
  • Hugging Face bert-base-uncased
  • Hugging Face roberta-base
  • Bidirectional LSTM with dot-product attention
  • Multilayer perceptron
  • Logistic regression
  • XGBoost
  • Random forest

Instruments and metrics

  • PANDORA user-level Big Five labels
  • Mean squared error (MSE)
  • Area under the receiver operating characteristic curve (AUROC)
  • Median binary split after min-max normalization
  • Cronbach alpha
  • Principal component analysis
  • LIWC-22 linguistic features
  • NRC emotion lexicon
  • NRC valence, arousal and dominance features
  • VADER sentiment
  • Lasso and mutual-information feature selection

Data used

  • PANDORA: approximately 17 million Reddit posts from 10,000 users in the source corpus
  • Big Five subset: 1,568 users and approximately 3 million posts before language and length filtering
  • English subset: 2,830,311 posts
  • Posts with at least 30 words: 1,008,627
  • Declared final set after deduplication: 935,102 posts
  • Declared label-method counts: 520,065 scores, 268,189 percentiles and 145,848 descriptions, totaling 934,102 rather than 935,102
  • Multimedia Appendix 1: hyperparameter search and six classification settings
  • PersonaClassifier public GitHub repository at commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3

Evidence and location

  • PANDORA corpus, 1.568 users, filtered and posts treated as independent samples: JMIR e75347, Methods, Data Collection and Preprocessing, p. 4
  • Counts by label type and declared embedding dimensions: JMIR e75347, Methods, pp. 4–5, Figure 1 and Table 1
  • BiLSTM, hidden units, 80:10:10 split, grid search, and zero-shot prompt: JMIR e75347, Persona Classifier and Training, pp. 5–6, Textbox 3
  • Classification, median dichotomization, five-fold claim, and erroneous TPR formula: JMIR e75347, RQ3 Training and Evaluation Metric, pp. 7–8
  • MSE, zero-shot comparison, 45% claim, and published alphas: JMIR e75347, Results RQ1–RQ2, Tables 3–4, p. 9
  • Correlations interpreted as convergence and divergence: JMIR e75347, Results RQ2, Figure 2 and Figures 3–8, pp. 9–16
  • AUROC by configuration and by embedding family: JMIR e75347, Results RQ3–RQ4, Tables 5–6, pp. 16–17; Multimedia Appendix 1, pp. 3–5
  • Limitations, high-impact risks, conclusions, IRB, and declared availability: JMIR e75347, Ethical Considerations p. 8; Discussion, Limitations and Data Availability, pp. 17–19
  • Contradictory hyperparameters, six settings, and 908.432 rows in variable selection: Official Multimedia Appendix 1, Tables S1–S2 and experiment results, pp. 1–5
  • Random row-by-row splits, author identifier removal, and main path without k-fold: PersonaClassifier commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3, run.py; PersonaClassifier.py; utils/DataProcessor.py
  • The alphas in Table 4 are calculated over linguistic variables and the embedding alpha is commented out: PersonaClassifier commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3, EMB_analysis/liwc_analysis.ipynb
  • Negative exploratory alpha on selected embedding dimensions: PersonaClassifier commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3, EMB_analysis/7_CFA.ipynb
  • Single-step BiLSTM input and trivial attention before the LSTM: PersonaClassifier commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3, utils/Models.py and run.py
  • Absence of data, environment, license, and complete artifacts: PersonaClassifier repository tree, README.md and .gitignore at commit 3dcd3bebe0d61142b49b20c54d4dca5b6006a8f3, audited 15 Jul 2026